Apps
Google pushes for its robots.txt parser to become internet standard
Google LLC is pushing for its decades-old Robots Exclusion Protocol to be certified as an official internet standard, so today it open-sourced its robots.txt parser as part of that effort. The REP, as it’s known,…