Published on [Permalink]
Posted in:

Google wants to make robots.txt an Internet standard

They have Martijn Koster, the original creator of the Robots Exclusion Standard, on-board. I'm hoping this is purely altruism on Google's part, and they won't try to exert influence through this initiative.

Reply by email