Google wants to make robots.txt an Internet standard
They have Martijn Koster, the original creator of the Robots Exclusion Standard, on-board. I’m hoping this is purely altruism on Google’s part, and they won’t try to exert influence through this initiative.