Google announced yesterday as part of its efforts to standardizing the robots exclusion protocol that it is open sourcing its robots.txt parser. That means how GoogleBot reads and listens to robots.txt files will be available for any crawler or coder to look at or use.
It is rare for Google to share anything they do in core search with the open source world – it is their secret sauce – but here Google has published it to Github for all to access.
Google wrote they “open sourced the C++ library that our production systems use for parsing and matching rules in robots.txt files. This library has been around for 20 years and it contains pieces of code that were written in the 90’s. Since then, the library evolved; we learned a lot about how webmasters write robots.txt files and corner cases that we had to cover for, and added what we learned over the years also to the internet draft when it made sense.”
It’s been awesome working with @methode and https://t.co/CPJfDQnxn1 on this. I am very happy that it is finally ready to be shared with you all! 😃 https://t.co/gyxvzrFLtp
— Edu Pereda (@epere4) July 1, 2019
If you have SERIOUS ideas about what else could be useful as OSS, leave a comment with the idea and an explanation how would you use that OSS https://t.co/cxxqhI9Nzo
— Gary “鯨理” Illyes (@methode) July 1, 2019
Forum discussion at Twitter.