Gary Illyes from Google added to the big crawl budget article he published on the webmaster blog in January 2017 a new question and answer. It basically says that if you disallow a URL in your robots.txt file, then those URLs do not affect your crawl budget.

He added this question and answer:

Q: Do URLs I disallowed through robots.txt affect my crawl budget in any way?

A: No, disallowed URLs do not affect the crawl budget.

He posted last night that he added the Q&A for a former Googler, Pierre Far:

More details:


Forum discussion at Twitter.

Original Source

Advertisement