I saw a discussion on Twitter this morning about the idea of having a feature in Robots.txt that would block both crawling AND indexing. It started with this tweet by Christian Thurston (@ct_oz):
“Hi John [Mueller], has Google considered making it so that the robots.txt file doesn’t just block crawling, but also blocks indexation? To quote @willcritchlow: “I can’t see many situations where I want to block crawling but don’t want to block indexing”.
“That would be a significant change in expectations (and yes, we do think about these things regardless). Do you have some examples where this would cause a visible improvement in search?” Mueller responded. “I’d like to avoid adding more directives. I’m still not aware of common issues caused by this documented functionality … SEOs worry about indexing, but usually these URLs only rank for site:-queries (or if there isn’t other, better content on the site), so it feels artificial?”
With over 20 years of experience in SEO, “I have never encountered a situation where a publisher wanted to have a page indexed that they block for crawling in robots.txt. Not even once have I seen that,” I tweeted in response. “It’s common practice for me to educate people that they have a choice: (1) block crawling, or: (2) prevent indexation, when what they want to do is both. Note: definitely more of an issue for larger sites where crawl budget is an issue.”
Will Critchlow of Search Pilot agreed, “100% agree. I can see the conservatism of not changing a long-standing standard, but I have never seen, and can’t think of a situation where you’d want to block crawling but allow indexation.”
But what do YOU think? Would it be helpful to have a feature in Robots.txt that allowed you to specify that pages you don’t want to have indexed? Take our quick three-question poll below and let us know what you think.
Please keep in mind that if you vote yes for doing this that you would need to accept ALL the risk that at some point your dev team might misapply the capability and NoIndex pages that you don’t want them to.
Opinions expressed in this article are those of the guest author and not necessarily Search Engine Land. Staff authors are listed here.
New on Search Engine Land
About The Author
Eric Enge is General Manager of Perficient Digital, a full-service, award-winning digital agency. Previously Eric was the founder and CEO of Stone Temple, also an award-winning digital marketing agency, which was acquired by Perficient in July 2018. He is the lead co-author of The Art of SEO, a 900+ page book that’s known in the industry as “the bible of SEO.” In 2016, Enge was awarded Search Engine Land’s Landy Award for Search Marketer of the Year, and US Search Awards Search Personality of the Year. He is a prolific writer, researcher, teacher and a sought-after keynote speaker and panelist at major industry conferences.