A Google spokesperson has confirmed that the search company will be testing the new IndexNow protocol first introduced by Microsoft Bing and Yandex a few weeks ago. Google said while its crawling efforts are efficient, Google will test to see if it can improve its overall sustainability efforts by leveraging the IndexNow protocol.
Google’s statement. Google told Search Engine Land “we take a holistic approach to sustainability at Google, including efforts to develop the most efficient and accurate index of the web.” “We have been carbon neutral since 2007 and we are on pace to be carbon-free by 2030 across both our data centers and campuses. We’re encouraged by work to make web crawling more efficient, and we will be testing the potential benefits of this protocol,” a Google spokesperson added.
What is IndexNow. IndexNow provides a method for websites owners to instantly inform search engines about latest content changes on their website. IndexNow is a simple ping protocol so that search engines know that a URL and its content has been added, updated, or deleted, allowing search engines to quickly reflect this change in their search results.
How it works. The protocol is very simple — all you need to do is create a key on your server, and then post a URL to the search engine to notify IndexNow-participating search engines of the change. The steps include:
- Generate a key supported by the protocol using the online key generation tool.
- Host the key in text file named with the value of the key at the root of your web site.
- Start submitting URLs when your URLs are added, updated, or deleted. You can submit one URL or a set of URLs per API call.
Submit one URL is easy as sending a simple HTTP request containing the URL changed and your key.
https://www.bing.com/IndexNow?url=url-changed&key=your-key and the same would work by using https://yandex.com/indexnow?url=url-changed&key=your-key
How does it work with Google. We asked Google if https://www.google.com/IndexNow?url=url-changed&key=your-key would work with the search engine and we await to hear back. The protocol works that if you submit it to Bing or Yandex and, one would assume now Google, that one submission to any of those search engines would send them to all the search engines participating in this protocol. So submitting it to Bing, would essentially also submit it to Yandex, Google and other participating search engines.
Google changed its mind? When this first was introduced, we were told that Google is aware of the IndexNow initiative and the company was asked to participate but that Google did not. I guess Google had a change of heart?
Google crawling is efficient. Google’s crawling mechanism is supposedly very efficient and Google has continued to improve its crawling efficiency. Last year Googlebot began supporting HTTP/2, the next generation of the fundamental data transfer protocol of the web. HTTP/2 is significantly more efficient than its predecessors and it saves resources for both Google and web sites. Google uses HTTP/2 in over half of all crawls.
Why we care. Like we said before, instant indexing is an SEO’s dream when it comes to giving search engines the most updated content on a site. Google has been very strict about its applications indexing API, used for job postings and livestream content only now. Google’s change of heart here can be an exciting change for SEOs and site owners.
The protocol is very simple and it requires very little developer effort to add this to your site, so it makes sense to implement this if you care about speedy indexing.
About The Author
Barry Schwartz a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry’s personal blog is named Cartoon Barry and he can be followed on Twitter here.