Microsoft and Yandex announced a new initiative today named IndexNow, a protocol that any search engine can participate in to enable site owners to have their pages and content instantly indexed by the search engine. Currently, Microsoft Bing and Yandex are the two search engines fully participating in the initiative but others are welcome to adopt this open protocol.
IndexNow allows “websites to easily notify search engines whenever their website content is created, updated, or deleted,” Microsoft wrote on its blog. The goal is to make for a “more efficient Internet,” the company said, by reducing the dependency on search engine spiders having to go out into the web and crawl each URL they find. Instead, the goal is for site owners to push these details and URL changes to the search engines directly. “By telling search engines whether an URL has been changed, website owners provide a clear signal helping search engines to prioritize crawl for these URLs, thereby limiting the need for exploratory crawl to test if the content has changed,” Microsoft wrote.
How it works. The protocol is very simple — all you need to do is create a key on your server, and then post a URL to the search engine to notify IndexNow-participating search engines of the change. The steps include:
- Generate a key supported by the protocol using the online key generation tool.
- Host the key in text file named with the value of the key at the root of your web site.
- Start submitting URLs when your URLs are added, updated, or deleted. You can submit one URL or a set of URLs per API call.
Submit one URL is easy as sending a simple HTTP request containing the URL changed and your key.
https://www.bing.com/IndexNow?url=url-changed&key=your-key and the same would work by using https://yandex.com/indexnow?url=url-changed&key=your-key
You can see more details instructions at the Microsoft Bing IndexNow site or the IndexNow protocol web site.
They work together. If you use the Bing method, then both Bing and Yandex (or other participating search engines) will get the update. You do not need to submit to both Bing and Yandex’s URLs, you just need to pick one and all search engines that are part of this initiative will pick up on the change.
The search engines are sharing this IndexNow system, so if you notify one, that search engine will immediately re-ping each other engine in the background, notifying them all. In fact, it is a requirement of IndexNow that any search engines adopting the IndexNow protocol must agree that submitted URLs will be automatically shared with all other participating search engines. To participate, search engines must have a noticeable presence in at least one market, Microsoft told Search Engine Land.
Similar to Bing URL submission API. Is this similar to the Bing URL submission API? Yes, in that the aim is to reduce crawling requirements and improve efficiency. But, it is different in that this is a completely different protocol. If you are using the Bing URL submission API or the Bing content submission API, technically Bing will get your URLs and content changes immediately but these two APIs do not work with the IndexNow protocol, so the other search engines won’t get the changes.
Will these APIs go away if and when the IndexNow initiative becomes more popular? That is unclear. The URL submission API would be somewhat redundant to IndexNow but the content submission API is unique.
Integrations. IndexNow is gaining support among third-party websites, like eBay, LinkedIn, MSN, GitHub and others to integrate with the IndexNow API. Microsoft said many have adopted the Microsoft Bing Webmaster URL submission API and are planning a migration to IndexNow.
Microsoft said it is encouraging all Web Content Management Systems to adopt IndexNow to help their users get their latest website content immediately indexed and minimize crawl load on their websites. In fact, Microsoft provided WordPress code it can use to integrate IndexNow into its CMS. Wix, Duda and others plan to integrate with IndexNow soon as well. CDNs like CloudFlare and Akamai are also working with the IndexNow protocol and so are SEO tools like Botify, OnCrawl and others.
What about Google. We were told that Google is aware of the IndexNow initiative and the company was asked to participate but at this point Google is not an active IndexNow participant.
Why we care. Instant indexing is an SEO’s dream when it comes to giving search engines the most updated content on a site. Google has been very strict about its applications indexing API, used for job postings and livestream content only now. So while it seems Google may not participate in IndexNow in the near future, search engines like Microsoft Bing and Yandex are aiming to push this initiative hard.
The protocol is very simple and it requires very little developer effort to add this to your site, so it makes sense to implement this if you care about speedy indexing. It seems more and more search engines will participate but in terms of the big one, Google, that remains unclear.
About The Author
Barry Schwartz a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry’s personal blog is named Cartoon Barry and he can be followed on Twitter here.