1.2. Launching Evergreen Googlebot
In May 2019, Google announced that they updated Chromium for rendering to the latest version and that they would continue to keep it up to date. It was huge:
The yellow shading shows CPU usage during the scripting process. For several seconds, it maxes out.
The same applies to Google and other search engines, which need to crawl and index billions of pages. Even a well-equipped server infrastructure needs to make a significant effort to render all pages on the web effectively. Because of this, Google delays rendering in many cases. Further crawling and content indexing were also postponed. Based on our observations, indexing might be delayed for weeks or even months.
2.1. 2019: Google Still Indexes Pages in Two Waves
Google says that they render almost all pages on the web. If they encounter a brand-new website, they try to render the page and compare the raw HTML with the most recently rendered version to see if there are significant differences between the source code and DOM. If they notice that much content and many new links appear on the site after rendering, it gives a clear hint to them that it should go through two waves of indexing. As a result, rendering and indexing of the rest of the subpages might be significantly deferred. However, many experiments we conducted show that the rendering process is not smooth and may lead to significant delays in indexing.
Google claims that they are working on closer indexing and rendering, so we can expect delays to get smaller and smaller. That would be a big thing that changes SEO as we know it, so I’m keeping my fingers crossed for it.
3. Our Experiment to Check if Google Is Catching Up
Note: The bars in the dark green show the sample where indexing is above 80%. If the indexing is below 60% of the whole sample, we mark it in yellow.
Now, let’s see how it looks after seven days.
How does it look after two weeks?
After additional seven days, the percentage of indexed URLs is almost the same, meaning that Google probably reached the natural level of indexing for the domains. Google doesn’t index 100% of URLs we submit due to low quality of the page or other technical SEO issues.
For many websites, fast indexing is crucial. If you have a website with offers that change every minute, any delay in indexing may impact the overall SEO performance of your site. We can see that Google doesn’t index 100% of the URLs you submit in the sitemap on the same day.
- Some dynamic rendering or (ideally) server-side rendering should be in use if you want to make sure your website won't suffer because of Google’s two-waves approach to indexing. If the website contains content that needs fast or instant indexing, you can’t rely on Client-Side Rendering.
Remember that Google is not the only search engine. Even if Google and Bing (Bingbot goes evergreen too) can render pages like a modern browser, we can’t expect the same of Yahoo or social media channels. Even if Google catches up in the future, keep in mind that some share of the traffic to your site may come from other search engines that are not well-skilled at rendering.