In order for web pages to be included within search results, they must be in Google’s index. Search engine indexing is a complex topic and dependent on a number of different factors. Our Hangout Notes on indexing cover a range of best practice advice to ensure your website’s important pages are indexed by search engines.

Google Being as Transparent as Possible About Visible Indexing Bugs

May 1, 2019 Source

Google is being as transparent as possible about their recent bugs, because they have had visible impacts on indexing. It is important for John and the team to keep everyone updated on these issues so that webmasters don’t take actions that could be unnecessary or detrimental to their websites.

Google Doesn’t Differentiate Between UGC & Other Content

May 1, 2019 Source

Google doesn’t differentiate between UGC and the other content on a website, so it is important to control how low-quality UGC is dealt with.

Indexing API Currently Limited to Jobs & Live Video Content in Limited Countries

April 18, 2019 Source

The Indexing API is currently limited to jobs and live video content and has only been rolled out to a limited number of countries.

Sites Should No Longer be Impacted by Google Deindexing Bug

April 18, 2019 Source

Google’s deindexing issue was fully fixed on 16th April. If pages still aren’t being indexed, then this will have been caused by a different issue.

Pages with Internally Duplicated Content Are Indexed Separately but Folded Together in Search

April 5, 2019 Source

Google will index pages with duplicate blocks of text separately but will work out which of those pages is most relevant to show for each query and will show just one of them in the search results.

Google Can Index Pages Blocked by Robots.txt

March 22, 2019 Source

Google can index pages blocked in robots.txt if they have internal links pointing to them. In a scenario like this, Google will likely use a title from some the internal links pointing to the page, but the page will rarely be shown in search because Google has very little information about it.

Replace Unnecessary URL Parameters with Fragment Identifiers

March 19, 2019 Source

John recommends replacing unnecessary URL parameters with fragment identifiers because anything after the # is usually dropped for indexing, whereas parameter URLs can be indexed separately.

Mobile-friendly Test Errors Not Necessarily Representative of Google’s Indexing Process

March 8, 2019 Source

The Mobile-friendly test might not pull all page resources because it is a live tool and needs to be time-efficient. However, this isn’t representative of Google’s indexing process which uses the cached version for some of a page’s resources.

Google Folds Together Same Language Country Versions & Swaps Out URLs Dependent on Searcher Location

February 19, 2019 Source

Google identifies different country versions of the same language as duplicates and folds them together for indexing. Google can then swap out the URLs depending on where the search is performed.

Related Topics

Crawling Crawl Budget Crawl Errors Crawl Rate Disallow Sitemaps Last Modified Nofollow Noindex RSS Canonicalization Fetch and Render