APIs & Crawl Budget: Don’t block API requests if they load important content
An attendee asked whether a website should disallow subdomains that are sending API requests, as they seemed to be taking up a lot of crawl budget. They also asked how API endpoints are discovered or used by Google.
John first clarified that API endpoints are normally used by JavaScript on a website. When Google renders the page, it will try to load the content served by the API and use it for rendering the page. It might be hard for Google to cache the API results, depending on your API and JavaScript set-up — which means Google may crawl a lot of the API requests to get a rendered version of your page for indexing.
You could help avoid crawl budget issues here by making sure the API results are cached well and don’t contain timestamps in the URL. If you don’t care about the content being returned to Google, you could block the API subdomains from being crawled, but you should test this out first to make sure it doesn’t stop critical content from being rendered.
John suggested making a test page that doesn’t crawl the API, or uses a broken URL for it, and see how the page renders in the browser (and for Google).
Are web components and Javascript-only content bad for SEO? (Testing is key!)
One user asked whether web components are bad from an SEO perspective. Most web components are implemented in Javascript frameworks and Google can process most forms of Javascript. John also mentions later in the video that sites that aren’t user-friendly if their JS were to be switched off typically aren’t a problem for Googlebot (as long as the relevant links and content are also available within the source code). However, John would always recommend testing a sample of pages using the URL Inspect tool before assuming your chosen JS frameworks are supported.
Image sitemaps can be useful for sites that use lazy loading
When “lazy loading” images on a page in a way that doesn’t include defined image elements, it’s recommended to have back-up in the form of structured data or an image sitemap. That way, Google will know to associate those images with the page even before they’re loaded.
Fetch & Render Tool in GSC Doesn’t Reflect Real Rendering
Getting ‘temporarily unreachable’ messages in the Fetch & Render tool doesn’t reflect how Google is rendering content for its index. Google’s rendering service has a longer cutoff time and uses caching.
Scroll Events Shouldn’t be Used in Isolation to Execute Lazy-loading
Scroll events aren’t always the best solution because they are expensive, users on desktop may resize their window to get more content which wouldn’t trigger a scroll event, and Google doesn’t scroll. Test lazy-loading is working by using Fetch & Render and Intersection Observer.
Text That’s Hidden by Default During Rendering is Fine For Google
Some sites will prevent content from being visible until the page has finished rendering to stop elements from jumping around the screen as they are loaded. This is fine for Google as long as the textual content is in the HTML, but check what Google can see with the mobile-friendly testing tool and Fetch & Render in GSC.
Google’s Cache Isn’t an Accurate View of Googlebot Rendering
The Fetch & Render tool in GSC and the Mobile-friendly Test tool show a more accurate view of how Googlebot is able to render a page than Google’s cache view, as this can easily be broken.
GSC Will Still Have the Option to Fetch Desktop Pages After Mobile-first Indexing
Google still wants website owners to be able to check how their desktop pages appear even if the content is being taken from mobile for mobile-first indexing, so there will still be an option to fetch both page versions in GSC.
Ensure Google is Shown the Same Title When the Page is Fetched & Rendered
If Google is switching the titles between individual URLs, then something with the back-end of the website may be wrong. Google should be able to get the same title when it initially fetches the page as when it is rendered.