Blocking Googlebot’s IP is The Best Way to Prevent Google From Crawling Your Site While Allowing Other Tools to Access It
If you want to block Googlebot from crawling a staging site, but want to allow other crawling tools access, John recommends whitelisting the IPs of the users and tools you need to view the site but disallowing Googlebot. This is because Google may crawl pages they find on a site, even if they have a noindex tag, or index pages without crawling them, even if they are blocked in robots.txt.
Rollout Stack Migrations Gradually in Stages
John recommends rolling out a migration between different stacks in batches, allowing enough time in between to test and track any negative performance impact before rolling it out to more sections of the site.
Block Staging Sites From Being Crawled by Google
You should block Google from indexing your staging site as it can cause problems. You can block access based on Googlebot’s user agent, or using robots.txt.
Expect to See Rankings Drop When Sites Are Merged
If you merge websites, you should expect to see some rankings drop, as Google doesn’t just add up all the signals, and the pages need to be re-evaluated on the new site. Ensuring you have correctly redirected every page should help the signals be moved over.
Google Re-Evaluates Pages if the URL Changes
If you move a page to a new URL, Google will re-evaluate it and the rankings may change temporarily.
Google Crawl Rate May Take Time to Build for New Websites
Googlebot’s maximum crawl rate is limited to prevent a site slowing down, but Google doesn’t crawl more than it needs to. New websites take take time to build trust to crawl and index new content as quickly as possible.
Splitting or Merging Websites May Change Rankings
If you’re moving a whole website to a different domain, Google can move the signals across for the whole site. But if you’re splitting off part of a site to a new domain, or merging multiple websites, then the rankings may change as Google has to revaluate the sites individually.
When Splitting a Category Page Into Two New Ones, Redirect to One of the New URLs & Update Internal Linking
When splitting a category page into two separate pages, John recommends redirecting the old URL to one of the new pages and then updating internal linking normally within your website’s structure.
Back-end Migrations Shouldn’t Cause Problems for Google if HTML & URLs Stay the Same
Migrating your site to a different CMS or platform in the back-end shouldn’t cause any problems for Google, because in these instances the HTML and URLs will stay the same. John recommends staggering this across site sections to monitor any potential performance decreases.