Subdomains Should Include Separate Robots.txt

December 13, 2019 Source

Robots.txt is per hostname and protocol. If a page contains content from a different subdomain or domain, Google would respect the robots.txt for the main domain for the primary content. For the embedded content, Google would look at the robots.txt for the domain or subdomain that it is from and respect that.


More Website Structure Topics