Skip to content
SeoLegacy.Org
Menu
  • About Us
  • Privacy Policy
  • Marketing
  • Seo tips
  • General Seo
  • News
  • Analytics & conversion
  • Link building
  • PPC
Menu

Google changes documentation on crawling and robots.txt

Posted on July 1, 2022

Google has updated the crawl status report documentation. According to this, if robots.txt is not available, Google will stop crawling after 30 days if the homepage of a website cannot be reached.

There are a number of interesting changes to Google’s documentation on the Search Console crawl status report. It is now even clearer how Google reacts in the event of an unreachable robots.txt and what the consequences are for crawling. There are also some important changes in how Googlebot responds. Spotted and shared on Twitter by SEO Brodie Clark:

The most important changes in the documentation are summarized below:

If Google has received a successful response to the robots.txt request that is more recent than 24 hours, then Google uses this robots.txt for crawling (the period of 24 hours has been added). The note was also added that a 404 when retrieving robots.txt is considered a successful retrieval. This is treated as if there were no robots.txt. Google can use it to crawl any URL on the website.

The following periods are also new: If robots.txt is not successfully retrieved, Google will stop crawling the website for 12 hours. After 12 hours and up to 30 days, Google will use the last successfully retrieved robots.txt to crawl. After 30 days, Google will crawl the entire site if the home page is available and act as if there is no robots.txt. If the site’s home page is not available, Google will stop crawling the site. However, Google will continue to attempt to retrieve the robots.txt on a regular basis.

Previously it was said that Google would crawl a website with unavailable robots.txt after 30 days if most of the website’s URLs were available. The last successfully retrieved robots.txt would be used.

In contrast to a 404, a 403 does not count as a successful retrieval of a robots.txt. This also and especially applies to 500 errors.

40

SHARES
Share on Facebook
Tweet
Follow us

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Recent Posts

  • The Google March 2023 General Core Update Has Been Released.
  • Should You Use The Disavow Tool – Now, After A Decade, There Is An Answer
  • Enhanced CPC Will Replace Manual CPC Campaigns On The Microsoft Audience Network.
  • Survey For 2023: Ranking Factors For Local Searches
  • How YMYL SEO’s Success Can Be Fueled By E-A-T Content And Link Building

Recent Comments

  1. joker123 on Top 7 Survey and Quiz Plugins for WordPress
  2. sbobet on Top 7 Survey and Quiz Plugins for WordPress
  3. yukslot88 on Top 7 Survey and Quiz Plugins for WordPress
  4. sv388 on Top 7 Survey and Quiz Plugins for WordPress
  5. 사설토토 on 4 Easy But Powerful SEO Tips to Boost Traffic to Your Website
©2023 SeoLegacy.Org | Design: Newspaperly WordPress Theme

We are using cookies to give you the best experience on our website.

We use tracking technologies like cookies to keep track of user activity on our Service and store some information.

Cookies are small data files that may contain an anonymous unique identifier. From a website, cookies are sent to your browser and stored on your device. Beacons, tags, and scripts are other tracking technologies that are utilized to collect and track data, as well as to enhance and analyze our Service.

You have the ability to tell your browser when a cookie is being sent or to reject all cookies.  However, if you do not accept cookies, you may not be able to use some portions of our Service.

 

You can find out more about which cookies we are using or switch them off in settings.

Powered by  GDPR Cookie Compliance
Privacy Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.

Strictly Necessary Cookies

Strictly Necessary Cookie should be enabled at all times so that we can save your preferences for cookie settings.

If you disable this cookie, we will not be able to save your preferences. This means that every time you visit this website you will need to enable or disable cookies again.