What is crawl budget?
Crawl budget is how many URLs Googlebot can and wants to crawl on a site. Fast, error-free responses raise the crawl capacity limit.
Crawl budget is the set of URLs Google can and wants to crawl on a site in a given time. Google describes it as the combination of crawl capacity, how much crawling a site can handle, and crawl demand, how much Google wants to crawl. It matters mostly for large sites or sites that change often.
Search Console -> Settings -> Crawl stats
average response time, host status, responses by typeGoogle raises the crawl capacity limit when a site responds quickly and consistently, and its documentation names latency and Time to First Byte. It lowers the limit when responses slow down or return server errors such as 5xx or 429.
- After a deploy, a cold cache sends crawler requests to the origin, with a higher TTFB.
- Cache warming keeps the pages crawlers fetch after a deploy in cache. It does not make Google crawl or index more pages.
- Googlebot crawls primarily from IP addresses in the United States, so the PoPs that serve crawlers may not be the ones that serve your visitors.
- An XML sitemap with accurate
lastmoddates helps crawlers find changed pages.
Related terms
References
- Google: Crawl budget management for large siteshttps://developers.google.com/crawling/docs/crawl-budget
- Google: Overview of Google crawlers and fetchershttps://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers
Warm your cache after every deploy
sitewarm requests the pages in your sitemap after a deploy and reports the cache state the CDN returns for each one, with the vantage point it was measured from.