The sitewarm bot
Last updated:
sitewarm requests the pages of a website and records the cache state the site's CDN returns for each page, such as a hit, a miss or a stale response. If you saw requests with sitewarm in the User-Agent, this page explains what they are and how to stop them.
Why it visited your site
The hosted sitewarm service scans a site only after someone has proven they control it, with a DNS record, a file or a meta tag. Every request the scanning engine sends for the hosted service stays on that verified host. Reading the verification file or meta tag and the sitemap can follow a redirect to another public host.
The scanning engine is also an open-source command-line tool. Anyone can run it from their own machine, and those requests do not come from sitewarm.
What it does
It reads your robots.txt and your sitemap, then sends ordinary GET requests for pages listed in the sitemap, up to the URL limit of the scan. It requests each page with a desktop and a mobile profile, and can request a page more than once to compare the cache before and after.
It reads response headers and the start of each page to report cache state and indexability. It does not log in, submit forms or change anything on your site.
By default it sends at most 5 requests per second and 4 at a time. The hosted service uses those defaults.
How to recognise it
Scan requests carry a Chrome User-Agent string (desktop or mobile) with sitewarm/<version> (+https://github.com/sitewarm/sitewarm) appended.
When the hosted service checks verification or reads robots.txt and the sitemap, it sends the User-Agent sitewarm-verifier/1.0 (+https://sitewarm.dev/bot).
It never presents itself as a search engine crawler, and it does not hide or change its sitewarm token.
robots.txt
It obeys the sitewarm group of your robots.txt, or the * group when there is no sitewarm group. A page that robots.txt disallows is not requested.
A robots.txt that returns a 4xx status allows everything, as RFC 9309 specifies. A 5xx status, a timeout or a network error skips every page on that host.
Reading the sitemap is separate: the hosted service fetches the sitemap files robots.txt lists, or /sitemap.xml when it lists none or cannot be read, and does not apply the allow and disallow rules to them.
The hosted service always checks robots.txt. The open-source tool has an option to skip it, which only the person running the tool can turn on.
To block it on your whole site, add this to your robots.txt:
User-agent: sitewarm
Disallow: /Ask us to stop
Add the robots.txt rule above, or write to support@sitewarm.dev. Include the hostname and, if you can, a timestamp and the source IP address from your logs.
If you own the site and asked for the scan but your firewall blocks it, allow the sitewarm token in the User-Agent. We will never ask you to turn your firewall off.
Source code and security
The scanning engine is open source: https://github.com/sitewarm/sitewarm. How the hosted service handles data is on the security page.