About our bot
When someone scans a page with GeoPageSpeed, our server fetches that page and a few files from the same site. This page explains what it fetches and how to block it.
How it identifies itself
User agent: GeoPageSpeedBot/1.0. robots.txt token: GeoPageSpeedBot.
What it fetches
Only when a person asks for a scan, never on a schedule. For each scan, at most:
- the scanned page (following up to 5 redirects);
/robots.txt,/llms.txtand the sitemap (at most two sitemap files);- an
http://request to check that it redirects to HTTPS; - one extra request for the page with an OAI-SearchBot user agent, to see whether your CDN treats AI crawlers differently. It is sent only when your robots.txt allows OAI-SearchBot for that page, and the report always says it is “a request identifying as OAI-SearchBot”, not the real crawler.
That is 12 requests at most, with a 4 s timeout each, a 7 s budget per scan and a 5 MB size cap. Repeat scans of the same page within 15 minutes are served from memory, and each site can be scanned at most 30 times per hour in total. The bot never runs JavaScript, submits forms or follows links beyond the files listed above.
How to block it
If robots.txt disallows GeoPageSpeedBot for a page, we don't fetch the page: the report then shows only the robots.txt results. Add this to your robots.txt:
User-agent: GeoPageSpeedBot
Disallow: /