AI-bot user agent gets the page

Good practice Moderate effortcrawl.bot_ua_access

Module
Crawlability & AI-bot access
Severity
Warning
Applies to
All pages

How to fix it

Your CDN/WAF may be blocking AI bots (Cloudflare blocks AI crawlers by default on new zones). Check your bot-management settings and server logs for real OAI-SearchBot / PerplexityBot requests, and allow verified AI search bots.

Evidence

Good practice. Widely recommended, but no platform has confirmed it matters for AI answers.

How it counts

Read the full methodology.

Test your page

Runs the Crawlability & AI-bot access module, including this check.

Related checks