AI search crawlers allowed
How to fix it
Allow AI search/retrieval bots. These fetch pages to cite them in answers.
Evidence
Confirmed. Documented by the search engine or AI operator itself.
- https://developers.openai.com/api/docs/bots
- https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- https://docs.perplexity.ai/guides/bots
How it counts
- Weight: 10 (critical) × 1 (evidence tier), within the Crawlability & AI-bot access module, which carries 25% of the overall score.
- Pass counts fully, warn counts half, fail counts nothing.
- A confirmed, high-confidence failure of this check caps the overall score at 49.