AI Product Index / AI crawlers not blocked

AI crawlers not blocked

A Disallow that names GPTBot or ClaudeBot is a decision to be absent from the answers those models give. Sometimes that is the right call — but it should be a decision, and it is very often inherited from a default nobody chose.

10
of 122 points
8.2%
of the total grade
2025
checklist generation

How to pass it

Remove the Disallow: / rules for the AI user-agents you want to be cited by, or name them with Allow: /.

Paste-ready

Replace {{ORIGIN}} with your own origin — or call the paid endpoint, which substitutes it for you and tells you which checks you actually failed.

# /robots.txt — allow the AI crawlers explicitly. A bare
# "User-agent: * / Disallow: /" hides you from every assistant.
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Check your own site

Free, no signup, same endpoint that produced every grade on the leaderboard:

curl "https://index.percall.dev/api/score?url=https://your-site.example"

It returns the letter grade and every check by name with pass/fail. The $0.05 call at POST /api/audit adds why each one failed and a snippet with your origin already in it.

The rest of the checklist

12 more 2025 checks, and 7 from the other generation.