AI Product Index / AI crawlers not blocked

AI crawlers not blocked

A Disallow that names GPTBot or ClaudeBot is a decision to be absent from the answers those models give. Sometimes that is the right call — but it should be a decision, and it is very often inherited from a default nobody chose.

10
of 122 points
8.2%
of the total grade
2025
checklist generation

Does this apply to your site?

This check concerns site information and access. Consider your intended audience and publishing policy before making a change.

This score measures coverage of a published technical checklist. It does not measure AI visibility, referrals, content understanding or successful agent tasks. Optional service interfaces may not apply to your site; missing them still affects this fixed checklist.

How to address it when relevant

Remove the Disallow: / rules for the AI user-agents you want to be cited by, or name them with Allow: /.

Example to adapt

Replace {{ORIGIN}} with your own origin — or call the paid endpoint, which substitutes it for you and tells you which checks failed. Review all placeholder names, paths, prices and capabilities before publishing; origin substitution does not verify them.

# /robots.txt — allow the AI crawlers explicitly. A bare
# "User-agent: * / Disallow: /" hides you from every assistant.
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Check your own site

Free, no signup, same endpoint that produced every grade on the leaderboard:

curl "https://index.percall.dev/api/score?url=https://your-site.example"

It returns the letter grade and every check by name with pass/fail. The $0.05 call at POST /api/audit adds why each one failed and a snippet with your origin already in it.

The rest of the checklist

12 more 2025 checks, and 7 from the other generation.