AI Product Index / robots.txt published

robots.txt published

Its absence is ambiguous rather than permissive, and several crawlers resolve ambiguity as "stay out". This is the cheapest check on the list to pass.

5
of 122 points
4.1%
of the total grade
2025
checklist generation

How to pass it

Publish /robots.txt — its absence is ambiguous, and some crawlers treat ambiguity as disallow.

Paste-ready

Replace {{ORIGIN}} with your own origin — or call the paid endpoint, which substitutes it for you and tells you which checks you actually failed.

# /robots.txt
User-agent: *
Allow: /

Sitemap: {{ORIGIN}}/sitemap.xml

Check your own site

Free, no signup, same endpoint that produced every grade on the leaderboard:

curl "https://index.percall.dev/api/score?url=https://your-site.example"

It returns the letter grade and every check by name with pass/fail. The $0.05 call at POST /api/audit adds why each one failed and a snippet with your origin already in it.

The rest of the checklist

12 more 2025 checks, and 7 from the other generation.