AI Product Index / robots.txt published

robots.txt published

Its absence is ambiguous rather than permissive, and several crawlers resolve ambiguity as "stay out". This is the cheapest check on the list to pass.

5
of 122 points
4.1%
of the total grade
2025
checklist generation

Does this apply to your site?

This check concerns site information and access. Consider your intended audience and publishing policy before making a change.

This score measures coverage of a published technical checklist. It does not measure AI visibility, referrals, content understanding or successful agent tasks. Optional service interfaces may not apply to your site; missing them still affects this fixed checklist.

How to address it when relevant

Publish /robots.txt — its absence is ambiguous, and some crawlers treat ambiguity as disallow.

Example to adapt

Replace {{ORIGIN}} with your own origin — or call the paid endpoint, which substitutes it for you and tells you which checks failed. Review all placeholder names, paths, prices and capabilities before publishing; origin substitution does not verify them.

# /robots.txt
User-agent: *
Allow: /

Sitemap: {{ORIGIN}}/sitemap.xml

Check your own site

Free, no signup, same endpoint that produced every grade on the leaderboard:

curl "https://index.percall.dev/api/score?url=https://your-site.example"

It returns the letter grade and every check by name with pass/fail. The $0.05 call at POST /api/audit adds why each one failed and a snippet with your origin already in it.

The rest of the checklist

12 more 2025 checks, and 7 from the other generation.