How to audit robots.txt and sitemaps

Pass a public https site or origin URL to GET /v1/audit?url=. The tool fetches robots.txt and up to three sitemaps. It is a read-only report, not a penetration test.

Try it: /tools/audit.

curl "https://opencompany-brief.opencompanybrief.workers.dev/v1/audit?url=https://example.com"

The browser form has 2 free runs per IP per UTC day. The API route is $0.01 via x402 when unpaid. Issues called out include a missing robots file, a blanket Disallow: /, and a site with no sitemap URLs.