How to audit robots.txt and sitemaps
Pass a public https site or origin URL to GET /v1/audit?url=. The tool fetches robots.txt and up to three sitemaps. It is a read-only report, not a penetration test.
Try it: /tools/audit.
curl "https://opencompany-brief.opencompanybrief.workers.dev/v1/audit?url=https://example.com"
The browser form has 2 free runs per IP per UTC day. The API route is $0.01 via x402 when unpaid. Issues called out include a missing robots file, a blanket Disallow: /, and a site with no sitemap URLs.