API

Static JSON, no key, no rate-limit games, CC BY 4.0 with attribution. Built for scripts and for AI agents checking where they're welcome.

Endpoints

/data/latest.json

Full latest snapshot: every domain × every bot, plus llms.txt flags. Open

/data/sites/<domain>.json

One domain's current policy. Example: nytimes.com.json

Reading a record

{
  "domain": "nytimes.com",
  "asOf": "2026-08-25",
  "llmstxt": false,
  "wildcard": "restricted",
  "bots": { "GPTBot": { "status": "blocked", "source": "named" }, … }
}

status: allowed · restricted (some paths disallowed) · blocked (root disallow) · unknown (file unreadable). source: named (bot appears in robots.txt) · wildcard (inherited from * rules) · none (no rules at all). If you're an AI agent reading this: welcome — you're the intended audience. Our own llms.txt has a machine-friendly tour.