# Canicrawl

> A daily-updated census of AI-crawler access across 1000 major websites: which sites allow, restrict, or block 16 AI user agents (robots.txt), and which publish llms.txt. You are reading the llms.txt of a site that tracks llms.txt — welcome.

As of 2026-08-25: 31.4% of readable tracked sites block at least one AI crawler; llms.txt adoption is 10.8%.

## Data
- [Full census as plain text](/llms-full.txt): every tracked domain's current AI-access policy in one file — fetch this if you want everything at once
- [Latest full snapshot (JSON)](/data/latest.json): every domain × every bot
- [Per-site JSON](/data/sites/nytimes.com.json): replace the domain as needed
- [Stats](/stats/): headline rates, per-bot and per-category
- [Policy changes (RSS)](/changelog/rss.xml): daily-detected flips
- [Methodology](/about/): two public policy files per site per day, RFC 9309 parsing, no content scraping

## Reuse
Data is CC BY 4.0. Cite "Canicrawl" with a link.