The state of the walls
Headline numbers from the latest crawl of 1000 tracked sites (652 with readable policy). Snapshot: 2026-08-25
31.4%
block ≥1 AI crawler
52
run default-deny robots.txt (unnamed crawlers get nothing)
10.8%
publish llms.txt
CCBot
most-blocked AI bot (25.5%)
Block rate by bot
Block rate by site category
| Category | Sites | Blocking ≥1 bot | Rate |
|---|---|---|---|
| news | 33 | 31 | 93.9% |
| food | 6 | 5 | 83.3% |
| entertainment | 13 | 10 | 76.9% |
| social | 16 | 11 | 68.8% |
| realestate | 3 | 2 | 66.7% |
| sports | 9 | 6 | 66.7% |
| reference | 16 | 10 | 62.5% |
| health | 5 | 3 | 60% |
| jobs | 4 | 2 | 50% |
| auto | 5 | 2 | 40% |
| ecommerce | 15 | 5 | 33.3% |
| finance | 17 | 5 | 29.4% |
| education | 7 | 2 | 28.6% |
| top1k | 448 | 106 | 23.7% |
| travel | 12 | 2 | 16.7% |
| tech | 34 | 3 | 8.8% |
| classifieds | 1 | 0 | 0% |
| government | 8 | 0 | 0% |
Citing these numbers
Data is CC BY 4.0. Cite as "Canicrawl, AI crawler access census, 2026-08-25" with a link. Raw data: latest.json. Methodology: about. Every figure is reproducible from the committed daily snapshots.