State of the Agent Web

A weekly read on how the web's doors are opening and closing to AI — written from our own daily census data. Subscribe via RSS.

#9 — Wiley switches on Cloudflare's AI block

2026-10-25

Wiley added Cloudflare's managed AI section and blocked 13 tracked crawlers, its first AI rule. Instagram reverted to its pre-October file, Facebook copied the same blocks, and Dotdash Meredith added Diffbot across four sites. Vercel deleted its Content-Signal line. 31.9% of 645 sites block at least one AI crawler.

#8 — AliExpress names six crawlers and blocks them

2026-10-18

AliExpress went from naming no AI crawler to blocking six, the only site this week that started blocking because of an edit. Cisco deleted the group that invited four training crawlers to its llms.txt and markdown pages, and Pinterest shut out Meta's on-demand fetcher. AT&T joined the Content Signals cohort. 31.8% of 647 sites block at least one AI crawler.

#7 — Instagram lets the training crawlers back in

2026-10-11

Instagram replaced outright blocks on six training crawlers with search-engine-style path rules, while still blocking the agents that fetch pages for a live answer. Time moved one line and produced 33 changelog entries. Pexels loosened two crawlers and tightened three until all five got the same rules, and CNET's longer blocklist caught GrokBot. 31.9% of 645 sites block at least one AI crawler.

#6 — The first open door with a name on it

2026-10-05

Check Point added named Allow groups for nine tracked AI crawlers, training crawlers included. Wiley's path rules came back, and MarkMonitor gained some, which accounts for 64 flips without any decision about AI. Airbnb blocked four training crawlers and kept path rules for retrieval agents. 31.7% of 646 sites block at least one AI crawler.

#5 — Blocklists that left with the plumbing

2026-09-22

Patreon and Kick switched off Cloudflare's managed AI block on the same morning. Wiley's 33 flips came from moving to a new platform, not from opening up. Weather.com's new AI policy says training crawlers stay blocked while the same edit unblocked GPTBot and Google-Extended. The block rate didn't move: 31.9% of 646, again.

#4 — Say one thing, serve another

2026-09-14

Corriere della Sera's new Content Signals header welcomes AI training while its crawler groups block every training bot it names. It and USA Today leave one kind of page open to the AI agents they block: the sponsored ones. USA Today's robots.txt comes in two versions that alternate. github.com names AI crawlers for the first time, to slow them rather than stop them. And a 33-flip day changed no block rate.

#3 — Most blocklists are not written, they are moved

2026-09-09

A CDN toggle, a copied community list and one publisher's fleet deployment - three ways an AI blocklist changes without anybody deciding anything. Plus the counter-example: one editor, one bot, one ticket number. And a caveat about our own growing denominator.

#2 — The web is blocking crawlers that barely exist

2026-08-30

Publishers are naming AI agents with almost no public footprint - FirecrawlAgent on 22 independent operators, a dataset name on 26 - while the newest bots stay the least-blocked. Plus four real policy flips, and a correction that removed half our changelog.

#1 — The bigger the brand, the higher the wall

2026-08-25

Founding census: 31% of the top 1,000 sites block an AI crawler — but among household names it's 49%, and among news sites it's 94%.