State of the Agent Web
A weekly read on how the web's doors are opening and closing to AI — written from our own daily census data. Subscribe via RSS.
#12 — Le Figaro takes OpenAI off its blocklist
2026-11-15
Le Figaro deleted every robots.txt group naming an OpenAI agent, and GPTBot, ChatGPT-User and OAI-SearchBot went from blocked to restricted. TechCrunch dropped three named blocks the same week. Harvard Business Review added fourteen and blocked ten more tracked crawlers. Wiley's file gained a second, longer copy of Cloudflare's managed section that blocks five retrieval agents. 32.5% of 646 sites block at least one AI crawler.
#11 — Delta closes the door it opened
2026-11-08
Delta deleted the “LLM / AI Crawlers - Explicitly Permitted” section it added a week earlier, and 17 crawlers went back from allowed to restricted. Corriere della Sera added 78 named blocklist groups and blocked 17 tracked crawlers. Seeking Alpha un-blocked seven retrieval agents while blocking Amazonbot, GPTBot and Applebot-Extended. Hostinger changed its Content-Signal from ai-train=no to ai-train=yes. 32.6% of 648 sites block at least one AI crawler.
#10 — Semafor puts its blocklist back, Delta opens up
2026-11-01
Semafor restored the robotstxt.com/ai blocklist it deleted in September and re-blocked 31 tracked crawlers. Delta rewrote its robots.txt to name 17 AI crawlers and allow each one everything, the largest loosening since September. Kick switched on Cloudflare's managed block, The Atlantic gave OpenAI's crawlers its ordinary path rules, and Motley Fool blocked three training crawlers. 32.4% of 645 sites block at least one AI crawler.
#9 — Wiley switches on Cloudflare's AI block
2026-10-25
Wiley added Cloudflare's managed AI section and blocked 13 tracked crawlers, its first AI rule. Instagram reverted to its pre-October file, Facebook copied the same blocks, and Dotdash Meredith added Diffbot across four sites. Vercel deleted its Content-Signal line. 31.9% of 645 sites block at least one AI crawler.
#8 — AliExpress names six crawlers and blocks them
2026-10-18
AliExpress went from naming no AI crawler to blocking six, the only site this week that started blocking because of an edit. Cisco deleted the group that invited four training crawlers to its llms.txt and markdown pages, and Pinterest shut out Meta's on-demand fetcher. AT&T joined the Content Signals cohort. 31.8% of 647 sites block at least one AI crawler.
#7 — Instagram lets the training crawlers back in
2026-10-11
Instagram replaced outright blocks on six training crawlers with search-engine-style path rules, while still blocking the agents that fetch pages for a live answer. Time moved one line and produced 33 changelog entries. Pexels loosened two crawlers and tightened three until all five got the same rules, and CNET's longer blocklist caught GrokBot. 31.9% of 645 sites block at least one AI crawler.
#6 — The first open door with a name on it
2026-10-05
Check Point added named Allow groups for nine tracked AI crawlers, training crawlers included. Wiley's path rules came back, and MarkMonitor gained some, which accounts for 64 flips without any decision about AI. Airbnb blocked four training crawlers and kept path rules for retrieval agents. 31.7% of 646 sites block at least one AI crawler.
#5 — Blocklists that left with the plumbing
2026-09-22
Patreon and Kick switched off Cloudflare's managed AI block on the same morning. Wiley's 33 flips came from moving to a new platform, not from opening up. Weather.com's new AI policy says training crawlers stay blocked while the same edit unblocked GPTBot and Google-Extended. The block rate didn't move: 31.9% of 646, again.
#4 — Say one thing, serve another
2026-09-14
Corriere della Sera's new Content Signals header welcomes AI training while its crawler groups block every training bot it names. It and USA Today leave one kind of page open to the AI agents they block: the sponsored ones. USA Today's robots.txt comes in two versions that alternate. github.com names AI crawlers for the first time, to slow them rather than stop them. And a 33-flip day changed no block rate.
#3 — Most blocklists are not written, they are moved
2026-09-09
A CDN toggle, a copied community list and one publisher's fleet deployment - three ways an AI blocklist changes without anybody deciding anything. Plus the counter-example: one editor, one bot, one ticket number. And a caveat about our own growing denominator.
#2 — The web is blocking crawlers that barely exist
2026-08-30
Publishers are naming AI agents with almost no public footprint - FirecrawlAgent on 22 independent operators, a dataset name on 26 - while the newest bots stay the least-blocked. Plus four real policy flips, and a correction that removed half our changelog.
#1 — The bigger the brand, the higher the wall
2026-08-25
Founding census: 31% of the top 1,000 sites block an AI crawler — but among household names it's 49%, and among news sites it's 94%.