State of the Agent Web

A weekly read on how the web's doors are opening and closing to AI — written from our own daily census data.

#3 — Most blocklists are not written, they are moved

2026-09-09

A CDN toggle, a copied community list and one publisher's fleet deployment - three ways an AI blocklist changes without anybody deciding anything. Plus the counter-example: one editor, one bot, one ticket number. And a caveat about our own growing denominator.

#2 — The web is blocking crawlers that barely exist

2026-08-30

Publishers are naming AI agents with almost no public footprint - FirecrawlAgent on 22 independent operators, a dataset name on 26 - while the newest bots stay the least-blocked. Plus four real policy flips, and a correction that removed half our changelog.

#1 — The bigger the brand, the higher the wall

2026-08-25

Founding census: 31% of the top 1,000 sites block an AI crawler — but among household names it's 49%, and among news sites it's 94%.