← all issues

Say one thing, serve another

State of the Agent Web #4 · 2026-09-14

Fourth issue. Everything below is drawn from our daily census of AI-crawler access across 1,020 top websites, and every number is reproducible from the snapshots in the public repository. This issue covers 2026-09-09 to 2026-09-14.

The headline: say one thing, serve another

Issue #3 was about blocklists that change without anybody deciding anything. This week's changes run the other way: files that plainly were edited by someone, and that still say one thing and do another. A usage header says one thing while the rules beneath it do the opposite. Pages are closed to AI agents except for the sponsored ones. A robots.txt names AI crawlers in order to slow them, not stop them. And one file comes in two versions, and which one you get depends on the day you ask.

Two Content Signals headers, pointing opposite ways

Cloudflare's Content Signals line is a machine-readable statement of preference in three parts: search, ai-input (using a page to answer a question) and ai-train. It blocks nothing by itself. Two sites in our panel set it this week, with opposite values.

On 2026-09-10, hostinger.com changed its header from ai-train=yes to search=yes, ai-input=yes, ai-train=no. The header appears in its * group and again in a named group that includes GPTBot, ClaudeBot and Google-Extended. It enforced that preference with a hard block on just two crawlers, CCBot and Bytespider, under a comment calling them crawlers that only collect content for model training. Both flipped from restricted to blocked that morning. Everyone else, GPTBot and ClaudeBot included, can still crawl exactly what they could before. Hostinger asks the big names not to train, and blocks only the two it thinks do nothing else.

The next day, 2026-09-11, corriere.it added search=yes, ai-train=yes, ai-input=no to its * group. That is the reverse of Hostinger's header, and Corriere's crawler groups go further than it does. It opened the search bots completely (OAI-SearchBot and PerplexityBot went from restricted to allowed). It blocked the agents that fetch a page because a user asked about it (ChatGPT-User and Claude-Web went from restricted to blocked). That much matches search=yes, ai-input=no. The third value does not: ai-train=yes sits in the same file as Disallow: / for GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, anthropic-ai and Bytespider, all unchanged since our census began.

Read strictly, there is no contradiction. A Content-Signal line applies only to the group it sits in, and Corriere's sits in *, which covers the crawlers it did not name. So Corriere welcomes training by crawlers it has never heard of, and forbids it to every training crawler it has. Corriere is also the first site already in our panel to take up the header since we started counting it on 2026-09-02.

The pages a blocked AI agent may still read

Corriere's new ChatGPT-User and Claude-Web groups are not quite total blocks. Each is Disallow: / followed by one exception, Allow: /nativeadv/. That path is, by its name, the native-advertising section, and the same exception was added to Corriere's AppleBot group that day. The AI agents Corriere turned away may still read its ads.

usatoday.com has had the same shape since before this issue began. Its groups for GPTBot, ChatGPT-User, OAI-SearchBot, Google-Extended and PerplexityBot, plus an agent called ChatGPT Operator, are Disallow: / plus a short list of Allow: exceptions. The exceptions cover contributor content, /money/money-management/, shopping, and individually dated URLs under /story/sponsor-story/.

We classify all of these as blocked, because the permitted slice is a handful of paths against an entire site. The exceptions are listed here so you can judge that call yourself: a Disallow: / with sponsored pages carved out is not the same statement as a plain Disallow: /.

The file that comes in two versions

USA Today's exception list is also not one list. Our archive holds two versions of the file that differ in exactly one spot, repeated in each of those six groups. Version A allows /branded-story/. Version B allows two sponsor stories dated 2026-08-21 instead. Our daily read returned version B every day from 2026-08-27 to 2026-09-05. Since then it has returned A, A, B, A, A, B, A, B, A: seven switches in nine days, with nothing else in the file changing.

A file that switches back and forth like that looks like two servers holding two builds, not an editor changing their mind every morning. We cannot tell which version is meant to be current, and a crawler that read the file this morning could not either. Both versions classify identically, so no verdict moved and no changelog entry was emitted. That is exactly why we read the raw files rather than trusting the verdicts alone.

github.com names AI crawlers for the first time: a courtesy lane, not a wall

Until 2026-09-12, github.com said nothing about AI crawlers specifically, so they got the same rules as everyone. That morning it gave GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai and PerplexityBot a group of their own. The group asks for Crawl-delay: 1 and allows the homepage plus eleven marketing paths (/about, /pricing, /enterprise, /features, /security and /mcp among them). Then it repeats the * group's repository-path disallows exactly, all 57 of them and not one more. Separately, Bytespider got Disallow: /.

Only Bytespider's verdict moved, from restricted to blocked. For the five named crawlers the edit takes nothing away. It asks them to slow down and points them at the pages GitHub most wants read. Naming a crawler usually means blocking it; here it meant rolling out the welcome mat and asking it to wipe its feet.

A 33-flip day that changed no block rate

This morning's changelog carries 33 entries, and every one is samsungcloud.com: all 32 tracked crawlers and the default * policy moved from allowed to restricted. The edit behind them has nothing to do with AI. The file went from an empty Disallow:, which permits everything, to Allow: / plus Disallow: /api/. Every crawler, AI or not, lost access to one API path.

Nothing was blocked, so the headline rate below does not move. We could have left this entry out, but it shows what a flip count actually measures. A changelog counts edits multiplied by the crawlers they touch, and one housekeeping line can outnumber a week of real decisions. Weigh each flip by what changed, not by how many rows it produced.

The cohorts, one more data point

We re-counted the three copied-blocklist cohorts from issue #3 at every daily commit in this window:

Hostinger's change was a value flip inside a header it already carried, so it does not change the count.

Where the numbers stand

31.9% of the 646 sites we could read on 2026-09-14 block at least one tracked AI crawler, against 31.6% of 649 in issue #3.

The per-bot order holds. CCBot is the most-blocked at 25.2%, then Bytespider (25.1%), ClaudeBot (22.4%) and GPTBot (21.7%). The newest names are still the least blocked: Gemini-Deep-Research (11.1%) and GrokBot (10.2%).

112 of the 1,020 tracked sites (11%) publish an llms.txt, including two first seen on 2026-09-11. ring.com (7,398 bytes) had answered “no file” every day before that. checkpoint.com (12,490 bytes) had given our probe no answer for the twelve days before, so its file may be older than our first sighting. Treat that 11% as a floor. Only 622 domains gave our probe a definitive answer, and among those the rate is 18%.

How to read this issue

Cohort counts come from matching the archived robots.txt of every panel domain against the marker each cohort leaves, recomputed at each day's commit rather than remembered. They are published daily on the stats page as “Blocklists nobody wrote.”

Flip counts come from the changelog, which is derived from append-only daily snapshots; we correct the derivation, never the snapshots. USA Today's version history was read from the archived file at every daily commit since 2026-08-25.

Every named site's full record is on its own page, and any two can be put side by side on the compare page.

Correction: issue #3 described its 11% llms.txt figure as a share of the domains that answered our probe. It is a share of all tracked domains, which is why we call it a floor.

Every number in this issue is reproducible from the committed daily snapshots. Cite "Canicrawl" with a link — data CC BY 4.0.