Le Figaro takes OpenAI off its blocklist
State of the Agent Web #12 · 2026-11-15
Twelfth issue, a week after the eleventh. Everything below is drawn from our daily census of AI-crawler access across 1,020 top websites, covering 2026-11-09 to 2026-11-15, and every number is reproducible from the snapshots in the public repository.
The headline: 15 tighter, 7 looser, and one missing day
The changelog gained 22 entries in the window, all of them crawler flips: no default-policy changes and no llms.txt entries. Fifteen flips went tighter and 7 went looser, and five sites account for all of them. Only three days recorded anything: 11-10, 11-13 and 11-14.
One caveat applies to the whole issue. We have no snapshot for 2026-11-09; that day's scheduled crawl did not complete, and a missed day cannot be re-crawled. The entries dated 11-10 were therefore compared with 11-08 and may have happened on either day. The changelog page states this under the 11-10 heading, and its coverage line now reads 82 daily snapshots with one day missing.
Le Figaro takes OpenAI off its blocklist
lefigaro.fr keeps a section of its robots.txt headed #LLM. Between 11-08 and 11-10 it deleted every group in that section that names an OpenAI agent, and nothing else: the edit removes 33 lines and adds none, apart from trailing spaces tidied on four other lines. GPTBot had a bare Disallow: /. ChatGPT-User and OAI-SearchBot each had Disallow: / under nine Allow lines for sections such as /voyages, /culture and /bons-plans; we printed both as blocked, with the caveat we attach to every carve-out of that shape. A fourth group, for ChatGPT Agent, which we don't track, went with them. All three tracked crawlers now inherit the User-agent: * group's path rules and read restricted. No OpenAI name is left in the file.
The rest of the section is untouched. Claude-SearchBot, Claude-User and MistralAI-User keep the same carve-out groups, and ClaudeBot, Google-Extended, CCBot and Amazonbot keep their plain blocks. Le Figaro blocked 18 tracked crawlers on 11-08 and blocks 15 now. The archive records what changed, not why.
TechCrunch drops three names
On 2026-11-13 techcrunch.com deleted seven lines from its alphabetical blocklist: the group for ChatGPT-User, the group for Google-Extended, and the Claude-Web line that had shared a group with ClaudeBot. All three moved from blocked to restricted and now follow the * group, which disallows only WordPress admin, API and search paths. The other nine blocks stand, GPTBot and ClaudeBot among them, so TechCrunch still refuses OpenAI's and Anthropic's training crawlers while no longer naming the agent that fetches a page when a ChatGPT user asks for it.
Harvard Business Review names ten more
The same day hbr.org went the other way. Its robots.txt had three AI groups, for GPTBot, CCBot and Google-Extended. On 2026-11-13 it appended fourteen more, each a bare Disallow: /. Ten are crawlers we track, each moving from restricted to blocked: ClaudeBot, anthropic-ai, Applebot-Extended, Bytespider, meta-externalagent, Meta-ExternalFetcher, omgili, omgilibot, Timpibot and AI2Bot. The other four are Ai2Bot-Dolma, SemrushBot-OCOB, Meta-ExternalAds and Meta-WebIndexer. HBR blocked 3 tracked crawlers before the edit and blocks 13 after it, the largest tightening of the window. The edit also added one line to the * group, Allow: /sponsored/. That group has no site-wide disallow, so the line changes no verdict and is not a carve-out of the kind described above.
Wiley gets Cloudflare's block a second time
Issue #9 covered wiley.com putting Cloudflare's managed section at the top of its robots.txt on 2026-10-24, which blocked 13 tracked crawlers and left the retrieval agents on path rules. On 2026-11-14 the file gained 174 lines and lost none: a second copy of the managed section, markers and Content-Signal preamble included, placed above the first. The two copies are not the same. The old one names 32 user agents under “# Training crawlers”. The new one names the same 32 and then adds a part headed “# AI Agents” with 15 more. Five of those are crawlers we track, each moving from restricted to blocked: ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User and Meta-ExternalFetcher. The rest include FirecrawlAgent, Google-Agent, Google-NotebookLM, Kimi-User and Amzn-User. Wiley now blocks 18 tracked crawlers. Both copies were still in the file on 11-15. It is one site, so it proves nothing about the other four members of the Cloudflare cohort, whose archived files do not carry the new part.
Corriere lets Amazonbot in
corriere.it, which added 78 named blocks in issue #11, changed one of its older ones between 11-08 and 11-10: the Amazonbot group went from Disallow: / to Allow: /, matching the Amzn-SearchBot group directly below it. Amazonbot moved from blocked to allowed, and Corriere now blocks 29 tracked crawlers, down from 30. On 11-11 the site added a group for User-agent: __usage_ai-training with Disallow: /, which changes no verdict; see below.
Not a flip
- The training-preference token reaches an eleventh file. Issue #11 counted ten robots.txt files carrying
__usage_ai-training, the IETF draft token that states a preference instead of naming a crawler. Corriere's is the eleventh, and the first from a news publisher. The other ten are unchanged. - Delta did not edit a third time. delta.com, which opened to 17 crawlers on 10-30 and closed again on 11-06, has the same archived robots.txt at the end of this window as at the start, with no change recorded on any day between.
- Seeking Alpha left its blocklist alone. After two rewrites in issue #11, the only change to seekingalpha.com's robots.txt was a
Sitemapline on 11-13. - The token radar. It now counts 2,138 distinct user-agent tokens across the archive, up from 2,135, and 31 AI-looking names we don't track, down from 32. Meta-WebIndexer appears in 46 files, up from 42; HBR's is one of the four.
The cohorts: one joiner
Cloudflare Managed Content stays at 5 and the robotstxt.com/ai list at 2, with the same members as on 11-08. Content Signals rises from 27 to 28: on 2026-11-11 surveymonkey.com put Content-Signal: search=yes, ai-input=yes at the top of its * group. It states no ai-train value at all, which under the signal's own terms neither grants nor restricts training. The 27 earlier members are all still present, checked name by name against the files archived on 11-08, and 11 of the 28 files say ai-train=yes, as before. The live counts are on the stats page.
Where the numbers stand
32.5% of the 646 sites we could read on 2026-11-15 block at least one tracked AI crawler (210), against 32.6% of 648 (211) in issue #11. None of that movement is an edit. Three sites stopped being readable: gosuslugi.ru and livejournal.com, which did not answer, and tplinkcloud.com, which refused us. One became readable: miit.gov.cn, which serves no robots.txt and has gone in and out of reach all series. gosuslugi.ru disallows everything, so its absence takes one site and 32 blocks out of the count; it joined the blocking set by becoming readable in issue #11 and leaves it the same way. Le Figaro, TechCrunch, HBR, Wiley and Corriere were in the blocking set before their edits and are in it after.
Bytespider is still the most-blocked crawler at 25.7%, ahead of CCBot at 25.2%, ClaudeBot (23.1%) and GPTBot (22.0%). The crawlers that lost the most ground are OpenAI's: GPTBot is blocked by 142 sites, down from 144, ChatGPT-User by 92, down from 94, and OAI-SearchBot by 77, down from 79. gosuslugi.ru dropping out of reach accounts for one site in each case; the rest is Le Figaro and TechCrunch, with Wiley's new block of ChatGPT-User offsetting one. Meta-ExternalFetcher is the only tracked crawler blocked by more sites than a week ago, 98 against 97, on HBR and Wiley. The newest names are still the least blocked: Gemini-Deep-Research (11.1%) and GrokBot (10.8%).
132 of the 1,020 tracked sites (12.9%) publish an llms.txt, up from 130, and neither addition is in the changelog. seekingalpha.com gave us a file for the first time on 11-10, 3,860 bytes, and has served it every day since; its last definite answer before that was “no file” on 08-29, with no answer to the probe on any day in between, too long a gap for us to date a publication. time.com published its file on 09-05 and answers our probe only now and then: 11-15 is the sixth day it has, so it is in today's count and may be out of tomorrow's. comcast.com, which dropped out of the count in issue #11, has still given no definite answer. As before, this figure is a floor.
How to read this issue
Flip counts come from the changelog, which is derived from append-only daily snapshots. Every file quoted above was read from our archive at the commit for the day in question. A flip records what a crawler that obeys robots.txt may fetch. It does not record why the file changed. Compare any two sites on the compare page: try TechCrunch against HBR.
Every number in this issue is reproducible from the committed daily snapshots. Cite "Canicrawl" with a link — data CC BY 4.0.