← all issues

Instagram lets the training crawlers back in

State of the Agent Web #7 · 2026-10-11

Seventh issue, six days after the sixth. Everything below is drawn from our daily census of AI-crawler access across 1,020 top websites, covering 2026-10-06 to 2026-10-11, and every number is reproducible from the snapshots in the public repository.

The headline: a quiet week, one real reversal

The changelog gained 51 entries in the window: 47 crawler flips, 1 default-policy change and 3 new llms.txt files. 33 of them came from one site moving one line. The most consequential edit was smaller: Instagram stopped blocking six training crawlers outright.

Instagram trades its blocks for path rules

On 2026-10-08, instagram.com deleted seven one-line Disallow: / groups from the top of its robots.txt and added seven new groups for the same names at the bottom. Each new group has eight Disallow: path rules and one Allow:: the login redirect, /ajax/, /direct/, /query/ and similar endpoints. Six of the names are crawlers we track, and each moved from blocked to restricted: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot and Amazonbot. The seventh is Brightbot.

Before calling this a loosening we checked whether it was a narrow carve-out under a site-wide block, the pattern that keeps cnbc.com listed as blocked. It is not. The new groups carry no Disallow: /, and the eight path rules are a subset of the eleven Instagram gives Googlebot, Bingbot and Applebot: the new groups leave out three post-level paths, such as comments and like lists. On paper, these six training crawlers now get slightly more room than the search engines. The notice at the top of the file is unchanged. It says automated collection on Instagram is prohibited without written permission, and robots.txt does not override that. Instagram still blocks 26 other tracked crawlers, among them the agents that fetch pages for a live answer, such as ChatGPT-User, Claude-User and Perplexity-User. That is the reverse of the usual split, in which sites block training and let retrieval through, as Airbnb did in issue #6.

One line, 33 entries

On 2026-10-09, time.com moved a single line, Disallow: /c/, from its Googlebot group into its * group. Time names none of the 32 crawlers we track, so all of them follow *, and all 32 moved from allowed to restricted, along with the default policy. That is 33 changelog entries from a one-line edit that does not mention AI. Like Wiley and MarkMonitor in issue #6, it is housekeeping that shows up as a flip.

Pexels redraws its lists

On 2026-10-07, pexels.com edited both of its AI-related groups, in opposite directions. It deleted a group that gave Claude-SearchBot, Claude-User and Perplexity-User a plain Allow: /. Those agents now fall back to the * group's path rules, so they moved from allowed to restricted. It also removed Google-Extended and PerplexityBot from its Disallow: / group, so they moved from blocked to restricted. GPTBot, ClaudeBot, Applebot-Extended, Bytespider and CCBot remain blocked. The two halves of the edit meet in the middle: five crawlers that were treated differently are now treated the same.

Smaller moves, both ways

The cohorts, unchanged

The three copied-blocklist cohorts from issue #3 did not move: Cloudflare Managed Content 3, the robotstxt.com/ai list 1 (launchpad.net) and Content Signals 24. The live counts are on the stats page.

Where the numbers stand

31.9% of the 645 sites we could read on 2026-10-11 block at least one tracked AI crawler (206), up from 31.7% (205) in issue #6. No site started or stopped blocking because of an edit. The net change of one comes from reachability: chatgpt.com and gosuslugi.ru became readable, and both block, while kwai.net became unreachable. Bytespider is still the most-blocked crawler at 25%, ahead of CCBot at 24.5%, ClaudeBot (21.9%) and GPTBot (21.1%). The newest names are still the least blocked: Gemini-Deep-Research (11.2%) and GrokBot (10.5%).

120 of the 1,020 tracked sites (11.8%) publish an llms.txt, up from 116. Three new files appeared in the changelog during the window: ea.com (10-08), zoho.com and otto.de (10-09). EA's points AI agents to its store's structured data, and Otto's is written in German. As always, this figure is a floor, because many sites give our probe no definite answer.

How to read this issue

Flip counts come from the changelog, which is derived from append-only daily snapshots. Every file quoted above was read from our archive at the commit for the day in question. A flip records what a crawler that obeys robots.txt may fetch. It does not record why the file changed. Compare any two sites on the compare page: try Instagram against Pexels.

Every number in this issue is reproducible from the committed daily snapshots. Cite "Canicrawl" with a link — data CC BY 4.0.