Changelog
Every AI-policy flip we observe lands here, newest first — detected by comparing daily snapshots. Subscribe via RSS.
Coverage: 82 daily snapshots from 2026-08-25 to 2026-11-15. Missing: 2026-11-09 — the scheduled crawl did not complete, and a missed day cannot be re-crawled after the fact.
2026-11-14
- wiley.com began blocking Meta-ExternalFetcher (restricted → blocked)
- wiley.com began blocking MistralAI-User (restricted → blocked)
- wiley.com began blocking Perplexity-User (restricted → blocked)
- wiley.com began blocking Claude-User (restricted → blocked)
- wiley.com began blocking ChatGPT-User (restricted → blocked)
2026-11-13
- hbr.org began blocking AI2Bot (restricted → blocked)
- hbr.org began blocking Meta-ExternalFetcher (restricted → blocked)
- hbr.org began blocking Timpibot (restricted → blocked)
- hbr.org began blocking omgilibot (restricted → blocked)
- hbr.org began blocking omgili (restricted → blocked)
- hbr.org began blocking anthropic-ai (restricted → blocked)
- hbr.org began blocking meta-externalagent (restricted → blocked)
- hbr.org began blocking Bytespider (restricted → blocked)
- hbr.org began blocking Applebot-Extended (restricted → blocked)
- hbr.org began blocking ClaudeBot (restricted → blocked)
- techcrunch.com stopped blocking Claude-Web (blocked → restricted)
- techcrunch.com stopped blocking Google-Extended (blocked → restricted)
- techcrunch.com stopped blocking ChatGPT-User (blocked → restricted)
2026-11-10
Covers 2026-11-09 too: no snapshot exists for that day, so these changes were detected against the last snapshot before it and may have happened on either day.
- corriere.it stopped blocking Amazonbot (blocked → allowed)
- lefigaro.fr stopped blocking OAI-SearchBot (blocked → restricted)
- lefigaro.fr stopped blocking ChatGPT-User (blocked → restricted)
- lefigaro.fr stopped blocking GPTBot (blocked → restricted)
2026-11-08
- seekingalpha.com began blocking Applebot-Extended (restricted → blocked)
- seekingalpha.com began blocking GPTBot (restricted → blocked)
2026-11-06
- delta.com changed access for Meta-ExternalFetcher (allowed → restricted)
- delta.com changed access for YouBot (allowed → restricted)
- delta.com changed access for Diffbot (allowed → restricted)
- delta.com changed access for cohere-ai (allowed → restricted)
- delta.com changed access for anthropic-ai (allowed → restricted)
- delta.com changed access for DuckAssistBot (allowed → restricted)
- delta.com changed access for Amazonbot (allowed → restricted)
- delta.com changed access for meta-externalagent (allowed → restricted)
- delta.com changed access for Bytespider (allowed → restricted)
- delta.com changed access for CCBot (allowed → restricted)
- delta.com changed access for PerplexityBot (allowed → restricted)
- delta.com changed access for Applebot-Extended (allowed → restricted)
- delta.com changed access for Google-Extended (allowed → restricted)
- delta.com changed access for ClaudeBot (allowed → restricted)
- delta.com changed access for OAI-SearchBot (allowed → restricted)
- delta.com changed access for ChatGPT-User (allowed → restricted)
- delta.com changed access for GPTBot (allowed → restricted)
2026-11-05
- lemonde.fr began blocking GrokBot (restricted → blocked)
- seekingalpha.com stopped blocking Meta-ExternalFetcher (blocked → restricted)
- seekingalpha.com stopped blocking YouBot (blocked → restricted)
- seekingalpha.com stopped blocking MistralAI-User (blocked → restricted)
- seekingalpha.com stopped blocking DuckAssistBot (blocked → restricted)
- seekingalpha.com began blocking Amazonbot (restricted → blocked)
- seekingalpha.com stopped blocking Perplexity-User (blocked → restricted)
- seekingalpha.com stopped blocking Claude-User (blocked → restricted)
- seekingalpha.com stopped blocking ChatGPT-User (blocked → restricted)
- digitalocean.com published an llms.txt — a welcome mat for AI readers
- apnews.com began blocking Perplexity-User (restricted → blocked)
2026-11-04
- hostinger.com stopped blocking CCBot (blocked → restricted)
- corriere.it began blocking PanguBot (restricted → blocked)
- corriere.it began blocking GrokBot (restricted → blocked)
- corriere.it began blocking DeepSeekBot (restricted → blocked)
- corriere.it began blocking AI2Bot (restricted → blocked)
- corriere.it began blocking ImagesiftBot (restricted → blocked)
- corriere.it began blocking omgilibot (restricted → blocked)
- corriere.it began blocking omgili (restricted → blocked)
- corriere.it began blocking Diffbot (restricted → blocked)
- corriere.it began blocking cohere-training-data-crawler (restricted → blocked)
- corriere.it began blocking cohere-ai (restricted → blocked)
- corriere.it began blocking MistralAI-User (restricted → blocked)
- corriere.it began blocking DuckAssistBot (restricted → blocked)
- corriere.it began blocking Perplexity-User (restricted → blocked)
- corriere.it began blocking PerplexityBot (allowed → blocked)
- corriere.it began blocking Claude-SearchBot (restricted → blocked)
- corriere.it began blocking Claude-User (restricted → blocked)
- corriere.it began blocking OAI-SearchBot (allowed → blocked)
2026-11-03
2026-11-02
- appsflyer.com published an llms.txt — a welcome mat for AI readers
- bbc.co.uk began blocking DuckAssistBot (restricted → blocked)
- bbc.co.uk began blocking Claude-SearchBot (restricted → blocked)
- researchgate.net began blocking Applebot-Extended (restricted → blocked)
- bbc.com began blocking DuckAssistBot (restricted → blocked)
- bbc.com began blocking Claude-SearchBot (restricted → blocked)
2026-10-30
- delta.com changed access for Meta-ExternalFetcher (restricted → allowed)
- delta.com changed access for YouBot (restricted → allowed)
- delta.com changed access for Diffbot (restricted → allowed)
- delta.com changed access for cohere-ai (restricted → allowed)
- delta.com changed access for anthropic-ai (restricted → allowed)
- delta.com changed access for DuckAssistBot (restricted → allowed)
- delta.com changed access for Amazonbot (restricted → allowed)
- delta.com changed access for meta-externalagent (restricted → allowed)
- delta.com changed access for Bytespider (restricted → allowed)
- delta.com changed access for CCBot (restricted → allowed)
- delta.com changed access for PerplexityBot (restricted → allowed)
- delta.com changed access for Applebot-Extended (restricted → allowed)
- delta.com changed access for Google-Extended (restricted → allowed)
- delta.com changed access for ClaudeBot (restricted → allowed)
- delta.com changed access for OAI-SearchBot (restricted → allowed)
- delta.com changed access for ChatGPT-User (restricted → allowed)
- delta.com changed access for GPTBot (restricted → allowed)
- newsweek.com stopped blocking Amazonbot (blocked → restricted)
- newsweek.com began blocking Perplexity-User (restricted → blocked)
2026-10-29
- kick.com began blocking omgili (restricted → blocked)
- kick.com began blocking Diffbot (restricted → blocked)
- kick.com began blocking cohere-ai (restricted → blocked)
- kick.com began blocking Claude-Web (restricted → blocked)
- kick.com began blocking anthropic-ai (restricted → blocked)
- kick.com began blocking Amazonbot (restricted → blocked)
- kick.com began blocking meta-externalagent (restricted → blocked)
- kick.com began blocking Bytespider (restricted → blocked)
- kick.com began blocking CCBot (restricted → blocked)
- kick.com began blocking Applebot-Extended (restricted → blocked)
- kick.com began blocking Google-Extended (restricted → blocked)
- kick.com began blocking ClaudeBot (restricted → blocked)
- kick.com began blocking GPTBot (restricted → blocked)
- dailymail.com began blocking cohere-training-data-crawler (restricted → blocked)
- cnbc.com began blocking Perplexity-User (restricted → blocked)
- cnbc.com began blocking Claude-SearchBot (restricted → blocked)
- dailymail.co.uk began blocking cohere-training-data-crawler (restricted → blocked)
- theatlantic.com changed access for DuckAssistBot (allowed → restricted)
- theatlantic.com changed access for OAI-SearchBot (allowed → restricted)
- theatlantic.com changed access for ChatGPT-User (allowed → restricted)
- theatlantic.com changed access for GPTBot (allowed → restricted)
2026-10-28
- cloudns.net published an llms.txt — a welcome mat for AI readers
- robinhood.com published an llms.txt — a welcome mat for AI readers
- fool.com began blocking CCBot (restricted → blocked)
- fool.com began blocking ClaudeBot (restricted → blocked)
- fool.com began blocking GPTBot (restricted → blocked)
- ikea.com began blocking Amazonbot (restricted → blocked)
- metacritic.com began blocking Meta-ExternalFetcher (restricted → blocked)
- metacritic.com began blocking meta-externalagent (restricted → blocked)
2026-10-27
- semafor.com began blocking Gemini-Deep-Research (restricted → blocked)
- semafor.com began blocking PanguBot (restricted → blocked)
- semafor.com began blocking GrokBot (restricted → blocked)
- semafor.com began blocking DeepSeekBot (restricted → blocked)
- semafor.com began blocking AI2Bot (restricted → blocked)
- semafor.com began blocking Meta-ExternalFetcher (restricted → blocked)
- semafor.com began blocking Timpibot (restricted → blocked)
- semafor.com began blocking ImagesiftBot (restricted → blocked)
- semafor.com began blocking YouBot (restricted → blocked)
- semafor.com began blocking omgilibot (restricted → blocked)
- semafor.com began blocking omgili (restricted → blocked)
- semafor.com began blocking Diffbot (restricted → blocked)
- semafor.com began blocking cohere-training-data-crawler (restricted → blocked)
- semafor.com began blocking cohere-ai (restricted → blocked)
- semafor.com began blocking Claude-Web (restricted → blocked)
- semafor.com began blocking anthropic-ai (restricted → blocked)
- semafor.com began blocking MistralAI-User (restricted → blocked)
- semafor.com began blocking DuckAssistBot (restricted → blocked)
- semafor.com began blocking Amazonbot (restricted → blocked)
- semafor.com began blocking meta-externalagent (restricted → blocked)
- semafor.com began blocking Bytespider (restricted → blocked)
- semafor.com began blocking CCBot (restricted → blocked)
- semafor.com began blocking Perplexity-User (restricted → blocked)
- semafor.com began blocking PerplexityBot (restricted → blocked)
- semafor.com began blocking Applebot-Extended (restricted → blocked)
- semafor.com began blocking Google-Extended (restricted → blocked)
- semafor.com began blocking Claude-SearchBot (restricted → blocked)
- semafor.com began blocking Claude-User (restricted → blocked)
- semafor.com began blocking ClaudeBot (restricted → blocked)
- semafor.com began blocking OAI-SearchBot (restricted → blocked)
- semafor.com began blocking ChatGPT-User (restricted → blocked)
2026-10-24
- wiley.com began blocking omgili (restricted → blocked)
- wiley.com began blocking Diffbot (restricted → blocked)
- wiley.com began blocking cohere-ai (restricted → blocked)
- wiley.com began blocking Claude-Web (restricted → blocked)
- wiley.com began blocking anthropic-ai (restricted → blocked)
- wiley.com began blocking Amazonbot (restricted → blocked)
- wiley.com began blocking meta-externalagent (restricted → blocked)
- wiley.com began blocking Bytespider (restricted → blocked)
- wiley.com began blocking CCBot (restricted → blocked)
- wiley.com began blocking Applebot-Extended (restricted → blocked)
- wiley.com began blocking Google-Extended (restricted → blocked)
- wiley.com began blocking ClaudeBot (restricted → blocked)
- wiley.com began blocking GPTBot (restricted → blocked)
2026-10-23
- soundcloud.com began blocking omgilibot (restricted → blocked)
- soundcloud.com began blocking omgili (restricted → blocked)
- soundcloud.com began blocking Claude-SearchBot (restricted → blocked)
- soundcloud.com began blocking Claude-User (restricted → blocked)
2026-10-22
- unpkg.com published an llms.txt — a welcome mat for AI readers
- nerdwallet.com stopped blocking Google-Extended (blocked → restricted)
- fool.com published an llms.txt — a welcome mat for AI readers
- nbcnews.com began blocking Meta-ExternalFetcher (restricted → blocked)
2026-10-21
2026-10-20
- people.com began blocking Diffbot (restricted → blocked)
- dailymail.com published an llms.txt — a welcome mat for AI readers
- paloaltonetworks.com published an llms.txt — a welcome mat for AI readers
- seriouseats.com began blocking Diffbot (restricted → blocked)
- allrecipes.com began blocking Diffbot (restricted → blocked)
- dailymail.co.uk published an llms.txt — a welcome mat for AI readers
2026-10-19
- paloaltonetworks.com removed its llms.txt
- instagram.com began blocking Amazonbot (restricted → blocked)
- instagram.com began blocking PerplexityBot (restricted → blocked)
- instagram.com began blocking Applebot-Extended (restricted → blocked)
- instagram.com began blocking ClaudeBot (restricted → blocked)
- instagram.com began blocking GPTBot (restricted → blocked)
- facebook.com began blocking Amazonbot (restricted → blocked)
- facebook.com began blocking PerplexityBot (restricted → blocked)
- facebook.com began blocking Applebot-Extended (restricted → blocked)
- facebook.com began blocking ClaudeBot (restricted → blocked)
- facebook.com began blocking GPTBot (restricted → blocked)
- investopedia.com began blocking Diffbot (restricted → blocked)
2026-10-17
- cisco.com changed access for CCBot (allowed → restricted)
- cisco.com changed access for Google-Extended (allowed → restricted)
- cisco.com changed access for ClaudeBot (allowed → restricted)
- cisco.com changed access for GPTBot (allowed → restricted)
- businessinsider.com published an llms.txt — a welcome mat for AI readers
2026-10-16
- seedtag.com published an llms.txt — a welcome mat for AI readers
- pinimg.com began blocking Meta-ExternalFetcher (restricted → blocked)
- fontawesome.com published an llms.txt — a welcome mat for AI readers
- pinterest.com began blocking Meta-ExternalFetcher (restricted → blocked)
2026-10-15
- pcgamer.com began blocking Applebot-Extended (restricted → blocked)
- facebook.com began blocking Google-Extended (restricted → blocked)
- aliexpress.com began blocking Amazonbot (restricted → blocked)
- aliexpress.com began blocking Bytespider (restricted → blocked)
- aliexpress.com began blocking CCBot (restricted → blocked)
- aliexpress.com began blocking Google-Extended (restricted → blocked)
- aliexpress.com began blocking ClaudeBot (restricted → blocked)
- aliexpress.com began blocking GPTBot (restricted → blocked)
- cnn.com stopped blocking Google-Extended (blocked → restricted)
2026-10-13
2026-10-10
2026-10-09
- otto.de published an llms.txt — a welcome mat for AI readers
- zoho.com published an llms.txt — a welcome mat for AI readers
- time.com changed access for Gemini-Deep-Research (allowed → restricted)
- time.com changed access for PanguBot (allowed → restricted)
- time.com changed access for GrokBot (allowed → restricted)
- time.com changed access for DeepSeekBot (allowed → restricted)
- time.com changed access for AI2Bot (allowed → restricted)
- time.com changed access for Meta-ExternalFetcher (allowed → restricted)
- time.com changed access for Timpibot (allowed → restricted)
- time.com changed access for ImagesiftBot (allowed → restricted)
- time.com changed access for YouBot (allowed → restricted)
- time.com changed access for omgilibot (allowed → restricted)
- time.com changed access for omgili (allowed → restricted)
- time.com changed access for Diffbot (allowed → restricted)
- time.com changed access for cohere-training-data-crawler (allowed → restricted)
- time.com changed access for cohere-ai (allowed → restricted)
- time.com changed access for Claude-Web (allowed → restricted)
- time.com changed access for anthropic-ai (allowed → restricted)
- time.com changed access for MistralAI-User (allowed → restricted)
- time.com changed access for DuckAssistBot (allowed → restricted)
- time.com changed access for Amazonbot (allowed → restricted)
- time.com changed access for meta-externalagent (allowed → restricted)
- time.com changed access for Bytespider (allowed → restricted)
- time.com changed access for CCBot (allowed → restricted)
- time.com changed access for Perplexity-User (allowed → restricted)
- time.com changed access for PerplexityBot (allowed → restricted)
- time.com changed access for Applebot-Extended (allowed → restricted)
- time.com changed access for Google-Extended (allowed → restricted)
- time.com changed access for Claude-SearchBot (allowed → restricted)
- time.com changed access for Claude-User (allowed → restricted)
- time.com changed access for ClaudeBot (allowed → restricted)
- time.com changed access for OAI-SearchBot (allowed → restricted)
- time.com changed access for ChatGPT-User (allowed → restricted)
- time.com changed access for GPTBot (allowed → restricted)
- time.com changed its default crawler policy (allowed → restricted)
2026-10-08
- ea.com published an llms.txt — a welcome mat for AI readers
- instagram.com stopped blocking Amazonbot (blocked → restricted)
- instagram.com stopped blocking PerplexityBot (blocked → restricted)
- instagram.com stopped blocking Applebot-Extended (blocked → restricted)
- instagram.com stopped blocking Google-Extended (blocked → restricted)
- instagram.com stopped blocking ClaudeBot (blocked → restricted)
- instagram.com stopped blocking GPTBot (blocked → restricted)
2026-10-07
- pexels.com changed access for Perplexity-User (allowed → restricted)
- pexels.com stopped blocking PerplexityBot (blocked → restricted)
- pexels.com stopped blocking Google-Extended (blocked → restricted)
- pexels.com changed access for Claude-SearchBot (allowed → restricted)
- pexels.com changed access for Claude-User (allowed → restricted)
- flashscore.com stopped blocking Meta-ExternalFetcher (blocked → restricted)
- flashscore.com stopped blocking meta-externalagent (blocked → restricted)
- flashscore.com stopped blocking CCBot (blocked → restricted)
2026-10-05
2026-10-03
2026-10-01
- trustpilot.com stopped blocking Meta-ExternalFetcher (blocked → restricted)
- trustpilot.com stopped blocking meta-externalagent (blocked → restricted)
- trustpilot.com stopped blocking CCBot (blocked → restricted)
- checkpoint.com changed access for Claude-Web (restricted → allowed)
- checkpoint.com changed access for CCBot (restricted → allowed)
- checkpoint.com changed access for Perplexity-User (restricted → allowed)
- checkpoint.com changed access for PerplexityBot (restricted → allowed)
- checkpoint.com changed access for Google-Extended (restricted → allowed)
- checkpoint.com changed access for ClaudeBot (restricted → allowed)
- checkpoint.com changed access for OAI-SearchBot (restricted → allowed)
- checkpoint.com changed access for ChatGPT-User (restricted → allowed)
- checkpoint.com changed access for GPTBot (restricted → allowed)
- figma.com began blocking Bytespider (restricted → blocked)
2026-09-29
2026-09-27
- seekingalpha.com stopped blocking Amazonbot (blocked → restricted)
- seekingalpha.com stopped blocking PerplexityBot (blocked → restricted)
- seekingalpha.com stopped blocking Applebot-Extended (blocked → restricted)
2026-09-26
- airbnb.com began blocking AI2Bot (restricted → blocked)
- airbnb.com began blocking Applebot-Extended (restricted → blocked)
- airbnb.com began blocking ClaudeBot (restricted → blocked)
- airbnb.com began blocking GPTBot (restricted → blocked)
- etsy.com published an llms.txt — a welcome mat for AI readers
2026-09-25
- markmonitor.com changed access for Gemini-Deep-Research (allowed → restricted)
- markmonitor.com changed access for PanguBot (allowed → restricted)
- markmonitor.com changed access for GrokBot (allowed → restricted)
- markmonitor.com changed access for DeepSeekBot (allowed → restricted)
- markmonitor.com changed access for AI2Bot (allowed → restricted)
- markmonitor.com changed access for Meta-ExternalFetcher (allowed → restricted)
- markmonitor.com changed access for Timpibot (allowed → restricted)
- markmonitor.com changed access for ImagesiftBot (allowed → restricted)
- markmonitor.com changed access for YouBot (allowed → restricted)
- markmonitor.com changed access for omgilibot (allowed → restricted)
- markmonitor.com changed access for omgili (allowed → restricted)
- markmonitor.com changed access for Diffbot (allowed → restricted)
- markmonitor.com changed access for cohere-training-data-crawler (allowed → restricted)
- markmonitor.com changed access for cohere-ai (allowed → restricted)
- markmonitor.com changed access for Claude-Web (allowed → restricted)
- markmonitor.com changed access for anthropic-ai (allowed → restricted)
- markmonitor.com changed access for MistralAI-User (allowed → restricted)
- markmonitor.com changed access for DuckAssistBot (allowed → restricted)
- markmonitor.com changed access for Amazonbot (allowed → restricted)
- markmonitor.com changed access for meta-externalagent (allowed → restricted)
- markmonitor.com changed access for Bytespider (allowed → restricted)
- markmonitor.com changed access for CCBot (allowed → restricted)
- markmonitor.com changed access for Perplexity-User (allowed → restricted)
- markmonitor.com changed access for PerplexityBot (allowed → restricted)
- markmonitor.com changed access for Applebot-Extended (allowed → restricted)
- markmonitor.com changed access for Google-Extended (allowed → restricted)
- markmonitor.com changed access for Claude-SearchBot (allowed → restricted)
- markmonitor.com changed access for Claude-User (allowed → restricted)
- markmonitor.com changed access for ClaudeBot (allowed → restricted)
- markmonitor.com changed access for OAI-SearchBot (allowed → restricted)
- markmonitor.com changed access for ChatGPT-User (allowed → restricted)
- markmonitor.com changed access for GPTBot (allowed → restricted)
- markmonitor.com changed its default crawler policy (allowed → restricted)
- name.com began blocking meta-externalagent (restricted → blocked)
2026-09-24
- fwmrm.net published an llms.txt — a welcome mat for AI readers
- aol.com stopped blocking PerplexityBot (blocked → restricted)
- aol.com stopped blocking ChatGPT-User (blocked → restricted)
- wiley.com changed access for Gemini-Deep-Research (allowed → restricted)
- wiley.com changed access for PanguBot (allowed → restricted)
- wiley.com changed access for GrokBot (allowed → restricted)
- wiley.com changed access for DeepSeekBot (allowed → restricted)
- wiley.com changed access for AI2Bot (allowed → restricted)
- wiley.com changed access for Meta-ExternalFetcher (allowed → restricted)
- wiley.com changed access for Timpibot (allowed → restricted)
- wiley.com changed access for ImagesiftBot (allowed → restricted)
- wiley.com changed access for YouBot (allowed → restricted)
- wiley.com changed access for omgilibot (allowed → restricted)
- wiley.com changed access for omgili (allowed → restricted)
- wiley.com changed access for Diffbot (allowed → restricted)
- wiley.com changed access for cohere-training-data-crawler (allowed → restricted)
- wiley.com changed access for cohere-ai (allowed → restricted)
- wiley.com changed access for Claude-Web (allowed → restricted)
- wiley.com changed access for anthropic-ai (allowed → restricted)
- wiley.com changed access for MistralAI-User (allowed → restricted)
- wiley.com changed access for DuckAssistBot (allowed → restricted)
- wiley.com changed access for Amazonbot (allowed → restricted)
- wiley.com changed access for meta-externalagent (allowed → restricted)
- wiley.com changed access for Bytespider (allowed → restricted)
- wiley.com changed access for CCBot (allowed → restricted)
- wiley.com changed access for Perplexity-User (allowed → restricted)
- wiley.com changed access for PerplexityBot (allowed → restricted)
- wiley.com changed access for Applebot-Extended (allowed → restricted)
- wiley.com changed access for Google-Extended (allowed → restricted)
- wiley.com changed access for Claude-SearchBot (allowed → restricted)
- wiley.com changed access for Claude-User (allowed → restricted)
- wiley.com changed access for ClaudeBot (allowed → restricted)
- wiley.com changed access for OAI-SearchBot (allowed → restricted)
- wiley.com changed access for ChatGPT-User (allowed → restricted)
- wiley.com changed access for GPTBot (allowed → restricted)
- wiley.com changed its default crawler policy (allowed → restricted)
2026-09-18
- wiley.com changed access for Gemini-Deep-Research (restricted → allowed)
- wiley.com changed access for PanguBot (restricted → allowed)
- wiley.com changed access for GrokBot (restricted → allowed)
- wiley.com changed access for DeepSeekBot (restricted → allowed)
- wiley.com changed access for AI2Bot (restricted → allowed)
- wiley.com changed access for Meta-ExternalFetcher (restricted → allowed)
- wiley.com changed access for Timpibot (restricted → allowed)
- wiley.com changed access for ImagesiftBot (restricted → allowed)
- wiley.com changed access for YouBot (restricted → allowed)
- wiley.com changed access for omgilibot (restricted → allowed)
- wiley.com changed access for omgili (restricted → allowed)
- wiley.com changed access for Diffbot (restricted → allowed)
- wiley.com changed access for cohere-training-data-crawler (restricted → allowed)
- wiley.com changed access for cohere-ai (restricted → allowed)
- wiley.com changed access for Claude-Web (restricted → allowed)
- wiley.com changed access for anthropic-ai (restricted → allowed)
- wiley.com changed access for MistralAI-User (restricted → allowed)
- wiley.com changed access for DuckAssistBot (restricted → allowed)
- wiley.com changed access for Amazonbot (restricted → allowed)
- wiley.com changed access for meta-externalagent (restricted → allowed)
- wiley.com changed access for Bytespider (restricted → allowed)
- wiley.com changed access for CCBot (restricted → allowed)
- wiley.com changed access for Perplexity-User (restricted → allowed)
- wiley.com changed access for PerplexityBot (restricted → allowed)
- wiley.com changed access for Applebot-Extended (restricted → allowed)
- wiley.com changed access for Google-Extended (restricted → allowed)
- wiley.com changed access for Claude-SearchBot (restricted → allowed)
- wiley.com changed access for Claude-User (restricted → allowed)
- wiley.com changed access for ClaudeBot (restricted → allowed)
- wiley.com changed access for OAI-SearchBot (restricted → allowed)
- wiley.com changed access for ChatGPT-User (restricted → allowed)
- wiley.com changed access for GPTBot (restricted → allowed)
- wiley.com changed its default crawler policy (restricted → allowed)
2026-09-16
- onet.pl began blocking Timpibot (restricted → blocked)
- onet.pl began blocking YouBot (restricted → blocked)
- onet.pl began blocking omgilibot (restricted → blocked)
- onet.pl began blocking Diffbot (restricted → blocked)
- onet.pl began blocking cohere-ai (restricted → blocked)
- onet.pl began blocking Amazonbot (restricted → blocked)
- onet.pl began blocking meta-externalagent (restricted → blocked)
- onet.pl began blocking Bytespider (restricted → blocked)
- kick.com stopped blocking Amazonbot (blocked → restricted)
- kick.com stopped blocking meta-externalagent (blocked → restricted)
- kick.com stopped blocking Bytespider (blocked → restricted)
- kick.com stopped blocking CCBot (blocked → restricted)
- kick.com stopped blocking Applebot-Extended (blocked → restricted)
- kick.com stopped blocking Google-Extended (blocked → restricted)
- kick.com stopped blocking ClaudeBot (blocked → restricted)
- kick.com stopped blocking GPTBot (blocked → restricted)
- patreon.com stopped blocking Amazonbot (blocked → restricted)
- patreon.com stopped blocking meta-externalagent (blocked → restricted)
- patreon.com stopped blocking Bytespider (blocked → restricted)
- patreon.com stopped blocking CCBot (blocked → restricted)
- patreon.com stopped blocking Applebot-Extended (blocked → restricted)
- patreon.com stopped blocking Google-Extended (blocked → restricted)
- patreon.com stopped blocking ClaudeBot (blocked → restricted)
- patreon.com stopped blocking GPTBot (blocked → restricted)
- azure.com removed its llms.txt
- weather.com began blocking YouBot (restricted → blocked)
- weather.com stopped blocking cohere-ai (blocked → restricted)
- weather.com stopped blocking Claude-Web (blocked → restricted)
- weather.com stopped blocking anthropic-ai (blocked → restricted)
- weather.com stopped blocking meta-externalagent (blocked → restricted)
- weather.com stopped blocking PerplexityBot (blocked → restricted)
- weather.com stopped blocking Applebot-Extended (blocked → restricted)
- weather.com stopped blocking Google-Extended (blocked → restricted)
- weather.com stopped blocking ClaudeBot (blocked → restricted)
- weather.com stopped blocking GPTBot (blocked → restricted)
2026-09-14
- samsungcloud.com changed access for Gemini-Deep-Research (allowed → restricted)
- samsungcloud.com changed access for PanguBot (allowed → restricted)
- samsungcloud.com changed access for GrokBot (allowed → restricted)
- samsungcloud.com changed access for DeepSeekBot (allowed → restricted)
- samsungcloud.com changed access for AI2Bot (allowed → restricted)
- samsungcloud.com changed access for Meta-ExternalFetcher (allowed → restricted)
- samsungcloud.com changed access for Timpibot (allowed → restricted)
- samsungcloud.com changed access for ImagesiftBot (allowed → restricted)
- samsungcloud.com changed access for YouBot (allowed → restricted)
- samsungcloud.com changed access for omgilibot (allowed → restricted)
- samsungcloud.com changed access for omgili (allowed → restricted)
- samsungcloud.com changed access for Diffbot (allowed → restricted)
- samsungcloud.com changed access for cohere-training-data-crawler (allowed → restricted)
- samsungcloud.com changed access for cohere-ai (allowed → restricted)
- samsungcloud.com changed access for Claude-Web (allowed → restricted)
- samsungcloud.com changed access for anthropic-ai (allowed → restricted)
- samsungcloud.com changed access for MistralAI-User (allowed → restricted)
- samsungcloud.com changed access for DuckAssistBot (allowed → restricted)
- samsungcloud.com changed access for Amazonbot (allowed → restricted)
- samsungcloud.com changed access for meta-externalagent (allowed → restricted)
- samsungcloud.com changed access for Bytespider (allowed → restricted)
- samsungcloud.com changed access for CCBot (allowed → restricted)
- samsungcloud.com changed access for Perplexity-User (allowed → restricted)
- samsungcloud.com changed access for PerplexityBot (allowed → restricted)
- samsungcloud.com changed access for Applebot-Extended (allowed → restricted)
- samsungcloud.com changed access for Google-Extended (allowed → restricted)
- samsungcloud.com changed access for Claude-SearchBot (allowed → restricted)
- samsungcloud.com changed access for Claude-User (allowed → restricted)
- samsungcloud.com changed access for ClaudeBot (allowed → restricted)
- samsungcloud.com changed access for OAI-SearchBot (allowed → restricted)
- samsungcloud.com changed access for ChatGPT-User (allowed → restricted)
- samsungcloud.com changed access for GPTBot (allowed → restricted)
- samsungcloud.com changed its default crawler policy (allowed → restricted)
2026-09-12
2026-09-11
- corriere.it began blocking Claude-Web (restricted → blocked)
- corriere.it changed access for PerplexityBot (restricted → allowed)
- corriere.it changed access for OAI-SearchBot (restricted → allowed)
- corriere.it began blocking ChatGPT-User (restricted → blocked)
- ring.com published an llms.txt — a welcome mat for AI readers
- checkpoint.com published an llms.txt — a welcome mat for AI readers
2026-09-10
- hostinger.com began blocking Bytespider (restricted → blocked)
- hostinger.com began blocking CCBot (restricted → blocked)
2026-09-09
- welt.de began blocking Amazonbot (restricted → blocked)
- intel.com published an llms.txt — a welcome mat for AI readers
- t-online.de stopped blocking Google-Extended (blocked → restricted)
2026-09-08
- opera-api2.com added to the index
- xerox.com added to the index
- tbank.ru added to the index
- pccc.com added to the index
- immedia-semi.com added to the index
- dnse2.com added to the index
- bytefcdn.com added to the index
- zenecn.net added to the index
- epam.com added to the index
- ryanair.com added to the index
- dnse0.com added to the index
- usercontent.goog added to the index
- actify.nl added to the index
- usps.com added to the index
- pendo.io added to the index
- bitdefender.net added to the index
- buydomains.com added to the index
- hostinger.com added to the index
- h264.io added to the index
- statsigapi.net added to the index
- lg.com changed access for PerplexityBot (allowed → restricted)
- lg.com changed access for Google-Extended (allowed → restricted)
- lg.com changed access for ClaudeBot (allowed → restricted)
- lg.com changed access for OAI-SearchBot (allowed → restricted)
- lg.com changed access for ChatGPT-User (allowed → restricted)
- lg.com changed access for GPTBot (allowed → restricted)
- intel.com changed access for anthropic-ai (restricted → allowed)
- intel.com changed access for PerplexityBot (restricted → allowed)
- intel.com changed access for Google-Extended (restricted → allowed)
- intel.com changed access for ClaudeBot (restricted → allowed)
- intel.com changed access for ChatGPT-User (restricted → allowed)
- intel.com changed access for GPTBot (restricted → allowed)
- bonappetit.com began blocking YouBot (restricted → blocked)
- epicurious.com began blocking YouBot (restricted → blocked)
- wired.com began blocking YouBot (restricted → blocked)
- newyorker.com began blocking YouBot (restricted → blocked)
2026-09-05
2026-09-04
2026-09-03
2026-09-02
- worldbank.org changed access for Gemini-Deep-Research (restricted → allowed)
- worldbank.org changed access for PanguBot (restricted → allowed)
- worldbank.org changed access for GrokBot (restricted → allowed)
- worldbank.org changed access for DeepSeekBot (restricted → allowed)
- worldbank.org changed access for AI2Bot (restricted → allowed)
- worldbank.org changed access for Meta-ExternalFetcher (restricted → allowed)
- worldbank.org changed access for Timpibot (restricted → allowed)
- worldbank.org changed access for ImagesiftBot (restricted → allowed)
- worldbank.org changed access for YouBot (restricted → allowed)
- worldbank.org changed access for omgilibot (restricted → allowed)
- worldbank.org changed access for omgili (restricted → allowed)
- worldbank.org changed access for Diffbot (restricted → allowed)
- worldbank.org changed access for cohere-training-data-crawler (restricted → allowed)
- worldbank.org changed access for cohere-ai (restricted → allowed)
- worldbank.org changed access for Claude-Web (restricted → allowed)
- worldbank.org changed access for anthropic-ai (restricted → allowed)
- worldbank.org changed access for MistralAI-User (restricted → allowed)
- worldbank.org changed access for DuckAssistBot (restricted → allowed)
- worldbank.org changed access for Amazonbot (restricted → allowed)
- worldbank.org changed access for meta-externalagent (restricted → allowed)
- worldbank.org changed access for Bytespider (restricted → allowed)
- worldbank.org changed access for CCBot (restricted → allowed)
- worldbank.org changed access for Perplexity-User (restricted → allowed)
- worldbank.org changed access for PerplexityBot (restricted → allowed)
- worldbank.org changed access for Applebot-Extended (restricted → allowed)
- worldbank.org changed access for Google-Extended (restricted → allowed)
- worldbank.org changed access for Claude-SearchBot (restricted → allowed)
- worldbank.org changed access for Claude-User (restricted → allowed)
- worldbank.org changed access for ClaudeBot (restricted → allowed)
- worldbank.org changed access for OAI-SearchBot (restricted → allowed)
- worldbank.org changed access for ChatGPT-User (restricted → allowed)
- worldbank.org changed access for GPTBot (restricted → allowed)
- worldbank.org changed its default crawler policy (restricted → allowed)
- semafor.com stopped blocking Gemini-Deep-Research (blocked → restricted)
- semafor.com stopped blocking PanguBot (blocked → restricted)
- semafor.com stopped blocking GrokBot (blocked → restricted)
- semafor.com stopped blocking DeepSeekBot (blocked → restricted)
- semafor.com stopped blocking AI2Bot (blocked → restricted)
- semafor.com stopped blocking Meta-ExternalFetcher (blocked → restricted)
- semafor.com stopped blocking Timpibot (blocked → restricted)
- semafor.com stopped blocking ImagesiftBot (blocked → restricted)
- semafor.com stopped blocking YouBot (blocked → restricted)
- semafor.com stopped blocking omgilibot (blocked → restricted)
- semafor.com stopped blocking omgili (blocked → restricted)
- semafor.com stopped blocking Diffbot (blocked → restricted)
- semafor.com stopped blocking cohere-training-data-crawler (blocked → restricted)
- semafor.com stopped blocking cohere-ai (blocked → restricted)
- semafor.com stopped blocking Claude-Web (blocked → restricted)
- semafor.com stopped blocking anthropic-ai (blocked → restricted)
- semafor.com stopped blocking MistralAI-User (blocked → restricted)
- semafor.com stopped blocking DuckAssistBot (blocked → restricted)
- semafor.com stopped blocking Amazonbot (blocked → restricted)
- semafor.com stopped blocking meta-externalagent (blocked → restricted)
- semafor.com stopped blocking Bytespider (blocked → restricted)
- semafor.com stopped blocking CCBot (blocked → restricted)
- semafor.com stopped blocking Perplexity-User (blocked → restricted)
- semafor.com stopped blocking PerplexityBot (blocked → restricted)
- semafor.com stopped blocking Applebot-Extended (blocked → restricted)
- semafor.com stopped blocking Google-Extended (blocked → restricted)
- semafor.com stopped blocking Claude-SearchBot (blocked → restricted)
- semafor.com stopped blocking Claude-User (blocked → restricted)
- semafor.com stopped blocking ClaudeBot (blocked → restricted)
- semafor.com stopped blocking OAI-SearchBot (blocked → restricted)
- semafor.com stopped blocking ChatGPT-User (blocked → restricted)
2026-09-01
- roblox.com stopped blocking Amazonbot (blocked → restricted)
- roblox.com stopped blocking meta-externalagent (blocked → restricted)
- roblox.com stopped blocking Bytespider (blocked → restricted)
- roblox.com stopped blocking CCBot (blocked → restricted)
- roblox.com stopped blocking Applebot-Extended (blocked → restricted)
- roblox.com stopped blocking Google-Extended (blocked → restricted)
- roblox.com stopped blocking ClaudeBot (blocked → restricted)
- roblox.com stopped blocking GPTBot (blocked → restricted)
- webmd.com stopped blocking ChatGPT-User (blocked → restricted)
2026-08-28
2026-08-27
- stackadapt.com changed access for Gemini-Deep-Research (allowed → restricted)
- stackadapt.com changed access for PanguBot (allowed → restricted)
- stackadapt.com changed access for GrokBot (allowed → restricted)
- stackadapt.com changed access for DeepSeekBot (allowed → restricted)
- stackadapt.com changed access for AI2Bot (allowed → restricted)
- stackadapt.com changed access for Meta-ExternalFetcher (allowed → restricted)
- stackadapt.com changed access for Timpibot (allowed → restricted)
- stackadapt.com changed access for ImagesiftBot (allowed → restricted)
- stackadapt.com changed access for YouBot (allowed → restricted)
- stackadapt.com changed access for omgilibot (allowed → restricted)
- stackadapt.com changed access for omgili (allowed → restricted)
- stackadapt.com changed access for Diffbot (allowed → restricted)
- stackadapt.com changed access for cohere-training-data-crawler (allowed → restricted)
- stackadapt.com changed access for cohere-ai (allowed → restricted)
- stackadapt.com changed access for Claude-Web (allowed → restricted)
- stackadapt.com changed access for anthropic-ai (allowed → restricted)
- stackadapt.com changed access for MistralAI-User (allowed → restricted)
- stackadapt.com changed access for DuckAssistBot (allowed → restricted)
- stackadapt.com changed access for Amazonbot (allowed → restricted)
- stackadapt.com changed access for meta-externalagent (allowed → restricted)
- stackadapt.com changed access for Bytespider (allowed → restricted)
- stackadapt.com changed access for CCBot (allowed → restricted)
- stackadapt.com changed access for Perplexity-User (allowed → restricted)
- stackadapt.com changed access for PerplexityBot (allowed → restricted)
- stackadapt.com changed access for Applebot-Extended (allowed → restricted)
- stackadapt.com changed access for Google-Extended (allowed → restricted)
- stackadapt.com changed access for Claude-SearchBot (allowed → restricted)
- stackadapt.com changed access for Claude-User (allowed → restricted)
- stackadapt.com changed access for ClaudeBot (allowed → restricted)
- stackadapt.com changed access for OAI-SearchBot (allowed → restricted)
- stackadapt.com changed access for ChatGPT-User (allowed → restricted)
- stackadapt.com changed access for GPTBot (allowed → restricted)
- stackadapt.com changed its default crawler policy (allowed → restricted)
- shein.com published an llms.txt — a welcome mat for AI readers
2026-08-26
- semafor.com began blocking Gemini-Deep-Research (restricted → blocked)
- semafor.com began blocking PanguBot (restricted → blocked)
- semafor.com began blocking GrokBot (restricted → blocked)
- semafor.com began blocking DeepSeekBot (restricted → blocked)
- semafor.com began blocking AI2Bot (restricted → blocked)
- semafor.com began blocking Meta-ExternalFetcher (restricted → blocked)
- semafor.com began blocking Timpibot (restricted → blocked)
- semafor.com began blocking ImagesiftBot (restricted → blocked)
- semafor.com began blocking YouBot (restricted → blocked)
- semafor.com began blocking omgilibot (restricted → blocked)
- semafor.com began blocking omgili (restricted → blocked)
- semafor.com began blocking Diffbot (restricted → blocked)
- semafor.com began blocking cohere-training-data-crawler (restricted → blocked)
- semafor.com began blocking cohere-ai (restricted → blocked)
- semafor.com began blocking Claude-Web (restricted → blocked)
- semafor.com began blocking anthropic-ai (restricted → blocked)
- semafor.com began blocking MistralAI-User (restricted → blocked)
- semafor.com began blocking DuckAssistBot (restricted → blocked)
- semafor.com began blocking Amazonbot (restricted → blocked)
- semafor.com began blocking meta-externalagent (restricted → blocked)
- semafor.com began blocking Bytespider (restricted → blocked)
- semafor.com began blocking CCBot (restricted → blocked)
- semafor.com began blocking Perplexity-User (restricted → blocked)
- semafor.com began blocking PerplexityBot (restricted → blocked)
- semafor.com began blocking Applebot-Extended (restricted → blocked)
- semafor.com began blocking Google-Extended (restricted → blocked)
- semafor.com began blocking Claude-SearchBot (restricted → blocked)
- semafor.com began blocking Claude-User (restricted → blocked)
- semafor.com began blocking ClaudeBot (restricted → blocked)
- semafor.com began blocking OAI-SearchBot (restricted → blocked)
- semafor.com began blocking ChatGPT-User (restricted → blocked)
2026-08-25
- Index expanded by 768 domains (Tranco top sites; infrastructure and adult domains excluded)
- Index founded. First crawl: 1020 domains, 32 AI user agents. 32.5% of readable sites block at least one AI crawler; llms.txt adoption 12.9%; most-blocked bot Bytespider (25.7%). Daily change tracking starts here.