# Canicrawl full census — 2026-10-25

> AI-crawler access policy for 1020 top websites, from each site's own robots.txt and llms.txt, crawled daily. 31.9% of the 645 readable sites block at least one of the 32 tracked AI bots (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research). llms.txt adoption: 125 sites. Most-blocked bot: Bytespider (25.1%). Data CC BY 4.0 — cite "Canicrawl" (https://canicrawl.com). Machine-readable JSON: https://canicrawl.com/data/latest.json

163.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
1rx.io — policy unreadable (domain does not resolve)
2gis.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
2mdn.net — policy unreadable (domain does not resolve)
33across.com — policy unreadable (refused (401/403))
360.cn — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
360safe.com — policy unreadable (host answered nothing)
360yield.com — blocks no tracked AI bots
3gppnetwork.org — policy unreadable (domain does not resolve)
3lift.com — policy unreadable (domain does not resolve)
404media.co — blocks: GPTBot — restricts: ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
4dex.io — policy unreadable (domain does not resolve)
a-mo.net — policy unreadable (host answered nothing)
a2z.com — policy unreadable (domain does not resolve)
aa.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
aboutads.info — policy unreadable (HTML instead of text)
abovedomains.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
academia.edu — blocks: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, PerplexityBot, DuckAssistBot, MistralAI-User, YouBot, Gemini-Deep-Research — default-deny robots.txt
accuweather.com — policy unreadable (refused (401/403))
achmea.nl — blocks no tracked AI bots
ack.de — blocks no tracked AI bots
actify.nl — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
ad-tech.ru — policy unreadable (domain does not resolve)
addtoany.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
adform.net — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
adgrx.com — policy unreadable (host answered nothing)
adidas.com — policy unreadable (refused (401/403))
adjust.com — policy unreadable (HTTP 429)
adobe.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
adobe.io — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
adobe.net — policy unreadable (host answered nothing)
adobedc.net — policy unreadable (domain does not resolve)
adriver.ru — blocks no tracked AI bots
adsafeprotected.com — blocks no tracked AI bots — publishes llms.txt
adtrafficquality.google — policy unreadable (domain does not resolve)
afternic.com — blocks no tracked AI bots
agkn.com — blocks no tracked AI bots
agoda.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
agora.io — blocks no tracked AI bots — publishes llms.txt
ailawandorder.com — policy unreadable (host answered nothing)
airbnb.com — blocks: GPTBot, ClaudeBot, Applebot-Extended, ImagesiftBot, AI2Bot — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, Timpibot, Meta-ExternalFetcher, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
aiv-delivery.net — policy unreadable (domain does not resolve)
akam.net — policy unreadable (domain does not resolve)
akamaihd.net — policy unreadable (domain does not resolve)
akamaitech.net — policy unreadable (domain does not resolve)
alibaba.com — blocks: GPTBot, Google-Extended — restricts: ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
alibabadns.com — policy unreadable (domain does not resolve)
alicdn.com — policy unreadable (host answered nothing)
alidns.com — blocks no tracked AI bots — publishes llms.txt
aliexpress.com — blocks: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Amazonbot — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
alipaydns.com — policy unreadable (domain does not resolve)
aliyun.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
aliyuncs.com — policy unreadable (domain does not resolve)
aliyuncsslbintl.com — policy unreadable (domain does not resolve)
allaboutcookies.org — policy unreadable (refused (401/403))
allawnos.com — policy unreadable (domain does not resolve)
allegro.pl — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
allrecipes.com — blocks: ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
alrosa.ru — blocks no tracked AI bots
amazon.ca — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: Applebot-Extended, Amazonbot, anthropic-ai, omgilibot
amazon.co.jp — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: Applebot-Extended, Amazonbot, anthropic-ai, omgilibot
amazon.co.uk — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: Applebot-Extended, Amazonbot, anthropic-ai, omgilibot
amazon.co.za — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: Applebot-Extended, Amazonbot, anthropic-ai, omgilibot
amazon.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: Applebot-Extended, Amazonbot, anthropic-ai, omgilibot
amazon.com.au — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: Applebot-Extended, Amazonbot, anthropic-ai, omgilibot
amazon.com.br — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: Applebot-Extended, Amazonbot, anthropic-ai, omgilibot
amazon.de — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: Applebot-Extended, Amazonbot, anthropic-ai, omgilibot
amazon.dev — policy unreadable (domain does not resolve)
amazon.es — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: Applebot-Extended, Amazonbot, anthropic-ai, omgilibot
amazon.fr — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: Applebot-Extended, Amazonbot, anthropic-ai, omgilibot
amazon.in — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: Applebot-Extended, Amazonbot, anthropic-ai, omgilibot
amazon.it — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: Applebot-Extended, Amazonbot, anthropic-ai, omgilibot
amazonalexa.com — policy unreadable (HTML instead of text)
amazontrust.com — blocks no tracked AI bots
amazonvideo.com — blocks: ChatGPT-User, Claude-User, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, Applebot-Extended, Amazonbot, anthropic-ai, Claude-Web, omgilibot
ameblo.jp — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
americanexpress.com — policy unreadable (host answered nothing)
amplitude.com — blocks: Bytespider — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
ampproject.org — blocks no tracked AI bots
amzn.to — blocks no tracked AI bots
ancestry.com — policy unreadable (refused (401/403))
android.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
anthropic.com — blocks no tracked AI bots
anydesk.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
aol.com — blocks: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, cohere-training-data-crawler, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
apache.org — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
apartments.com — policy unreadable (refused (401/403))
apnews.com — blocks: GPTBot, ClaudeBot, Claude-User, Claude-SearchBot, Applebot-Extended, PerplexityBot, CCBot, Amazonbot, anthropic-ai, Claude-Web, cohere-ai, Timpibot — restricts: ChatGPT-User, OAI-SearchBot, Google-Extended, Perplexity-User, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
app-analytics-services.com — policy unreadable (domain does not resolve)
appcenter.ms — blocks no tracked AI bots
apple.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
applovin.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
appsflyer.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
appspot.com — policy unreadable (HTML instead of text)
arcgis.com — blocks no tracked AI bots
archive.org — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
arstechnica.com — blocks: ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgilibot, YouBot, Timpibot, Meta-ExternalFetcher, PanguBot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, omgili, ImagesiftBot, AI2Bot, DeepSeekBot, GrokBot, Gemini-Deep-Research
arubanetworks.com — policy unreadable (refused (401/403))
arxiv.org — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
asus.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
atlassian.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
atlassian.net — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
att.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
att.net — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
autodesk.com — policy unreadable (refused (401/403))
autotrader.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
auvik.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
avast.com — policy unreadable (refused (401/403))
avcdn.net — policy unreadable (host answered nothing)
avito.ru — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
avsxappcaptiveportal.com — policy unreadable (domain does not resolve)
aws.dev — policy unreadable (domain does not resolve)
awsglobalaccelerator.com — policy unreadable (domain does not resolve)
awswaf.com — policy unreadable (domain does not resolve)
ax-msedge.net — policy unreadable (domain does not resolve)
axios.com — blocks: CCBot, Bytespider, Amazonbot, Diffbot, ImagesiftBot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, omgili, omgilibot, YouBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
azure-devices.net — policy unreadable (domain does not resolve)
azure.com — policy unreadable (HTML instead of text)
azurewebsites.net — policy unreadable (host answered nothing)
b-msedge.net — policy unreadable (domain does not resolve)
baidu.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
bamgrid.com — policy unreadable (domain does not resolve)
bandcamp.com — blocks: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, meta-externalagent, Amazonbot, Diffbot, omgili — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Applebot-Extended, PerplexityBot, Perplexity-User, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, omgilibot, YouBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
bankofamerica.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
bankrate.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
bbc.co.uk — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgili, omgilibot, YouBot — restricts: Claude-User, Claude-SearchBot, DuckAssistBot, MistralAI-User, cohere-training-data-crawler, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
bbc.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgili, omgilibot, YouBot — restricts: Claude-User, Claude-SearchBot, DuckAssistBot, MistralAI-User, cohere-training-data-crawler, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
bdydns.com — policy unreadable (domain does not resolve)
behance.net — blocks: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, anthropic-ai, Claude-Web, cohere-training-data-crawler, Diffbot, omgili, omgilibot, ImagesiftBot, Timpibot, AI2Bot, PanguBot — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Amazonbot, DuckAssistBot, MistralAI-User, cohere-ai, YouBot, Meta-ExternalFetcher, DeepSeekBot, GrokBot, Gemini-Deep-Research
beian.gov.cn — policy unreadable (host answered nothing)
berkeley.edu — blocks no tracked AI bots
bestbuy.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
beyondwickedmapping.org — policy unreadable (domain does not resolve)
biblegateway.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
bidr.io — policy unreadable (domain does not resolve)
bidswitch.net — policy unreadable (domain does not resolve)
bild.de — blocks: Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Amazonbot, DuckAssistBot, MistralAI-User, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, AI2Bot, PanguBot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, meta-externalagent, anthropic-ai, Claude-Web, Meta-ExternalFetcher, DeepSeekBot, GrokBot, Gemini-Deep-Research
bilibili.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
bing.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
bit.ly — blocks no tracked AI bots
bitdefender.net — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
bitrix24.ru — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
blackberry.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
blackhub.team — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
bleacherreport.com — blocks: GPTBot, Google-Extended, CCBot, anthropic-ai — restricts: ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Applebot-Extended, PerplexityBot, Perplexity-User, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
blogger.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
blogspot.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
bloomberg.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, anthropic-ai, Claude-Web, cohere-ai, Diffbot, YouBot, Meta-ExternalFetcher — restricts: Perplexity-User, MistralAI-User, cohere-training-data-crawler, omgili, omgilibot, ImagesiftBot, Timpibot, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
bluehost.com — policy unreadable (refused (401/403))
bonappetit.com — blocks: ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, cohere-ai, cohere-training-data-crawler, Diffbot, YouBot, Timpibot, Meta-ExternalFetcher, PanguBot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, anthropic-ai, Claude-Web, omgili, omgilibot, ImagesiftBot, AI2Bot, DeepSeekBot, GrokBot, Gemini-Deep-Research
booking.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
box.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
brave.com — blocks no tracked AI bots
braze.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
britannica.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
browser-intake-datadoghq.com — policy unreadable (domain does not resolve)
bsky.app — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
bugsnag.com — blocks no tracked AI bots
businessinsider.com — blocks: ClaudeBot, CCBot, Bytespider, anthropic-ai, Diffbot, Timpibot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, Claude-Web, cohere-ai, cohere-training-data-crawler, omgili, omgilibot, YouBot, ImagesiftBot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
buydomains.com — policy unreadable (refused (401/403))
bx-msedge.net — policy unreadable (domain does not resolve)
bytedns1.com — policy unreadable (domain does not resolve)
bytefcdn-oversea.com — policy unreadable (domain does not resolve)
bytefcdn.com — policy unreadable (domain does not resolve)
byteglb.com — policy unreadable (domain does not resolve)
byteoversea.net — policy unreadable (domain does not resolve)
ca.gov — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
caixa.gov.br — policy unreadable (host answered nothing)
calendly.com — blocks: CCBot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
cambridge.org — blocks: ChatGPT-User — restricts: GPTBot, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
canva.com — blocks: GPTBot, ClaudeBot, Applebot-Extended, CCBot, Bytespider, meta-externalagent, anthropic-ai, Claude-Web, Diffbot, omgili, ImagesiftBot, Timpibot, AI2Bot — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Google-Extended, Amazonbot, DuckAssistBot, MistralAI-User, cohere-ai, cohere-training-data-crawler, omgilibot, YouBot, Meta-ExternalFetcher, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
capcut.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, DuckAssistBot, Meta-ExternalFetcher — publishes llms.txt
capcutapi.com — policy unreadable (domain does not resolve)
capgemini.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
capitalone.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
carfax.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
cars.com — policy unreadable (refused (401/403))
carvana.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
cbsnews.com — blocks: GPTBot — restricts: ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
cbssports.com — blocks: GPTBot — restricts: ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
cdc.gov — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
cdn-apple.com — policy unreadable (domain does not resolve)
cdn-vk.ru — policy unreadable (domain does not resolve)
cdn20.com — policy unreadable (domain does not resolve)
cdnbuild.net — policy unreadable (domain does not resolve)
cdngslb.com — policy unreadable (host answered nothing)
cdnhwc1.com — policy unreadable (host answered nothing)
cdnvideo.ru — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
change.org — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
character.ai — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
chase.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
chatgpt.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
checkpoint.com — blocks no tracked AI bots — restricts: Claude-User, Claude-SearchBot, Applebot-Extended, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
chegg.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
chess.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
chewy.com — blocks: Amazonbot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
chinamobile.com — policy unreadable (host answered nothing)
cisco.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
clarity.ms — blocks no tracked AI bots
claude.ai — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, Google-Extended — restricts: ClaudeBot, Claude-User, Claude-SearchBot, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
clevelandclinic.org — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
cloud.microsoft — policy unreadable (HTML instead of text)
cloudflare.com — blocks no tracked AI bots — publishes llms.txt
cloudns.net — blocks no tracked AI bots
cloudsink.net — policy unreadable (domain does not resolve)
cmediahub.ru — policy unreadable (refused (401/403))
cnbc.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, CCBot, Bytespider, meta-externalagent, Amazonbot, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgili, omgilibot — restricts: Claude-SearchBot, Perplexity-User, DuckAssistBot, MistralAI-User, cohere-training-data-crawler, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
cnet.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot — restricts: Google-Extended, Gemini-Deep-Research
cnn.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, AI2Bot, PanguBot — restricts: Google-Extended, meta-externalagent, Meta-ExternalFetcher, DeepSeekBot, GrokBot, Gemini-Deep-Research
coinbase.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
comcast.com — blocks no tracked AI bots — publishes llms.txt
comcast.net — blocks no tracked AI bots
consultant.ru — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
cookiedatabase.org — policy unreadable (host answered nothing)
cornell.edu — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
corriere.it — blocks: GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, Amazonbot, anthropic-ai, Claude-Web, YouBot, Timpibot — restricts: Claude-User, Claude-SearchBot, Perplexity-User, DuckAssistBot, MistralAI-User, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, ImagesiftBot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
costco.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
coupang.com — policy unreadable (refused (401/403))
coursera.org — blocks: meta-externalagent — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
cpanel.net — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
craigslist.org — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
creativecdn.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
creativecommons.org — blocks no tracked AI bots
crpt.ru — blocks no tracked AI bots
crwdcntrl.net — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
curseforge.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
cursor.sh — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
dailymail.co.uk — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, PanguBot, Gemini-Deep-Research — restricts: cohere-training-data-crawler, GrokBot — publishes llms.txt
dailymail.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, PanguBot, Gemini-Deep-Research — restricts: cohere-training-data-crawler, GrokBot — publishes llms.txt
dailymotion.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
datadoghq.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
daum.net — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
dbankcloud.com — policy unreadable (refused (401/403))
dbankcloud.ru — policy unreadable (domain does not resolve)
ddnss.de — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
debian.org — blocks no tracked AI bots
deepl.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
deepseek.com — blocks no tracked AI bots
dell.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
deloitte.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
delta.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
developer.mozilla.org — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
deviantart.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
dictionary.com — blocks: GPTBot, Applebot-Extended, CCBot — restricts: ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
digicert.com — blocks no tracked AI bots — restricts: Applebot-Extended, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
digitalocean.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
digitaloceanspaces.com — policy unreadable (HTML instead of text)
discogs.com — policy unreadable (refused (401/403))
discord.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
discord.gg — blocks no tracked AI bots
discord.media — policy unreadable (domain does not resolve)
discordapp.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
disneyplus.com — blocks: Perplexity-User, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, ImagesiftBot, Timpibot, AI2Bot, PanguBot, Gemini-Deep-Research — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, YouBot, Meta-ExternalFetcher, DeepSeekBot, GrokBot — default-deny robots.txt
disqus.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
dns-parking.com — policy unreadable (domain does not resolve)
dnse0.com — policy unreadable (domain does not resolve)
dnse2.com — policy unreadable (domain does not resolve)
dnsmadeeasy.com — blocks no tracked AI bots
dnsowl.com — policy unreadable (domain does not resolve)
dnspod.net — policy unreadable (host answered nothing)
docker.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
docker.io — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
doi.org — blocks no tracked AI bots
doordash.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
dotaplabs.net — policy unreadable (domain does not resolve)
dotomi.com — policy unreadable (host answered nothing)
douyincdn.com — policy unreadable (host answered nothing)
dreamhost.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
drom.ru — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
dropbox.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
dropcatch.com — policy unreadable (HTML instead of text)
dual-s-msedge.net — policy unreadable (domain does not resolve)
duckdns.org — blocks no tracked AI bots
duckduckgo.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
duolingo.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
dv.tech — policy unreadable (domain does not resolve)
dyingbirds.com — policy unreadable (domain does not resolve)
dynatrace.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
dyndns.org — policy unreadable (HTML instead of text)
dzen.ru — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
e2ro.com — policy unreadable (domain does not resolve)
ea.com — blocks: Bytespider, meta-externalagent, cohere-ai, cohere-training-data-crawler, AI2Bot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
easebar.com — policy unreadable (domain does not resolve)
ebay.co.uk — blocks: GPTBot, ClaudeBot, Applebot-Extended, PerplexityBot, CCBot, Bytespider, meta-externalagent, Amazonbot, anthropic-ai — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Google-Extended, Perplexity-User, DuckAssistBot, MistralAI-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
ebay.com — blocks: GPTBot, ClaudeBot, Applebot-Extended, PerplexityBot, CCBot, Bytespider, meta-externalagent, Amazonbot, anthropic-ai — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Google-Extended, Perplexity-User, DuckAssistBot, MistralAI-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
ebay.de — blocks: GPTBot, ClaudeBot, Applebot-Extended, PerplexityBot, CCBot, Bytespider, meta-externalagent, Amazonbot, anthropic-ai — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Google-Extended, Perplexity-User, DuckAssistBot, MistralAI-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
economist.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, anthropic-ai, Claude-Web — restricts: meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
edgcdn.net — policy unreadable (domain does not resolve)
edgecdn.ru — policy unreadable (host answered nothing)
edmunds.com — blocks: GPTBot, ClaudeBot, Bytespider, meta-externalagent, cohere-training-data-crawler, AI2Bot, DeepSeekBot — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, GrokBot, PanguBot, Gemini-Deep-Research
eeroup.com — blocks no tracked AI bots
elasticbeanstalk.com — policy unreadable (host answered nothing)
elpais.com — blocks: ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, CCBot, Bytespider, Amazonbot, DuckAssistBot, MistralAI-User, omgilibot, DeepSeekBot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, Google-Extended, Applebot-Extended, meta-externalagent, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, GrokBot, PanguBot, Gemini-Deep-Research
enacdn.net — policy unreadable (domain does not resolve)
epa.gov — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
epam.com — policy unreadable (refused (401/403))
epicgames.com — policy unreadable (refused (401/403))
epicurious.com — blocks: ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, cohere-ai, cohere-training-data-crawler, Diffbot, YouBot, Timpibot, Meta-ExternalFetcher, PanguBot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, anthropic-ai, Claude-Web, omgili, omgilibot, ImagesiftBot, AI2Bot, DeepSeekBot, GrokBot, Gemini-Deep-Research
eset.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
espn.com — blocks: GPTBot, ChatGPT-User, Google-Extended, CCBot, Bytespider, anthropic-ai, omgili, omgilibot — restricts: OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
etsy.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
eu-1-id5-sync.com — policy unreadable (domain does not resolve)
eu.com — blocks no tracked AI bots
europa.eu — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
eventbrite.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
example.com — blocks no tracked AI bots
exp-tas.com — policy unreadable (domain does not resolve)
expedia.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
expireddomains.com — blocks no tracked AI bots
eye4.cn — policy unreadable (HTTP 500)
ezviz7.com — policy unreadable (host answered nothing)
ezvizlife.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
f5.com — blocks no tracked AI bots
facebook.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
facebook.net — policy unreadable (domain does not resolve)
fandom.com — policy unreadable (refused (401/403))
fast.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
fastly-edge.com — policy unreadable (domain does not resolve)
fb.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
fbsbx.com — blocks no tracked AI bots
fda.gov — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
featureassets.org — policy unreadable (host answered nothing)
fidelity.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
fifa.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
figma.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, CCBot, Bytespider, cohere-ai, omgili, omgilibot — restricts: Applebot-Extended, Perplexity-User, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-training-data-crawler, Diffbot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
firefox.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
firetvcaptiveportal.com — policy unreadable (host answered nothing)
fiverr.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
flashscore.com — blocks: Bytespider, DuckAssistBot, cohere-ai, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, AI2Bot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, meta-externalagent, Amazonbot, MistralAI-User, anthropic-ai, Claude-Web, cohere-training-data-crawler, Meta-ExternalFetcher, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
flashtalking.com — policy unreadable (host answered nothing)
flickr.com — blocks: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, PerplexityBot — default-deny robots.txt
focus.de — policy unreadable (refused (401/403))
fontawesome.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
foodnetwork.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
fool.com — blocks: Bytespider — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
forbes.com — blocks: GPTBot, ClaudeBot, Applebot-Extended, PerplexityBot, CCBot, Bytespider, meta-externalagent, Amazonbot, anthropic-ai, cohere-ai, Diffbot, omgili, omgilibot, ImagesiftBot — restricts: ChatGPT-User, Google-Extended, Perplexity-User, DuckAssistBot, MistralAI-User, Claude-Web, cohere-training-data-crawler, YouBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
force.com — policy unreadable (refused (401/403))
forms.gle — policy unreadable (host answered nothing)
forter.com — blocks: CCBot, Bytespider, meta-externalagent, Diffbot, omgili, omgilibot, ImagesiftBot — restricts: DuckAssistBot, Claude-Web, cohere-training-data-crawler, YouBot, Timpibot, Meta-ExternalFetcher, AI2Bot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
foxnews.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
free.fr — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
frontiersin.org — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
ft.com — blocks: ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgili, omgilibot, Meta-ExternalFetcher — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Amazonbot, DuckAssistBot, MistralAI-User, cohere-training-data-crawler, YouBot, ImagesiftBot, Timpibot, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
fwmrm.net — blocks no tracked AI bots — publishes llms.txt
g.co — blocks no tracked AI bots
g.page — policy unreadable (HTML instead of text)
gamepass.com — policy unreadable (host answered nothing)
gamespot.com — policy unreadable (refused (401/403))
gandi-ns.fr — policy unreadable (domain does not resolve)
gandi.net — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
gcdn.co — policy unreadable (host answered nothing)
genius.com — policy unreadable (refused (401/403))
geobasket.ru — policy unreadable (domain does not resolve)
github.com — blocks: Bytespider — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
github.io — policy unreadable (HTML instead of text)
gitlab.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
glassdoor.com — blocks: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgili, omgilibot, YouBot — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, meta-externalagent, DuckAssistBot, MistralAI-User, cohere-training-data-crawler, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
globalsign.com — blocks: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
globo.com — blocks: GPTBot, Google-Extended, CCBot — restricts: ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Applebot-Extended, PerplexityBot, Perplexity-User, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
gmail.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
gnu.org — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
go-mpulse.net — policy unreadable (domain does not resolve)
go.com — blocks: GPTBot, Google-Extended — restricts: ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
godaddy.com — policy unreadable (refused (401/403))
goo.gl — blocks no tracked AI bots
goodreads.com — blocks: GPTBot, CCBot — restricts: ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
google.cn — blocks no tracked AI bots
google.co.uk — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
google.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
google.com.br — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
google.com.hk — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
google.de — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
googleblog.com — blocks no tracked AI bots
googledomains.com — policy unreadable (HTML instead of text)
googletagservices.com — blocks no tracked AI bots
googlezip.net — policy unreadable (domain does not resolve)
goskope.com — policy unreadable (domain does not resolve)
gosuslugi.ru — policy unreadable (host answered nothing)
gov.uk — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
grammarly.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
grammarly.io — policy unreadable (domain does not resolve)
gumgum.com — blocks no tracked AI bots — publishes llms.txt
gwfb.net — policy unreadable (domain does not resolve)
h264.io — policy unreadable (domain does not resolve)
harvard.edu — blocks no tracked AI bots
hbr.org — blocks: GPTBot, Google-Extended, CCBot — restricts: ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Applebot-Extended, PerplexityBot, Perplexity-User, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
hcaptcha.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
healthline.com — blocks: GPTBot, ClaudeBot, Applebot-Extended, CCBot, Bytespider, meta-externalagent, Amazonbot, anthropic-ai, Diffbot, omgili, Timpibot — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, DuckAssistBot, MistralAI-User, Claude-Web, cohere-ai, cohere-training-data-crawler, omgilibot, YouBot, ImagesiftBot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
herokuapp.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
herokudns.com — policy unreadable (domain does not resolve)
heytapdl.com — policy unreadable (domain does not resolve)
heytapmobile.com — policy unreadable (domain does not resolve)
hichina.com — policy unreadable (HTML instead of text)
hicloud.com — policy unreadable (HTML instead of text)
hicloudcam.com — blocks no tracked AI bots
hihonorcloud.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
hilton.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
hm.com — policy unreadable (refused (401/403))
homedepot.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
hostgator.com — policy unreadable (refused (401/403))
hostgator.com.br — policy unreadable (refused (401/403))
hostinger.com — blocks: CCBot, Bytespider — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
hotels.com — policy unreadable (host answered nothing)
hotjar.com — blocks no tracked AI bots
hp.com — blocks no tracked AI bots — restricts: Applebot-Extended, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
hstgr.net — policy unreadable (host answered nothing)
huawei.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
hubspot.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
huffpost.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgilibot, Timpibot, Meta-ExternalFetcher — restricts: MistralAI-User, cohere-training-data-crawler, omgili, YouBot, ImagesiftBot, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
huggingface.co — blocks no tracked AI bots
hulu.com — blocks: Perplexity-User, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, ImagesiftBot, Timpibot, AI2Bot, PanguBot, Gemini-Deep-Research — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, YouBot, Meta-ExternalFetcher, DeepSeekBot, GrokBot — default-deny robots.txt
ibm.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
ibyteimg.com — policy unreadable (domain does not resolve)
icloud-content.com — policy unreadable (HTML instead of text)
icloud.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
id5-sync.com — policy unreadable (HTML instead of text)
ieee.org — policy unreadable (HTTP 202)
ietf.org — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
ign.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot — restricts: Google-Extended, omgilibot, Gemini-Deep-Research
iiko.it — policy unreadable (domain does not resolve)
ikea.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
imcmdb.net — policy unreadable (domain does not resolve)
imdb.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: Amazonbot, DuckAssistBot — default-deny robots.txt
imgsmail.ru — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
imgur.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
immedia-semi.com — policy unreadable (domain does not resolve)
impervadns.net — policy unreadable (domain does not resolve)
imrworldwide.com — policy unreadable (domain does not resolve)
indeed.com — blocks: omgili, omgilibot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
independent.co.uk — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
indiatimes.com — blocks: meta-externalagent — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
inmobi.com — blocks no tracked AI bots
inner-active.mobi — policy unreadable (host answered nothing)
instacart.com — blocks: GPTBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: ChatGPT-User, OAI-SearchBot — default-deny robots.txt
instagram.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
intel.com — blocks no tracked AI bots — restricts: OAI-SearchBot, Claude-User, Claude-SearchBot, Applebot-Extended, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
intercom.io — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
internetwarriors.net — policy unreadable (HTML instead of text)
intuit.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
investopedia.com — blocks: ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
ioref.io — policy unreadable (domain does not resolve)
ip-api.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
ipify.org — blocks no tracked AI bots
ipinfo.io — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
ipv4only.arpa — policy unreadable (domain does not resolve)
irs.gov — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
iso.org — policy unreadable (refused (401/403))
issuu.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
it.com — blocks no tracked AI bots
itch.io — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
ivi.ru — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
jd.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
jetbrains.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
jomodns.com — policy unreadable (host answered nothing)
jotform.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
jquery.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
kaggle.com — blocks no tracked AI bots — publishes llms.txt
kaspersky-labs.com — policy unreadable (host answered nothing)
kaspersky.com — blocks no tracked AI bots — restricts: Claude-SearchBot, Bytespider, meta-externalagent, Amazonbot, anthropic-ai, cohere-ai, cohere-training-data-crawler, omgili, omgilibot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
kayak.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
kbb.com — blocks: Amazonbot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
keenetic.io — policy unreadable (domain does not resolve)
khanacademy.org — policy unreadable (HTML instead of text)
kick.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
klaviyo.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
kleinanzeigen.de — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
kotaku.com — blocks: CCBot, Bytespider, meta-externalagent, cohere-ai, Diffbot, omgilibot, ImagesiftBot, Timpibot, AI2Bot, PanguBot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-training-data-crawler, omgili, YouBot, Meta-ExternalFetcher, DeepSeekBot, GrokBot, Gemini-Deep-Research
kroger.com — policy unreadable (host answered nothing)
ks-cdn.com — policy unreadable (domain does not resolve)
ksyuncdn.com — policy unreadable (domain does not resolve)
kunluncan.com — policy unreadable (host answered nothing)
kwai-pro.com — blocks no tracked AI bots
kwai.com — blocks: Bytespider — publishes llms.txt
kwai.net — policy unreadable (host answered nothing)
kwaipros.com — blocks no tracked AI bots
latimes.com — blocks: GPTBot, ClaudeBot, Bytespider, Diffbot, omgili, omgilibot, ImagesiftBot — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, YouBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
launchdarkly.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
launchpad.net — blocks: GPTBot, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, PanguBot — restricts: ChatGPT-User, Perplexity-User, MistralAI-User, anthropic-ai, Claude-Web, DeepSeekBot, GrokBot, Gemini-Deep-Research
lefigaro.fr — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, CCBot, Amazonbot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgili, omgilibot, YouBot, Gemini-Deep-Research — restricts: Applebot-Extended, PerplexityBot, Perplexity-User, Bytespider, meta-externalagent, DuckAssistBot, cohere-training-data-crawler, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot
leiniao.com — blocks no tracked AI bots
lemonde.fr — blocks: ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, Timpibot, AI2Bot, DeepSeekBot, PanguBot, Gemini-Deep-Research — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, Perplexity-User, meta-externalagent, ImagesiftBot, Meta-ExternalFetcher, GrokBot
lencr.org — policy unreadable (HTML instead of text)
lenovo.com — policy unreadable (refused (401/403))
lg.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
lgtvcommon.com — policy unreadable (domain does not resolve)
liadm.com — policy unreadable (domain does not resolve)
libp2p.direct — policy unreadable (domain does not resolve)
life360.com — policy unreadable (refused (401/403))
liftoff.io — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
lijit.com — policy unreadable (refused (401/403))
line.me — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
linkedin.com — blocks: GPTBot, ChatGPT-User, ClaudeBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: OAI-SearchBot, Claude-SearchBot — default-deny robots.txt
linktr.ee — blocks: Bytespider, meta-externalagent, anthropic-ai, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, AI2Bot, DeepSeekBot, GrokBot, PanguBot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, CCBot, Amazonbot, DuckAssistBot, MistralAI-User, Claude-Web, Meta-ExternalFetcher, Gemini-Deep-Research — default-deny robots.txt — publishes llms.txt
linode.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
list-manage.com — policy unreadable (host answered nothing)
live-video.net — policy unreadable (domain does not resolve)
live.com — policy unreadable (HTML instead of text)
live.net — policy unreadable (domain does not resolve)
livejournal.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
ln-msedge.net — policy unreadable (domain does not resolve)
loc.gov — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
lowes.com — policy unreadable (refused (401/403))
lsrelayaccess.com — policy unreadable (host answered nothing)
lwsdns.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
macromedia.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
magnit.ru — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
mail.ru — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
mailchi.mp — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
mailchimp.com — blocks no tracked AI bots — publishes llms.txt
mangosip.ru — policy unreadable (host answered nothing)
marketwatch.com — blocks: ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: GPTBot, ChatGPT-User, OAI-SearchBot — default-deny robots.txt
markmonitor.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
marriott.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
max.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
mayoclinic.org — policy unreadable (refused (401/403))
mcafee.com — policy unreadable (host answered nothing)
mckinsey.com — policy unreadable (host answered nothing)
mdpi.com — policy unreadable (refused (401/403))
me.com — policy unreadable (HTML instead of text)
media-amazon.com — policy unreadable (domain does not resolve)
media.net — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
mediafire.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
mediatek.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
medium.com — blocks: GPTBot, ClaudeBot, Applebot-Extended, Bytespider, meta-externalagent, Amazonbot — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, CCBot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
mega.co.nz — blocks no tracked AI bots
meraki.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: CCBot — default-deny robots.txt
mercadolibre.com.ar — blocks: GPTBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Perplexity-User, Amazonbot — restricts: OAI-SearchBot, Claude-SearchBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
mercadolivre.com.br — blocks: GPTBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Perplexity-User, Amazonbot — restricts: OAI-SearchBot, Claude-SearchBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
merriam-webster.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
meta.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
metacritic.com — blocks: GPTBot, OAI-SearchBot, CCBot, ImagesiftBot — restricts: ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
mhverifier.ru — policy unreadable (refused (401/403))
mi.com — policy unreadable (refused (401/403))
microsoft.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
microsoftonline.com — policy unreadable (domain does not resolve)
miit.gov.cn — policy unreadable (host answered nothing)
mikrotik.com — blocks no tracked AI bots
minecraft.net — policy unreadable (host answered nothing)
mistral.ai — blocks no tracked AI bots — publishes llms.txt
mit.edu — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
miui.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
miwifi.com — blocks no tracked AI bots
mixpanel.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
mlb.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
mobile.de — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
mobolize.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
moloco.com — blocks: Bytespider — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
monster.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
mozilla.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
mozilla.org — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
msftauth.net — policy unreadable (domain does not resolve)
msftconnecttest.com — policy unreadable (host answered nothing)
msftncsi.com — policy unreadable (host answered nothing)
msidentity.com — policy unreadable (domain does not resolve)
msn.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, PanguBot — restricts: GrokBot, Gemini-Deep-Research
mtgglobals.com — blocks no tracked AI bots
mts.ru — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
my.com — policy unreadable (domain does not resolve)
mybluehost.me — policy unreadable (host answered nothing)
myfritz.net — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
myhuaweicloud.com — policy unreadable (host answered nothing)
mynetname.net — policy unreadable (domain does not resolve)
myqcloud.com — policy unreadable (domain does not resolve)
myshopify.com — blocks no tracked AI bots
myspace.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
mysql.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
mzstatic.com — policy unreadable (host answered nothing)
name-services.com — policy unreadable (domain does not resolve)
name.com — blocks: meta-externalagent — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
namebrightdns.com — policy unreadable (domain does not resolve)
namecheap.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
nasa.gov — blocks no tracked AI bots
nationalgeographic.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
nature.com — blocks: GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot, CCBot, Bytespider, meta-externalagent, Amazonbot, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot — restricts: OAI-SearchBot, Claude-User, Claude-SearchBot, Perplexity-User, DuckAssistBot, MistralAI-User, cohere-training-data-crawler, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
naver.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
nba.com — blocks: GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot, CCBot, Amazonbot, anthropic-ai, Claude-Web — restricts: Claude-User, Claude-SearchBot, Perplexity-User, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
nbcnews.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgili, omgilibot, YouBot, Meta-ExternalFetcher — restricts: DuckAssistBot, cohere-training-data-crawler, ImagesiftBot, Timpibot, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
nease.net — policy unreadable (host answered nothing)
nel.goog — policy unreadable (domain does not resolve)
nerdwallet.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
netangels.ru — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
netease.com — blocks no tracked AI bots
netflix.com — blocks: Applebot-Extended, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot — default-deny robots.txt
netflix.net — blocks: Applebot-Extended, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot — default-deny robots.txt
netlify.app — blocks no tracked AI bots — publishes llms.txt
netlify.com — blocks no tracked AI bots — publishes llms.txt
networkadvertising.org — blocks no tracked AI bots
newrelic.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
newsweek.com — blocks: GPTBot, ChatGPT-User, ClaudeBot, Applebot-Extended, PerplexityBot, CCBot, Bytespider, Amazonbot, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgili, omgilibot — restricts: OAI-SearchBot, Claude-User, Claude-SearchBot, Google-Extended, Perplexity-User, meta-externalagent, DuckAssistBot, MistralAI-User, cohere-training-data-crawler, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
newyorker.com — blocks: ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, cohere-ai, cohere-training-data-crawler, Diffbot, YouBot, Timpibot, Meta-ExternalFetcher, PanguBot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, anthropic-ai, Claude-Web, omgili, omgilibot, ImagesiftBot, AI2Bot, DeepSeekBot, GrokBot, Gemini-Deep-Research
nexusmods.com — policy unreadable (refused (401/403))
nfl.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
nflximg.com — policy unreadable (host answered nothing)
nflxso.net — policy unreadable (domain does not resolve)
nflxvideo.net — policy unreadable (host answered nothing)
ngenix.net — policy unreadable (refused (401/403))
nginx.com — policy unreadable (HTML instead of text)
nginx.org — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
nhl.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
nic.direct — policy unreadable (host answered nothing)
nic.ru — policy unreadable (HTML instead of text)
nih.gov — policy unreadable (refused (401/403))
nike.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
nikkei.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, Timpibot, Meta-ExternalFetcher, AI2Bot — restricts: MistralAI-User, ImagesiftBot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
nintendo.com — blocks: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, Timpibot, AI2Bot, PanguBot — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, YouBot, ImagesiftBot, Meta-ExternalFetcher, DeepSeekBot, GrokBot, Gemini-Deep-Research
nintendo.net — policy unreadable (domain does not resolve)
nist.gov — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
nmrodam.com — policy unreadable (domain does not resolve)
no-ip.com — policy unreadable (host answered nothing)
noaa.gov — policy unreadable (refused (401/403))
nominetdns.uk — policy unreadable (domain does not resolve)
note.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
notion.so — blocks: Amazonbot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
npmjs.com — policy unreadable (refused (401/403))
npr.org — policy unreadable (host answered nothing)
nr-data.net — policy unreadable (domain does not resolve)
nstld.com — policy unreadable (domain does not resolve)
ntp.org — blocks no tracked AI bots
nvidia.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
nypost.com — blocks: ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, CCBot, Bytespider, meta-externalagent, Amazonbot, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgili, omgilibot, ImagesiftBot, Meta-ExternalFetcher — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, Perplexity-User, DuckAssistBot, MistralAI-User, cohere-training-data-crawler, YouBot, Timpibot, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
nytimes.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher — restricts: Amazonbot, MistralAI-User, cohere-training-data-crawler, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
odoo.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
office.com — policy unreadable (HTML instead of text)
office365.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
ok.ru — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
okcdn.ru — policy unreadable (domain does not resolve)
okta.com — blocks no tracked AI bots — publishes llms.txt
on.aws — policy unreadable (domain does not resolve)
one.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
onelink.me — policy unreadable (HTML instead of text)
onet.pl — blocks: Bytespider, meta-externalagent, Amazonbot, cohere-ai, Diffbot, omgilibot, YouBot, Timpibot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-training-data-crawler, omgili, ImagesiftBot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
onetag-sys.com — policy unreadable (domain does not resolve)
online-metrix.net — policy unreadable (domain does not resolve)
openai.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
openstreetmap.org — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
opentable.com — policy unreadable (host answered nothing)
opera-api.com — policy unreadable (domain does not resolve)
opera-api2.com — policy unreadable (domain does not resolve)
opera.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
optimizely.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
oracle.com — policy unreadable (refused (401/403))
oraclecloud.com — policy unreadable (refused (401/403))
orderbox-dns.com — policy unreadable (domain does not resolve)
otto.de — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
oup.com — policy unreadable (HTTP 202)
outlook.com — policy unreadable (HTML instead of text)
ovh.net — policy unreadable (HTML instead of text)
ovscdns.com — policy unreadable (domain does not resolve)
oxylabs.io — blocks no tracked AI bots — publishes llms.txt
ozon.ru — policy unreadable (host answered nothing)
ozone.ru — policy unreadable (domain does not resolve)
pages.dev — policy unreadable (HTTP 522)
palmplaystore.com — blocks no tracked AI bots
paloaltonetworks.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
pangle.io — policy unreadable (domain does not resolve)
paramountplus.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
parklogic.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
patreon.com — blocks: PerplexityBot, Timpibot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
paypal.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
pccc.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
pcgamer.com — blocks: Applebot-Extended, Bytespider, meta-externalagent, Amazonbot, Diffbot, omgili, YouBot, AI2Bot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, CCBot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, omgilibot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
pendo.io — blocks no tracked AI bots — publishes llms.txt
people.com — blocks: ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
perplexity.ai — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
pexels.com — blocks: GPTBot, ClaudeBot, Applebot-Extended, CCBot, Bytespider — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
php.net — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
pikabu.ru — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
pinimg.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: DuckAssistBot — default-deny robots.txt
pinterest.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: DuckAssistBot — default-deny robots.txt
pixabay.com — blocks: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
pki.goog — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
playrix.com — policy unreadable (HTML instead of text)
playstation.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
playstation.net — policy unreadable (domain does not resolve)
plesk.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
politico.com — policy unreadable (refused (401/403))
polygon.com — blocks: GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot, CCBot, Bytespider, anthropic-ai, Claude-Web, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot — restricts: OAI-SearchBot, Claude-User, Claude-SearchBot, Perplexity-User, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, cohere-ai, cohere-training-data-crawler, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
porkbun.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
presage.io — policy unreadable (domain does not resolve)
primevideo.com — blocks: ChatGPT-User, Claude-User, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, Applebot-Extended, Amazonbot, anthropic-ai, Claude-Web, omgilibot
princeton.edu — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
prnewswire.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
prodregistryv2.org — policy unreadable (host answered nothing)
pushy.io — blocks no tracked AI bots
pv-cdn.net — policy unreadable (domain does not resolve)
pvp.net — policy unreadable (domain does not resolve)
pypi.org — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
python.org — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
qlivecdn.com — policy unreadable (domain does not resolve)
qq.com — blocks no tracked AI bots
qualtrics.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
quantserve.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
queniuaa.com — policy unreadable (host answered nothing)
quickconnect.to — blocks no tracked AI bots
quizlet.com — policy unreadable (refused (401/403))
quora.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: Google-Extended, Amazonbot — default-deny robots.txt
rakuten.co.jp — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
rakuten.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
rbxcdn.com — policy unreadable (domain does not resolve)
readthedocs.io — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
realtor.com — blocks: meta-externalagent, Meta-ExternalFetcher — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
recaptcha.net — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
reddit.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
redfin.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
redhat.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
reg.ru — policy unreadable (HTML instead of text)
registrar-servers.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
repubblica.it — blocks: GPTBot, Google-Extended, Applebot-Extended, PerplexityBot, CCBot, Bytespider, DuckAssistBot, anthropic-ai, cohere-ai, Diffbot, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, PanguBot — restricts: ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Perplexity-User, meta-externalagent, MistralAI-User, Claude-Web, cohere-training-data-crawler, omgili, AI2Bot, DeepSeekBot, GrokBot, Gemini-Deep-Research — publishes llms.txt
researchgate.net — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
resolver.arpa — policy unreadable (domain does not resolve)
reuters.com — blocks: GPTBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: ChatGPT-User, OAI-SearchBot — default-deny robots.txt
ring.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
ripn.net — policy unreadable (domain does not resolve)
rlcdn.com — blocks no tracked AI bots
robinhood.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
roblox.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
rocket-cdn.com — policy unreadable (host answered nothing)
rocketmoney.dev — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
roku.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
rottentomatoes.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
rt.ru — policy unreadable (HTML instead of text)
run.app — policy unreadable (host answered nothing)
ryanair.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
safebrowsing.apple — policy unreadable (domain does not resolve)
sagepub.com — blocks: CCBot, Bytespider, meta-externalagent, Amazonbot, cohere-ai, Diffbot, omgili, omgilibot, ImagesiftBot — restricts: OAI-SearchBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, DuckAssistBot, MistralAI-User, Claude-Web, cohere-training-data-crawler, YouBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
salesforce.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
samsung.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
samsungacr.com — policy unreadable (domain does not resolve)
samsungapps.com — policy unreadable (domain does not resolve)
samsungcloud.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
samsungcloudsolution.com — policy unreadable (domain does not resolve)
samsungiotcloud.com — policy unreadable (domain does not resolve)
samsungosp.com — blocks no tracked AI bots
samsungqbe.com — policy unreadable (domain does not resolve)
sberbank.ru — policy unreadable (host answered nothing)
sc-cdn.net — policy unreadable (domain does not resolve)
sc-gw.com — policy unreadable (domain does not resolve)
scdn.co — blocks no tracked AI bots
schwab.com — policy unreadable (HTML instead of text)
sciencedirect.com — policy unreadable (refused (401/403))
scribd.com — policy unreadable (HTML instead of text)
sec.gov — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
sedo.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
seedtag.com — blocks no tracked AI bots — publishes llms.txt
seekingalpha.com — blocks: ChatGPT-User, ClaudeBot, Claude-User, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, PanguBot — restricts: GPTBot, OAI-SearchBot, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Amazonbot, anthropic-ai, Claude-Web, DeepSeekBot, GrokBot, Gemini-Deep-Research
segment.io — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
selectel.ru — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
semafor.com — blocks: GPTBot — restricts: ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
sentinelone.net — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
sentry.io — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
sephora.com — policy unreadable (refused (401/403))
seriouseats.com — blocks: ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
service.gov.uk — policy unreadable (domain does not resolve)
seznam.cz — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
sfx.ms — policy unreadable (domain does not resolve)
shalltry.com — blocks no tracked AI bots
share-dns.com — policy unreadable (HTTP 409)
share.google — blocks no tracked AI bots
sharepoint.com — policy unreadable (host answered nothing)
sharethrough.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
shein.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
shiabank.com — policy unreadable (host answered nothing)
shifen.com — policy unreadable (host answered nothing)
shop.app — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
shopee.co.id — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
shopee.com.br — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
shopeemobile.com — policy unreadable (host answered nothing)
shopify.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
shopifysvc.com — policy unreadable (domain does not resolve)
sina.com.cn — blocks no tracked AI bots
singtelchina.cn — policy unreadable (domain does not resolve)
singular.net — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
skyhigh.cloud — policy unreadable (refused (401/403))
skype.com — policy unreadable (HTML instead of text)
skyscanner.com — blocks: meta-externalagent — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
slack.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
slideshare.net — policy unreadable (HTML instead of text)
smartadserver.com — blocks no tracked AI bots
smartthings.com — blocks no tracked AI bots
snapchat.com — blocks: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, Amazonbot, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, PanguBot — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, DuckAssistBot, MistralAI-User, DeepSeekBot, GrokBot, Gemini-Deep-Research
snopes.com — blocks: GPTBot, ChatGPT-User, ClaudeBot, Applebot-Extended, PerplexityBot, CCBot, Bytespider, meta-externalagent, Amazonbot, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot
sohu.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: Bytespider — default-deny robots.txt
sophos.com — blocks: Amazonbot, DeepSeekBot — publishes llms.txt
soundcloud.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, CCBot, Bytespider, meta-externalagent, Amazonbot, anthropic-ai, Claude-Web, cohere-ai, omgili, omgilibot — restricts: Perplexity-User, DuckAssistBot, MistralAI-User, cohere-training-data-crawler, Diffbot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
sourceforge.net — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
southwest.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
spaceweb.pro — policy unreadable (host answered nothing)
speedtest.net — blocks no tracked AI bots
spiegel.de — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Applebot-Extended, CCBot, Bytespider, meta-externalagent, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, AI2Bot, DeepSeekBot, PanguBot — restricts: Google-Extended, PerplexityBot, Perplexity-User, Amazonbot, DuckAssistBot, Meta-ExternalFetcher, GrokBot, Gemini-Deep-Research
spo-msedge.net — policy unreadable (domain does not resolve)
spotify.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
spotifycdn.com — blocks no tracked AI bots
spov-msedge.net — policy unreadable (domain does not resolve)
springer.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
squarespace.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
squarespacedns.com — policy unreadable (domain does not resolve)
ssl-images-amazon.com — policy unreadable (domain does not resolve)
stackadapt.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
stackoverflow.com — policy unreadable (HTTP 418)
stanford.edu — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
starlink.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
state.gov — policy unreadable (refused (401/403))
static.microsoft — policy unreadable (domain does not resolve)
statista.com — blocks: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, PanguBot — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, MistralAI-User, DeepSeekBot, GrokBot, Gemini-Deep-Research
statsigapi.net — policy unreadable (domain does not resolve)
stbid.ru — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
steamcommunity.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
steampowered.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
steamserver.net — policy unreadable (domain does not resolve)
steamstatic.com — policy unreadable (domain does not resolve)
stripe.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
substack.com — policy unreadable (refused (401/403))
supercell.com — blocks no tracked AI bots
supertms.com — policy unreadable (host answered nothing)
surveymonkey.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
svc.ms — blocks no tracked AI bots
synology.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
t-mobile.com — policy unreadable (refused (401/403))
t-msedge.net — policy unreadable (domain does not resolve)
t-online.de — blocks: GPTBot, ClaudeBot, Applebot-Extended, PerplexityBot, CCBot, Bytespider, meta-externalagent — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Google-Extended, Perplexity-User, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
t-s1-msedge.net — policy unreadable (domain does not resolve)
t.co — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
t.me — blocks no tracked AI bots
tailscale.com — blocks no tracked AI bots — publishes llms.txt
tandfonline.com — blocks: GPTBot — restricts: ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
taobao.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
target.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
tbank.ru — policy unreadable (host answered nothing)
tbcache.com — policy unreadable (host answered nothing)
teads.tv — policy unreadable (HTML instead of text)
teamviewer.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
techcrunch.com — blocks: GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgili — restricts: OAI-SearchBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, cohere-training-data-crawler, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
ted.com — blocks: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, Amazonbot, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, ImagesiftBot, Timpibot, DeepSeekBot, GrokBot, PanguBot — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, DuckAssistBot, MistralAI-User, YouBot, Meta-ExternalFetcher, AI2Bot, Gemini-Deep-Research
telecid.ru — policy unreadable (refused (401/403))
telegram.me — policy unreadable (HTML instead of text)
telegram.org — blocks no tracked AI bots
telegraph.co.uk — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgili, omgilibot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, PanguBot — restricts: Google-Extended, cohere-training-data-crawler, YouBot, GrokBot, Gemini-Deep-Research
telekom.net — policy unreadable (HTML instead of text)
telephony.goog — policy unreadable (domain does not resolve)
temu.com — blocks: Bytespider — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot
tencent-cloud.net — policy unreadable (domain does not resolve)
tencent.com — blocks no tracked AI bots
theathletic.com — blocks: GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Bytespider, Amazonbot, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgili, omgilibot — restricts: OAI-SearchBot, Claude-User, Claude-SearchBot, Applebot-Extended, Perplexity-User, meta-externalagent, DuckAssistBot, MistralAI-User, cohere-training-data-crawler, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
theatlantic.com — blocks: ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgili, omgilibot, Timpibot, Meta-ExternalFetcher — restricts: cohere-training-data-crawler, YouBot, ImagesiftBot, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
theguardian.com — blocks: ClaudeBot, Claude-User, Claude-SearchBot, Applebot-Extended, PerplexityBot, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, anthropic-ai, YouBot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, Google-Extended, Perplexity-User, MistralAI-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
theinformation.com — policy unreadable (refused (401/403))
thenai.org — blocks no tracked AI bots
theverge.com — blocks: ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot — restricts: OAI-SearchBot, GrokBot, PanguBot, Gemini-Deep-Research
threads.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
tiktok.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, anthropic-ai, AI2Bot, Gemini-Deep-Research — restricts: Amazonbot, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, DeepSeekBot, GrokBot, PanguBot
tiktokcdn-eu.com — policy unreadable (domain does not resolve)
tiktokcdn-us.com — policy unreadable (domain does not resolve)
tiktokcdn.com — policy unreadable (domain does not resolve)
tiktokv.com — blocks no tracked AI bots
tiktokv.eu — policy unreadable (domain does not resolve)
tiktokv.us — policy unreadable (domain does not resolve)
tiktokw.us — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
time.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
timeweb.ru — policy unreadable (HTML instead of text)
tinyurl.com — blocks no tracked AI bots
tld-servers.ru — policy unreadable (domain does not resolve)
tm-azurefd.net — policy unreadable (domain does not resolve)
tp-link.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
tplinkcloud.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
tplinknbu.com — policy unreadable (domain does not resolve)
tradingview.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
translate.goog — blocks no tracked AI bots
trendmicro.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
tripadvisor.com — blocks: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, Amazonbot, cohere-ai, YouBot, PanguBot — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-training-data-crawler, Diffbot, omgili, omgilibot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, Gemini-Deep-Research
triplinkintl.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
trueconf.net — policy unreadable (host answered nothing)
trustpilot.com — blocks: Applebot-Extended, Perplexity-User, Bytespider, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, CCBot, meta-externalagent, Meta-ExternalFetcher — default-deny robots.txt
tsyndicate.com — policy unreadable (domain does not resolve)
ttdns2.com — policy unreadable (domain does not resolve)
ttvnw.net — policy unreadable (domain does not resolve)
tumblr.com — blocks: ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent, Amazonbot, anthropic-ai, omgili, omgilibot, YouBot, ImagesiftBot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Bytespider, DuckAssistBot, MistralAI-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
turn.com — policy unreadable (domain does not resolve)
twc.com — policy unreadable (host answered nothing)
twitch.tv — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
twitter.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: Google-Extended, meta-externalagent, Meta-ExternalFetcher — default-deny robots.txt
uber.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
ubereats.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
ubi.com — policy unreadable (HTML instead of text)
ubnt.com — blocks no tracked AI bots
ubuntu.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
udemy.com — policy unreadable (refused (401/403))
ui-dns.com — policy unreadable (domain does not resolve)
ui.com — blocks no tracked AI bots
uk.com — blocks no tracked AI bots
umich.edu — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
un.org — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
unesco.org — policy unreadable (refused (401/403))
united.com — policy unreadable (host answered nothing)
unity3d.com — blocks: Bytespider — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
unpkg.com — blocks no tracked AI bots — publishes llms.txt
unsplash.com — blocks: ClaudeBot, Claude-SearchBot, Bytespider, ImagesiftBot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
uol.com.br — blocks: ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, PanguBot, Gemini-Deep-Research — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, DeepSeekBot, GrokBot
ups.com — policy unreadable (HTML instead of text)
urbandictionary.com — blocks no tracked AI bots
usa.gov — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
usatoday.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, PanguBot, Gemini-Deep-Research — restricts: GrokBot
usda.gov — policy unreadable (refused (401/403))
userapi.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
usercontent.goog — blocks no tracked AI bots
usgovcloudapi.net — policy unreadable (domain does not resolve)
usps.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
uspto.gov — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
vanguard.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
vecdnlb.com — policy unreadable (domain does not resolve)
vedcdnlb.com — policy unreadable (domain does not resolve)
vedsalb.com — policy unreadable (domain does not resolve)
vercel-dns-016.com — blocks no tracked AI bots
vercel-dns-017.com — blocks no tracked AI bots
vercel-dns-3.com — policy unreadable (domain does not resolve)
vercel.app — blocks no tracked AI bots — publishes llms.txt
vercel.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
verisign.com — blocks no tracked AI bots
viber.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
vidaahub.com — policy unreadable (domain does not resolve)
vimeo.com — blocks: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, anthropic-ai, Claude-Web, Diffbot, omgili, omgilibot, ImagesiftBot — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, cohere-ai, cohere-training-data-crawler, YouBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
virginm.net — policy unreadable (domain does not resolve)
visma.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
visualstudio.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
vivo.com.cn — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
vivoglobal.com — blocks no tracked AI bots
vk.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
vk.ru — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
vkontakte.ru — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
vkuser.net — policy unreadable (domain does not resolve)
vkuserphoto.ru — policy unreadable (HTTP 418)
volcfcdndvs.com — policy unreadable (domain does not resolve)
volcgslb.com — policy unreadable (domain does not resolve)
vox.com — blocks: ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot — restricts: OAI-SearchBot, GrokBot, PanguBot, Gemini-Deep-Research
vrbo.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
vungle.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
w3.org — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
wa.me — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
wac-msedge.net — policy unreadable (domain does not resolve)
walmart.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
warnerbros.com — blocks no tracked AI bots
washington.edu — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
washingtonpost.com — blocks: ClaudeBot, Applebot-Extended, PerplexityBot, CCBot, Bytespider, meta-externalagent, Amazonbot, anthropic-ai, Claude-Web, cohere-training-data-crawler, Diffbot, omgili, omgilibot, ImagesiftBot, Meta-ExternalFetcher — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Google-Extended, Perplexity-User, DuckAssistBot, MistralAI-User, cohere-ai, YouBot, Timpibot, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
wattpad.com — blocks: GPTBot — restricts: ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
wayfair.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
wb.ru — policy unreadable (HTTP 498)
wbx2.com — policy unreadable (host answered nothing)
wcdnga.com — policy unreadable (domain does not resolve)
weather.com — blocks: CCBot, Bytespider, Amazonbot, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot — restricts: GPTBot, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
weather.gov — blocks no tracked AI bots
webempresa.eu — blocks no tracked AI bots
webex.com — policy unreadable (refused (401/403))
webhostbox.net — policy unreadable (domain does not resolve)
webmd.com — blocks: GPTBot, ClaudeBot, CCBot — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
weebly.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
weforum.org — policy unreadable (refused (401/403))
weibo.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
wellsfargo.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
welt.de — blocks: Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Amazonbot, DuckAssistBot, MistralAI-User, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, AI2Bot, PanguBot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, meta-externalagent, anthropic-ai, Claude-Web, Meta-ExternalFetcher, DeepSeekBot, GrokBot, Gemini-Deep-Research
whatsapp.com — blocks: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot, Amazonbot — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
whecloud.com — policy unreadable (domain does not resolve)
who.int — blocks: CCBot
wikihow.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Applebot-Extended, PerplexityBot, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgili, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot — restricts: Google-Extended, Perplexity-User, cohere-training-data-crawler, omgilibot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
wikimedia.org — blocks no tracked AI bots
wikipedia.org — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
wildberries.ru — policy unreadable (HTTP 498)
wiley.com — blocks: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, Amazonbot, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgili — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, DuckAssistBot, MistralAI-User, cohere-training-data-crawler, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
windows.com — policy unreadable (HTML instead of text)
wired.com — blocks: ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, cohere-ai, cohere-training-data-crawler, Diffbot, YouBot, Timpibot, Meta-ExternalFetcher, PanguBot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, anthropic-ai, Claude-Web, omgili, omgilibot, ImagesiftBot, AI2Bot, DeepSeekBot, GrokBot, Gemini-Deep-Research
withgoogle.com — policy unreadable (HTML instead of text)
wix.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
wixsite.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
wordpress.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
wordpress.org — blocks no tracked AI bots — restricts: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Perplexity-User, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
workbox.dk — policy unreadable (domain does not resolve)
workers.dev — policy unreadable (HTML instead of text)
worldbank.org — blocks no tracked AI bots
worldnic.com — policy unreadable (host answered nothing)
wp.pl — blocks: GPTBot, CCBot — restricts: ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
wpguardian.com — blocks no tracked AI bots
wpguardian.io — blocks no tracked AI bots
wps.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
wsdvs.com — policy unreadable (domain does not resolve)
wsj.com — blocks: ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: GPTBot, ChatGPT-User, OAI-SearchBot — default-deny robots.txt
wswebcdn.com — policy unreadable (host answered nothing)
www.gov.br — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
www.gov.uk — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
wyzecam.com — blocks no tracked AI bots — restricts: ChatGPT-User, Claude-User, Claude-SearchBot, Applebot-Extended, Perplexity-User, CCBot, Bytespider, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
x.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: Google-Extended, meta-externalagent, Meta-ExternalFetcher — default-deny robots.txt
xbox.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
xboxlive.com — policy unreadable (host answered nothing)
xcal.tv — policy unreadable (domain does not resolve)
xerox.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
xhamster46.desi — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
xiaomi.com — policy unreadable (refused (401/403))
xiaomi.net — policy unreadable (host answered nothing)
ya.ru — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
yahoo.co.jp — blocks: GPTBot, Google-Extended
yahoo.com — blocks: GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Bytespider, anthropic-ai, Claude-Web, cohere-ai, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot — restricts: OAI-SearchBot, Claude-User, Claude-SearchBot, Applebot-Extended, Perplexity-User, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, cohere-training-data-crawler, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
yahoodns.net — policy unreadable (domain does not resolve)
yandex.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
yandex.net — blocks no tracked AI bots
yandex.ru — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
yandexcloud.net — policy unreadable (domain does not resolve)
yccdn.ru — policy unreadable (domain does not resolve)
yellowblue.io — policy unreadable (domain does not resolve)
yelp.com — blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — default-deny robots.txt
yieldmo.com — blocks no tracked AI bots — publishes llms.txt
youku.com — policy unreadable (HTML instead of text)
youtu.be — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
youtube-nocookie.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
youtube.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
ys7.com — policy unreadable (host answered nothing)
zendesk.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
zenecn.net — policy unreadable (domain does not resolve)
zillow.com — blocks: Applebot-Extended, meta-externalagent, Amazonbot, YouBot, AI2Bot — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
ziprecruiter.com — blocks: GPTBot, Applebot-Extended, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — restricts: ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot — default-deny robots.txt — publishes llms.txt
zoho.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research — publishes llms.txt
zoom.com — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
zoom.us — blocks no tracked AI bots — restricts: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-User, anthropic-ai, Claude-Web, cohere-ai, cohere-training-data-crawler, Diffbot, omgili, omgilibot, YouBot, ImagesiftBot, Timpibot, Meta-ExternalFetcher, AI2Bot, DeepSeekBot, GrokBot, PanguBot, Gemini-Deep-Research
zpath.net — policy unreadable (domain does not resolve)