← all sites

netlify.com

netlify.com currently blocks none of the 16 AI crawlers we track. Snapshot: 2026-08-25

Category
tech
robots.txt
readable — view live file
Default (*) policy
allowed
llms.txt
published — view (an on-ramp written for AI readers)
Machine-readable
JSON for this site

All 16 tracked AI bots

BotOperatorPurposeStatusHow it's set
GPTBotOpenAItrainingallowednamed explicitly
ChatGPT-UserOpenAIfetchallowednamed explicitly
OAI-SearchBotOpenAIsearchallowednamed explicitly
ClaudeBotAnthropictrainingallowednamed explicitly
Claude-UserAnthropicfetchalloweddefault (*) rules
Claude-SearchBotAnthropicsearchalloweddefault (*) rules
Google-ExtendedGoogletrainingallowednamed explicitly
Applebot-ExtendedAppletrainingalloweddefault (*) rules
PerplexityBotPerplexitysearchallowednamed explicitly
Perplexity-UserPerplexityfetchalloweddefault (*) rules
CCBotCommon Crawltrainingalloweddefault (*) rules
BytespiderByteDancetrainingalloweddefault (*) rules
meta-externalagentMetatrainingalloweddefault (*) rules
AmazonbotAmazontrainingalloweddefault (*) rules
DuckAssistBotDuckDuckGosearchalloweddefault (*) rules
MistralAI-UserMistralfetchalloweddefault (*) rules

Policy history

Tracking began 2026-08-25 (index founding). Changes to this site's AI policy will appear here as daily crawls accumulate.