Tech sites vs AI crawlers

37 tracked tech sites, 34 with readable policy. Snapshot: 2026-08-25

8.8%

block ≥1 AI crawler (3/34)

GPTBot

most-blocked bot here (2 sites)

publish llms.txt

Site	llms.txt	GPTBot	ClaudeBot	Google-Extended	PerplexityBot	CCBot	Bytespider
figma.com	—	blocked	blocked	blocked	blocked	blocked	restricted^*
canva.com	—	blocked	blocked	restricted^*	allowed	blocked	blocked
notion.so	yes	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
anthropic.com	—	allowed	allowed	allowed	allowed	allowed	allowed
apple.com	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
atlassian.com	yes	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
cloudflare.com	yes	allowed	allowed	allowed	allowed	allowed	allowed
deepseek.com	—	allowed	allowed	allowed	allowed	allowed	allowed
developer.mozilla.org	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
digitalocean.com	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
dropbox.com	yes	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
github.com	yes	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
gitlab.com	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
godaddy.com	—	unknown	unknown	unknown	unknown	unknown	unknown
google.com	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
hubspot.com	yes	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
huggingface.co	—	allowed	allowed	allowed	allowed	allowed	allowed
ibm.com	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
kaggle.com	yes	allowed	allowed	allowed	allowed	allowed	allowed
meta.com	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
microsoft.com	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
mistral.ai	yes	allowed	allowed	allowed	allowed	allowed	allowed
netlify.com	yes	allowed	allowed	allowed	allowed	allowed	allowed
npmjs.com	—	unknown	unknown	unknown	unknown	unknown	unknown
nvidia.com	yes	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
openai.com	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
oracle.com	—	unknown	unknown	unknown	unknown	unknown	unknown
perplexity.ai	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
pypi.org	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
salesforce.com	yes	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
slack.com	yes	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
squarespace.com	yes	restricted	restricted	restricted	restricted^*	restricted	restricted
vercel.com	yes	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
wix.com	yes	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
wordpress.com	yes	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
zendesk.com	yes	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*
zoom.us	—	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*	restricted^*