← all sites
Tech sites vs AI crawlers
37 tracked tech sites, 34 with readable policy. Snapshot: 2026-09-14
11.8%
block ≥1 AI crawler (4/34)
GPTBot
most-blocked bot here (2 sites)
17
publish llms.txt
| Site | llms.txt | GPTBot | ClaudeBot | Google-Extended | PerplexityBot | CCBot | Bytespider |
|---|---|---|---|---|---|---|---|
| canva.com | ? | blocked | blocked | restricted | allowed | blocked | blocked |
| figma.com | — | blocked | blocked | blocked | blocked | blocked | restricted |
| github.com | yes | restricted | restricted | restricted | restricted | restricted | blocked |
| notion.so | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| anthropic.com | — | allowed | allowed | allowed | allowed | allowed | allowed |
| apple.com | — | restricted | restricted | restricted | restricted | restricted | restricted |
| atlassian.com | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| cloudflare.com | yes | allowed | allowed | allowed | allowed | allowed | allowed |
| deepseek.com | — | allowed | allowed | allowed | allowed | allowed | allowed |
| developer.mozilla.org | — | restricted | restricted | restricted | restricted | restricted | restricted |
| digitalocean.com | — | restricted | restricted | restricted | restricted | restricted | restricted |
| dropbox.com | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| gitlab.com | ? | restricted | restricted | restricted | restricted | restricted | restricted |
| godaddy.com | ? | unknown | unknown | unknown | unknown | unknown | unknown |
| google.com | — | restricted | restricted | restricted | restricted | restricted | restricted |
| hubspot.com | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| huggingface.co | — | allowed | allowed | allowed | allowed | allowed | allowed |
| ibm.com | — | restricted | restricted | restricted | restricted | restricted | restricted |
| kaggle.com | yes | allowed | allowed | allowed | allowed | allowed | allowed |
| meta.com | — | restricted | restricted | restricted | restricted | restricted | restricted |
| microsoft.com | — | restricted | restricted | restricted | restricted | restricted | restricted |
| mistral.ai | yes | allowed | allowed | allowed | allowed | allowed | allowed |
| netlify.com | yes | allowed | allowed | allowed | allowed | allowed | allowed |
| npmjs.com | ? | unknown | unknown | unknown | unknown | unknown | unknown |
| nvidia.com | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| openai.com | ? | restricted | restricted | restricted | restricted | restricted | restricted |
| oracle.com | ? | unknown | unknown | unknown | unknown | unknown | unknown |
| perplexity.ai | ? | restricted | restricted | restricted | restricted | restricted | restricted |
| pypi.org | — | restricted | restricted | restricted | restricted | restricted | restricted |
| salesforce.com | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| slack.com | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| squarespace.com | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| vercel.com | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| wix.com | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| wordpress.com | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| zendesk.com | yes | restricted | restricted | restricted | restricted | restricted | restricted |
| zoom.us | — | restricted | restricted | restricted | restricted | restricted | restricted |