Which websites let AI in?
A daily census of AI-crawler access. We read the robots.txt and llms.txt of 232 major sites and record which AI bots each one allows, restricts, or blocks — with history, so you can see exactly when the web's doors open and close. Snapshot: 2026-08-25
48.5%
of sites block at least one AI crawler (99/204)
29.4%
block GPTBot (OpenAI training)
34.3%
block ClaudeBot (Anthropic training)
14.7%
publish an llms.txt (34 sites)
| Site | Category | llms.txt | GPTBot | ClaudeBot | Google-Extended | PerplexityBot | CCBot | Bytespider |
|---|---|---|---|---|---|---|---|---|
| 404media.co | news | — | blocked | restricted* | restricted* | restricted* | restricted* | restricted* |
| aa.com | travel | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| accuweather.com | reference | — | unknown | unknown | unknown | unknown | unknown | unknown |
| adidas.com | ecommerce | — | unknown | unknown | unknown | unknown | unknown | unknown |
| airbnb.com | travel | — | restricted | restricted | restricted* | restricted | restricted* | restricted* |
| aliexpress.com | ecommerce | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| allrecipes.com | food | — | restricted | blocked | blocked | blocked | blocked | blocked |
| amazon.com | ecommerce | — | blocked | blocked | blocked | blocked | blocked | blocked |
| americanexpress.com | finance | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| anthropic.com | tech | — | allowed | allowed | allowed | allowed | allowed | allowed |
| apartments.com | realestate | — | unknown | unknown | unknown | unknown | unknown | unknown |
| apnews.com | news | — | blocked | blocked | restricted* | blocked | blocked | restricted* |
| apple.com | tech | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| archive.org | reference | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| arstechnica.com | news | — | restricted* | blocked | blocked | blocked | blocked | blocked |
| atlassian.com | tech | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| autotrader.com | auto | — | restricted | restricted* | restricted* | restricted | restricted | restricted* |
| axios.com | news | — | restricted* | restricted* | restricted | restricted* | blocked | blocked |
| bankofamerica.com | finance | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| bankrate.com | finance | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| bbc.com | news | — | blocked | blocked | blocked | blocked | blocked | blocked |
| bestbuy.com | ecommerce | — | restricted | restricted | restricted | restricted | restricted | restricted |
| bleacherreport.com | sports | — | blocked | restricted* | blocked | restricted* | blocked | restricted* |
| bloomberg.com | news | — | blocked | blocked | blocked | blocked | blocked | blocked |
| bonappetit.com | food | — | restricted* | blocked | blocked | blocked | blocked | blocked |
| booking.com | travel | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| britannica.com | reference | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| bsky.app | social | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| businessinsider.com | news | — | restricted* | blocked | restricted* | restricted* | blocked | blocked |
| canva.com | tech | — | blocked | blocked | restricted* | allowed | blocked | blocked |
| capitalone.com | finance | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| carfax.com | auto | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| cars.com | auto | — | unknown | unknown | unknown | unknown | unknown | unknown |
| carvana.com | auto | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| cbsnews.com | news | — | blocked | restricted* | restricted* | restricted* | restricted* | restricted* |
| cbssports.com | sports | — | blocked | restricted* | restricted* | restricted* | restricted* | restricted* |
| cdc.gov | health | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| chase.com | finance | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| chegg.com | education | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| chewy.com | ecommerce | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| clevelandclinic.org | health | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| cloudflare.com | tech | yes | allowed | allowed | allowed | allowed | allowed | allowed |
| cnbc.com | finance | — | blocked | blocked | blocked | blocked | blocked | blocked |
| cnn.com | news | — | blocked | blocked | blocked | blocked | blocked | blocked |
| coinbase.com | finance | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| costco.com | ecommerce | — | unknown | unknown | unknown | unknown | unknown | unknown |
| coursera.org | education | yes | restricted | restricted | restricted | restricted | restricted | restricted* |
| craigslist.org | classifieds | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| dailymail.co.uk | news | — | blocked | blocked | blocked | blocked | blocked | blocked |
| deepseek.com | tech | — | allowed | allowed | allowed | allowed | allowed | allowed |
| delta.com | travel | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| developer.mozilla.org | tech | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| dictionary.com | reference | — | blocked | restricted* | restricted* | restricted* | blocked | restricted* |
| digitalocean.com | tech | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| discord.com | social | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| disneyplus.com | entertainment | — | restricted | restricted | restricted | restricted | restricted | restricted |
| doordash.com | ecommerce | — | unknown | unknown | unknown | unknown | unknown | unknown |
| dropbox.com | tech | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| duolingo.com | education | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| ebay.com | ecommerce | — | blocked | blocked | restricted* | blocked | blocked | blocked |
| economist.com | news | — | blocked | blocked | blocked | blocked | blocked | blocked |
| edmunds.com | auto | — | blocked | blocked | restricted* | restricted* | restricted* | blocked |
| epicgames.com | entertainment | — | unknown | unknown | unknown | unknown | unknown | unknown |
| epicurious.com | food | — | restricted* | blocked | blocked | blocked | blocked | blocked |
| espn.com | sports | — | blocked | restricted* | blocked | restricted* | blocked | blocked |
| etsy.com | ecommerce | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| europa.eu | government | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| expedia.com | travel | yes | restricted* | restricted* | restricted* | restricted | restricted* | restricted* |
| facebook.com | social | — | restricted | restricted | restricted | restricted | blocked* | blocked* |
| fidelity.com | finance | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| figma.com | tech | — | blocked | blocked | blocked | blocked | blocked | restricted* |
| flashscore.com | sports | — | restricted* | restricted* | restricted* | restricted* | blocked | blocked |
| foodnetwork.com | food | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| fool.com | finance | — | restricted* | restricted* | restricted* | restricted* | restricted* | blocked |
| forbes.com | news | — | blocked | blocked | restricted* | blocked | blocked | blocked |
| foxnews.com | news | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| ft.com | news | — | restricted* | blocked | blocked | blocked | blocked | blocked |
| gamespot.com | entertainment | — | blocked | blocked | blocked | allowed | blocked | blocked |
| genius.com | reference | — | blocked | blocked | blocked | restricted* | blocked | blocked |
| github.com | tech | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| gitlab.com | tech | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| glassdoor.com | jobs | — | blocked | blocked | blocked | restricted* | blocked | blocked |
| godaddy.com | tech | — | unknown | unknown | unknown | unknown | unknown | unknown |
| goodreads.com | reference | — | blocked | restricted* | restricted* | restricted* | blocked | restricted* |
| google.com | tech | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| gov.uk | government | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| harvard.edu | education | — | allowed | allowed | allowed | allowed | allowed | allowed |
| healthline.com | health | — | blocked | blocked | restricted* | restricted* | blocked | blocked |
| hilton.com | travel | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| homedepot.com | ecommerce | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| hotels.com | travel | — | unknown | unknown | unknown | unknown | unknown | unknown |
| hubspot.com | tech | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| huffpost.com | news | — | blocked | blocked | blocked | blocked | blocked | blocked |
| huggingface.co | tech | — | allowed | allowed | allowed | allowed | allowed | allowed |
| hulu.com | entertainment | — | restricted | restricted | restricted | restricted | restricted | restricted |
| ibm.com | tech | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| ign.com | entertainment | — | blocked | blocked | restricted* | blocked | blocked | blocked |
| ikea.com | ecommerce | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| imdb.com | reference | — | blocked* | blocked* | blocked* | blocked* | blocked* | blocked* |
| indeed.com | jobs | — | restricted | restricted | restricted | restricted | restricted | restricted |
| instacart.com | ecommerce | — | blocked* | blocked* | blocked* | blocked | blocked* | blocked* |
| instagram.com | social | — | blocked | blocked | blocked | blocked | blocked* | blocked* |
| investopedia.com | reference | — | restricted | blocked | blocked | blocked | blocked | blocked |
| irs.gov | government | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| kaggle.com | tech | yes | allowed | allowed | allowed | allowed | allowed | allowed |
| kayak.com | travel | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| kbb.com | auto | yes | restricted* | restricted* | restricted* | restricted | restricted | restricted* |
| khanacademy.org | education | — | unknown | unknown | unknown | unknown | unknown | unknown |
| kotaku.com | entertainment | — | restricted* | restricted* | restricted* | restricted* | blocked | blocked |
| kroger.com | ecommerce | — | unknown | unknown | unknown | unknown | unknown | unknown |
| latimes.com | news | — | blocked | blocked | restricted* | restricted* | restricted* | blocked |
| linkedin.com | social | — | blocked | blocked | blocked | blocked | blocked | blocked |
| lowes.com | ecommerce | — | unknown | unknown | unknown | unknown | unknown | unknown |
| marketwatch.com | finance | — | restricted | blocked* | blocked* | blocked* | blocked* | blocked* |
| marriott.com | travel | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| max.com | entertainment | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| mayoclinic.org | health | — | unknown | unknown | unknown | unknown | unknown | unknown |
| medium.com | social | — | blocked | blocked | restricted* | restricted* | restricted* | blocked |
| merriam-webster.com | reference | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| meta.com | tech | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| metacritic.com | reference | — | blocked | restricted* | restricted* | restricted* | blocked | restricted* |
| microsoft.com | tech | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| minecraft.net | entertainment | — | unknown | unknown | unknown | unknown | unknown | unknown |
| mistral.ai | tech | yes | allowed | allowed | allowed | allowed | allowed | allowed |
| mit.edu | education | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| mlb.com | sports | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| monster.com | jobs | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| nasa.gov | government | — | allowed | allowed | allowed | allowed | allowed | allowed |
| nba.com | sports | — | blocked | blocked | blocked | blocked | blocked | restricted* |
| nbcnews.com | news | — | blocked | blocked | blocked | blocked | blocked | blocked |
| nerdwallet.com | finance | — | restricted* | restricted* | blocked | restricted* | restricted* | restricted* |
| netflix.com | entertainment | — | restricted | restricted | restricted | restricted | blocked* | blocked* |
| netlify.com | tech | yes | allowed | allowed | allowed | allowed | allowed | allowed |
| newsweek.com | news | — | blocked | blocked | restricted* | blocked | blocked | blocked |
| newyorker.com | news | — | restricted* | blocked | blocked | blocked | blocked | blocked |
| nfl.com | sports | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| nhl.com | sports | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| nih.gov | health | — | unknown | unknown | unknown | unknown | unknown | unknown |
| nike.com | ecommerce | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| notion.so | tech | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| npmjs.com | tech | — | unknown | unknown | unknown | unknown | unknown | unknown |
| npr.org | news | — | unknown | unknown | unknown | unknown | unknown | unknown |
| nvidia.com | tech | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| nypost.com | news | — | restricted* | blocked | blocked | blocked | blocked | blocked |
| nytimes.com | news | — | blocked | blocked | blocked | blocked | blocked | blocked |
| openai.com | tech | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| opentable.com | food | — | unknown | unknown | unknown | unknown | unknown | unknown |
| oracle.com | tech | — | unknown | unknown | unknown | unknown | unknown | unknown |
| paypal.com | finance | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| pcgamer.com | entertainment | — | restricted* | restricted* | restricted* | restricted* | restricted* | blocked |
| perplexity.ai | tech | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| pinterest.com | social | — | blocked* | blocked* | blocked* | blocked* | blocked* | blocked* |
| politico.com | news | — | unknown | unknown | unknown | unknown | unknown | unknown |
| polygon.com | entertainment | — | blocked | blocked | blocked | blocked | blocked | blocked |
| pypi.org | tech | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| quizlet.com | education | — | unknown | unknown | unknown | unknown | unknown | unknown |
| quora.com | reference | — | blocked | blocked | restricted | blocked | blocked* | blocked |
| realtor.com | realestate | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| reddit.com | social | — | blocked* | blocked* | blocked* | blocked* | blocked* | blocked* |
| redfin.com | realestate | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| reuters.com | news | — | blocked* | blocked* | blocked* | blocked* | blocked* | blocked* |
| robinhood.com | finance | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| roblox.com | entertainment | — | blocked | blocked | blocked | restricted* | blocked | blocked |
| rottentomatoes.com | reference | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| salesforce.com | tech | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| schwab.com | finance | — | unknown | unknown | unknown | unknown | unknown | unknown |
| sec.gov | government | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| seekingalpha.com | finance | — | restricted* | blocked | restricted* | blocked | blocked | blocked |
| semafor.com | news | — | blocked | restricted* | restricted* | restricted* | restricted* | restricted* |
| sephora.com | ecommerce | — | unknown | unknown | unknown | unknown | unknown | unknown |
| seriouseats.com | food | — | restricted | blocked | blocked | blocked | blocked | blocked |
| shein.com | ecommerce | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| skyscanner.com | travel | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| slack.com | tech | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| snapchat.com | social | — | blocked | blocked | blocked | restricted | blocked | blocked |
| snopes.com | reference | — | blocked | blocked | allowed | blocked | blocked | blocked |
| soundcloud.com | entertainment | — | blocked | blocked | blocked | blocked | blocked | blocked |
| southwest.com | travel | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| spotify.com | entertainment | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| squarespace.com | tech | yes | restricted | restricted | restricted | restricted* | restricted | restricted |
| stackoverflow.com | reference | — | unknown | unknown | unknown | unknown | unknown | unknown |
| stanford.edu | education | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| steampowered.com | entertainment | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| stripe.com | finance | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| substack.com | social | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| target.com | ecommerce | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| techcrunch.com | news | — | blocked | blocked | blocked | restricted* | blocked | blocked |
| ted.com | education | — | blocked | blocked | blocked | restricted | blocked | blocked |
| temu.com | ecommerce | — | restricted | restricted | restricted | restricted | allowed | blocked |
| theathletic.com | sports | — | blocked | blocked | blocked | blocked | blocked | blocked |
| theatlantic.com | news | — | allowed | blocked | blocked | blocked | blocked | blocked |
| theguardian.com | news | — | restricted* | blocked | restricted* | blocked | blocked | blocked |
| theinformation.com | news | — | unknown | unknown | unknown | unknown | unknown | unknown |
| theverge.com | news | — | allowed | blocked | blocked | blocked | blocked | blocked |
| threads.com | social | — | blocked | blocked | blocked | blocked | blocked* | blocked* |
| tiktok.com | social | — | blocked | blocked | blocked | blocked | blocked | blocked |
| time.com | news | — | allowed | allowed | allowed | allowed | allowed | allowed |
| tripadvisor.com | travel | — | blocked | blocked | blocked | restricted | blocked | blocked |
| tumblr.com | social | — | restricted* | blocked | blocked | restricted* | blocked | restricted* |
| twitch.tv | social | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| ubereats.com | ecommerce | — | unknown | unknown | unknown | unknown | unknown | unknown |
| udemy.com | education | — | unknown | unknown | unknown | unknown | unknown | unknown |
| united.com | travel | — | unknown | unknown | unknown | unknown | unknown | unknown |
| urbandictionary.com | reference | — | allowed | allowed | allowed | allowed | allowed | allowed |
| usa.gov | government | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| usatoday.com | news | — | blocked | blocked | blocked | blocked | blocked | blocked |
| uspto.gov | government | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| vanguard.com | finance | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| vercel.com | tech | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| vox.com | news | — | allowed | blocked | blocked | blocked | blocked | blocked |
| vrbo.com | travel | — | restricted* | restricted* | restricted* | restricted | restricted* | restricted* |
| walmart.com | ecommerce | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| washingtonpost.com | news | — | restricted* | blocked | restricted* | blocked | blocked | blocked |
| wayfair.com | ecommerce | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| weather.com | reference | yes | blocked | blocked | blocked | blocked | blocked | blocked |
| weather.gov | government | — | allowed | allowed | allowed | allowed | allowed | allowed |
| webmd.com | health | — | blocked | blocked | restricted* | restricted* | blocked | restricted* |
| wellsfargo.com | finance | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| who.int | health | — | allowed | allowed | allowed | allowed | blocked | allowed |
| wikihow.com | reference | — | blocked | blocked | restricted* | blocked | blocked | blocked |
| wikipedia.org | reference | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| wired.com | news | — | restricted* | blocked | blocked | blocked | blocked | blocked |
| wix.com | tech | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| wordpress.com | tech | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| wsj.com | news | — | restricted | blocked* | blocked* | blocked* | blocked* | blocked* |
| x.com | social | — | blocked* | blocked* | restricted | blocked* | blocked* | blocked* |
| yelp.com | food | — | blocked | blocked | blocked | blocked | blocked | blocked* |
| youtube.com | social | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| zendesk.com | tech | yes | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| zillow.com | realestate | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
| ziprecruiter.com | jobs | yes | blocked | restricted | restricted | restricted | blocked | blocked* |
| zoom.us | tech | — | restricted* | restricted* | restricted* | restricted* | restricted* | restricted* |
Chips read from each site's robots.txt: allowed no rule stops this bot · restricted some paths disallowed · blocked fully disallowed · unknown we couldn't read the file. An asterisk means the verdict is inherited from the site's default (*) rules rather than naming the bot. Every site page shows all 16 tracked bots.