Which websites let AI in?

A daily census of AI-crawler access. We read the robots.txt and llms.txt of 232 major sites and record which AI bots each one allows, restricts, or blocks — with history, so you can see exactly when the web's doors open and close. Snapshot: 2026-08-25

48.5%
of sites block at least one AI crawler (99/204)
29.4%
block GPTBot (OpenAI training)
34.3%
block ClaudeBot (Anthropic training)
14.7%
publish an llms.txt (34 sites)
SiteCategoryllms.txtGPTBotClaudeBotGoogle-ExtendedPerplexityBotCCBotBytespider
404media.conews blockedrestricted*restricted*restricted*restricted*restricted*
aa.comtravel restricted*restricted*restricted*restricted*restricted*restricted*
accuweather.comreference unknownunknownunknownunknownunknownunknown
adidas.comecommerce unknownunknownunknownunknownunknownunknown
airbnb.comtravel restrictedrestrictedrestricted*restrictedrestricted*restricted*
aliexpress.comecommerce restricted*restricted*restricted*restricted*restricted*restricted*
allrecipes.comfood restrictedblockedblockedblockedblockedblocked
amazon.comecommerce blockedblockedblockedblockedblockedblocked
americanexpress.comfinance yesrestricted*restricted*restricted*restricted*restricted*restricted*
anthropic.comtech allowedallowedallowedallowedallowedallowed
apartments.comrealestate unknownunknownunknownunknownunknownunknown
apnews.comnews blockedblockedrestricted*blockedblockedrestricted*
apple.comtech restricted*restricted*restricted*restricted*restricted*restricted*
archive.orgreference restricted*restricted*restricted*restricted*restricted*restricted*
arstechnica.comnews restricted*blockedblockedblockedblockedblocked
atlassian.comtech yesrestricted*restricted*restricted*restricted*restricted*restricted*
autotrader.comauto restrictedrestricted*restricted*restrictedrestrictedrestricted*
axios.comnews restricted*restricted*restrictedrestricted*blockedblocked
bankofamerica.comfinance restricted*restricted*restricted*restricted*restricted*restricted*
bankrate.comfinance yesrestricted*restricted*restricted*restricted*restricted*restricted*
bbc.comnews blockedblockedblockedblockedblockedblocked
bestbuy.comecommerce restrictedrestrictedrestrictedrestrictedrestrictedrestricted
bleacherreport.comsports blockedrestricted*blockedrestricted*blockedrestricted*
bloomberg.comnews blockedblockedblockedblockedblockedblocked
bonappetit.comfood restricted*blockedblockedblockedblockedblocked
booking.comtravel restricted*restricted*restricted*restricted*restricted*restricted*
britannica.comreference restricted*restricted*restricted*restricted*restricted*restricted*
bsky.appsocial restricted*restricted*restricted*restricted*restricted*restricted*
businessinsider.comnews restricted*blockedrestricted*restricted*blockedblocked
canva.comtech blockedblockedrestricted*allowedblockedblocked
capitalone.comfinance restricted*restricted*restricted*restricted*restricted*restricted*
carfax.comauto restricted*restricted*restricted*restricted*restricted*restricted*
cars.comauto unknownunknownunknownunknownunknownunknown
carvana.comauto yesrestricted*restricted*restricted*restricted*restricted*restricted*
cbsnews.comnews blockedrestricted*restricted*restricted*restricted*restricted*
cbssports.comsports blockedrestricted*restricted*restricted*restricted*restricted*
cdc.govhealth restricted*restricted*restricted*restricted*restricted*restricted*
chase.comfinance restricted*restricted*restricted*restricted*restricted*restricted*
chegg.comeducation restricted*restricted*restricted*restricted*restricted*restricted*
chewy.comecommerce restricted*restricted*restricted*restricted*restricted*restricted*
clevelandclinic.orghealth yesrestricted*restricted*restricted*restricted*restricted*restricted*
cloudflare.comtech yesallowedallowedallowedallowedallowedallowed
cnbc.comfinance blockedblockedblockedblockedblockedblocked
cnn.comnews blockedblockedblockedblockedblockedblocked
coinbase.comfinance yesrestricted*restricted*restricted*restricted*restricted*restricted*
costco.comecommerce unknownunknownunknownunknownunknownunknown
coursera.orgeducation yesrestrictedrestrictedrestrictedrestrictedrestrictedrestricted*
craigslist.orgclassifieds restricted*restricted*restricted*restricted*restricted*restricted*
dailymail.co.uknews blockedblockedblockedblockedblockedblocked
deepseek.comtech allowedallowedallowedallowedallowedallowed
delta.comtravel restricted*restricted*restricted*restricted*restricted*restricted*
developer.mozilla.orgtech restricted*restricted*restricted*restricted*restricted*restricted*
dictionary.comreference blockedrestricted*restricted*restricted*blockedrestricted*
digitalocean.comtech restricted*restricted*restricted*restricted*restricted*restricted*
discord.comsocial restricted*restricted*restricted*restricted*restricted*restricted*
disneyplus.comentertainment restrictedrestrictedrestrictedrestrictedrestrictedrestricted
doordash.comecommerce unknownunknownunknownunknownunknownunknown
dropbox.comtech yesrestricted*restricted*restricted*restricted*restricted*restricted*
duolingo.comeducation restricted*restricted*restricted*restricted*restricted*restricted*
ebay.comecommerce blockedblockedrestricted*blockedblockedblocked
economist.comnews blockedblockedblockedblockedblockedblocked
edmunds.comauto blockedblockedrestricted*restricted*restricted*blocked
epicgames.comentertainment unknownunknownunknownunknownunknownunknown
epicurious.comfood restricted*blockedblockedblockedblockedblocked
espn.comsports blockedrestricted*blockedrestricted*blockedblocked
etsy.comecommerce restricted*restricted*restricted*restricted*restricted*restricted*
europa.eugovernment restricted*restricted*restricted*restricted*restricted*restricted*
expedia.comtravel yesrestricted*restricted*restricted*restrictedrestricted*restricted*
facebook.comsocial restrictedrestrictedrestrictedrestrictedblocked*blocked*
fidelity.comfinance restricted*restricted*restricted*restricted*restricted*restricted*
figma.comtech blockedblockedblockedblockedblockedrestricted*
flashscore.comsports restricted*restricted*restricted*restricted*blockedblocked
foodnetwork.comfood restricted*restricted*restricted*restricted*restricted*restricted*
fool.comfinance restricted*restricted*restricted*restricted*restricted*blocked
forbes.comnews blockedblockedrestricted*blockedblockedblocked
foxnews.comnews yesrestricted*restricted*restricted*restricted*restricted*restricted*
ft.comnews restricted*blockedblockedblockedblockedblocked
gamespot.comentertainment blockedblockedblockedallowedblockedblocked
genius.comreference blockedblockedblockedrestricted*blockedblocked
github.comtech yesrestricted*restricted*restricted*restricted*restricted*restricted*
gitlab.comtech restricted*restricted*restricted*restricted*restricted*restricted*
glassdoor.comjobs blockedblockedblockedrestricted*blockedblocked
godaddy.comtech unknownunknownunknownunknownunknownunknown
goodreads.comreference blockedrestricted*restricted*restricted*blockedrestricted*
google.comtech restricted*restricted*restricted*restricted*restricted*restricted*
gov.ukgovernment restricted*restricted*restricted*restricted*restricted*restricted*
harvard.edueducation allowedallowedallowedallowedallowedallowed
healthline.comhealth blockedblockedrestricted*restricted*blockedblocked
hilton.comtravel restricted*restricted*restricted*restricted*restricted*restricted*
homedepot.comecommerce restricted*restricted*restricted*restricted*restricted*restricted*
hotels.comtravel unknownunknownunknownunknownunknownunknown
hubspot.comtech yesrestricted*restricted*restricted*restricted*restricted*restricted*
huffpost.comnews blockedblockedblockedblockedblockedblocked
huggingface.cotech allowedallowedallowedallowedallowedallowed
hulu.comentertainment restrictedrestrictedrestrictedrestrictedrestrictedrestricted
ibm.comtech restricted*restricted*restricted*restricted*restricted*restricted*
ign.comentertainment blockedblockedrestricted*blockedblockedblocked
ikea.comecommerce restricted*restricted*restricted*restricted*restricted*restricted*
imdb.comreference blocked*blocked*blocked*blocked*blocked*blocked*
indeed.comjobs restrictedrestrictedrestrictedrestrictedrestrictedrestricted
instacart.comecommerce blocked*blocked*blocked*blockedblocked*blocked*
instagram.comsocial blockedblockedblockedblockedblocked*blocked*
investopedia.comreference restrictedblockedblockedblockedblockedblocked
irs.govgovernment restricted*restricted*restricted*restricted*restricted*restricted*
kaggle.comtech yesallowedallowedallowedallowedallowedallowed
kayak.comtravel yesrestricted*restricted*restricted*restricted*restricted*restricted*
kbb.comauto yesrestricted*restricted*restricted*restrictedrestrictedrestricted*
khanacademy.orgeducation unknownunknownunknownunknownunknownunknown
kotaku.comentertainment restricted*restricted*restricted*restricted*blockedblocked
kroger.comecommerce unknownunknownunknownunknownunknownunknown
latimes.comnews blockedblockedrestricted*restricted*restricted*blocked
linkedin.comsocial blockedblockedblockedblockedblockedblocked
lowes.comecommerce unknownunknownunknownunknownunknownunknown
marketwatch.comfinance restrictedblocked*blocked*blocked*blocked*blocked*
marriott.comtravel restricted*restricted*restricted*restricted*restricted*restricted*
max.comentertainment restricted*restricted*restricted*restricted*restricted*restricted*
mayoclinic.orghealth unknownunknownunknownunknownunknownunknown
medium.comsocial blockedblockedrestricted*restricted*restricted*blocked
merriam-webster.comreference restricted*restricted*restricted*restricted*restricted*restricted*
meta.comtech restricted*restricted*restricted*restricted*restricted*restricted*
metacritic.comreference blockedrestricted*restricted*restricted*blockedrestricted*
microsoft.comtech restricted*restricted*restricted*restricted*restricted*restricted*
minecraft.netentertainment unknownunknownunknownunknownunknownunknown
mistral.aitech yesallowedallowedallowedallowedallowedallowed
mit.edueducation restricted*restricted*restricted*restricted*restricted*restricted*
mlb.comsports restricted*restricted*restricted*restricted*restricted*restricted*
monster.comjobs yesrestricted*restricted*restricted*restricted*restricted*restricted*
nasa.govgovernment allowedallowedallowedallowedallowedallowed
nba.comsports blockedblockedblockedblockedblockedrestricted*
nbcnews.comnews blockedblockedblockedblockedblockedblocked
nerdwallet.comfinance restricted*restricted*blockedrestricted*restricted*restricted*
netflix.comentertainment restrictedrestrictedrestrictedrestrictedblocked*blocked*
netlify.comtech yesallowedallowedallowedallowedallowedallowed
newsweek.comnews blockedblockedrestricted*blockedblockedblocked
newyorker.comnews restricted*blockedblockedblockedblockedblocked
nfl.comsports restricted*restricted*restricted*restricted*restricted*restricted*
nhl.comsports restricted*restricted*restricted*restricted*restricted*restricted*
nih.govhealth unknownunknownunknownunknownunknownunknown
nike.comecommerce restricted*restricted*restricted*restricted*restricted*restricted*
notion.sotech yesrestricted*restricted*restricted*restricted*restricted*restricted*
npmjs.comtech unknownunknownunknownunknownunknownunknown
npr.orgnews unknownunknownunknownunknownunknownunknown
nvidia.comtech yesrestricted*restricted*restricted*restricted*restricted*restricted*
nypost.comnews restricted*blockedblockedblockedblockedblocked
nytimes.comnews blockedblockedblockedblockedblockedblocked
openai.comtech restricted*restricted*restricted*restricted*restricted*restricted*
opentable.comfood unknownunknownunknownunknownunknownunknown
oracle.comtech unknownunknownunknownunknownunknownunknown
paypal.comfinance yesrestricted*restricted*restricted*restricted*restricted*restricted*
pcgamer.comentertainment restricted*restricted*restricted*restricted*restricted*blocked
perplexity.aitech restricted*restricted*restricted*restricted*restricted*restricted*
pinterest.comsocial blocked*blocked*blocked*blocked*blocked*blocked*
politico.comnews unknownunknownunknownunknownunknownunknown
polygon.comentertainment blockedblockedblockedblockedblockedblocked
pypi.orgtech restricted*restricted*restricted*restricted*restricted*restricted*
quizlet.comeducation unknownunknownunknownunknownunknownunknown
quora.comreference blockedblockedrestrictedblockedblocked*blocked
realtor.comrealestate restricted*restricted*restricted*restricted*restricted*restricted*
reddit.comsocial blocked*blocked*blocked*blocked*blocked*blocked*
redfin.comrealestate restricted*restricted*restricted*restricted*restricted*restricted*
reuters.comnews blocked*blocked*blocked*blocked*blocked*blocked*
robinhood.comfinance restricted*restricted*restricted*restricted*restricted*restricted*
roblox.comentertainment blockedblockedblockedrestricted*blockedblocked
rottentomatoes.comreference restricted*restricted*restricted*restricted*restricted*restricted*
salesforce.comtech yesrestricted*restricted*restricted*restricted*restricted*restricted*
schwab.comfinance unknownunknownunknownunknownunknownunknown
sec.govgovernment restricted*restricted*restricted*restricted*restricted*restricted*
seekingalpha.comfinance restricted*blockedrestricted*blockedblockedblocked
semafor.comnews blockedrestricted*restricted*restricted*restricted*restricted*
sephora.comecommerce unknownunknownunknownunknownunknownunknown
seriouseats.comfood restrictedblockedblockedblockedblockedblocked
shein.comecommerce yesrestricted*restricted*restricted*restricted*restricted*restricted*
skyscanner.comtravel restricted*restricted*restricted*restricted*restricted*restricted*
slack.comtech yesrestricted*restricted*restricted*restricted*restricted*restricted*
snapchat.comsocial blockedblockedblockedrestrictedblockedblocked
snopes.comreference blockedblockedallowedblockedblockedblocked
soundcloud.comentertainment blockedblockedblockedblockedblockedblocked
southwest.comtravel restricted*restricted*restricted*restricted*restricted*restricted*
spotify.comentertainment restricted*restricted*restricted*restricted*restricted*restricted*
squarespace.comtech yesrestrictedrestrictedrestrictedrestricted*restrictedrestricted
stackoverflow.comreference unknownunknownunknownunknownunknownunknown
stanford.edueducation restricted*restricted*restricted*restricted*restricted*restricted*
steampowered.comentertainment restricted*restricted*restricted*restricted*restricted*restricted*
stripe.comfinance yesrestricted*restricted*restricted*restricted*restricted*restricted*
substack.comsocial restricted*restricted*restricted*restricted*restricted*restricted*
target.comecommerce yesrestricted*restricted*restricted*restricted*restricted*restricted*
techcrunch.comnews blockedblockedblockedrestricted*blockedblocked
ted.comeducation blockedblockedblockedrestrictedblockedblocked
temu.comecommerce restrictedrestrictedrestrictedrestrictedallowedblocked
theathletic.comsports blockedblockedblockedblockedblockedblocked
theatlantic.comnews allowedblockedblockedblockedblockedblocked
theguardian.comnews restricted*blockedrestricted*blockedblockedblocked
theinformation.comnews unknownunknownunknownunknownunknownunknown
theverge.comnews allowedblockedblockedblockedblockedblocked
threads.comsocial blockedblockedblockedblockedblocked*blocked*
tiktok.comsocial blockedblockedblockedblockedblockedblocked
time.comnews allowedallowedallowedallowedallowedallowed
tripadvisor.comtravel blockedblockedblockedrestrictedblockedblocked
tumblr.comsocial restricted*blockedblockedrestricted*blockedrestricted*
twitch.tvsocial restricted*restricted*restricted*restricted*restricted*restricted*
ubereats.comecommerce unknownunknownunknownunknownunknownunknown
udemy.comeducation unknownunknownunknownunknownunknownunknown
united.comtravel unknownunknownunknownunknownunknownunknown
urbandictionary.comreference allowedallowedallowedallowedallowedallowed
usa.govgovernment restricted*restricted*restricted*restricted*restricted*restricted*
usatoday.comnews blockedblockedblockedblockedblockedblocked
uspto.govgovernment restricted*restricted*restricted*restricted*restricted*restricted*
vanguard.comfinance restricted*restricted*restricted*restricted*restricted*restricted*
vercel.comtech yesrestricted*restricted*restricted*restricted*restricted*restricted*
vox.comnews allowedblockedblockedblockedblockedblocked
vrbo.comtravel restricted*restricted*restricted*restrictedrestricted*restricted*
walmart.comecommerce restricted*restricted*restricted*restricted*restricted*restricted*
washingtonpost.comnews restricted*blockedrestricted*blockedblockedblocked
wayfair.comecommerce restricted*restricted*restricted*restricted*restricted*restricted*
weather.comreference yesblockedblockedblockedblockedblockedblocked
weather.govgovernment allowedallowedallowedallowedallowedallowed
webmd.comhealth blockedblockedrestricted*restricted*blockedrestricted*
wellsfargo.comfinance restricted*restricted*restricted*restricted*restricted*restricted*
who.inthealth allowedallowedallowedallowedblockedallowed
wikihow.comreference blockedblockedrestricted*blockedblockedblocked
wikipedia.orgreference restricted*restricted*restricted*restricted*restricted*restricted*
wired.comnews restricted*blockedblockedblockedblockedblocked
wix.comtech yesrestricted*restricted*restricted*restricted*restricted*restricted*
wordpress.comtech yesrestricted*restricted*restricted*restricted*restricted*restricted*
wsj.comnews restrictedblocked*blocked*blocked*blocked*blocked*
x.comsocial blocked*blocked*restrictedblocked*blocked*blocked*
yelp.comfood blockedblockedblockedblockedblockedblocked*
youtube.comsocial restricted*restricted*restricted*restricted*restricted*restricted*
zendesk.comtech yesrestricted*restricted*restricted*restricted*restricted*restricted*
zillow.comrealestate restricted*restricted*restricted*restricted*restricted*restricted*
ziprecruiter.comjobs yesblockedrestrictedrestrictedrestrictedblockedblocked*
zoom.ustech restricted*restricted*restricted*restricted*restricted*restricted*

Chips read from each site's robots.txt: allowed no rule stops this bot · restricted some paths disallowed · blocked fully disallowed · unknown we couldn't read the file. An asterisk means the verdict is inherited from the site's default (*) rules rather than naming the bot. Every site page shows all 16 tracked bots.