← all bots

AI2Bot

Crawls for AI2's open language-model research (e.g. the Dolma corpus). Operator docs.

Operator
Allen Institute for AI
Purpose
training
Blocked by
96 of 650 readable sites (14.8%) — 46 name it explicitly, 50 block it via default-deny rules

Sites blocking AI2Bot

SiteCategoryHow
academia.edutop1kdefault-deny robots.txt
adform.nettop1kdefault-deny robots.txt
amazon.catop1knamed explicitly
amazon.co.jptop1knamed explicitly
amazon.co.uktop1knamed explicitly
amazon.co.zatop1knamed explicitly
amazon.comecommercenamed explicitly
amazon.com.autop1knamed explicitly
amazon.com.brtop1knamed explicitly
amazon.detop1knamed explicitly
amazon.estop1knamed explicitly
amazon.frtop1knamed explicitly
amazon.intop1knamed explicitly
amazon.ittop1knamed explicitly
amazonvideo.comtop1knamed explicitly
baidu.comtop1kdefault-deny robots.txt
behance.nettop1knamed explicitly
bild.detop1knamed explicitly
blackhub.teamtop1kdefault-deny robots.txt
canva.comtechnamed explicitly
chatgpt.comtop1kdefault-deny robots.txt
cnet.comtop1knamed explicitly
cnn.comnewsnamed explicitly
creativecdn.comtop1kdefault-deny robots.txt
crwdcntrl.nettop1kdefault-deny robots.txt
dailymail.co.uknewsnamed explicitly
dailymail.comtop1knamed explicitly
daum.nettop1kdefault-deny robots.txt
disneyplus.comentertainmentdefault-deny robots.txt
ea.comtop1knamed explicitly
edmunds.comautonamed explicitly
facebook.comsocialdefault-deny robots.txt
fb.comtop1kdefault-deny robots.txt
flashscore.comsportsnamed explicitly
flickr.comtop1kdefault-deny robots.txt
hihonorcloud.comtop1kdefault-deny robots.txt
hulu.comentertainmentdefault-deny robots.txt
ign.comentertainmentnamed explicitly
imdb.comreferencedefault-deny robots.txt
imgsmail.rutop1kdefault-deny robots.txt
instacart.comecommercedefault-deny robots.txt
instagram.comsocialdefault-deny robots.txt
kotaku.comentertainmentnamed explicitly
launchpad.nettop1knamed explicitly
lemonde.frtop1knamed explicitly
linkedin.comsocialdefault-deny robots.txt
linktr.eetop1kdefault-deny robots.txt
mail.rutop1kdefault-deny robots.txt
mailchi.mptop1kdefault-deny robots.txt
marketwatch.comfinancedefault-deny robots.txt
meraki.comtop1kdefault-deny robots.txt
msn.comtop1knamed explicitly
myfritz.nettop1kdefault-deny robots.txt
naver.comtop1kdefault-deny robots.txt
netflix.comentertainmentdefault-deny robots.txt
netflix.nettop1kdefault-deny robots.txt
nikkei.comtop1knamed explicitly
nintendo.comtop1knamed explicitly
office365.comtop1kdefault-deny robots.txt
pcgamer.comentertainmentnamed explicitly
pinimg.comtop1kdefault-deny robots.txt
pinterest.comsocialdefault-deny robots.txt
primevideo.comtop1knamed explicitly
quantserve.comtop1kdefault-deny robots.txt
quora.comreferencedefault-deny robots.txt
reddit.comsocialdefault-deny robots.txt
registrar-servers.comtop1kdefault-deny robots.txt
reuters.comnewsdefault-deny robots.txt
seekingalpha.comfinancenamed explicitly
sentinelone.nettop1kdefault-deny robots.txt
sharethrough.comtop1knamed explicitly
snapchat.comsocialnamed explicitly
sohu.comtop1kdefault-deny robots.txt
spiegel.detop1knamed explicitly
statista.comtop1knamed explicitly
stbid.rutop1kdefault-deny robots.txt
t.cotop1kdefault-deny robots.txt
telegraph.co.uktop1knamed explicitly
theverge.comnewsnamed explicitly
threads.comsocialdefault-deny robots.txt
tiktok.comsocialnamed explicitly
tiktokw.ustop1kdefault-deny robots.txt
trustpilot.comtop1kdefault-deny robots.txt
twitter.comtop1kdefault-deny robots.txt
uol.com.brtop1knamed explicitly
usatoday.comnewsnamed explicitly
vkontakte.rutop1kdefault-deny robots.txt
vox.comnewsnamed explicitly
weibo.comtop1kdefault-deny robots.txt
welt.detop1knamed explicitly
wikihow.comreferencenamed explicitly
wsj.comnewsdefault-deny robots.txt
x.comsocialdefault-deny robots.txt
yelp.comfooddefault-deny robots.txt
zillow.comrealestatenamed explicitly
ziprecruiter.comjobsdefault-deny robots.txt