← all bots

CCBot

Builds the open Common Crawl corpus, historically a major source of AI training data. Operator docs.

Operator
Common Crawl
Purpose
training
Blocked by
166 of 652 readable sites (25.5%) — 124 name it explicitly, 42 block it via default-deny rules

Sites blocking CCBot

SiteCategoryHow
academia.edutop1kdefault-deny robots.txt
adform.nettop1kdefault-deny robots.txt
allrecipes.comfoodnamed explicitly
amazon.catop1knamed explicitly
amazon.co.jptop1knamed explicitly
amazon.co.uktop1knamed explicitly
amazon.co.zatop1knamed explicitly
amazon.comecommercenamed explicitly
amazon.com.autop1knamed explicitly
amazon.com.brtop1knamed explicitly
amazon.detop1knamed explicitly
amazon.estop1knamed explicitly
amazon.frtop1knamed explicitly
amazon.intop1knamed explicitly
amazon.ittop1knamed explicitly
amazonvideo.comtop1knamed explicitly
aol.comtop1knamed explicitly
apnews.comnewsnamed explicitly
arstechnica.comnewsnamed explicitly
axios.comnewsnamed explicitly
baidu.comtop1kdefault-deny robots.txt
bandcamp.comtop1knamed explicitly
bbc.co.uktop1knamed explicitly
bbc.comnewsnamed explicitly
behance.nettop1knamed explicitly
bild.detop1knamed explicitly
blackhub.teamtop1kdefault-deny robots.txt
bleacherreport.comsportsnamed explicitly
bloomberg.comnewsnamed explicitly
bonappetit.comfoodnamed explicitly
businessinsider.comnewsnamed explicitly
calendly.comtop1knamed explicitly
canva.comtechnamed explicitly
chatgpt.comtop1knamed explicitly
cnbc.comfinancenamed explicitly
cnet.comtop1knamed explicitly
cnn.comnewsnamed explicitly
corriere.ittop1knamed explicitly
creativecdn.comtop1kdefault-deny robots.txt
crwdcntrl.nettop1kdefault-deny robots.txt
dailymail.co.uknewsnamed explicitly
dailymail.comtop1knamed explicitly
daum.nettop1kdefault-deny robots.txt
dictionary.comreferencenamed explicitly
ebay.co.uktop1knamed explicitly
ebay.comecommercenamed explicitly
ebay.detop1knamed explicitly
economist.comnewsnamed explicitly
elpais.comtop1knamed explicitly
epicurious.comfoodnamed explicitly
espn.comsportsnamed explicitly
facebook.comsocialdefault-deny robots.txt
fb.comtop1kdefault-deny robots.txt
figma.comtechnamed explicitly
flashscore.comsportsnamed explicitly
flickr.comtop1kdefault-deny robots.txt
forbes.comnewsnamed explicitly
forter.comtop1knamed explicitly
ft.comnewsnamed explicitly
gamespot.comentertainmentnamed explicitly
genius.comreferencenamed explicitly
glassdoor.comjobsnamed explicitly
globalsign.comtop1knamed explicitly
globo.comtop1knamed explicitly
goodreads.comreferencenamed explicitly
gosuslugi.rutop1kdefault-deny robots.txt
hbr.orgtop1knamed explicitly
healthline.comhealthnamed explicitly
hihonorcloud.comtop1kdefault-deny robots.txt
huffpost.comnewsnamed explicitly
ign.comentertainmentnamed explicitly
imdb.comreferencedefault-deny robots.txt
imgsmail.rutop1kdefault-deny robots.txt
instacart.comecommercedefault-deny robots.txt
instagram.comsocialdefault-deny robots.txt
investopedia.comreferencenamed explicitly
kick.comtop1knamed explicitly
kotaku.comentertainmentnamed explicitly
launchpad.nettop1knamed explicitly
lefigaro.frtop1knamed explicitly
lemonde.frtop1knamed explicitly
linkedin.comsocialnamed explicitly
mail.rutop1kdefault-deny robots.txt
mailchi.mptop1kdefault-deny robots.txt
marketwatch.comfinancedefault-deny robots.txt
metacritic.comreferencenamed explicitly
msn.comtop1knamed explicitly
myfritz.nettop1kdefault-deny robots.txt
nature.comtop1knamed explicitly
naver.comtop1kdefault-deny robots.txt
nba.comsportsnamed explicitly
nbcnews.comnewsnamed explicitly
netflix.comentertainmentdefault-deny robots.txt
netflix.nettop1kdefault-deny robots.txt
newsweek.comnewsnamed explicitly
newyorker.comnewsnamed explicitly
nexusmods.comtop1knamed explicitly
nikkei.comtop1knamed explicitly
nintendo.comtop1knamed explicitly
nypost.comnewsnamed explicitly
nytimes.comnewsnamed explicitly
patreon.comtop1knamed explicitly
people.comtop1knamed explicitly
pexels.comtop1knamed explicitly
pinimg.comtop1kdefault-deny robots.txt
pinterest.comsocialdefault-deny robots.txt
pixabay.comtop1knamed explicitly
polygon.comentertainmentnamed explicitly
primevideo.comtop1knamed explicitly
quantserve.comtop1kdefault-deny robots.txt
quora.comreferencedefault-deny robots.txt
reddit.comsocialdefault-deny robots.txt
registrar-servers.comtop1kdefault-deny robots.txt
repubblica.ittop1knamed explicitly
reuters.comnewsdefault-deny robots.txt
roblox.comentertainmentnamed explicitly
rocketmoney.devtop1kdefault-deny robots.txt
sagepub.comtop1knamed explicitly
scribd.comtop1knamed explicitly
seekingalpha.comfinancenamed explicitly
sentinelone.nettop1kdefault-deny robots.txt
seriouseats.comfoodnamed explicitly
sharethrough.comtop1knamed explicitly
slideshare.nettop1knamed explicitly
snapchat.comsocialnamed explicitly
snopes.comreferencenamed explicitly
sohu.comtop1kdefault-deny robots.txt
soundcloud.comentertainmentnamed explicitly
spiegel.detop1knamed explicitly
statista.comtop1knamed explicitly
stbid.rutop1kdefault-deny robots.txt
t-online.detop1knamed explicitly
t.cotop1kdefault-deny robots.txt
techcrunch.comnewsnamed explicitly
ted.comeducationnamed explicitly
telegraph.co.uktop1knamed explicitly
theathletic.comsportsnamed explicitly
theatlantic.comnewsnamed explicitly
theguardian.comnewsnamed explicitly
theverge.comnewsnamed explicitly
threads.comsocialdefault-deny robots.txt
tiktok.comsocialnamed explicitly
tiktokw.ustop1kdefault-deny robots.txt
tripadvisor.comtravelnamed explicitly
trustpilot.comtop1knamed explicitly
tumblr.comsocialnamed explicitly
twitter.comtop1kdefault-deny robots.txt
uol.com.brtop1knamed explicitly
usatoday.comnewsnamed explicitly
vimeo.comtop1knamed explicitly
vkontakte.rutop1kdefault-deny robots.txt
vox.comnewsnamed explicitly
washingtonpost.comnewsnamed explicitly
weather.comreferencenamed explicitly
webmd.comhealthnamed explicitly
weibo.comtop1kdefault-deny robots.txt
welt.detop1knamed explicitly
who.inthealthnamed explicitly
wikihow.comreferencenamed explicitly
wired.comnewsnamed explicitly
wp.pltop1knamed explicitly
wsj.comnewsdefault-deny robots.txt
x.comsocialdefault-deny robots.txt
yahoo.comtop1knamed explicitly
yelp.comfoodnamed explicitly
ziprecruiter.comjobsnamed explicitly