← all bots

omgilibot

Webz.io crawler token (sibling of omgili).

Operator
Webz.io
Purpose
training
Blocked by
110 of 650 readable sites (16.9%) — 62 name it explicitly, 48 block it via default-deny rules

Sites blocking omgilibot

SiteCategoryHow
academia.edutop1kdefault-deny robots.txt
adform.nettop1kdefault-deny robots.txt
allrecipes.comfoodnamed explicitly
aol.comtop1knamed explicitly
arstechnica.comnewsnamed explicitly
baidu.comtop1kdefault-deny robots.txt
bbc.co.uktop1knamed explicitly
bbc.comnewsnamed explicitly
behance.nettop1knamed explicitly
bild.detop1knamed explicitly
blackhub.teamtop1kdefault-deny robots.txt
chatgpt.comtop1knamed explicitly
cnbc.comfinancenamed explicitly
cnet.comtop1knamed explicitly
cnn.comnewsnamed explicitly
creativecdn.comtop1kdefault-deny robots.txt
crwdcntrl.nettop1kdefault-deny robots.txt
dailymail.co.uknewsnamed explicitly
dailymail.comtop1knamed explicitly
daum.nettop1kdefault-deny robots.txt
disneyplus.comentertainmentdefault-deny robots.txt
elpais.comtop1knamed explicitly
espn.comsportsnamed explicitly
facebook.comsocialdefault-deny robots.txt
fb.comtop1kdefault-deny robots.txt
flashscore.comsportsnamed explicitly
flickr.comtop1kdefault-deny robots.txt
forbes.comnewsnamed explicitly
forter.comtop1knamed explicitly
ft.comnewsnamed explicitly
glassdoor.comjobsnamed explicitly
hihonorcloud.comtop1kdefault-deny robots.txt
huffpost.comnewsnamed explicitly
hulu.comentertainmentdefault-deny robots.txt
imdb.comreferencedefault-deny robots.txt
imgsmail.rutop1kdefault-deny robots.txt
indeed.comjobsnamed explicitly
instacart.comecommercedefault-deny robots.txt
instagram.comsocialdefault-deny robots.txt
investopedia.comreferencenamed explicitly
kotaku.comentertainmentnamed explicitly
latimes.comnewsnamed explicitly
launchpad.nettop1knamed explicitly
lefigaro.frtop1knamed explicitly
lemonde.frtop1knamed explicitly
linkedin.comsocialnamed explicitly
linktr.eetop1kdefault-deny robots.txt
mail.rutop1kdefault-deny robots.txt
mailchi.mptop1kdefault-deny robots.txt
marketwatch.comfinancedefault-deny robots.txt
meraki.comtop1kdefault-deny robots.txt
msn.comtop1knamed explicitly
myfritz.nettop1kdefault-deny robots.txt
nature.comtop1knamed explicitly
naver.comtop1kdefault-deny robots.txt
nbcnews.comnewsnamed explicitly
netflix.comentertainmentdefault-deny robots.txt
netflix.nettop1kdefault-deny robots.txt
newsweek.comnewsnamed explicitly
nikkei.comtop1knamed explicitly
nintendo.comtop1knamed explicitly
nypost.comnewsnamed explicitly
nytimes.comnewsnamed explicitly
office365.comtop1kdefault-deny robots.txt
people.comtop1knamed explicitly
pinimg.comtop1kdefault-deny robots.txt
pinterest.comsocialdefault-deny robots.txt
polygon.comentertainmentnamed explicitly
quantserve.comtop1kdefault-deny robots.txt
quora.comreferencedefault-deny robots.txt
reddit.comsocialdefault-deny robots.txt
registrar-servers.comtop1kdefault-deny robots.txt
repubblica.ittop1knamed explicitly
reuters.comnewsdefault-deny robots.txt
sagepub.comtop1knamed explicitly
seekingalpha.comfinancenamed explicitly
sentinelone.nettop1kdefault-deny robots.txt
seriouseats.comfoodnamed explicitly
sharethrough.comtop1kdefault-deny robots.txt
snapchat.comsocialnamed explicitly
snopes.comreferencenamed explicitly
sohu.comtop1kdefault-deny robots.txt
spiegel.detop1knamed explicitly
statista.comtop1knamed explicitly
stbid.rutop1kdefault-deny robots.txt
t.cotop1kdefault-deny robots.txt
ted.comeducationnamed explicitly
telegraph.co.uktop1knamed explicitly
theathletic.comsportsnamed explicitly
theatlantic.comnewsnamed explicitly
theverge.comnewsnamed explicitly
threads.comsocialdefault-deny robots.txt
tiktokw.ustop1kdefault-deny robots.txt
trustpilot.comtop1knamed explicitly
tumblr.comsocialnamed explicitly
twitter.comtop1kdefault-deny robots.txt
uol.com.brtop1knamed explicitly
usatoday.comnewsnamed explicitly
vimeo.comtop1knamed explicitly
vkontakte.rutop1kdefault-deny robots.txt
vox.comnewsnamed explicitly
washingtonpost.comnewsnamed explicitly
weather.comreferencenamed explicitly
weibo.comtop1kdefault-deny robots.txt
welt.detop1knamed explicitly
wsj.comnewsdefault-deny robots.txt
x.comsocialdefault-deny robots.txt
yahoo.comtop1knamed explicitly
yelp.comfooddefault-deny robots.txt
ziprecruiter.comjobsdefault-deny robots.txt