← all sites

Education sites vs AI crawlers

10 tracked education sites, 7 with readable policy. Snapshot: 2026-08-25

28.6%
block ≥1 AI crawler (2/7)
meta-externalagent
most-blocked bot here (2 sites)
1
publish llms.txt
Sitellms.txtGPTBotClaudeBotGoogle-ExtendedPerplexityBotCCBotBytespider
ted.com — blockedblockedblockedrestrictedblockedblocked
coursera.org yes restrictedrestrictedrestrictedrestrictedrestrictedrestricted*
chegg.com — restricted*restricted*restricted*restricted*restricted*restricted*
duolingo.com — restricted*restricted*restricted*restricted*restricted*restricted*
harvard.edu — allowedallowedallowedallowedallowedallowed
khanacademy.org — unknownunknownunknownunknownunknownunknown
mit.edu — restricted*restricted*restricted*restricted*restricted*restricted*
quizlet.com — unknownunknownunknownunknownunknownunknown
stanford.edu — restricted*restricted*restricted*restricted*restricted*restricted*
udemy.com — unknownunknownunknownunknownunknownunknown