Can I Crawl a Website?

We're ready to help you find out.

Just type a domain name in the search box above and we'll help you out by reading the domain's robots.txt file and interpretting it for you.

If you want to understand more about how our tool works and what you should use it for, please visit the links below: