Disclaimer : Can I Crawl a site?
This website was created to assist in reading and interperting robots.txt files on other websites, and helping humans understand whether or not they should use robots on these websites. While robots.txt files are useful and fairly standard on the web, they are not the only factor to be considered when asking yourself whether or not you should crawl a website.
Other factors to consider when crawling are listed below (but are not limited to):
- Copyright notices
- End User Agreements
- Robots meta tags
- Relevant laws in your country
- Relevant laws in the website's country of origin
- Any legal statements posted on the site pertaining to crawling or robots
At the end of the day, some level of judgment comes into play when considering whether a site should be crawled or not, especially since your actions could have legal repurcussions. Just because you can crawl a website doesn't necessarily mean that you should, especially if the website's owners have expressed (in any way) the desire to not be crawled.
We do not accept any responsibility or liability for the decisions you make based on this tool since our site focuses exclusively on one single factor of crawling websites, and does not attempt to evaluate other considerations beyond the robots.txt file.
Bottom line: We are not responsible for your actions, or what you choose to do based on this tool. It is up to you to crawl the web responsibly and not engage in illegal activity.