> Retrieving Robots.txt file from craigslist.org/robots.txt
> Retrieval Complete
> craigslist.org responded with HTTP status code 200 OK

It Depends... :-/


The robots.txt file contains "Disallow" instructions. Whether or not you can crawl this site depends on who you are and what folders you want to crawl.



We've broken out the individual declarations from the robots.txt file below. Hover any section for more detail about what the declaration means. We've also hyperlinked any URL's and paths that were included in the file for your convenience.

**Please read our disclaimer to learn about other considerations before crawling a website.**
  1. User-agent: *
    ⇧ This row specifies that all user agents (a.k.a. "*") should obey the instructions on the following rows.
    Disallow: /reply
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /fb/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /suggest
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /flag
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /mf
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /eaf
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
User-agent: *
Disallow: /reply
Disallow: /fb/
Disallow: /suggest
Disallow: /flag
Disallow: /mf
Disallow: /eaf