> Retrieving Robots.txt file from alibaba.com/robots.txt
> Retrieval Complete
> alibaba.com responded with HTTP status code 200

It Depends... :-/


The robots.txt file contains "Disallow" instructions. Whether or not you can crawl this site depends on who you are and what folders you want to crawl.



We've broken out the individual declarations from the robots.txt file below. Hover any section for more detail about what the declaration means. We've also hyperlinked any URL's and paths that were included in the file for your convenience.

**Please read our disclaimer to learn about other considerations before crawling a website.**
  1. User-agent: *
    ⇧ This row specifies that all user agents (a.k.a. "*") should obey the instructions on the following rows.
    Allow: /*s=p
    ⇧ This row contains instructions for folders that are allowed access to the user agents specified above.
    Allow: /*?_escaped_fragment_
    ⇧ This row contains instructions for folders that are allowed access to the user agents specified above.
    Allow: *?src=sem_ggl
    ⇧ This row contains instructions for folders that are allowed access to the user agents specified above.
    Allow: /catalog/*?
    ⇧ This row contains instructions for folders that are allowed access to the user agents specified above. ⇧ This row contains instructions for folders that are allowed access to the user agents specified above.
    Allow: /products/
    ⇧ This row contains instructions for folders that are allowed access to the user agents specified above.
    Allow: *?spm=
    ⇧ This row contains instructions for folders that are allowed access to the user agents specified above.
    Disallow: /*?
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /bin/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /apps/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /buy/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /memberhome/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /minisite/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /corporations/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /buyers/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /trade/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /product/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /rss/*/*.rss
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /activities/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /member/*/contactinfo.html
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /member/*/company_profile.html
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /member/*/productlist.html
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /searchweb/*.html
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /magellan/*.html
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /newsuppliers/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /trends/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /city/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /countrysearch/*/*-importer.html
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /countrysearch/*/*/importers/*.html
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /search/buyeroffers/*.html
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /search/buyeroffers/*/*.html
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /buyeroffers/*/*.html
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /buyeroffers/*/*/*.html
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /detail/ajax/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    User-agent: bingbot
    ⇧ This row specifies a user agent pattern. Any robots or browsers that match this pattern should obey the instructions on the following rows.
    Allow: /*s=p
    ⇧ This row contains instructions for folders that are allowed access to the user agents specified above.
    Allow: /*?_escaped_fragment_
    ⇧ This row contains instructions for folders that are allowed access to the user agents specified above.
    Allow: *?src=sem_ggl
    ⇧ This row contains instructions for folders that are allowed access to the user agents specified above.
    Allow: /catalog/*?
    ⇧ This row contains instructions for folders that are allowed access to the user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /products/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /*?
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /bin/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /apps/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /buy/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /memberhome/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /minisite/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /corporations/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /buyers/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /trade/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /product/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /rss/*/*.rss
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /activities/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /member/*/contactinfo.html
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /member/*/company_profile.html
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /member/*/productlist.html
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /searchweb/*.html
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /magellan/*.html
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /newsuppliers/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /trends/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /city/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /countrysearch/*/*-importer.html
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /countrysearch/*/*/importers/*.html
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /search/buyeroffers/*.html
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /search/buyeroffers/*/*.html
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /buyeroffers/*/*.html
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /buyeroffers/*/*/*.html
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /detail/ajax/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
User-agent: *
Allow: /*s=p
Allow: /*?_escaped_fragment_
Allow: *?src=sem_ggl
Allow: /catalog/*?
Allow: /catalogs/products/
Allow: /products/
Allow: *?spm=
Disallow: /*?
Disallow: /bin/
Disallow: /apps/
Disallow: /buy/
Disallow: /memberhome/
Disallow: /minisite/
Disallow: /corporations/
Disallow: /buyers/
Disallow: /trade/
Disallow: /product/
Disallow: /catalogs/corporations/
Disallow: /rss/*/*.rss
Disallow: /activities/
Disallow: /productshowimg/
Disallow: /member/*/contactinfo.html
Disallow: /member/*/company_profile.html
Disallow: /member/*/productlist.html
Disallow: /searchweb/*.html
Disallow: /magellan/*.html
Disallow: /newsuppliers/
Disallow: /wholesalecheckout/
Disallow: /buyofferdetail/
Disallow: /trends/
Disallow: /industry-promotion/
Disallow: /city/
Disallow: /countrysearch/*/*-importer.html
Disallow: /countrysearch/*/*/importers/*.html
Disallow: /search/buyeroffers/*.html
Disallow: /search/buyeroffers/*/*.html
Disallow: /buyeroffers/*/*.html
Disallow: /buyeroffers/*/*/*.html
Disallow: /detail/ajax/
Disallow: /ajax/seoRequest/

User-agent: bingbot
Allow: /*s=p
Allow: /*?_escaped_fragment_
Allow: *?src=sem_ggl
Allow: /catalog/*?
Disallow: /catalogs/products/
Disallow: /products/
Disallow: /*?
Disallow: /bin/
Disallow: /apps/
Disallow: /buy/
Disallow: /memberhome/
Disallow: /minisite/
Disallow: /corporations/
Disallow: /buyers/
Disallow: /trade/
Disallow: /product/
Disallow: /catalogs/corporations/
Disallow: /rss/*/*.rss
Disallow: /activities/
Disallow: /productshowimg/
Disallow: /member/*/contactinfo.html
Disallow: /member/*/company_profile.html
Disallow: /member/*/productlist.html
Disallow: /searchweb/*.html
Disallow: /magellan/*.html
Disallow: /newsuppliers/
Disallow: /wholesalecheckout/
Disallow: /buyofferdetail/
Disallow: /trends/
Disallow: /industry-promotion/
Disallow: /city/
Disallow: /countrysearch/*/*-importer.html
Disallow: /countrysearch/*/*/importers/*.html
Disallow: /search/buyeroffers/*.html
Disallow: /search/buyeroffers/*/*.html
Disallow: /buyeroffers/*/*.html
Disallow: /buyeroffers/*/*/*.html
Disallow: /detail/ajax/
Disallow: /ajax/seoRequest/