> Retrieving Robots.txt file from bing.com/robots.txt
> Retrieval Complete
> bing.com responded with HTTP status code 200 OK

It Depends... :-/


The robots.txt file contains "Disallow" instructions. Whether or not you can crawl this site depends on who you are and what folders you want to crawl.



We've broken out the individual declarations from the robots.txt file below. Hover any section for more detail about what the declaration means. We've also hyperlinked any URL's and paths that were included in the file for your convenience.

**Please read our disclaimer to learn about other considerations before crawling a website.**
  1. User-agent: msnbot-media
    ⇧ This row specifies a user agent pattern. Any robots or browsers that match this pattern should obey the instructions on the following rows.
    Disallow: /
    ⇧ This row declares that the specified user agents are disallowed access to the site's root (a.k.a. "/")
    Allow: /th?
    ⇧ This row contains instructions for folders that are allowed access to the user agents specified above.
    User-agent: Twitterbot
    ⇧ This row specifies a user agent pattern. Any robots or browsers that match this pattern should obey the instructions on the following rows.
    Disallow:
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    User-agent: *
    ⇧ This row specifies that all user agents (a.k.a. "*") should obey the instructions on the following rows.
    Disallow: /account/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /amp/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /bfp/search
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /blogs/search/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /fd/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /history
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /hotels/search
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /images?
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /images/search?
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /local
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /notifications/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /ping
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /Proxy.ashx
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /results
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /rewardsapp/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /search
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /Search
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /settings
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /shenghuo
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /shop$
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /shop?
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /shop/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /social/search?
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /spbasic
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /spresults
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /static/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /th?
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /th$
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /travel/css
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /travel/scripts
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /travel/secure
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /url
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /videos?
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /videos/?
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /videos/search?
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /widget/cr
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /widget/render
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /work
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /fun/g/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /fun/api/
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /hotel/reviews?
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /hpm
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above.
    Disallow: /hpmob
    ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains instructions to disallow robot access to a particular folder on the site for user agents specified above. ⇧ This row contains a path to a sitemap file that contains URLs that are crawlable. ⇧ This row contains a path to a sitemap file that contains URLs that are crawlable. ⇧ This row contains a path to a sitemap file that contains URLs that are crawlable. ⇧ This row contains a path to a sitemap file that contains URLs that are crawlable. ⇧ This row contains a path to a sitemap file that contains URLs that are crawlable. ⇧ This row contains a path to a sitemap file that contains URLs that are crawlable.
User-agent: msnbot-media
Disallow: /
Allow: /th?

User-agent: Twitterbot
Disallow:

User-agent: *
Disallow: /account/
Disallow: /amp/
Disallow: /bfp/search
Disallow: /bing-site-safety
Disallow: /blogs/search/
Disallow: /entities/search
Disallow: /fd/
Disallow: /history
Disallow: /hotels/search
Disallow: /images?
Disallow: /images/search?
Disallow: /images/search/?
Disallow: /images/searchbyimage
Disallow: /local
Disallow: /maps/adsendpoint
Disallow: /notifications/
Disallow: /offers/proxy/dealsserver/api/log
Disallow: /offers/proxy/dealsserver/buy
Disallow: /ping
Disallow: /profile/history?
Disallow: /proFile/history?
Disallow: /Proxy.ashx
Disallow: /results
Disallow: /rewardsapp/
Disallow: /search
Disallow: /Search
Disallow: /settings
Disallow: /shenghuo
Disallow: /shop$
Disallow: /shop?
Disallow: /shop/
Disallow: /social/search?
Disallow: /spbasic
Disallow: /spresults
Disallow: /static/
Disallow: /th?
Disallow: /th$
Disallow: /translator/?ref=
Disallow: /travel/css
Disallow: /travel/flight/flightSearch
Disallow: /travel/flight/flightSearchAction
Disallow: /travel/flight/search?
Disallow: /travel/flight/search/?
Disallow: /travel/hotel/hotelMiniSearchRequest
Disallow: /travel/hotel/hotelSearch
Disallow: /travel/hotels/search?
Disallow: /travel/hotels/search/?
Disallow: /travel/scripts
Disallow: /travel/secure
Disallow: /url
Disallow: /videos?
Disallow: /videos/?
Disallow: /videos/search?
Disallow: /videos/search/?
Disallow: /widget/cr
Disallow: /widget/entity/search/?
Disallow: /widget/render
Disallow: /widget/snapshot
Disallow: /work
Disallow: /academic/search
Disallow: /academic/profile
Disallow: /fun/g/
Disallow: /fun/api/
Disallow: /merchant/reviews?
Disallow: /product/reviews?
Disallow: /hotel/reviews?
Disallow: /hpm
Disallow: /hpmob
Disallow: /HpImageArchive.aspx


Sitemap: http://cn.bing.com/dict/sitemap-index.xml
Sitemap: http://cached.blob.core.windows.net/tmp/sitemap_all_v2.xml
Sitemap: https://www.bing.com/api/maps/mapcontrol/isdk/sitemap.xml
Sitemap: https://www.bing.com/travelguide/sitemaps/sitemap.xml
Sitemap: https://www.bing.com/maps/sitemap.xml
Sitemap: https://bingknowsblob.blob.core.windows.net/bingknows/sitemaps/sitemapIndex.xml.gz