User-agent: * Disallow: /cgi/ Disallow: /cart/ Disallow: /acc_mgt/ Disallow: /manage/ Disallow: /email/ Disallow: /searches/ Disallow: /sort/ #Baiduspider User-agent: Baiduspider Disallow: / #Xenu User-agent: xenu link sleuth/1.3.8 Disallow: / User-agent: xenu Disallow: / #Python Bots User-agent: python-requests Disallow: /