# /robots.txt file for http://webcrawler.com/ # mail webmaster@webcrawler.com for constructive criticism User-agent: * Disallow: /tmp/ Disallow: /logs/ Disallow: /vsadmin/ Disallow: /GeneratedItems/ Disallow: /cgi-bin/ Disallow: /diets/ Disallow: /forum/admin/ Disallow: /forum/includes/ Disallow: /forum/language/ Disallow: /forum/docs/ Disallow: /forum/db/ Disallow: /baghera1389/ Disallow: /phpMySQLAutoBackup/ Disallow: https://secure.dietresourcesenter.com allow: http://healthblog.dietresourcesenter.com Sitemap: http://www.dietresourcesenter.com/sitemap.xml Disallow: http://healthblog.dietresourcesenter.com/wp-admin