# A robots.txt is a file placed on your server to tell the various search engine spiders not to crawl or index certain sections or pages of your site. You can use it to prevent indexing totally, prevent certain areas of your site from being indexes or to issue individual indexing instructions to specific search engines. # robots.txt file for http://www.certstation.com # The following bans all search engines from the directories we do not want indexed at all. User-agent: * Disallow: /_private/ Disallow: /_vti_bin/ Disallow: /_vti_cnf/ Disallow: /_vti_log/ Disallow: /_vti_pvt/ Disallow: /_vti_txt/ Disallow: /0xmluser/ Disallow: /demo_arpatech/ Disallow: /picture_library/ Disallow: /plesk-stat/ Disallow: /qa-test/ Disallow: /text_file/ Disallow: /flash/ Disallow: /rssdemo/ # The following prevents the Google image bot from indexing any of our images, no matter where they are on the site. I haven't specified the '/images/' directory since some images lie outside of it. User-Agent: Googlebot-Image Disallow: /