How to hide web pages, folders, or your entire site from search engines: robots.txt
Good search engines (Yahoo, Google, etc.) will leave designated parts of your web site alone if you ask them nicely.
This DOES NOT HIDE your web content from BAD searches - all of your content is still publicly-accessible on your site. In fact, by including this file, you could be pointing bad searchers in the direction of your private content.
- Create a text file in the root folder of the site where you want to "hide" content and use robots.txt for its filename.
- Within the robots.txt file, place the code that tells good searchers (A.K.A. spiders) where NOT to look.
To prevent good spiders from viewing your entire web site, use this code:
To prevent good spiders from viewing a specific folder on your web site, use this code:
To prevent good spiders from viewing a file on your web site, use this code:
For more details, see the Web Robots page.