1.Take notepad and copy paste the following code
2. save the file as robots.txt
3. put the file to website root
All finished. This file tells web robots to do not to visit any page of the website.
Click the following link to download the sample robots.txt
Webmasters handle the robots.txt file to give direction about their website to web robots. This process is called The Robots Exclusion Protocol.
Working of robots.txt
Web robot wants to visit a Web site URL, say http://www.smarttips.in/videos. Before it makes so, first checks for http://www.smarttips.in/robots.txt and will find the following code.
The symbol “*” denotes, this for all robots. The “Disallow: /” instruct the robot that it should not visit any pages on the site.
how do I hide my website from search engines?
how to make website invisible to search engines?
prevent search engines from indexing?
block search engine?