1.Take notepad and copy paste the following code
User-agent: *
Disallow: /
2. save the file as robots.txt
3. put the file to website root
All finished. This file tells web robots to do not to visit any page of the website.
Click the following link to download the sample robots.txt
Download
Webmasters handle the robots.txt file to give direction about their website to web robots. This process is called The Robots Exclusion Protocol.
Working of robots.txt
Web robot wants to visit a Web site URL, say https://www.smarttips.in/videos. Before it makes so, first checks for https://www.smarttips.in/robots.txt and will find the following code.
User-agent: *
Disallow: /
The symbol “*” denotes, this for all robots. The “Disallow: /” instruct the robot that it should not visit any pages on the site.
Related Questions:
how do I hide my website from search engines?
how to make website invisible to search engines?
prevent search engines from indexing?
block search engine?