Robots Site Settings
Some sites use the robots.txt file or the ROBOTS meta tag to tell spiders and others which parts of the Web site they may access. Be careful not to misuse this feature because it is quite easy to accidentally remove your site completely from search engine listings.
To allow spiders and robots to access all content on your site, include the following meta tag:
<meta name="ROBOTS" content="allow;all">
To prevent them from accessing any content on your site, include the following meta tag:
<meta name="ROBOTS" content="disallow;all">