Add your site to google robot.txt

You can use a robots.txt file to indicate which robots you don't want crawling your site, and which files or directories you don't want crawled.

1: Choose your default robot access

Allow all robots (recommended — you can fine-tune this rule in the next step)
Block all robots

2: Specify any additional rules (optional)

Action	Robot	Files or directories
Allow	All robots	/2009/05/bullet-seo-by-dexter.html		Edit	Delete
Allow	All robots	/2009/05/how-to-built-good-site.html		Edit	Delete
Action... Allow Block Error	Type the name of a robot ("user-agent") Robot... All robots Googlebot Googlebot-Mobile Googlebot-Image Mediapartners-Google Adsbot-Google Other robot... Error	If you want this rule to affect all files and directories, type /. To specify individual files or directories, type them one per line. Example: /directory/file.html /directory/subdirectory/ Error	Add

3: Save your robots.txt file

To save your robots.txt file, click Download, or paste the text below into a .txt file. Name the file robots.txt, and save it to your site's top-level directory.