可以通过聪明的机器人元标签，来与机器人爬虫交互，告知它怎么爬你的网站，用法如下：

<meta name=”robots” content=”noindex, nofollow, noodp, noydir, nosnippet”>

可以放到content中的关键字有以下几个:

• “noindex”– this will prevent search engines from indexing the page
• “nofollow” — this will prevent search engines from following any link on the page
• “noarchive” — this will prevent search engines from keeping a cached version of the page in search results
• “nosnippet” — this will prevent a description from appearing on your site’s listing on search engine results pages
• “noodp” — this will prevent search engines from displaying the Open Directory title or description on your site’s listing
• “noydir” — this will prevent Yahoo! from displaying the Yahoo! Directory title or description on your sites listing
• “none” – this is the equivalent of “noindex, nofollow”
• “noimageindex” – this will prevent search engines from indexing images on the page

参考：http://www.yourseoplan.com/meta-tags/