[转载]爬虫的自我解剖(抓取网页HtmlUnit)

网络爬虫第一个要面临的问题，就是如何抓取网页，抓取其实很容易，没你想的那么复杂，一个开源HtmlUnit包，4行代码就OK啦，例子如下：

java">

1 2 3 4

java keyword" style="border:0px;color:#0000ff;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">final  java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">WebClient webClient= java keyword" style="border:0px;color:#0000ff;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">new  java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">WebClient(); java keyword" style="border:0px;color:#0000ff;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">final  java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">HtmlPage page=webClient.getPage( java string" style="border:0px;color:#0000FF;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">"http://www.yanyulin.info" java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">); java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">System.out.println(page.asText()); java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">webClient.closeAllWindows();

在程序中写上上面的4行代码，运行，就可以得到烟雨林博客首页的全部内容，上面代码在运行的过程中会出现很多警告，出现这些警告的主要原因是由于以下两点:

1、HtmlUnit对Javascript的支持不是很好

2、HtmlUnit对CSS的支持不是很好

明白了上面的两点后，将代码重新改写一下，该禁用的就禁用，同时禁用一些不必要的功能，也有利于提高程序的运行效率,再者说网络爬虫也不需要CSS的支持滴

java">

1 2 3 4 5 6

java keyword" style="border:0px;color:#0000ff;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">final  java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">WebClient webClient= java keyword" style="border:0px;color:#0000ff;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">new  java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">WebClient(); java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">webClient.getOptions().setCssEnabled( java keyword" style="border:0px;color:#0000ff;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">false java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">); java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">webClient.getOptions().setJavaScriptEnabled( java keyword" style="border:0px;color:#0000ff;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">false java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">); java keyword" style="border:0px;color:#0000ff;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">final  java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">HtmlPage page=webClient.getPage( java string" style="border:0px;color:#0000FF;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">"http://www.yanyulin.info" java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">); java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">System.out.println(page.asText()); java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">webClient.closeAllWindows();

HtmlUnit的使用: 简介:HtmlUnit说白了就是一个浏览器，这个浏览器是用Java写的无界面的浏览器，正因为其没有界面,因此执行的速度还是可以滴，HtmlUnit提供了一系列的API,这些API可以干的功能比较多，如表单的填充，表单的提交，模仿点击链接，由于内置了Rhinojs引擎，因此可以执行Javascript

作用:web的自动化测试(最初的目的),浏览器，网络爬虫

重要API的使用 在介绍API的使用之前要先明白的一个问题是，WebClient,WebWindow,Page三者之间的关系，所有的页面最终都是在一个WebWindow对象里面，WebClient在创建时会自动的创建一个WebWindow对象，当调用getPage时会将新页面加载到WebWindow里，你可以理解成WebClient就是IE内核，WebWindow就是呈现页面的浏览器窗口，三者之间的关系图如下图所示：

1、模拟特定浏览器，也可以指定浏览器的相应版本(HtmlUnit最新版2.13现在可以模拟的浏览器有Chrome/FireFox/IE)

java">

1 2

java comments" style="border:0px;color:#008200;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">//模拟chorme浏览器，其他浏览器请修改BrowserVersion.后面 java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">WebClient  webClient= java keyword" style="border:0px;color:#0000ff;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">new  java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">WebClient(BrowserVersion.CHROME);

2、查找特定元素，通过get或者XPath可以从HtmlPage中获得特定的Html元素，如下例子

方法一，通过get方法获取

java">

1 2 3

java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">HtmlPage page=webClient.getPage( java string" style="border:0px;color:#0000FF;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">"http://www.yanyulin.info" java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">); java comments" style="border:0px;color:#008200;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">//从[烟雨林博客]上获取标签hed的内容 java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">HtmlDivision div=(HtmlDivision)page.getElementById( java string" style="border:0px;color:#0000FF;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">"hed" java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">);

方法二，通过XPath获取，XPath通常用于无法通过Id搜索，或者需要更为复杂的搜索时，XPath的相关教程

XPath相关教程请查看我的个人博客

java">

1 2 3 4

java comments" style="border:0px;color:#008200;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">//同样可以打印出hed的内容,//div中//表示搜索整个文档中的div,并将这些div java comments" style="border:0px;color:#008200;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">//放入list中，然后获取第一个div java keyword" style="border:0px;color:#0000ff;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">final  java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">HtmlDivision div = (HtmlDivision) page.getByXPath( java string" style="border:0px;color:#0000FF;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">"//div" java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">).get( java value" style="border:0px;color:#009900;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">0 java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">); java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">System.out.println(div.asXml());

3、代理服务器的配置，代理的配置很简单，只需要配置好地址，端口，用户名与密码即可

java">

1 2 3

java keyword" style="border:0px;color:#0000ff;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">final  java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">WebClient webClient = java keyword" style="border:0px;color:#0000ff;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">new  java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">WebClient(BrowserVersion.CHROME, java string" style="border:0px;color:#0000FF;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">"http://127.0.0.1" java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">, java value" style="border:0px;color:#009900;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">8087 java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">); java keyword" style="border:0px;color:#0000ff;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">final  java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">DefaultCredentialsProvider credentialsProvider = (DefaultCredentialsProvider) webClient.getCredentialsProvider(); java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">credentialsProvider.addCredentials( java string" style="border:0px;color:#0000FF;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">"username" java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">, java string" style="border:0px;color:#0000FF;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">"password" java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">);

4、模拟表单的提交

java">

1 2 3 4 5 6 7 8 9

java comments" style="border:0px;color:#008200;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">//获取表单  java keyword" style="border:0px;color:#0000ff;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">final  java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">HtmlForm form = page.getFormByName( java string" style="border:0px;color:#0000FF;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">"form" java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">); java comments" style="border:0px;color:#008200;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">//获取提交按扭 java keyword" style="border:0px;color:#0000ff;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">final  java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">HtmlSubmitInput button = form.getInputByName( java string" style="border:0px;color:#0000FF;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">"submit" java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">); java comments" style="border:0px;color:#008200;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">//一会得输入的 java keyword" style="border:0px;color:#0000ff;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">final  java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">HtmlTextInput textField = form.getInputByName( java string" style="border:0px;color:#0000FF;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">"userid" java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">); java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">textField.setValueAttribute( java string" style="border:0px;color:#0000FF;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">"test" java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">); java comments" style="border:0px;color:#008200;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">//点击提交表单 java keyword" style="border:0px;color:#0000ff;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">final  java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">HtmlPage page = button.click();

API的使用就介绍到这，网络爬虫中主要目的就是获取页中所有的链接，代码如下：

java">

1 2 3 4

java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">java.util.List<HtmlAnchor> achList=page.getAnchors(); java keyword" style="border:0px;color:#0000ff;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">for java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">(HtmlAnchor ach:achList){ java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">System.out.println(ach.getHrefAttribute()); java plain" style="border:0px;line-height:1.8em;font-family:Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace;vertical-align:baseline;">}

最后来个例子，HtmlUnit模拟浏览器登录小米网站帐户，程运运行的截图如下，红色方框表示登录成功跳转到下一个页面出现的帐号：

本博文源代码下载以及相关jar包下载，请到我的个人博客中下载,如果遇到什么问题或本文有什么问题，请到我的个人博客中给我留言

本博文源代码下载:

源码下载

Java皮肤库下载

HtmlUnit源码包下载

HtmlUnit的Jar包下载

本文地址: http://www.yanyulin.info/pages/2013/11/fetchPage.html