2019独角兽企业重金招聘Python工程师标准>>> 
关于数据过滤，scrapy提供xpath和css两种过滤器（selector），一般xpath使用的较多，另外我对css也不算熟。这里主要是xpath。
关于xpath，是一种专门在 XML 文档中查找信息的语言。详细教程可以看这里：http://www.w3school.com.cn/xpath/，不过对于刚入门的人来说不用那么复杂。官网的tutorial给的一些示例足够基本入门的。
以下为示例：
shell;toolbar: true; auto-links: false;">/html/head/title: selects the <title> element, inside the <head>element of a HTML document （选择html头部的标题元素）
/html/head/title/text(): selects the text inside the aforementioned<title> element.（选择上例中元素的文本）
//td: selects all the <td> elements（选择所有td项目）
//div[@class="mine"]: selects all div elements which contain an attribute class="mine"（选择class="mine"的div标签）
为了方便调式，scrapy提供了交互式的方式对网站进行抓取分析（注意一定要有""，否则。。。）：
shell;toolbar: true; auto-links: false;">scrapy shell "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/"
正常的话会出现以下内容：
[ ... Scrapy log here ... ]
2014-01-23 17:11:42-0400 [default] DEBUG: Crawled (200) <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> (referer: None)
[s] Available Scrapy objects:
[s] crawler <scrapy.crawler.Crawler object at 0x3636b50>
[s] item {}
[s] request <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
[s] response <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
[s] settings <scrapy.settings.Settings object at 0x3fadc50>
[s] spider <Spider 'default' at 0x3cebf50>
[s] Useful shortcuts:
[s] shelp() Shell help (print this help)
[s] fetch(req_or_url) Fetch request (or URL) and update local objects
[s] view(response) View response in a browser
In [1]:这样你就获得了crawler等六个类，现在我们只关注response类，正如字面意思，response代表返回的网页内容，使用view（response）就可以调用默认的浏览器查看抓取的页面（我的是firefox）。response.body存放着html的源代码，你可以在交互程序下查看。不过一般没有换行和对齐，十分惨烈。。。
response中包含了四个基本方法，用于数据过滤：
xpath(): returns a list of selectors, each of them representing the nodes selected by the xpath expression given as argument.
css(): returns a list of selectors, each of them representing the nodes selected by the CSS expression given as argument.
extract(): returns a unicode string with the selected data.
re(): returns a list of unicode strings extracted by applying the regular expression given as argument.
其中xpath()就是选择器，而extract()方法则会返回html标签之间的unicode内容，re()则是调用正则的接口（我恨正则）。
示例如下：
In [1]: response.xpath('//title')
Out[1]: [<Selector xpath='//title' data=u'<title>Open Directory - Computers: Progr'>]
In [2]: response.xpath('//title').extract()
Out[2]: [u'<title>Open Directory - Computers: Programming: Languages: Python: Books</title>']
In [3]: response.xpath('//title/text()')
Out[3]: [<Selector xpath='//title/text()' data=u'Open Directory - Computers: Programming:'>]
In [4]: response.xpath('//title/text()').extract()
Out[4]: [u'Open Directory - Computers: Programming: Languages: Python: Books']
In [5]: response.xpath('//title/text()').re('(\w+):')
Out[5]: [u'Computers', u'Programming', u'Languages', u'Python']response.xpath()将会返回selector的一个列表，你可以对其中的元素再次调用xpath, extract()等函数。
就像这样：
python">>>> response.xpath('//title')[0].xpath('text()').extract()
[u'DMOZ - Computers: Programming: Languages: Python: Books']剩下的事就是把元素存入items之中了，然后通过pipeline存放到你想要的数据格式：
import scrapyfrom tutorial.items import DmozItemclass DmozSpider(scrapy.Spider):
name = "dmoz"
allowed_domains = ["dmoz.org"]
start_urls = [
"http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
"http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
]
def parse(self, response):
for sel in response.xpath('//ul/li'):
item = DmozItem()
item['title'] = sel.xpath('a/text()').extract()
item['link'] = sel.xpath('a/@href').extract()
item['desc'] = sel.xpath('text()').extract()
yield item这里说一下yield这个关键字，yield在python中被称为generator，作用是将循环转化为iterator并返回，详情可以看这里：http://www.ibm.com/developerworks/cn/opensource/os-cn-python-yield/
