Scrapy爬取hupu論壇標(biāo)題統(tǒng)計(jì)數(shù)量并生成wordcloud

renweihub 發(fā)布于2019-07-30 14:26 / 1567人閱讀

摘要：爬取數(shù)據(jù)提取標(biāo)題最終爬取了頁(yè)多個(gè)標(biāo)題分詞并統(tǒng)計(jì)詞的數(shù)量這里我使用了這個(gè)庫(kù)來(lái)分詞分出了個(gè)詞然后統(tǒng)計(jì)數(shù)量這里我統(tǒng)計(jì)了兩個(gè)詞以下和兩個(gè)詞以上的詞的量分配如圖生成詞云以及其他數(shù)據(jù)圖表這里我是用的庫(kù)生

爬取數(shù)據(jù)

huputitle_spiders.py

#coding:utf-8    
import scrapy   
from huputitle.items import HuputitleItem

from scrapy.crawler import CrawlerProcess 

class hupuSpider(scrapy.Spider):
    name = "huputitle"    
    allowed_domains = ["bbs.hupu.com"]    
    start_urls = ["https://bbs.hupu.com/bxj"]    
         
    def parse(self, response):    
        item = HuputitleItem()    
        item["titles"] = response.xpath("http://a[@id=""]/text()").extract()#提取標(biāo)題
        # print "titles",item["titles"]  
        yield item 
        new_url = "https://bbs.hupu.com" + response.xpath("http://a[@id="j_next"]/@href").extract_first()
        if new_url:    
            yield scrapy.Request(new_url,callback=self.parse)

items.py

# -*- coding: utf-8 -*-
import scrapy


class HuputitleItem(scrapy.Item):
    # define the fields for your item here like:
    titles = scrapy.Field()

pipelines.py

# -*- coding: utf-8 -*-
import os    
import urllib   
  
from huputitle import settings  

import sys  
reload(sys)  
sys.setdefaultencoding( "utf-8" )  

class HuputitlePipeline(object):
    def process_item(self, item, spider):
        for title in item["titles"]:
            # print "title",title
            fo = open("foo.txt", "a")
            fo.write("".join(title)+"
")
        fo.close()
        return item

settings.py

BOT_NAME = "huputitle"

SPIDER_MODULES = ["huputitle.spiders"]
NEWSPIDER_MODULE = "huputitle.spiders"

ITEM_PIPELINES = {  
    "huputitle.pipelines.HuputitlePipeline": 1,  
}

USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.54 Safari/536.5"

最終爬取了100頁(yè)2W多個(gè)標(biāo)題

分詞并統(tǒng)計(jì)詞的數(shù)量

這里我使用了 jieba 這個(gè)庫(kù)來(lái)分詞
hupudivide.py

#encoding=utf-8  
import jieba  
import sys
reload(sys)
sys.setdefaultencoding("utf-8")

fo = open("hupu.txt", "r")
fi = open("hupudi.txt", "w")
lines = fo.readlines()
for line in lines:
    seg_list = jieba.cut_for_search(line)
    fi.write(" 
".join(seg_list))

分出了17w個(gè)詞

然后統(tǒng)計(jì)數(shù)量
huPuCounter.py

#encoding=utf-8  
import jieba  
import jieba.analyse
import time
from collections import Counter
import sys
reload(sys)
sys.setdefaultencoding("utf-8")

fo = open("hupudi.txt", "r")
fi = open("hupunum.txt", "w")
fl = open("hupunumword.txt", "w")
f = open("hupuword.txt", "w")

lines = fo.readlines()

d = {}

for line in lines:
    if line not in d:
        d[line] = 1
    else:
        d[line] = d[line] + 1

d = sorted(d.items(),key=lambda item:item[1],reverse=True)

for k in d:
    fi.write("%s%d
" % (k[0][:-1].encode("utf-8"),k[1]))
    if len(k[0][:-1].encode("utf-8")) >= 6:
        fl.write("%s%d
" % (k[0][:-1].encode("utf-8"),k[1]))
        f.write("%s" % (k[0][:-1].encode("utf-8")))

這里我統(tǒng)計(jì)了兩個(gè)詞以下和兩個(gè)詞以上的詞的量分配如圖

生成詞云以及其他數(shù)據(jù)圖表

makeHupuCloud.py

#encoding=utf-8 
import matplotlib.pyplot as plt
from wordcloud import WordCloud
import jieba

text_from_file_with_apath = open("foo.txt").read()

wordlist_after_jieba = jieba.cut(text_from_file_with_apath, cut_all = False)
wl_space_split = " ".join(wordlist_after_jieba)

backgroud_Image = plt.imread("huputag.jpg")
my_wordcloud = WordCloud(background_color = "white",    
                mask = backgroud_Image,  ).generate(wl_space_split)

plt.imshow(my_wordcloud)
plt.axis("off")
plt.show()

這里我是用python的wordcloud庫(kù)生成的詞云，圖片是hupu的logo

使用jieba的分詞分出詞性生成的圖表