小白爬蟲(chóng)scrapy第五篇

I_Am 發(fā)布于2019-07-30 15:23 / 956人閱讀

摘要：接下來(lái)將數(shù)據(jù)保存到數(shù)據(jù)庫(kù)中在項(xiàng)目路徑下創(chuàng)建一個(gè)目錄名字叫做以便區(qū)分創(chuàng)建將其他目錄的文件拷貝一份放入該目錄然后可以正式開(kāi)始吧爬取的數(shù)據(jù)如何存放在數(shù)據(jù)庫(kù)中了客觀代碼來(lái)了打開(kāi)文件在最下面添加數(shù)據(jù)庫(kù)地址數(shù)據(jù)庫(kù)用戶名數(shù)據(jù)庫(kù)密碼數(shù)據(jù)庫(kù)端口數(shù)據(jù)庫(kù)中庫(kù)由于

接下來(lái)將數(shù)據(jù)保存到數(shù)據(jù)庫(kù)中
在項(xiàng)目路徑下創(chuàng)建一個(gè)目錄名字叫做:mysqlpipelines以便區(qū)分
創(chuàng)建將其他目錄的__init__.py文件拷貝一份放入該目錄
然后可以正式開(kāi)始吧爬取的數(shù)據(jù)如何存放在數(shù)據(jù)庫(kù)中了,客觀代碼來(lái)了

打開(kāi)settings.py文件在最下面添加
# configure MySQL
MYSQL_HOSTS = "127.0.0.1"#數(shù)據(jù)庫(kù)地址
MYSQL_USER = "root"#數(shù)據(jù)庫(kù)用戶名
MYSQL_PASSWORD = "root"#數(shù)據(jù)庫(kù)密碼
MYSQL_PORT = "3306"#數(shù)據(jù)庫(kù)端口
MYSQL_DB = "test"#數(shù)據(jù)庫(kù)中test庫(kù)

由于我的數(shù)據(jù)庫(kù)是本地的那就所以直接醬紫
之后呢,在mysqlpipelines目錄中新建一個(gè)sql.py
代碼如下

import mysql.connector
from dingdian import settings
# 引用配置文件
MYSQL_HOSTS = settings.MYSQL_HOSTS
MYSQL_USER = settings.MYSQL_USER
MYSQL_PASSWORD = settings.MYSQL_PASSWORD
MYSQL_PORT = settings.MYSQL_PORT
MYSQL_DB = settings.MYSQL_DB
# 初始化MYSQL游標(biāo)操作
cnx = mysql.connector.connect(user=MYSQL_USER, password=MYSQL_PASSWORD, host=MYSQL_HOSTS, database=MYSQL_DB)
cur = cnx.cursor(buffered=True)

# 定義一個(gè)sql類(lèi)
class Sql:
    # @classmethod 做類(lèi)修飾,相當(dāng)于靜態(tài)類(lèi)
    # 定義函數(shù),將函數(shù)中的變量保存到數(shù)據(jù)庫(kù)中
    @classmethod
    def insert_dd_name(cls, video_name, video_time, video_imgurl, video_url):
        sql = "INSERT INTO video_info(video_name, video_time, video_imgurl, video_url)
            VALUES(%(video_name)s, %(video_time)s, %(video_imgurl)s, %(video_url)s)"
        value = {"video_name" : video_name,
                 "video_time": video_time ,
                 "video_imgurl": video_imgurl ,
                 "video_url": video_url }
        print(sql, value)
        cur.execute(sql, value)
        cnx.commit()
        pass
        
    # 查找是否有重復(fù)的小說(shuō)編號(hào)有則返回1 沒(méi)有則返回0
    @classmethod
    def select_name(cls, video_name):
        sql = "SELECT EXISTS(SELECT 1 FROM video_info WHERE video_name= %(video_name)s)"
        value = {
            "video_name": video_name
        }
        cur.execute(sql, value)
        return cur.fetchall()[0]

上面代碼中import mysql.connector可能會(huì)報(bào)錯(cuò),如果報(bào)錯(cuò)的話就去百度python如何下載數(shù)據(jù)庫(kù)連接包吧
然后繼續(xù)在mysqlpipelines創(chuàng)建一個(gè)pipelines.py
代碼如下:

from .sql import Sql
from AiQuer.items import AiquerItem

class DingdianPipeline(object):

    def process_item(self, item, spider):
        if isinstance(item, AiquerItem):
            video_name= item["video_name"]
            ret = Sql.select_name(video_name)
            if ret[0] == 1:
                print(u"已存在")
            else:
                video_name= item["video_name"]
                video_time= item["video_time"]
                video_imgurl= item["video_imgurl"]
                video_url= item["video_url"]
                
                Sql.insert_dd_name(video_name, video_time, video_imgurl, video_url)
                print(u"開(kāi)始存視頻信息")
        return item

最后在settings.py里面去注冊(cè)一下DingdianPipeline
找到下面這一段

# Configure item pipelines
# See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
    "AiQuer.mysqlpipelines.pipelines.AiquerPipeline": 300,
}

注:我在這里面沒(méi)有建立數(shù)據(jù)庫(kù)表請(qǐng)同學(xué)們自己去創(chuàng)建表吧
啟用后你會(huì)發(fā)現(xiàn)數(shù)據(jù)庫(kù)有你選擇用戶的所有視頻,哈哈哈~
python的就到這兒了.如果要詳細(xì)了解進(jìn)階的話就看python的scrapy文檔吧!
附上鏈接Scrapy入門(mén)教程.拜拜~