爬蟲基礎(chǔ)練習(xí)－python批量下載圖片之達(dá)蓋爾的旗幟

pepperwang 發(fā)布于2019-07-25 11:30 / 1184人閱讀

摘要：三個(gè)重點(diǎn)，我隱藏了的地址，本爬蟲只適用于的抓取。每個(gè)帖子以帖子名為文件名新建一個(gè)文件并把帖子內(nèi)圖片保存下來(lái)。

三個(gè)重點(diǎn)，我隱藏了1024的地址，本爬蟲只適用于1024的抓取。每個(gè)帖子以帖子名為文件名新建一個(gè)文件并把帖子內(nèi)圖片保存下來(lái)。

url_start設(shè)置起始頁(yè) url_end設(shè)置結(jié)束頁(yè)
有問(wèn)題的留言，我看到就會(huì)回復(fù)

1編碼 2文章頁(yè)鏈接匹配 3文件夾操作


import requests
import re
import time
from bs4 import BeautifulSoup
import os

url_start = "url1"
url_end = "url2"


# 獲取圖片鏈接并保存到文件夾的函數(shù)
def getIMG(article_url):
    # time.sleep(1)
    urls = []
    try:
        html = requests.get(article_url)
        html.encoding = "gbk"
        soup = BeautifulSoup(html.text, "html.parser")
        part_picURL = re.findall("src="http://img(.+?.jpg)"",html.text,re.S)
        for each in part_picURL:
            picURL = "http://img" + each
            urls.append(picURL)
        i=0
        for each in urls:
            try:
                pic = requests.get(each, timeout = 10)
                folder_name = soup.select("h4")[0].text
                if os.path.isdir(folder_name):
                    pass
                else:
                    os.mkdir(folder_name)
                    print("文件夾"+ "$ " + folder_name + "$" + "創(chuàng)建完成")
                file_name = folder_name+"/" + folder_name + str(i) + ".jpg"
                fp = open(file_name,"wb")
                fp.write(pic.content)
                fp.close()
                i += 1
            except:
                pass
        print("圖片下載完成")
    except:
        pass
    return urls


url_list = []
#獲取當(dāng)前頁(yè)面文章列表鏈接并翻頁(yè)
def getlist(url_Start):
    headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36"}
    req = requests.get(url_Start)
    req.encoding = "gbk"
    url_index = re.findall(""打開新窗口" href="htm_(.+?.html)" target="_blank">",req.text,re.S)
    for p in url_index:
        full_url = "http://cl.gtta.pw/htm_" + p
        url_list.append(full_url)
    #判斷是否要翻頁(yè)
    urls_next = re.findall("false;}">下一頁(yè)",req.text,re.S)[0]
    url_next = "http://cl.gtta.pw/" + urls_next
    if url_next != url_end:
        getlist(url_next)
    else:
        print("已到達(dá)末頁(yè)")
    return url_list


lists = getlist(url_start)
print(len(lists))
for list in lists:
    img = getIMG(list)
    print(img)