作者：cxuzhen

这几天爬了京东和苏宁易购的全站，京东450万左右的sku,苏宁易购280多万sku。虽然会有些漏网之鱼，但是肯定不多。两个网站都是调用的移动端APP的接口，fiddler抓包即可，只是需要注意构造的url是否合理，不然很容易漏掉很多sku。

记得京东某个副总裁说京东有500多万sku，和我这个爬虫结果基本一致，毕竟要考虑程序的错误率和反爬。京东对IP的封杀比较厉害，爬取全站估计需要十万左右的IP才够用，苏宁相对来说比较容易，一天就能爬完所有商品。

先记录下苏宁易购的爬虫过程吧。

目的：通过APP接口爬取全站商品

fiddler可以方便地抓取app接口，手机root后和PC在同一个WIFI下，手机wifi设置代理，服务器主机名填写PC的IP（局域网IP，ipconfig /all可以查询），端口填写8888（fiddler的默认端口）。这里需要注意的是，如果你PC打开了jupyter notebook，一定要确保jupyter notebook不能使用8888端口，jupyter notebook –port=7777 这样可以使用不同端口。

fiddler的具体使用google下即可，我的博客也曾经提到过。这里的接口：

https://search.suning.com/emall/mobile/clientSearch.jsonp?set=5&ps=120&channelId=MOBILE&keyword=空调免息&st=0&ci=&cf=&sc=&cityId=459&cp=26&iv=-1&ct=-1&sp=&spf=&prune=1&cat1_大家电_cat2_空调_cat3_空调免息
需要注意的是，这里用的关键词搜索，也可以用ci这个分类。这个分类代码并不是每一个三级分类都有，有的只能用关键词

苏宁爬虫的关键就是抓包构造url，这里贴出代码来：

import json
import pprint
import re
import redis
from urllib import parse

cat3_urls = []
pp = pprint.PrettyPrinter(indent=4)
with open(r'E:\splash\suning\suning_cat.json', 'r') as f:
    json_dict = json.load(f)    
#     pp.pprint(json_dict['rs'])
    print('num of cat1 categories:', len(json_dict['rs']))
    
    for i in range(1, len(json_dict['rs'])-3):
        category_1_name = json_dict['rs'][i]['dirName']
        print('category_1: ', category_1_name)
        print('category_1_id: ', json_dict['rs'][i]['id'])
        print(f'num of cat2 categories under {category_1_name}:', len(json_dict['rs'][i]))
        for j in range(len(json_dict['rs'][i]['children'])):
            try:
#                 print('name of category_2:', json_dict['rs'][i]['children'][j])
                category_2_name = json_dict['rs'][i]['children'][j]['dirName']
                print('category_2_id:', json_dict['rs'][i]['children'][j]['id'])
                num_cat3 = len(json_dict['rs'][i]['children'][j]['children'])
                print(f'num of categories in category_2_{category_2_name}:', num_cat3)  
                print(f'num_{j} cat2_{category_2_name} under category_1_{category_1_name}: ')
                pp.pprint(json_dict['rs'][i]['children'][j]['children'])
                # pp.pprint(json_dict['rs'][i]['children'][j]['children'])
                print('\n\ncategory_3:')
                for k in range(0, len(json_dict['rs'][i]['children'][j]['children'])):
                    print(f'num{k} cat3 under cat1_{category_1_name}_cat2_{category_2_name}')
                    cat_3 = json_dict['rs'][i]['children'][j]['children'][k]
                    category_3_name = cat_3['dirName']
                    category_3_id = cat_3['id']
                    try:
                        category_3_pcCi = cat_3['pcCi']
                        if '图书音像' not in category_1_name:
                            
                            cat3_url = 'https://search.suning.com/emall/mobile/clientSearch.jsonp?set=5&ps=120&channelId=MOBILE&keyword=&st=0&ci=' + category_3_pcCi + '&cf=&sc=&cityId=459&cp=0&iv=-1&ct=-1&sp=&spf=&prune=1'+ f'&cat1_{parse.quote(category_1_name)}_cat2_{parse.quote(category_2_name)}_cat3_{parse.quote(category_3_name)}'
                        else:
                            # '图书音像'大类比较特殊，不能使用pcCi
                            cat3_url = 'https://search.suning.com/emall/mobile/clientSearch.jsonp?set=5&ps=120&channelId=MOBILE&keyword=' + parse.quote(category_3_name) + '&st=0&ci=&cf=&sc=&cityId=459&cp=0&iv=-1&ct=-1&sp=&spf=&prune=1' + f'&cat1_{parse.quote(category_1_name)}_cat2_{parse.quote(category_2_name)}_cat3_{parse.quote(category_3_name)}' 
                    except KeyError:
                        category_3__gotoApp = cat_3['gotoApp']
                        if 'search' in category_3__gotoApp:
                            print(f'cat3_{category_3_name} search by keyword')
                            cat3_url = 'https://search.suning.com/emall/mobile/clientSearch.jsonp?set=5&ps=120&channelId=MOBILE&keyword=' + parse.quote(category_3_name) + '&st=0&ci=&cf=&sc=&cityId=459&cp=0&iv=-1&ct=-1&sp=&spf=&prune=1' + f'&cat1_{parse.quote(category_1_name)}_cat2_{parse.quote(category_2_name)}_cat3_{parse.quote(category_3_name)}' 
                        elif '100020_' in category_3__gotoApp:
                            pattern = '100020_\d+'
                            adId = re.search(pattern, category_3__gotoApp)[0]
                            print(adId)
                            if adId:
                                ci = adId.split('_')[1]
                                cat3_url = 'https://ebuy.suning.com/mobile/clientSearch?ch=' + '100020' + '&iv=-1&keyword=&cityId=459&ci=' + ci + '&cp=0&ps=120&st=0&cf=&ct=-1&sp=&v=1.6'  + '&cf=&sc=&cityId=459&cp=0&iv=-1&ct=-1&sp=&spf=&prune=1' + f'&cat1_{parse.quote(category_1_name)}_cat2_{parse.quote(category_2_name)}_cat3_{parse.quote(category_3_name)}'
                            else:
                                cat3_url = 'https://search.suning.com/emall/mobile/clientSearch.jsonp?set=5&ps=120&channelId=MOBILE&keyword=' + parse.quote(category_3_name) + '&cf=&sc=0&cityId=459&ci=' + '&cf=&sc=0&cityId=459&cp=0&iv=-1&ct=-1&sp=&spf=&prune=1' + f'&cat1_{parse.quote(category_1_name)}_cat2_{parse.quote(category_2_name)}_cat3_{parse.quote(category_3_name)}'                       
                                
                        else:
                            print('#'*100, '----Warning!')
                            cat3_url = 'https://search.suning.com/emall/mobile/clientSearch.jsonp?set=5&ps=120&channelId=MOBILE&keyword=' + parse.quote(category_3_name) + '&st=0&ci=&cf=&sc=&cityId=459&cp=0&iv=-1&ct=-1&sp=&spf=&prune=1' + f'&cat1_{parse.quote(category_1_name)}_cat2_{parse.quote(category_2_name)}_cat3_{parse.quote(category_3_name)}' 
                    finally:
                        if 'cat2_品牌出版社_cat3_' not in parse.unquote(cat3_url):
                            print(cat3_url)
                            cat3_urls.append(cat3_url)
#                     else:
#                         print('#'*100, cat_3)
                    pp.pprint(cat_3)
                    print('\n\n')
                
            except Exception as e:
                print(e, i, j, category_2_name )

r = redis.Redis(host=REDIS_HOST, port=7379, db=0, password=REDIS_PASSWD, encoding='utf-8', decode_responses=True)
set_redis = {""}
set_redis.update(cat3_urls)
type(set_redis)
r.lpush('suning:start_urls', *set_redis)                
print(f'num of cat3_urls: {len(cat3_urls)}')            
for i in cat3_urls:
    if 'cat2_品牌出版社_cat3_' not in parse.unquote(i):
        print(parse.unquote(i))

这个分类json文件可以通过抓包抓取，代码如下：

import requests
from requests.exceptions import ConnectionError, ReadTimeout
import pprint
import json

url = 'https://search.suning.com/emall/mobile/clientSearch.jsonp?set=5&ps=120&channelId=MOBILE&keyword=11513369305'
cat_url = 'https://ds.suning.com/ds/terminal/categoryInfo/v1/99999998-.jsonp'
header = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36'
}
pp = pprint.PrettyPrinter(indent=4)
# try:
#     response = requests.get(url, timeout=10, headers=header)
#     if response.status_code == 200:
#         print('yes connected')
# #         print(response.text)
#         json_dict = response.json()
#         pp.pprint(json_dict['goods'])
        
# except (ConnectionError, ReadTimeout):
#     print('connect failed')
    
try:
    response = requests.get(cat_url, timeout=10, headers=header)
    if response.status_code == 200:
        print('yes connected')
#         print(response.text)
        json_dict = response.json()
        pp.pprint(json_dict['rs'])
        json.dumps('suning_cat.json')
        with open(r'E:\splash\suning\suning_cat.json', 'w') as f:
            json.dump(json_dict, f, ensure_ascii=False, indent=4, sort_keys=True)        
except (ConnectionError, ReadTimeout):
    print('connect failed')    

url构造好之后lpush到redis即可分布式爬虫了。

工具：scrapy-redis, mongodb, adsl vps自建代理池, gerapy分布式爬虫管理和监控

二话不说，直接上关键代码：

# # -*- coding: utf-8 -*-
import scrapy
import json
from scrapy_redis.spiders import RedisSpider
try:
    from urllib.parse import urlparse
except ImportError:
    from urlparse import urlparse
from suning.log import logger
from suning.items import SuningItem
import re
from urllib import parse

class SuningspdSpider(RedisSpider):
    name = 'suningspd'
    allowed_domains = ['suning.com']
    redis_key = 'suning:start_urls'
    
    def __init__(self, *args, **kwargs):
        super(SuningspdSpider, self).__init__(*args, **kwargs)
        
    def parse(self, response):
        url = response.url
        items = []
        if len(response.text) < 50 or response.status==402:
            logger.info(f'返回response为空，可能是最后一页或者超时了: {response.url}')
        else:    
            goods = json.loads(response.text)['goods']
            logger.info(f'goods: {parse.unquote(url)}')
            goodsCount = json.loads(response.text)["goodsCount"]
            logger.info(f'goodsCount: {goodsCount}')
            # 抓取的这个接口最多显示50页，每页120个商品，和PC端的限制是一样的
            max_page = min(int(goodsCount/120) + 1, 50)
            for good in goods:
                item = SuningItem()
                # logger.info(f'good: {good}\n\n')
                good_id = good['partnumberVendorId']
                good_url = 'https://product.suning.com/' + good_id.split('_')[1] +'/' + good_id.split('_')[0]  + '.html'
                # logger.info(good_id)
                item['_id'] = good_id
                item['good_sku_id'] = good_id
                item['good_name'] = good["catentdesc"]
                item['url'] = good_url
                item['cat1'] = url.split('&')[-1].split('_')[1]
                item['cat2'] = url.split('&')[-1].split('_')[3]
                item['cat3'] = url.split('&')[-1].split('_')[5]
                item['good_info'] = json.dumps(good, ensure_ascii=False)
                # logger.info(f'check item: {item}') 
                items.append(item)
                logger.warning(f'check num of items: {len(set(items))}')
                yield item
            page = re.search('cp=\d+', url)[0].split('=')[1]        
            if int(page) < max_page-1:            
                next_page = int(page) + 1
                logger.info(f'总共{max_page}页，还有下一页。继续采集第{next_page}页')
                next_url = re.sub(f'cp={page}', f'cp={next_page}', url)
                logger.info(f'next_url: {parse.unquote(next_url)}')
                # yield response.follow(next_url, self.parse)
                yield scrapy.Request(url=next_url, callback=self.parse, dont_filter=False)   
      

即使移动端接口也有每页120个商品，单分类下50页的限制，所以对于一些热门的分类，最多也只能爬取6000个SKU，但是这里我用了细分类，一共2397个，漏网之鱼不多。一个服务器加上六七个adsl vps拨号代理服务器，一天就能爬完全站商品，最后抓取了2843855个SKU，漏网之鱼肯定不少，毕竟我没有去PC端按照细分类去遍历所有的SKU，如果需要全部SKU，则需要去PC端去按照细分类+品牌分类的方式去遍历SKU，爬取京东450万SKU我就是用的这种方式。

获取的商品sku信息如下：

 'good_name': '宏碁Acer 暗影骑士·擎 15.6英寸电竞游戏本RGB背光学生笔记本电脑（i7-10750H 32G 1TB+1TBSSD '
              'GTX1660Ti 6G 144Hz）定制',
 'good_sku_id': '11961839381_0070174045',
 'url': 'https://product.suning.com/0070174045/11961839381.html'}}, {'index': 33, 'code': 11000, 'keyPattern': {'_id': 1}, 'keyValue': {'_id': '12200322581_0070640451'}, 'errmsg': 'E11000 duplicate key error collection: suning.product index: _id_ dup key: { _id: "12200322581_0070640451" }', 'op': {'_id': '12200322581_0070640451',
 'cat1': '%E4%BA%8C%E6%89%8B%E4%BC%98%E5%93%81',
 'cat2': '%E7%94%B5%E8%84%91%E5%8A%9E%E5%85%AC',
 'cat3': '%E7%AC%94%E8%AE%B0%E6%9C%AC',
 'good_info': '{"inventory": "0", "auxdescription": "北京 上海 南京 郑州 '
              '武汉都有库房，就近安排，即送大礼包，详情请咨询客服", "catentdesc": "联想ThinkPad T14 15CD '
              '英特尔酷睿i5 14英寸轻薄笔记本电脑（i5-10210U 32G 1TSSD固态 Win10）高分屏 红外摄像头 定制", '
              '"catentryId": "12200322581", "countOfarticle": "0", '
              '"partnumber": "12200322581", "totalCount": "0", "price": '
              '"7699.0", "brandId": "000052450", "saleStatus": 0, '
              '"contractInfos": "", "snFlag": false, "author": "", '
              '"suningSale": false, "praiseRate": "", "salesCode": '
              '"0070640451", "salesName": "麦田电脑旗舰店", "salesCode10": '
              '"0070640451", "beancurdFlag": "0", "goodsType": "1", '
              '"priceType": "2", "filters": [{}, {}, {}], "filterAttr": true, '
              '"isFav": true, "hwgLable": false, "spsLable": false, '
              '"baoguangHwg": "0", "dynamicImg": '
              '"//imgservice4.suning.cn/uimg1/b2c/image/Ews79nBjIFzU-qZ2hLdLoA.jpg", '
              '"docType": 1, "specificUrl": "http://18181818.suning.com", '
              '"salesUrl": "", "threeGroupId": "258004@507196", '
              '"threeGroupName": "创意设计笔记本,笔记本", "priority": "2", "orderType": '
              '"0", "snpmDity": "2", "shortBrandId": "2450", "isNew": "1", '
              '"partnumberVendorId": "12200322581_0070640451", "msList": [], '
              '"salesCount": "0", "jlfDirGroupIdList": [], "xdGroupIDCopy": '
              '[], "mdGoodType": "", "catalog": '
              '"NPC,ALL,NO,XDALL,10051,SNZB,NGD,ftzm,fnaep,wx001,vedio,wx002,XDC,wx003,NH", '
              '"isSnPharmacy": false, "extenalFileds": {"lpg_activeId": '
              '"null", "brandNameZh": "ThinkPad", "detailsUrl": '
              '"https://m4.pptvyun.com/pvod/e11a0/cjKfRo_ggIlgD6XsDEgSWDXrrCs/eyJkbCI6MTU5MDQ3OTg4NywiZXMiOjYwNDgwMCwiaWQiOiIwYTJkb3FpZHBLZWRucS1MNEsyZG9hZmhvNmljb0thY29hayIsInYiOiIxLjAifQ/0a2doqidpKednq-L4K2doafho6icoKacoak.mp4", '
              '"ZYLY": "0", "goodType": "Z001", "attrShow": [{"attrAppDesc": '
              '"", "attrAppTrueValue": "集成显卡", "attrAppValue": "集成显卡", '
              '"attrId": "solr_1855_attrId", "attrName": "显卡类型", "attrValue": '
              '"集成显卡", "attrValueId": "24087", "sort": "92.0"}, '
              '{"attrAppDesc": "", "attrAppTrueValue": "Intel i5", '
              '"attrAppValue": "Intel i5", "attrId": "solr_6160_attrId", '
              '"attrName": "CPU类型", "attrValue": "Intel i5", "attrValueId": '
              '"attrAppValue": "Intel i5", "attrId": "solr_6160_attrId", '
              '"attrName": "CPU类型", "attrValue": "Intel i5", "attrValueId": '
              '"45969", "sort": "91.0"}, {"attrAppDesc": "", '
              '"attrAppTrueValue": "1TB", "attrAppValue": "1TB", "attrId": '
              '"solr_2088_attrId", "attrName": "硬盘容量", "attrValue": "1TB", '
              '"attrValueId": "2280366", "sort": "0.0"}], "mdmGroupId": '
              '"R1502001", "appAttrTitle": ["商务办公", "轻薄便捷", "红外摄像头"], '
              '"groupIDCopy": ["157122:258003:258004", '
              '"157122:258003:507196"], "specificUrl": '
              '"http://18181818.suning.com", "groupIDCombination": '
              '["157122@A@电脑/办公/外设:258003@A@电脑整机:258004@A@笔记本", '
              '"157122@A@电脑/办公/外设:258003@A@电脑整机:507196@A@创意设计笔记本"], '
              '"activationFlag": "1", "commentShow": "0", "auxdescription": '
              '"北京 上海 南京 郑州 武汉都有库房，就近安排，即送大礼包，详情请咨询客服", "paramValue": "T14"}}',

如果需要评论数据，也很简单，需要两步：

• 抓取移动端甚至PC端评论接口，非常简单
• 去PC端遍历每个SKU，获取clusterID，这个参数是必须的，而且是js生成的，如果你有时间去研究它的JS代码应该也能找到这个参数的生成算法，但是苏宁反爬不厉害，不如去遍历每个商品，返回的response.text里有script，script里就含有这个关键字，正则提取即可。我的服务器硬盘空间快满了，得等段时间才能采集了。评论数量特别大，非常占空间。