轻型微博爬虫¶
weibo_crawler参考【nghuyong/WeiboSpider】 对代码用法进行了简化，可以做轻度的微博数据采集。
- 用户信息抓取
- 用户微博抓取(全量/指定时间段)
- 用户社交关系抓取(粉丝/关注)
- 微博评论抓取
- 基于关键词和时间段(粒度到小时)的微博抓取
- 微博转发抓取
安装¶
In [2]:
pip install weibo-crawler
Requirement already satisfied: weibo-crawler in /opt/anaconda3/lib/python3.7/site-packages (1.0) Requirement already satisfied: pyquery in /opt/anaconda3/lib/python3.7/site-packages (from weibo-crawler) (1.4.3) Requirement already satisfied: requests in /opt/anaconda3/lib/python3.7/site-packages (from weibo-crawler) (2.24.0) Requirement already satisfied: cssselect>0.7.9 in /opt/anaconda3/lib/python3.7/site-packages (from pyquery->weibo-crawler) (1.1.0) Requirement already satisfied: lxml>=2.1 in /opt/anaconda3/lib/python3.7/site-packages (from pyquery->weibo-crawler) (4.6.1) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/anaconda3/lib/python3.7/site-packages (from requests->weibo-crawler) (1.25.11) Requirement already satisfied: idna<3,>=2.5 in /opt/anaconda3/lib/python3.7/site-packages (from requests->weibo-crawler) (2.8) Requirement already satisfied: certifi>=2017.4.17 in /opt/anaconda3/lib/python3.7/site-packages (from requests->weibo-crawler) (2019.11.28) Requirement already satisfied: chardet<4,>=3.0.2 in /opt/anaconda3/lib/python3.7/site-packages (from requests->weibo-crawler) (3.0.4) Note: you may need to restart the kernel to use updated packages.
获取cookie¶
- 使用chrome浏览器打开手机微博 https://weibo.cn 登录
- 右键inspect（即打开开发者模式）
- 查看network内容
- 获取html文件header中的cookie信息
- 其中可能需要SSOLoginState字段
In [3]:
from weibo_crawler import Profile
In [6]:
# 如果程序失败，需要传入你的微博cookies
cookies='_T_WM=9b80727fa0cc3b6b6c374b9262ff084d; SUB=_2A25MjZmZDeRhGeNI4lYX-S7FwjWIHXVscSfRrDV6PUJbktAKLXSmkW1NSAJ40ykLq1lxtFqpHJ4BRMiY1XKHNT6g; SUBP=0033WrSXqPxfM725Ws9jqgMF55529P9D9WhkPe1HSir85xF8hwHpTZa75NHD95QfSo.XSo.71K.4Ws4DqcjT9s8Xqgpyqoz7eK-t; SSOLoginState=1636428233'
# csv文件路径
prof=Profile(csvfile='./data/weibo-chenkun-intro.csv', delay=1, cookies=cookies)
prof.get_profile(userid='1087770692') # 陈坤微博的id
{'userid': '1087770692', 'nickname': '陈坤', 'gender': '男', 'province': '重庆', 'introduction': '莫失己道，莫扰他心。', 'birthday': '0001-00-00', 'vip_level': '7级送Ta会员', 'authentication': '演员，代表作《龙门飞甲》《画皮》等，行走的力量发起者', 'labels': '演员'}
采集完毕，请查看 ./data/weibo-chenkun-intro.csv 内的数据