百度不收录哪些网站,管理咨询的主体包括哪些,电影网站做静态是不是好一些,最好的营销型网站案例urllib提供了一系列用于操作URL的功能Geturllib的request模块可以非常方便的抓取URL的内容, 也就是发送一个GET请求到制定的页面, 然后返回HTTP的响应:例如, 对豆瓣的一个URLhttps://api.douban.com/v2/book/2129650进行抓取, 并返回响应:from urllib import requestwith reque…urllib提供了一系列用于操作URL的功能Geturllib的request模块可以非常方便的抓取URL的内容, 也就是发送一个GET请求到制定的页面, 然后返回HTTP的响应:例如, 对豆瓣的一个URLhttps://api.douban.com/v2/book/2129650进行抓取, 并返回响应:from urllib import requestwith request.urlopen(https://api.douban.com/v2/book/2129650) as f:data f.read()print(Status:, f.status, f.reason)for k, v in f.getheaders():print(%s: %s % (k, v))print(Data:, data.decode(utf-8))可以看到HTTP响应的头和JSON数据Status: 200 OKServer: nginxDate: Tue, 26 May 2015 10:02:27 GMTContent-Type: application/json; charsetutf-8Content-Length: 2049Connection: closeExpires: Sun, 1 Jan 2006 01:00:00 GMTPragma: no-cacheCache-Control: must-revalidate, no-cache, privateX-DAE-Node: pidl1Data: {rating:{max:10,numRaters:16,average:7.4,min:0},subtitle:,author:[廖雪峰编著],pubdate:2007-6,...}如果我们要想模拟浏览器发送GET请求就需要使用Request对象通过往Request对象添加HTTP头我们就可以把请求伪装成浏览器。例如模拟iPhone 6去请求豆瓣首页from urllib import requestreq request.Request(http://www.douban.com/)req.add_header(User-Agent, Mozilla/6.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/8.0 Mobile/10A5376e Safari/8536.25)with request.urlopen(req) as f:print(Status:, f.status, f.reason)for k, v in f.getheaders():print(%s: %s % (k, v))print(Data:, f.read().decode(utf-8))这样豆瓣会返回适合iPhone的移动版网页......Post如果要以POST发送一个请求只需要把参数data以bytes形式传入。我们模拟一个微博登录先读取登录的邮箱和口令然后按照weibo.cn的登录页的格式以usernamexxxpasswordxxx的编码传入from urllib import request, parseprint(Login to weibo.cn...)email input(Email: )passwd input(Password: )login_data parse.urlencode([(username, email),(password, passwd),(entry, mweibo),(client_id, ),(savestate, 1),(ec, ),(pagerefer, https://passport.weibo.cn/signin/welcome?entrymweiborhttp%3A%2F%2Fm.weibo.cn%2F)])req request.Request(https://passport.weibo.cn/sso/login)req.add_header(Origin, https://passport.weibo.cn)req.add_header(User-Agent, Mozilla/6.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/8.0 Mobile/10A5376e Safari/8536.25)req.add_header(Referer, https://passport.weibo.cn/signin/login?entrymweiboreswelwm3349rhttp%3A%2F%2Fm.weibo.cn%2F)with request.urlopen(req, datalogin_data.encode(utf-8)) as f:print(Status:, f.status, f.reason)for k, v in f.getheaders():print(%s: %s % (k, v))print(Data:, f.read().decode(utf-8))如果登录成功我们获得的响应如下Status: 200 OKServer: nginx/1.2.0...Set-Cookie: SSOLoginState1432620126; path/; domainweibo.cn...Data: {retcode:20000000,msg:,data:{...,uid:1658384301}}如果登录失败我们获得的响应如下...Data: {retcode:50011015,msg:\u7528\u6237\u540d\u6216\u5bc6\u7801\u9519\u8bef,data:{username:examplepython.org,errline:536}}利用urllib读取JSON然后将JSON解析为Python对象import jsondef fetch_data(url):return json.load(request.urlopen(url))
相关文章：
