宁波网站优化技术,淘宝店铺,温州网站设计制作课程,最近一周国内热点新闻Requests是Python的非常常用的HTTP的库#xff0c;主要用于网络爬虫和接口自动化测试。下面使用Requests最新版本#xff0c;通过pip install requests安装。pip install requestsCollecting requests Downloading requests-2.25.0-py2.py3-none-any.whl (61 kB) |███… Requests是Python的非常常用的HTTP的库主要用于网络爬虫和接口自动化测试。下面使用Requests最新版本通过pip install requests安装。pip install requestsCollecting requests Downloading requests-2.25.0-py2.py3-none-any.whl (61 kB) |████████████████████████████████| 61 kB 99 kB/s Requirement already satisfied: urllib31.27,1.21.1 in ./venv/lib/python3.9/site-packages (from requests) (1.26.2)Collecting certifi2017.4.17 Downloading certifi-2020.12.5-py2.py3-none-any.whl (147 kB) |████████████████████████████████| 147 kB 6.7 kB/s Collecting chardet4,3.0.2 Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)Collecting idna3,2.5 Downloading idna-2.10-py2.py3-none-any.whl (58 kB) |████████████████████████████████| 58 kB 5.2 kB/s Installing collected packages: idna, chardet, certifi, requestsSuccessfully installed certifi-2020.12.5 chardet-3.0.4 idna-2.10 requests-2.25.0查看本地安装版本 import requests requests.__version__2.25.0我们来编写一个请求百度首页分为带参数请求和不带参数请求的https://www.baidu.com/https://www.baidu.com/s?wdsuv%E6%B1%BD%E8%BD%A6(suv汽车)Requests实现Get请求2个例子 url https://www.baidu.com/s?wdSpringboot payload {} headers { Connection: keep-alive, Accept: */*, is_xhr: 1, X-Requested-With: XMLHttpRequest, User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36, is_pbs: %E5%BE%AE%E4%BF%A1%E5%85%AC%E4%BC%97%E5%B9%B3%E5%8F%B0, Sec-Fetch-Site: same-origin, Sec-Fetch-Mode: cors, Sec-Fetch-Dest: empty, Referer: https://www.baidu.com/s?ieutf-8f8rsv_bp1rsv_idx1tnbaiduwd%E5%BE%AE%E4%BF%A1%E5%85%AC%E4%BC%97%E5%B9%B3%E5%8F%B0fenlei256oq%25E5%25BE%25AE%25E4%25BF%25A1%25E5%2585%25AC%25E4%25BC%2597%25E5%25B9%25B3%25E5%258F%25B0rsv_pqdbe6f9780003d2c2rsv_td1ddM3MDeEzN0o9%2BO4RzSnpDa8%2Bpu3avFyNpR4YZC3hodmvp3wBjm9N5k0srqlangcnrsv_enter0rsv_dltbrsv_btypetinputT5425rsv_sug35rsv_sug14rsv_sug7100rsv_sug412430rsv_sug1, Accept-Language: en-US,en;q0.9, Cookie: BIDUPSID54C0826149B7299E360B557AB5A497A6; PSTM1602209293; BAIDUID54C0826149B7299ED5ED2B69EA7EAE01:FG1; BD_UPN12314753; MCITY-%3A; BDORZB490B5EBF6F3CD402E515D22BCDA1598; sug3; sugstore1; ORIGIN0; bdime0; BDSFRCVIDDUCOJeC62ZuoAecrzNDIhH2qlmwUBOTTH6aotFZDEgCmWtkRMW_mEG0P8f8g0KAbGdi6ogKK3mOTHR8F_2uxOjjg8UtVJeC6EG0Ptf8g0f5; H_BDCLCKID_SFtbCq_IKafCP3HJ84q465bPPJqxbXqM3P02OZ0l8KtfchDpjN-n50Xx0PM-AHb47LbaTG2MbmWIQHDPbDWxOpbUtD3pOWQfrUXKj4KKJxfnLWeIJoLt5nb-cBhUJiB5JLBan7bDnIXKohJh7FM4tW3J0ZyxomtfQxtNRJ0DnjtnLhbRO4-TFKjj5LjUK; BAIDUID_BFESS54C0826149B7299ED5ED2B69EA7EAE01:FG1; __yjsv5_shitong1.0_7_98da4595c8bb2361eb889d3a64fd8fc8e3d5_300_1607497358701_112.5.168.233_fb665284; delPer0; BD_CK_SAM1; BD_HOME1; PSINO6; H_PS_PSSID1466_33225_33058_33259_33236_33099_33101_26350_33199_33144_33148; H_PS_645ECd1ddM3MDeEzN0o9%2BO4RzSnpDa8%2Bpu3avFyNpR4YZC3hodmvp3wBjm9N5k0s; BA_HECTOR2k0k2l2l84al2k8llc1ft394d0q; COOKIE_SESSION82578_0_8_4_2_2_0_0_8_2_1_0_0_0_0_0_1607423260_0_1607574668%7C9%2326670_117_1607417652%7C9; Hm_lvt_aec699bb6442ba076c8981c6dc4907711606455659,1606789969,1607417005,1607574727; Hm_lpvt_aec699bb6442ba076c8981c6dc4907711607574727; BDSVRTM0; WWW_ST1607574827950 } response requests.request(GET, url, headersheaders, datapayload) print(response.headers)请求的打印的Header{Bdpagetype: 3, Bdqid: 0x8e8822890000da2a, Cache-Control: private, Ckpacknum: 2, Ckrndstr: 90000da2a, Connection: keep-alive, Content-Encoding: gzip, Content-Type: text/html;charsetutf-8, Date: Thu, 10 Dec 2020 05:39:06 GMT, Server: BWS/1.1, Set-Cookie: delPer0; path/; domain.baidu.com, BD_CK_SAM1;path/, PSINO6; domain.baidu.com; path/, BDSVRTM19; path/, H_PS_PSSID1466_33225_33058_33259_33236_33099_33101_26350_33199_33144_33149; path/; domain.baidu.com, Strict-Transport-Security: max-age172800, Traceid: 1607578746025443994610270496922024335914, Vary: Accept-Encoding, X-Ua-Compatible: IEEdge,chrome1, Transfer-Encoding: chunked}Requests 复杂的方法复杂的请求方式通常带有请求头、代理IP、证书验证和Cookies等功能。Requests将这一系列复杂的请求做了简化将这些功能在发送请求中以参数的形式传递并作用到请求中。1添加请求头请求头以字典的形式表示然后在发送请求中设置headers参数。请求中设置请求头相当于把程序伪装成浏览器来向网站发送请求主要设置User-Agent和Referer的内容因为很多网站反爬虫都是根据这两个内容来判断当前请求是否合法。 url https://www.baidu.com/s?wdSpringboot payload {} headers { Connection: keep-alive, X-Requested-With: XMLHttpRequest, User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36, } response requests.request(GET, url, headersheaders, datapayload) print(response.text)2使用代理IP使用方法与请求头的使用方法一致只需设置proxies参数即可。proxies以字典的形式表示字典的key主要有http和https这是两种不同的HTTP协议字典的value是一个可访问的IP地址免费的代理IP可以网上搜索不过很多都是无法使用。代理IP的实现代码如下3证书验证网站中出现证书不合法的时候只需设置verifyFalse等于关闭证书验证。参数verify的默认值为True。如果需要设置证书文件那么可将参数verify值设为证书所在的路径。4超时设置发送请求后由于网络、服务器等因素从请求到响应会有一个时间差。如果不想程序等待时间过长或者延长等待时间可以设定参数timeout的等待秒数超过这个等待时间就会停止等待响应并引发一个异常。使用代码如下 requests.get(https://www.baidu.com/, timeout1) requests.post(https://www.baidu.com/, timeout1)5使用Cookies在请求过程中使用Cookies也只需设置参数Cookies即可。Cookies的作用是标识用户身份在Requests中以字典或RequestsCookieJar对象作为参数。获取方式主要从浏览器读取或通过程序运行产生。下面的例子进一步讲解如何使用Cookies。 print(r.utils.dict_from_cookiejar(response.cookies))
