w3resource

Python Web Scraping: Verify SSL certificates for HTTPS requests using requests module

Python Web Scraping: Exercise-27 with Solution

Write a Python program to verify SSL certificates for HTTPS requests using requests module.

Note: Requests verifies SSL certificates for HTTPS requests, just like a web browser. By default, SSL verification is enabled, and Requests will throw a SSLError if it’s unable to verify the certificate

Sample Solution:

Python Code:

import requests
print(requests.get('https://w3resource.com'))
print(requests.get('https://wayback.com'))
  
 

Sample Output:

<Response [200]>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 839, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 364, in connect
    _match_hostname(cert, self.assert_hostname or server_hostname)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 374, in _match_hostname
    match_hostname(cert, asserted_hostname)
  File "/usr/lib/python3.6/ssl.py", line 327, in match_hostname
    % (hostname, ', '.join(map(repr, dnsnames))))
ssl.CertificateError: hostname 'wayback.com' doesn't match either of '*.prod.iad2.secureserver.net', 'prod.iad2.secureserver.net'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/dist-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='wayback.com', port=443): Max retries exceeded with url: / (Caused by SSLError(CertificateError("hostname 'wayback.com' doesn't match either of '*.prod.iad2.secureserver.net', 'prod.iad2.secureserver.net'",),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/sessions/d2a193285f0c778a/main.py", line 3, in <module>
    print(requests.get('https://wayback.com'))
  File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='wayback.com', port=443): Max retries exceeded with url: / (Caused by SSLError(CertificateError("hostname 'wayback.com' doesn't match either of '*.prod.iad2.secureserver.net', 'prod.iad2.secureserver.net'",),))

Flowchart:

Python Web Scraping Flowchart: Verifiy SSL certificates for HTTPS requests using requests module

Python Code Editor:


Have another way to solve this solution? Contribute your code (and comments) through Disqus.

Previous: Write a Python program to display the contains of different attributes like status_code, headers, url, history, encoding, reason, cookies, elapsed, request and content of a specified resource.

What is the difficulty level of this exercise?



Become a Patron!

Follow us on Facebook and Twitter for latest update.

It will be nice if you may share this link in any developer community or anywhere else, from where other developers may find this content. Thanks.

https://www.w3resource.com/python-exercises/web-scraping/web-scraping-exercise-27.php