0

I use urllib.request.urlopen to fetch data from server over HTTPS. The function is called a lot to the same server, often to the exact same url. However, unlike standard web browsers which perform a handshake on initial request, calling separate urlopen(url)'s will result in a new handshake for each call. This is very slow on high-latency networks. Is there a way to perform handshake once and reuse the existing connection for further communications?

I cannot modify server code to utilise sockets or other protocols.

Mirac7
  • 1,566
  • 4
  • 26
  • 44

2 Answers2

0

You are opening a new connection for every request. To reuse the connection, you either need to use an http.client:

>>> import http.client
>>> conn = http.client.HTTPSConnection("www.python.org")
>>> conn.request("GET", "/")
>>> r1 = conn.getresponse()
>>> print(r1.status, r1.reason)
200 OK
>>> data1 = r1.read()  # This will return entire content.
>>> # The following example demonstrates reading data in chunks.
>>> conn.request("GET", "/")
>>> r1 = conn.getresponse()
>>> while not r1.closed:
...     print(r1.read(200)) # 200 bytes
b'<!doctype html>\n<!--[if"...
...
>>> # Example of an invalid request
>>> conn.request("GET", "/parrot.spam")
>>> r2 = conn.getresponse()
>>> print(r2.status, r2.reason)
404 Not Found
>>> data2 = r2.read()
>>> conn.close()

Or use the recommended python Requests Package, which has session objects that make use of persistent connections (using urllib3).

JimB
  • 104,193
  • 13
  • 262
  • 255
-1

You should open a stream for it, because HTTP/(s) is stateless its open new socket to the server for each connection.

So there is no way with this logic, but I just searched around for opening the persistent connection. I just see that hope it will help. It mentions about urllib2

Persistent HTTPS Connections in Python

Community
  • 1
  • 1
FZE
  • 1,587
  • 12
  • 35