Proxy detection in Python3

Question

The script below works fine when I am using script at home (same PC!):

import urllib.request

x = urllib.request.urlopen('https://www.google.com/')
print(x.read())

the same does not work using the same script when I am connected at work. I do not know proxy address or IP, so my script should use the same way as IE or anything else on this PC.

I found some suggestions about using proxy , but the point it I do not know proxy IP or details. When I move the script to another PC it might have different proxy, so I think hardcoding it is not good approach. Can I somehow inform Python to autodetect proxy settings?

Answer 1

1. Going by your eample, I am assuming you are doing a https call over proxy. The urllib documentation hints its not supported. So, instead you may have to settle down with http.

2. In order to validate that there is nothing wrong with your setup, you may try to do open the IP directly:

    import urllib
    # IP address for `http://www.google.com` is `216.58.205.196`
    x = urllib.urlopen('http://216.58.205.196')
    print x.read()

3. A. There are lots of complaints about Python's trippy auto-detect proxy settings in various other threads. I had this issue only once years ago and I opted for setting a fixed proxy instead of trying to configure auto-detect. To know your proxy, you can go to chrome url chrome://net-internals/#proxy or run netstat -an | grep EST command.

   B. Once you have proxy address, you can use following code:

    import urllib
    # IP address for `http://www.google.com` is `216.58.205.196`
    x = urllib.urlopen('http://216.58.205.196', 
                        proxies={'http': 'http://www.someproxy.com:3128'})
    print x.read()

4. If you cannot avoid https, then you may consider requests library. I didn't test this, but requests documentation looks quite promising. This is how it can be done!

    import requests
    proxies = {
      'http': 'http://10.10.1.10:3128',
      'https': 'http://10.10.1.10:1080',
    }
    requests.get('https://example.org', proxies=proxies)

Edit:

1: You may need to setup proxy authentication in order for 3.B. to work

2: For Special characters, you would need to have the password in unicode: 'p@ssw0rd'.decode('utf-8')

Hope this helps!