Getting errors when requesting some HTTPS URLs

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Post Reply
LucasC
Posts: 3
Joined: Wed Apr 06, 2022 12:01 pm

Getting errors when requesting some HTTPS URLs

Post by LucasC » Wed Apr 06, 2022 12:19 pm

I'm currently unable to read the content from some websites that have HTTPS. An example of an URL that gives this error is https://letsencrypt.org/documents/isrg-cps-v4.2/.
It happens when I use urllib.urequest.urlopen:

Code: Select all

import urllib
f=urllib.urequest.urlopen('https://letsencrypt.org/documents/isrg-cps-v4.2/')
I can also replicate this error with urequests.get:

Code: Select all

import urequests
response = urequests.get('https://letsencrypt.org/documents/isrg-cps-v4.2/')
In both cases, this error appears:

Code: Select all

File "<stdin>", line 1, in <module>
OSError: -40
I looked at the source code from the urllib.urequest.urlopen to know when the error happens, and it looks like it's a problem on the function "ussl.wrap_socket".

The only pattern I could find between the links that give this error is that they use a SSL certificate from Let's Encrypt. But I tried to connect to "https://www.freepik.com" and it is working, even though it uses the same type of certificate.

My os.uname() info is:

Code: Select all

(sysname='esp8266', nodename='esp8266', release='2.2.0-dev(9422289)', version='v
1.18 on 2022-01-17', machine='ESP module with ESP8266')   
Is there any other alternative function to connect to these websites? Is this a bug?

Edit:

Looks like this github issue might be related.

tepalia02
Posts: 99
Joined: Mon Mar 21, 2022 5:13 am

Re: Getting errors when requesting some HTTPS URLs

Post by tepalia02 » Wed Apr 13, 2022 12:12 pm

Hi, if you write

Code: Select all

import urequests as requests
do you see any improvement?

LucasC
Posts: 3
Joined: Wed Apr 06, 2022 12:01 pm

Re: Getting errors when requesting some HTTPS URLs

Post by LucasC » Tue Jun 28, 2022 4:06 pm

tepalia02 wrote:
Wed Apr 13, 2022 12:12 pm
Hi, if you write

Code: Select all

import urequests as requests
do you see any improvement?
Hey, sorry for the big delay. But it doesn't work still.

Code: Select all

import urequests as requests
response = requests.get('https://letsencrypt.org/documents/isrg-cps-v4.2/')
gives this error:

Code: Select all

OSError: -40
I tried in the ESP8266, and on the Micropython for Unix version too.
I think there's something wrong with dealing with the SSL certificate of these websites?

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Getting errors when requesting some HTTPS URLs

Post by jimmo » Wed Jun 29, 2022 12:30 am

tepalia02 wrote:
Wed Apr 13, 2022 12:12 pm
Hi, if you write
import urequests as requests
This won't change anything, just renames the module on import.
LucasC wrote:
Tue Jun 28, 2022 4:06 pm
I think there's something wrong with dealing with the SSL certificate of these websites?
Yes, the github issue you linked to (and also https://github.com/micropython/micropython/issues/3198 and viewtopic.php?t=9466&p=53058) explain what's happening here.

The ESP8266 and Unix port both use axTLS as their TLS libary, and this only supports RSA keys (more importantly, EC is not supported, see http://axtls.sourceforge.net/)

If you look at the SSL handshake, you'll see that letsencrypt uses ECDSA.

Code: Select all

$ openssl s_client -showcerts -connect letsencrypt.org:443

...
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
...
It's kind of just a coincidence that the Unix port still uses axTLS (because support was added at the same time as the ESP8266 port). We should migrate it to use mbedtls. However, for the ESP8266 I'm not sure this is something that we can easily fix. See https://github.com/micropython/micropyt ... -690840497 for the latest.

LucasC
Posts: 3
Joined: Wed Apr 06, 2022 12:01 pm

Re: Getting errors when requesting some HTTPS URLs

Post by LucasC » Wed Jun 29, 2022 5:28 pm

jimmo wrote:
Wed Jun 29, 2022 12:30 am
The ESP8266 and Unix port both use axTLS as their TLS libary, and this only supports RSA keys (more importantly, EC is not supported, see http://axtls.sourceforge.net/)

If you look at the SSL handshake, you'll see that letsencrypt uses ECDSA.

Code: Select all

$ openssl s_client -showcerts -connect letsencrypt.org:443

...
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
...
It's kind of just a coincidence that the Unix port still uses axTLS (because support was added at the same time as the ESP8266 port). We should migrate it to use mbedtls. However, for the ESP8266 I'm not sure this is something that we can easily fix. See https://github.com/micropython/micropyt ... -690840497 for the latest.
Thank you for the answer! So the Unix port can use mbedtls in the future, but it's more complicated for the ESP8266 port.
Do you know if the ESP32 port already use mbedtls? Should all of this be documented?

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Getting errors when requesting some HTTPS URLs

Post by jimmo » Wed Jun 29, 2022 11:23 pm

LucasC wrote:
Wed Jun 29, 2022 5:28 pm
Do you know if the ESP32 port already use mbedtls? Should all of this be documented?
Yes, ESP32 uses mbedtls (and I have confirmed that using urequests to fetch that letsencrypt.org URL above works on ESP32).

Yes, documentation would be good. And definitely we should move Unix over to mbedtls to match the majority of the ports.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Getting errors when requesting some HTTPS URLs

Post by jimmo » Mon Jul 04, 2022 5:46 am

LucasC wrote:
Wed Jun 29, 2022 5:28 pm
Should all of this be documented?
I just came across https://docs.micropython.org/en/latest/ ... imitations

(This needs to be improved though, it's not easily discoverable)

Post Reply