upip not working with IDF4 builds

All ESP32 boards running MicroPython.
Target audience: MicroPython users with an ESP32 board.
chibill
Posts: 47
Joined: Thu Oct 31, 2019 10:44 pm

upip not working with IDF4 builds

Post by chibill » Wed Nov 06, 2019 6:22 pm

I am using builds esp32-idf3-20191106-v1.11-563-g4be316fb0 and esp32-idf4-20191106-v1.11-563-g4be316fb0 for this.

On both builds I first connect to my wifi then try to use upip.

I was trying to upip install urequests using

Code: Select all

upip.install("micropython-urequests")
On the IDF3 based builds the install works perfectly.

On the IDF4 based builds it throws an error.
>>> upip.install("micropython-urequests")
Installing to: /lib/
Error installing 'micropython-urequests': [Errno 12] ENOMEM, packages may be partially installed
I already asked in IRC but I figured it would be best to also post here for anyone else running into this problem.

chibill
Posts: 47
Joined: Thu Oct 31, 2019 10:44 pm

Re: upip not working with IDF4 builds

Post by chibill » Thu Nov 07, 2019 4:00 am

More details on the IDF4 build running

Code: Select all

print(gc.mem_free())
before upip.install gives 103408 then after upip fails it returns 100480.

I am not going to bother running it on the IDF3 build as it works fine.

Running gc.collect() as suggested on IRC doesn't seem to help at all either.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: upip not working with IDF4 builds

Post by jimmo » Thu Nov 07, 2019 4:47 am

Thanks for the report.

I can repro the issue here, but only on non-psram devices (i.e. the generic build).

The ENOMEM is actually not related to the MicroPython heap running out of memory. The failure is coming out of mbedtls.

I'll look into it. I wonder if the allocator changes in IDF4 somehow such that heap_caps_get_largest_free_block ends up being too greedy and not leaving enough RAM left over for other functionality.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: upip not working with IDF4 builds

Post by jimmo » Thu Nov 07, 2019 5:34 am


chibill
Posts: 47
Joined: Thu Oct 31, 2019 10:44 pm

Re: upip not working with IDF4 builds

Post by chibill » Thu Nov 07, 2019 3:24 pm

Interesting. I wonder why in IDF4 they decided to change how memory was segmented. Surprised this wasn't listed in the logs on Espressif's for the IDF4 vs IDF3 changes. (I looked once you mentioned something in IDF was likely causing it.

chibill
Posts: 47
Joined: Thu Oct 31, 2019 10:44 pm

Re: upip not working with IDF4 builds

Post by chibill » Thu Nov 07, 2019 6:47 pm

I am also going to try your edit locally by building micropython. As that is how I found this bug originally and at first thought I had did something wrong while building it. Then found it to also affect the pre-built firmware.

chibill
Posts: 47
Joined: Thu Oct 31, 2019 10:44 pm

Re: upip not working with IDF4 builds

Post by chibill » Fri Nov 08, 2019 12:18 am

Seems adding that patch to my local build doesn't fix the upip error. Is there a way to compile it to get extra debug info?

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: upip not working with IDF4 builds

Post by jimmo » Fri Nov 08, 2019 1:54 am

Have a look at build-GENERIC/sdkconfig.h, with the patch you should see

Code: Select all

#define CONFIG_MBEDTLS_ASYMMETRIC_CONTENT_LEN 1
#define CONFIG_MBEDTLS_SSL_IN_CONTENT_LEN 16384
#define CONFIG_MBEDTLS_SSL_OUT_CONTENT_LEN 4096
whereas before that change, it would just be

Code: Select all

#define CONFIG_MBEDTLS_SSL_MAX_CONTENT_LEN 16384
If you want to debug this further, put

Code: Select all

heap_caps_print_heap_info(MALLOC_CAP_INTERNAL | MALLOC_CAP_DEFAULT);
immediately before the call to

Code: Select all

ret = mbedtls_ssl_setup(&o->ssl, &o->conf);
in socket_new() in modussl_mbedtls.c

chibill
Posts: 47
Joined: Thu Oct 31, 2019 10:44 pm

Re: upip not working with IDF4 builds

Post by chibill » Sat Nov 09, 2019 10:11 pm

Seems to work although it does output

Code: Select all

>>> upip.install("micropython-urequests")
Installing to: /lib/
Warning: micropython.org SSL certificate is not validated
Installing micropython-urequests 0.6 from https://micropython.org/pi/urequests/urequests-0.6.tar.gz
I don't remember the warning on the IDF3 version

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: upip not working with IDF4 builds

Post by jimmo » Sun Nov 10, 2019 11:44 pm

I'm fairly sure the "Warning: micropython.org SSL certificate is not validated" was there before (or if it wasn't, it definitely wasn't validated before either)

Post Reply