ESP32 port now uses CMake

All ESP32 boards running MicroPython.
Target audience: MicroPython users with an ESP32 board.
User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

ESP32 port now uses CMake

Post by jimmo » Mon Feb 15, 2021 6:57 am

https://github.com/micropython/micropython/pull/6892 was merged today which will affect anyone who is building ESP32 firmware from source. If you're using the released binaries (either nightly builds or versioned releases) this probably won't affect you (**).

Summary: Please see the updated build instructions in https://github.com/micropython/micropyt ... /README.md

On ESP32 we use the Espressif IDF which provides the toolchain and SDK for the ESP32 family of chips. Originally the MicroPython port had to use quite a complicated workaround to integrate with their build system. The IDF's build system was rewritten in IDF 4.0 to use cmake, and as of later versions, only cmake is supported.

As other MicroPython ports are also using cmake (such as the Raspberry Pi Pico), MicroPython now supports integrating with cmake, and this is now the only supported build configuration for MicroPython on ESP32 (the Makefile-based build has been removed). Overall this makes the ESP32 build process much simpler, and importantly goes a long way towards adding support for newer chips such as the ESP32-S2 and others.

There are two main things to be aware of:
- User C Modules (such as ulab) will no longer work until a cmake-based replacement for this mechanism is implemented. In the meantime, you can use the v1.14 tag of the MicroPython repository.
- (**) As the v4 IDF changed the API for the wired LAN driver (and MicroPython has not yet been updated), there will no longer be nightly builds supporting wired LAN. MicroPython will continue to make the v3 IDF based builds for past releases available, and hopefully the MicroPython bindings will be updated at some stage to add this back.

Jeizooo
Posts: 10
Joined: Mon Jan 04, 2021 10:14 am

Re: ESP32 port now uses CMake

Post by Jeizooo » Mon Feb 15, 2021 11:08 am

Hi,

Thanks for all the work around this improvement. It works!
jimmo wrote:
Mon Feb 15, 2021 6:57 am
- User C Modules (such as ulab) will no longer work until a cmake-based replacement for this mechanism is implemented. In the meantime, you can use the v1.14 tag of the MicroPython repository.
I have a simple C module. Is there a way to adapt it to compile with cmake? Or it's just impossible now without more work on the port modules compilation process?

Here is the current .mk file:

Code: Select all

ST7789_MOD_DIR := $(USERMOD_DIR)
SRC_USERMOD += $(addprefix $(ST7789_MOD_DIR)/, \
	st7789.c \
)
CFLAGS_USERMOD += -I$(ST7789_MOD_DIR) -DMODULE_ST7789_ENABLED=1
CFLAGS_USERMOD += -DEXPOSE_EXTRA_METHODS=1
Thanks, read you.

uraich
Posts: 56
Joined: Tue Mar 26, 2019 12:24 pm

Re: ESP32 port now uses CMake

Post by uraich » Mon Feb 15, 2021 5:30 pm

The build instructions say that you can replace version 4.0.2 by any supported version. How do I know, which version is supported?

Jeizooo
Posts: 10
Joined: Mon Jan 04, 2021 10:14 am

Re: ESP32 port now uses CMake

Post by Jeizooo » Mon Feb 15, 2021 5:45 pm

Look at the commits: https://github.com/micropython/micropyt ... its/master

4.0.2, 4.1.1, 4.2

Hope it helps.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: ESP32 port now uses CMake

Post by jimmo » Mon Feb 15, 2021 11:43 pm

Jeizooo wrote:
Mon Feb 15, 2021 11:08 am
I have a simple C module. Is there a way to adapt it to compile with cmake? Or it's just impossible now without more work on the port modules compilation process?
Unfortunately it's not just a matter of adapting your module.

Support needs to be added to the MicroPython cmake scripts. I don't imagine it's terribly difficult, just someone needs to do it.

Jeizooo
Posts: 10
Joined: Mon Jan 04, 2021 10:14 am

Re: ESP32 port now uses CMake

Post by Jeizooo » Tue Feb 16, 2021 10:36 am

Ok thanks. I will wait, I have no clue how to do that!
J.

michael.o
Posts: 15
Joined: Sun Oct 25, 2020 12:38 am

Re: ESP32 port now uses CMake

Post by michael.o » Fri Feb 19, 2021 5:57 am

I have a part way solution for getting user modules to compile with esp-idf and cmake.

I found out how to create idf components and then how to include this into the build in a similar way to how USER_C_MODULES used to be passed in.

In the current system each c module needs to have a GNU Makefile snippet which sets up the various source files, compile flags, linker options, etc.

With what I've found each c module will need to have a CMakeLists.txt file part. At the moment I've only run where the snippet is defining the c module as an idf component but it should be possible to interact at a lower lever like a library.

I have a use case where I need to link to an existing static library: https://docs.espressif.com/projects/esp ... components

Code: Select all

$ mkdir my-micropython-c-modules

$ cd my-micropytohn-c-modules

$ idf.py create-component c-component-1

$ idf.py create-component c-component-2
in micropython/ports/esp32 set the EXTRA_COMPONENT_DIRS to the space seperated directories that contain the idf components (user c modules) you want to also include in the firmware build.

https://docs.espressif.com/projects/esp ... ialization
(EXTRA_COMPONENT_DIRS is the second last bullet from the bottom)

Code: Select all

 idf.py -DEXTRA_COMPONENT_DIRS="/src/my-micropython-c-modules/c-component-1 /src/my-micropython-c-modules/c-component-1" build

-- Components: app_trace app_update asio bootloader bootloader_support bt cbor cmock coap console cxx c-component-1 c-component-2 driver efuse esp-
tls esp32 esp_adc_cal esp_common esp_eth esp_event esp_gdbstub esp_hid esp_http_client esp_http_server esp_https_ota esp_https_server
esp_hw_support esp_ipc esp_local_ctrl esp_netif esp_pm esp_ringbuf esp_rom esp_serial_slave_link esp_system esp_timer esp_websocket_cl
ient esp_wifi espcoredump esptool_py expat fatfs freemodbus freertos hal heap idf_test jsmn json libsodium log lwip main mbedtls mdns mqtt newlib nghttp nvs_flash openssl partition_table perfmon protobuf-c protocomm pthread sdmmc soc spi_flash spiffs tcp_tra
nsport tcpip_adapter tinyusb ulp unity vfs wear_levelling wifi_provisioning wpa_supplicant xtensa
-- Component paths: /opt/esp/idf/components/app_trace /opt/esp/idf/components/app_update /opt/esp/idf/components/asio /src/my-micropython-c-modules/c-component-1 /opt/esp/idf/components/bootloader /opt/esp/idf/components/bootloader_support /opt/esp/idf/comp
onents/bt /opt/esp/idf/components/cbor /opt/esp/idf/components/cmock /opt/esp/idf/components/coap /opt/esp/idf/components/console /opt
/esp/idf/components/cxx /opt/esp/idf/components/driver /opt/esp/idf/components/efuse /opt/esp/idf/components/esp-tls /opt/esp/idf/comp
onents/esp32 /opt/esp/idf/components/esp_adc_cal /opt/esp/idf/components/esp_common /opt/esp/idf/components/esp_eth /opt/esp/idf/compo
nents/esp_event /opt/esp/idf/components/esp_gdbstub /opt/esp/idf/components/esp_hid /opt/esp/idf/components/esp_http_client /opt/esp/i
df/components/esp_http_server /opt/esp/idf/components/esp_https_ota /opt/esp/idf/components/esp_https_server /opt/esp/idf/components/e
sp_hw_support /opt/esp/idf/components/esp_ipc /opt/esp/idf/components/esp_local_ctrl /opt/esp/idf/components/esp_netif /opt/esp/idf/co
mponents/esp_pm /opt/esp/idf/components/esp_ringbuf /opt/esp/idf/components/esp_rom /opt/esp/idf/components/esp_serial_slave_link /opt
/esp/idf/components/esp_system /opt/esp/idf/components/esp_timer /opt/esp/idf/components/esp_websocket_client /opt/esp/idf/components/
esp_wifi /opt/esp/idf/components/espcoredump /opt/esp/idf/components/esptool_py /opt/esp/idf/components/expat /opt/esp/idf/components/
fatfs /opt/esp/idf/components/freemodbus /opt/esp/idf/components/freertos /opt/esp/idf/components/hal /opt/esp/idf/components/heap /op
t/esp/idf/components/idf_test /opt/esp/idf/components/jsmn /opt/esp/idf/components/json /opt/esp/idf/components/libsodium /opt/esp/idf
/components/log /opt/esp/idf/components/lwip /src/micropython/ports/esp32/main /opt/esp/idf/components/mbedtls /opt/esp/idf/components
/mdns /src/my-micropython-c-modules/c-component-2 /opt/esp/idf/components/mqtt /opt/esp/idf/components/newlib /opt/esp/idf/components
/nghttp /opt/esp/idf/components/nvs_flash /opt/esp/idf/components/openssl /opt/esp/idf/components/partition_table /opt/esp/idf/compone
nts/perfmon /opt/esp/idf/components/protobuf-c /opt/esp/idf/components/protocomm /opt/esp/idf/components/pthread /opt/esp/idf/componen
ts/sdmmc /opt/esp/idf/components/soc /opt/esp/idf/components/spi_flash /opt/esp/idf/components/spiffs /opt/esp/idf/components/tcp_tran
sport /opt/esp/idf/components/tcpip_adapter /opt/esp/idf/components/tinyusb /opt/esp/idf/components/ulp /opt/esp/idf/components/unity
/opt/esp/idf/components/vfs /opt/esp/idf/components/wear_levelling /opt/esp/idf/components/wifi_provisioning /opt/esp/idf/components/w
pa_supplicant /opt/esp/idf/components/xtensa
-- Configuring done

The two modules are present in this list which makes me think they would be included in the firmware.

Tomorrow I'll continue the experiment to see if I can convert my micropython.mk build files into cmake equivalents and end up with a working custom firmware.

I was testing this inside of the espressif/idf:release-v4.3 docker container.

bertel
Posts: 27
Joined: Tue Feb 09, 2021 3:55 pm
Location: Tokyo

Re: ESP32 port now uses CMake

Post by bertel » Sun Feb 21, 2021 3:42 pm

I’m trying to get the new version going. build-GENERIC works, but build-GENERIC_SPIRAM is throwing me a curve ball. When I use the proper partitions.csv , and update Makefile with

Code: Select all

BOARD ?= GENERIC_SPIRAM
FLASH_SIZE ?= 16MB
I get the following error:

Code: Select all


FAILED: partition_table/partition-table.bin
cd /usr/local/bin/micropython/micropython/ports/esp32/build-GENERIC_SPIRAM/esp-idf/partition_table && /root/.espressif/python_env/idf4.1_py3.8_env/bin/python /root/esp4/esp-idf-v4.1.1/components/partition_table/gen_esp32part.py -q --offset 0x8000 --flash-size 4MB /usr/local/bin/micropython/micropython/ports/esp32/partitions.csv /usr/local/bin/micropython/micropython/ports/esp32/build-GENERIC_SPIRAM/partition_table/partition-table.bin
Partitions defined in '/usr/local/bin/micropython/micropython/ports/esp32/partitions.csv' occupy 16.0MB of flash (16777216 bytes) which does not fit in configured flash size 4MB. Change the flash size in menuconfig under the 'Serial Flasher Config' menu.
[79/1136] Building C object esp-idf/driver/CMakeFiles/__idf_driver.dir/i2s.c.obj
ninja: build stopped: subcommand failed.
ninja failed with exit code 1
Apparently, make doesn’t understand the 16MB directive. I go into menuconfig (great addition btw) and change the 'Serial Flasher Config' to 16MB, but make still thinks I only want code for 4MB flash.
I tried all kinds of settings, commandline, a makefile (lowercase m) with FLASH_SIZE ?= 16MB, but to no avail.

How do I tell the compiler that I want code for 16MB?

Jeizooo
Posts: 10
Joined: Mon Jan 04, 2021 10:14 am

Re: ESP32 port now uses CMake

Post by Jeizooo » Tue Feb 23, 2021 12:46 pm

michael.o wrote:
Fri Feb 19, 2021 5:57 am
I have a part way solution for getting user modules to compile with esp-idf and cmake.
Nice! Did you find something more?

I am going to try you solution for now.

Thanks.

michael.o
Posts: 15
Joined: Sun Oct 25, 2020 12:38 am

Re: ESP32 port now uses CMake

Post by michael.o » Wed Feb 24, 2021 6:17 am

I went around in circles for a few days on this but finally got something working.

I just filed a pull request for it: https://github.com/micropython/micropython/pull/6956

I got it to work with the simple c example but more work is needed to pass through more flags and options.

I will try this approach with my actual custom firmware modules and probably add more to the pull request. I have some c++ and also use external static library cases.

At first I tried having the module be a seperate idf component except there are issues with how the qstr generation logic works with that approach.

build -> main -> qstr build

extra module -> main (unable to append module data into qstr build)

I moved the mkrules.cmake include from the main module to the top level esp32 cmake file and was able to get the qstr part working right but other parts broke because they expected to be used in the main module context.

Post Reply