Trouble with CMake and flash > 4 MB

All ESP32 boards running MicroPython.
Target audience: MicroPython users with an ESP32 board.
Post Reply
bertel
Posts: 23
Joined: Tue Feb 09, 2021 3:55 pm
Location: Tokyo

Trouble with CMake and flash > 4 MB

Post by bertel » Mon Feb 22, 2021 2:36 pm

I am still (see my comment in https://forum.micropython.org/viewtopi ... 18&t=9820 battling with the new Cmake version when it comes to larger flash sizes. It simply won’t compile for anything larger than 4MB
Previously, I put these into Makefile

Code: Select all

BOARD ?= GENERIC_SPIRAM
FLASH_SIZE ?= 16MB
My partitions.csv looked like this:

Code: Select all

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 0x180000,
vfs,      data, fat,     0x190000, 0xE00000
It happily compiled.
The Cmake version does not seem to understand the FLASH_SIZE ?= 16MB instruction. With the settings above, it complains that the partition “does not fit in configured flash size 4MB. Change the flash size in menuconfig under the 'Serial Flasher Config' menu.”
I have done so repeatedly, without luck. The Cmake setup does not seem to pick up on the menuconfig instructions.
If I set flash to 16MB in menuconfig, …/build/sdkconfig dutifully reports:

Code: Select all

# CONFIG_ESPTOOLPY_FLASHSIZE_1MB is not set
# CONFIG_ESPTOOLPY_FLASHSIZE_2MB is not set
# CONFIG_ESPTOOLPY_FLASHSIZE_4MB is not set
# CONFIG_ESPTOOLPY_FLASHSIZE_8MB is not set
CONFIG_ESPTOOLPY_FLASHSIZE_16MB=y
CONFIG_ESPTOOLPY_FLASHSIZE="16MB"

I do a make clean, which leaves …/build-GENERIC_SPIRAM empty
make produces this:

Code: Select all

idf.py -D MICROPY_BOARD=GENERIC_SPIRAM -B build-GENERIC_SPIRAM build
Executing action: all (aliases: build)
Running cmake in directory /usr/local/bin/micropython/micropython_1.14/ports/esp32/build-GENERIC_SPIRAM
Executing "cmake -G Ninja -DPYTHON_DEPS_CHECKED=1 -DESP_PLATFORM=1 -DMICROPY_BOARD=GENERIC_SPIRAM -DCCACHE_ENABLE=0 /usr/local/bin/micropython/micropython_1.14/ports/esp32"...
-- Found Git: /usr/bin/git (found version "2.25.1")
-- IDF_TARGET not set, using default target: esp32
-- The C compiler identification is GNU 8.4.0
-- The CXX compiler identification is GNU 8.4.0
-- The ASM compiler identification is GNU
-- Found assembler: /root/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /root/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /root/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Project version: v1.14-80-g75db0b907-dirty
-- Building ESP-IDF components for target esp32
-- Project sdkconfig file /usr/local/bin/micropython/micropython_1.14/ports/esp32/build-GENERIC_SPIRAM/sdkconfig
Loading defaults file /usr/local/bin/micropython/micropython_1.14/ports/esp32/build-GENERIC_SPIRAM/sdkconfig.combined...
/usr/local/bin/micropython/micropython_1.14/ports/esp32/build-GENERIC_SPIRAM/sdkconfig.combined:9 CONFIG_COMPILER_OPTIMIZATION_LEVEL_RELEASE was replaced with CONFIG_COMPILER_OPTIMIZATION_SIZE
/tmp/confgen_tmphk6dbqrs:61 line was updated to CONFIG_BTDM_CTRL_MODE_BR_EDR_ONLY=n
/tmp/confgen_tmphk6dbqrs:62 line was updated to CONFIG_BTDM_CTRL_MODE_BTDM=n
CMake Deprecation Warning at /root/esp4/esp-idf-v4.1.1/components/mbedtls/mbedtls/CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- Found PythonInterp: /root/.espressif/python_env/idf4.1_py3.8_env/bin/python (found version "3.8.5")
-- Found Perl: /usr/bin/perl (found version "5.30.0")
-- Adding linker script /usr/local/bin/micropython/micropython_1.14/ports/esp32/build-GENERIC_SPIRAM/esp-idf/esp32/esp32_out.ld
-- Adding linker script /root/esp4/esp-idf-v4.1.1/components/esp32/ld/esp32.project.ld.in
-- Adding linker script /root/esp4/esp-idf-v4.1.1/components/esp32/ld/esp32.peripherals.ld
-- Adding linker script /root/esp4/esp-idf-v4.1.1/components/esp_rom/esp32/ld/esp32.rom.ld
-- Adding linker script /root/esp4/esp-idf-v4.1.1/components/esp_rom/esp32/ld/esp32.rom.libgcc.ld
-- Adding linker script /root/esp4/esp-idf-v4.1.1/components/esp_rom/esp32/ld/esp32.rom.newlib-data.ld
-- Adding linker script /root/esp4/esp-idf-v4.1.1/components/esp_rom/esp32/ld/esp32.rom.syscalls.ld
-- Found Python3: /usr/local/bin/micropython/micropython_1.14/ports/esp32/build-venv/bin/python3 (found version "3.8.5") found components: Interpreter
-- Components: app_trace app_update asio bootloader bootloader_support bt cbor coap console cxx driver efuse esp-tls esp32 esp_adc_cal esp_common esp_eth esp_event esp_gdbstub esp_http_client esp_http_server esp_https_ota esp_https_server esp_local_ctrl esp_netif esp_ringbuf esp_rom esp_serial_slave_link esp_websocket_client esp_wifi espcoredump esptool_py expat fatfs freemodbus freertos heap idf_test jsmn json libsodium log lwip main mbedtls mdns mqtt newlib nghttp nvs_flash openssl partition_table perfmon protobuf-c protocomm pthread sdmmc soc spi_flash spiffs tcp_transport tcpip_adapter ulp unity vfs wear_levelling wifi_provisioning wpa_supplicant xtensa
-- Component paths: /root/esp4/esp-idf-v4.1.1/components/app_trace /root/esp4/esp-idf-v4.1.1/components/app_update /root/esp4/esp-idf-v4.1.1/components/asio /root/esp4/esp-idf-v4.1.1/components/bootloader /root/esp4/esp-idf-v4.1.1/components/bootloader_support /root/esp4/esp-idf-v4.1.1/components/bt /root/esp4/esp-idf-v4.1.1/components/cbor /root/esp4/esp-idf-v4.1.1/components/coap /root/esp4/esp-idf-v4.1.1/components/console /root/esp4/esp-idf-v4.1.1/components/cxx /root/esp4/esp-idf-v4.1.1/components/driver /root/esp4/esp-idf-v4.1.1/components/efuse /root/esp4/esp-idf-v4.1.1/components/esp-tls /root/esp4/esp-idf-v4.1.1/components/esp32 /root/esp4/esp-idf-v4.1.1/components/esp_adc_cal /root/esp4/esp-idf-v4.1.1/components/esp_common /root/esp4/esp-idf-v4.1.1/components/esp_eth /root/esp4/esp-idf-v4.1.1/components/esp_event /root/esp4/esp-idf-v4.1.1/components/esp_gdbstub /root/esp4/esp-idf-v4.1.1/components/esp_http_client /root/esp4/esp-idf-v4.1.1/components/esp_http_server /root/esp4/esp-idf-v4.1.1/components/esp_https_ota /root/esp4/esp-idf-v4.1.1/components/esp_https_server /root/esp4/esp-idf-v4.1.1/components/esp_local_ctrl /root/esp4/esp-idf-v4.1.1/components/esp_netif /root/esp4/esp-idf-v4.1.1/components/esp_ringbuf /root/esp4/esp-idf-v4.1.1/components/esp_rom /root/esp4/esp-idf-v4.1.1/components/esp_serial_slave_link /root/esp4/esp-idf-v4.1.1/components/esp_websocket_client /root/esp4/esp-idf-v4.1.1/components/esp_wifi /root/esp4/esp-idf-v4.1.1/components/espcoredump /root/esp4/esp-idf-v4.1.1/components/esptool_py /root/esp4/esp-idf-v4.1.1/components/expat /root/esp4/esp-idf-v4.1.1/components/fatfs /root/esp4/esp-idf-v4.1.1/components/freemodbus /root/esp4/esp-idf-v4.1.1/components/freertos /root/esp4/esp-idf-v4.1.1/components/heap /root/esp4/esp-idf-v4.1.1/components/idf_test /root/esp4/esp-idf-v4.1.1/components/jsmn /root/esp4/esp-idf-v4.1.1/components/json /root/esp4/esp-idf-v4.1.1/components/libsodium /root/esp4/esp-idf-v4.1.1/components/log /root/esp4/esp-idf-v4.1.1/components/lwip /usr/local/bin/micropython/micropython_1.14/ports/esp32/main /root/esp4/esp-idf-v4.1.1/components/mbedtls /root/esp4/esp-idf-v4.1.1/components/mdns /root/esp4/esp-idf-v4.1.1/components/mqtt /root/esp4/esp-idf-v4.1.1/components/newlib /root/esp4/esp-idf-v4.1.1/components/nghttp /root/esp4/esp-idf-v4.1.1/components/nvs_flash /root/esp4/esp-idf-v4.1.1/components/openssl /root/esp4/esp-idf-v4.1.1/components/partition_table /root/esp4/esp-idf-v4.1.1/components/perfmon /root/esp4/esp-idf-v4.1.1/components/protobuf-c /root/esp4/esp-idf-v4.1.1/components/protocomm /root/esp4/esp-idf-v4.1.1/components/pthread /root/esp4/esp-idf-v4.1.1/components/sdmmc /root/esp4/esp-idf-v4.1.1/components/soc /root/esp4/esp-idf-v4.1.1/components/spi_flash /root/esp4/esp-idf-v4.1.1/components/spiffs /root/esp4/esp-idf-v4.1.1/components/tcp_transport /root/esp4/esp-idf-v4.1.1/components/tcpip_adapter /root/esp4/esp-idf-v4.1.1/components/ulp /root/esp4/esp-idf-v4.1.1/components/unity /root/esp4/esp-idf-v4.1.1/components/vfs /root/esp4/esp-idf-v4.1.1/components/wear_levelling /root/esp4/esp-idf-v4.1.1/components/wifi_provisioning /root/esp4/esp-idf-v4.1.1/components/wpa_supplicant /root/esp4/esp-idf-v4.1.1/components/xtensa
-- Configuring done
-- Generating done
-- Build files have been written to: /usr/local/bin/micropython/micropython_1.14/ports/esp32/build-GENERIC_SPIRAM
Running ninja in directory /usr/local/bin/micropython/micropython_1.14/ports/esp32/build-GENERIC_SPIRAM
Executing "ninja all"...
[7/1206] Generating ../../partition_table/partition-table.bin
FAILED: partition_table/partition-table.bin
cd /usr/local/bin/micropython/micropython_1.14/ports/esp32/build-GENERIC_SPIRAM/esp-idf/partition_table && /root/.espressif/python_env/idf4.1_py3.8_env/bin/python /root/esp4/esp-idf-v4.1.1/components/partition_table/gen_esp32part.py -q --offset 0x8000 --flash-size 4MB /usr/local/bin/micropython/micropython_1.14/ports/esp32/partitions.csv /usr/local/bin/micropython/micropython_1.14/ports/esp32/build-GENERIC_SPIRAM/partition_table/partition-table.bin
Partitions defined in '/usr/local/bin/micropython/micropython_1.14/ports/esp32/partitions.csv' occupy 16.0MB of flash (16777216 bytes) which does not fit in configured flash size 4MB. Change the flash size in menuconfig under the 'Serial Flasher Config' menu.
[72/1206] Building C object esp-idf/mbedtls/mbedtls/library/CMakeFiles/mbedtls.dir/ssl_tls.c.obj
ninja: build stopped: subcommand failed.
ninja failed with exit code 1
make: *** [Makefile:26: all] Error 2

According to the output above, the project sdkconfig file sits in
…/build-GENERIC_SPIRAM/sdkconfig
However, the pertinent part of the sdkconfig file now reads like this:

Code: Select all

# CONFIG_ESPTOOLPY_FLASHSIZE_1MB is not set
# CONFIG_ESPTOOLPY_FLASHSIZE_2MB is not set
CONFIG_ESPTOOLPY_FLASHSIZE_4MB=y
# CONFIG_ESPTOOLPY_FLASHSIZE_8MB is not set
# CONFIG_ESPTOOLPY_FLASHSIZE_16MB is not set
CONFIG_ESPTOOLPY_FLASHSIZE="4MB"
CONFIG_ESPTOOLPY_FLASHSIZE_DETECT=y

Apparently, when the sdkconfig file is fetched from the …/build directory, the flash size is changed to 4MB. Most likely, the sdkconfig file produced by menuconfig is ignored, and a default file is copied from somewhere else.

Declare me stumped. Any help would be appreciated.

bertel
Posts: 23
Joined: Tue Feb 09, 2021 3:55 pm
Location: Tokyo

Re: Trouble with CMake and flash > 4 MB

Post by bertel » Mon Feb 22, 2021 4:10 pm

I found what is happening, but I don’t why:

/esp32/boards/sdkconfig.base is being used instead of the sdkconfig file produced by menuconfig.
In /esp32/boards/sdkconfig.base, if I make this edit, the compile finishes successfully:

Code: Select all

#CONFIG_ESPTOOLPY_FLASHSIZE_4MB=y
CONFIG_ESPTOOLPY_FLASHSIZE_16MB=y
The above, and other observations, make me doubt that any changes made via menuconfig are being used.

User avatar
jimmo
Posts: 2244
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Trouble with CMake and flash > 4 MB

Post by jimmo » Wed Feb 24, 2021 12:48 am

bertel wrote:
Mon Feb 22, 2021 4:10 pm
The above, and other observations, make me doubt that any changes made via menuconfig are being used.
I think this is the case. You will need to copy whatever menuconfig generates for you into sdkconfig.board

bertel
Posts: 23
Joined: Tue Feb 09, 2021 3:55 pm
Location: Tokyo

Re: Trouble with CMake and flash > 4 MB

Post by bertel » Wed Feb 24, 2021 5:26 am

jimmo wrote:
Wed Feb 24, 2021 12:48 am
I think this is the case. You will need to copy whatever menuconfig generates for you into sdkconfig.board
Thank you. I have copied the menuconfig file to .../ports/esp32/boards/GENERIC_SPIRAM/sdkconfig.board (I guess that's where it's supposed to go) which contains the proper instructions:

Code: Select all

 
# CONFIG_ESPTOOLPY_FLASHSIZE_4MB is not set
# CONFIG_ESPTOOLPY_FLASHSIZE_8MB is not set
CONFIG_ESPTOOLPY_FLASHSIZE_16MB=y
CONFIG_ESPTOOLPY_FLASHSIZE="16MB"
CONFIG_ESPTOOLPY_FLASHSIZE_DETECT=y
However, make remains unimpressed. It still fails with

Code: Select all

Partitions defined in '/usr/local/bin/micropython/micropython_1.14/ports/esp32/partitions.csv' occupy 16.0MB of flash (16777216 bytes) which does not fit in configured flash size 4MB. Change the flash size in menuconfig under the 'Serial Flasher Config' menu.
No unsurprisingly, .../esp32/build-GENERIC_SPIRAM/sdkconfig.combined contains

Code: Select all

# For cmake build
CONFIG_ESPTOOLPY_FLASHSIZE_4MB=y
#CONFIG_ESPTOOLPY_FLASHSIZE_16MB=y
CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions.csv"
And .../esp32/build-GENERIC_SPIRAM/sdkconfig likewise contains

Code: Select all

CONFIG_ESPTOOLPY_FLASHSIZE_4MB=y
# CONFIG_ESPTOOLPY_FLASHSIZE_8MB is not set
# CONFIG_ESPTOOLPY_FLASHSIZE_16MB is not set
CONFIG_ESPTOOLPY_FLASHSIZE="4MB"
It could just be a documentation issue, but "Change the flash size in menuconfig under the 'Serial Flasher Config' menu" clearly needs some additional documentation.

My current workaround is to replace .../esp32/boards.skdconfig.base with the output of menuconfig, which seems to work, but I'm not sure how well.

User avatar
jimmo
Posts: 2244
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Trouble with CMake and flash > 4 MB

Post by jimmo » Thu Feb 25, 2021 12:42 am

I was about to suggest raising an issue but I see you've done exactly that :)

https://github.com/micropython/micropython/issues/6950

Post Reply