Trouble with CMake and flash > 4 MB

All ESP32 boards running MicroPython.
Target audience: MicroPython users with an ESP32 board.
bertel
Posts: 27
Joined: Tue Feb 09, 2021 3:55 pm
Location: Tokyo

Trouble with CMake and flash > 4 MB

Post by bertel » Mon Feb 22, 2021 2:36 pm

I am still (see my comment in https://forum.micropython.org/viewtopi ... 18&t=9820 battling with the new Cmake version when it comes to larger flash sizes. It simply won’t compile for anything larger than 4MB
Previously, I put these into Makefile

Code: Select all

BOARD ?= GENERIC_SPIRAM
FLASH_SIZE ?= 16MB
My partitions.csv looked like this:

Code: Select all

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 0x180000,
vfs,      data, fat,     0x190000, 0xE00000
It happily compiled.
The Cmake version does not seem to understand the FLASH_SIZE ?= 16MB instruction. With the settings above, it complains that the partition “does not fit in configured flash size 4MB. Change the flash size in menuconfig under the 'Serial Flasher Config' menu.”
I have done so repeatedly, without luck. The Cmake setup does not seem to pick up on the menuconfig instructions.
If I set flash to 16MB in menuconfig, …/build/sdkconfig dutifully reports:

Code: Select all

# CONFIG_ESPTOOLPY_FLASHSIZE_1MB is not set
# CONFIG_ESPTOOLPY_FLASHSIZE_2MB is not set
# CONFIG_ESPTOOLPY_FLASHSIZE_4MB is not set
# CONFIG_ESPTOOLPY_FLASHSIZE_8MB is not set
CONFIG_ESPTOOLPY_FLASHSIZE_16MB=y
CONFIG_ESPTOOLPY_FLASHSIZE="16MB"

I do a make clean, which leaves …/build-GENERIC_SPIRAM empty
make produces this:

Code: Select all

idf.py -D MICROPY_BOARD=GENERIC_SPIRAM -B build-GENERIC_SPIRAM build
Executing action: all (aliases: build)
Running cmake in directory /usr/local/bin/micropython/micropython_1.14/ports/esp32/build-GENERIC_SPIRAM
Executing "cmake -G Ninja -DPYTHON_DEPS_CHECKED=1 -DESP_PLATFORM=1 -DMICROPY_BOARD=GENERIC_SPIRAM -DCCACHE_ENABLE=0 /usr/local/bin/micropython/micropython_1.14/ports/esp32"...
-- Found Git: /usr/bin/git (found version "2.25.1")
-- IDF_TARGET not set, using default target: esp32
-- The C compiler identification is GNU 8.4.0
-- The CXX compiler identification is GNU 8.4.0
-- The ASM compiler identification is GNU
-- Found assembler: /root/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /root/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /root/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Project version: v1.14-80-g75db0b907-dirty
-- Building ESP-IDF components for target esp32
-- Project sdkconfig file /usr/local/bin/micropython/micropython_1.14/ports/esp32/build-GENERIC_SPIRAM/sdkconfig
Loading defaults file /usr/local/bin/micropython/micropython_1.14/ports/esp32/build-GENERIC_SPIRAM/sdkconfig.combined...
/usr/local/bin/micropython/micropython_1.14/ports/esp32/build-GENERIC_SPIRAM/sdkconfig.combined:9 CONFIG_COMPILER_OPTIMIZATION_LEVEL_RELEASE was replaced with CONFIG_COMPILER_OPTIMIZATION_SIZE
/tmp/confgen_tmphk6dbqrs:61 line was updated to CONFIG_BTDM_CTRL_MODE_BR_EDR_ONLY=n
/tmp/confgen_tmphk6dbqrs:62 line was updated to CONFIG_BTDM_CTRL_MODE_BTDM=n
CMake Deprecation Warning at /root/esp4/esp-idf-v4.1.1/components/mbedtls/mbedtls/CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- Found PythonInterp: /root/.espressif/python_env/idf4.1_py3.8_env/bin/python (found version "3.8.5")
-- Found Perl: /usr/bin/perl (found version "5.30.0")
-- Adding linker script /usr/local/bin/micropython/micropython_1.14/ports/esp32/build-GENERIC_SPIRAM/esp-idf/esp32/esp32_out.ld
-- Adding linker script /root/esp4/esp-idf-v4.1.1/components/esp32/ld/esp32.project.ld.in
-- Adding linker script /root/esp4/esp-idf-v4.1.1/components/esp32/ld/esp32.peripherals.ld
-- Adding linker script /root/esp4/esp-idf-v4.1.1/components/esp_rom/esp32/ld/esp32.rom.ld
-- Adding linker script /root/esp4/esp-idf-v4.1.1/components/esp_rom/esp32/ld/esp32.rom.libgcc.ld
-- Adding linker script /root/esp4/esp-idf-v4.1.1/components/esp_rom/esp32/ld/esp32.rom.newlib-data.ld
-- Adding linker script /root/esp4/esp-idf-v4.1.1/components/esp_rom/esp32/ld/esp32.rom.syscalls.ld
-- Found Python3: /usr/local/bin/micropython/micropython_1.14/ports/esp32/build-venv/bin/python3 (found version "3.8.5") found components: Interpreter
-- Components: app_trace app_update asio bootloader bootloader_support bt cbor coap console cxx driver efuse esp-tls esp32 esp_adc_cal esp_common esp_eth esp_event esp_gdbstub esp_http_client esp_http_server esp_https_ota esp_https_server esp_local_ctrl esp_netif esp_ringbuf esp_rom esp_serial_slave_link esp_websocket_client esp_wifi espcoredump esptool_py expat fatfs freemodbus freertos heap idf_test jsmn json libsodium log lwip main mbedtls mdns mqtt newlib nghttp nvs_flash openssl partition_table perfmon protobuf-c protocomm pthread sdmmc soc spi_flash spiffs tcp_transport tcpip_adapter ulp unity vfs wear_levelling wifi_provisioning wpa_supplicant xtensa
-- Component paths: /root/esp4/esp-idf-v4.1.1/components/app_trace /root/esp4/esp-idf-v4.1.1/components/app_update /root/esp4/esp-idf-v4.1.1/components/asio /root/esp4/esp-idf-v4.1.1/components/bootloader /root/esp4/esp-idf-v4.1.1/components/bootloader_support /root/esp4/esp-idf-v4.1.1/components/bt /root/esp4/esp-idf-v4.1.1/components/cbor /root/esp4/esp-idf-v4.1.1/components/coap /root/esp4/esp-idf-v4.1.1/components/console /root/esp4/esp-idf-v4.1.1/components/cxx /root/esp4/esp-idf-v4.1.1/components/driver /root/esp4/esp-idf-v4.1.1/components/efuse /root/esp4/esp-idf-v4.1.1/components/esp-tls /root/esp4/esp-idf-v4.1.1/components/esp32 /root/esp4/esp-idf-v4.1.1/components/esp_adc_cal /root/esp4/esp-idf-v4.1.1/components/esp_common /root/esp4/esp-idf-v4.1.1/components/esp_eth /root/esp4/esp-idf-v4.1.1/components/esp_event /root/esp4/esp-idf-v4.1.1/components/esp_gdbstub /root/esp4/esp-idf-v4.1.1/components/esp_http_client /root/esp4/esp-idf-v4.1.1/components/esp_http_server /root/esp4/esp-idf-v4.1.1/components/esp_https_ota /root/esp4/esp-idf-v4.1.1/components/esp_https_server /root/esp4/esp-idf-v4.1.1/components/esp_local_ctrl /root/esp4/esp-idf-v4.1.1/components/esp_netif /root/esp4/esp-idf-v4.1.1/components/esp_ringbuf /root/esp4/esp-idf-v4.1.1/components/esp_rom /root/esp4/esp-idf-v4.1.1/components/esp_serial_slave_link /root/esp4/esp-idf-v4.1.1/components/esp_websocket_client /root/esp4/esp-idf-v4.1.1/components/esp_wifi /root/esp4/esp-idf-v4.1.1/components/espcoredump /root/esp4/esp-idf-v4.1.1/components/esptool_py /root/esp4/esp-idf-v4.1.1/components/expat /root/esp4/esp-idf-v4.1.1/components/fatfs /root/esp4/esp-idf-v4.1.1/components/freemodbus /root/esp4/esp-idf-v4.1.1/components/freertos /root/esp4/esp-idf-v4.1.1/components/heap /root/esp4/esp-idf-v4.1.1/components/idf_test /root/esp4/esp-idf-v4.1.1/components/jsmn /root/esp4/esp-idf-v4.1.1/components/json /root/esp4/esp-idf-v4.1.1/components/libsodium /root/esp4/esp-idf-v4.1.1/components/log /root/esp4/esp-idf-v4.1.1/components/lwip /usr/local/bin/micropython/micropython_1.14/ports/esp32/main /root/esp4/esp-idf-v4.1.1/components/mbedtls /root/esp4/esp-idf-v4.1.1/components/mdns /root/esp4/esp-idf-v4.1.1/components/mqtt /root/esp4/esp-idf-v4.1.1/components/newlib /root/esp4/esp-idf-v4.1.1/components/nghttp /root/esp4/esp-idf-v4.1.1/components/nvs_flash /root/esp4/esp-idf-v4.1.1/components/openssl /root/esp4/esp-idf-v4.1.1/components/partition_table /root/esp4/esp-idf-v4.1.1/components/perfmon /root/esp4/esp-idf-v4.1.1/components/protobuf-c /root/esp4/esp-idf-v4.1.1/components/protocomm /root/esp4/esp-idf-v4.1.1/components/pthread /root/esp4/esp-idf-v4.1.1/components/sdmmc /root/esp4/esp-idf-v4.1.1/components/soc /root/esp4/esp-idf-v4.1.1/components/spi_flash /root/esp4/esp-idf-v4.1.1/components/spiffs /root/esp4/esp-idf-v4.1.1/components/tcp_transport /root/esp4/esp-idf-v4.1.1/components/tcpip_adapter /root/esp4/esp-idf-v4.1.1/components/ulp /root/esp4/esp-idf-v4.1.1/components/unity /root/esp4/esp-idf-v4.1.1/components/vfs /root/esp4/esp-idf-v4.1.1/components/wear_levelling /root/esp4/esp-idf-v4.1.1/components/wifi_provisioning /root/esp4/esp-idf-v4.1.1/components/wpa_supplicant /root/esp4/esp-idf-v4.1.1/components/xtensa
-- Configuring done
-- Generating done
-- Build files have been written to: /usr/local/bin/micropython/micropython_1.14/ports/esp32/build-GENERIC_SPIRAM
Running ninja in directory /usr/local/bin/micropython/micropython_1.14/ports/esp32/build-GENERIC_SPIRAM
Executing "ninja all"...
[7/1206] Generating ../../partition_table/partition-table.bin
FAILED: partition_table/partition-table.bin
cd /usr/local/bin/micropython/micropython_1.14/ports/esp32/build-GENERIC_SPIRAM/esp-idf/partition_table && /root/.espressif/python_env/idf4.1_py3.8_env/bin/python /root/esp4/esp-idf-v4.1.1/components/partition_table/gen_esp32part.py -q --offset 0x8000 --flash-size 4MB /usr/local/bin/micropython/micropython_1.14/ports/esp32/partitions.csv /usr/local/bin/micropython/micropython_1.14/ports/esp32/build-GENERIC_SPIRAM/partition_table/partition-table.bin
Partitions defined in '/usr/local/bin/micropython/micropython_1.14/ports/esp32/partitions.csv' occupy 16.0MB of flash (16777216 bytes) which does not fit in configured flash size 4MB. Change the flash size in menuconfig under the 'Serial Flasher Config' menu.
[72/1206] Building C object esp-idf/mbedtls/mbedtls/library/CMakeFiles/mbedtls.dir/ssl_tls.c.obj
ninja: build stopped: subcommand failed.
ninja failed with exit code 1
make: *** [Makefile:26: all] Error 2

According to the output above, the project sdkconfig file sits in
…/build-GENERIC_SPIRAM/sdkconfig
However, the pertinent part of the sdkconfig file now reads like this:

Code: Select all

# CONFIG_ESPTOOLPY_FLASHSIZE_1MB is not set
# CONFIG_ESPTOOLPY_FLASHSIZE_2MB is not set
CONFIG_ESPTOOLPY_FLASHSIZE_4MB=y
# CONFIG_ESPTOOLPY_FLASHSIZE_8MB is not set
# CONFIG_ESPTOOLPY_FLASHSIZE_16MB is not set
CONFIG_ESPTOOLPY_FLASHSIZE="4MB"
CONFIG_ESPTOOLPY_FLASHSIZE_DETECT=y

Apparently, when the sdkconfig file is fetched from the …/build directory, the flash size is changed to 4MB. Most likely, the sdkconfig file produced by menuconfig is ignored, and a default file is copied from somewhere else.

Declare me stumped. Any help would be appreciated.

bertel
Posts: 27
Joined: Tue Feb 09, 2021 3:55 pm
Location: Tokyo

Re: Trouble with CMake and flash > 4 MB

Post by bertel » Mon Feb 22, 2021 4:10 pm

I found what is happening, but I don’t why:

/esp32/boards/sdkconfig.base is being used instead of the sdkconfig file produced by menuconfig.
In /esp32/boards/sdkconfig.base, if I make this edit, the compile finishes successfully:

Code: Select all

#CONFIG_ESPTOOLPY_FLASHSIZE_4MB=y
CONFIG_ESPTOOLPY_FLASHSIZE_16MB=y
The above, and other observations, make me doubt that any changes made via menuconfig are being used.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Trouble with CMake and flash > 4 MB

Post by jimmo » Wed Feb 24, 2021 12:48 am

bertel wrote:
Mon Feb 22, 2021 4:10 pm
The above, and other observations, make me doubt that any changes made via menuconfig are being used.
I think this is the case. You will need to copy whatever menuconfig generates for you into sdkconfig.board

bertel
Posts: 27
Joined: Tue Feb 09, 2021 3:55 pm
Location: Tokyo

Re: Trouble with CMake and flash > 4 MB

Post by bertel » Wed Feb 24, 2021 5:26 am

jimmo wrote:
Wed Feb 24, 2021 12:48 am
I think this is the case. You will need to copy whatever menuconfig generates for you into sdkconfig.board
Thank you. I have copied the menuconfig file to .../ports/esp32/boards/GENERIC_SPIRAM/sdkconfig.board (I guess that's where it's supposed to go) which contains the proper instructions:

Code: Select all

 
# CONFIG_ESPTOOLPY_FLASHSIZE_4MB is not set
# CONFIG_ESPTOOLPY_FLASHSIZE_8MB is not set
CONFIG_ESPTOOLPY_FLASHSIZE_16MB=y
CONFIG_ESPTOOLPY_FLASHSIZE="16MB"
CONFIG_ESPTOOLPY_FLASHSIZE_DETECT=y
However, make remains unimpressed. It still fails with

Code: Select all

Partitions defined in '/usr/local/bin/micropython/micropython_1.14/ports/esp32/partitions.csv' occupy 16.0MB of flash (16777216 bytes) which does not fit in configured flash size 4MB. Change the flash size in menuconfig under the 'Serial Flasher Config' menu.
No unsurprisingly, .../esp32/build-GENERIC_SPIRAM/sdkconfig.combined contains

Code: Select all

# For cmake build
CONFIG_ESPTOOLPY_FLASHSIZE_4MB=y
#CONFIG_ESPTOOLPY_FLASHSIZE_16MB=y
CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions.csv"
And .../esp32/build-GENERIC_SPIRAM/sdkconfig likewise contains

Code: Select all

CONFIG_ESPTOOLPY_FLASHSIZE_4MB=y
# CONFIG_ESPTOOLPY_FLASHSIZE_8MB is not set
# CONFIG_ESPTOOLPY_FLASHSIZE_16MB is not set
CONFIG_ESPTOOLPY_FLASHSIZE="4MB"
It could just be a documentation issue, but "Change the flash size in menuconfig under the 'Serial Flasher Config' menu" clearly needs some additional documentation.

My current workaround is to replace .../esp32/boards.skdconfig.base with the output of menuconfig, which seems to work, but I'm not sure how well.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Trouble with CMake and flash > 4 MB

Post by jimmo » Thu Feb 25, 2021 12:42 am

I was about to suggest raising an issue but I see you've done exactly that :)

https://github.com/micropython/micropython/issues/6950

User avatar
RobH
Posts: 91
Joined: Fri Mar 23, 2018 3:37 pm
Location: Netherlands
Contact:

Re: Trouble with CMake and flash > 4 MB

Post by RobH » Wed Mar 03, 2021 2:02 pm

@bertel: I'm also trying to enable 8MB PSRAM.
My current workaround is to replace .../esp32/boards.skdconfig.base with the output of menuconfig, which seems to work,
Could you be somewhat more specific about this, please?
Which 'output of menuconfig' has to replace .../esp32/boards/skdconfig.base ?

bertel
Posts: 27
Joined: Tue Feb 09, 2021 3:55 pm
Location: Tokyo

Re: Trouble with CMake and flash > 4 MB

Post by bertel » Fri Mar 05, 2021 3:56 am

@robh:

With a properly set-up IDF4 environment,

Code: Select all

idf.py menuconfig
will (well, should) take you into a menu structure that allows you to make various settings before the compile. “Serial flasher config” for instance allows you to choose FLASH size up to 16MB.
Menuconfig produces an sdkconfig file in the ../esp32/build directory. However, that file is ignored until it is renamed to sdkconfig.base, and placed into the …/esp32/boards directory. At least, that’s my workaround, see above.
I could not find anything regarding RAM in these settings. RAM is dealt with in the CONFIG_SPIRAM board, which calls sdkconfig.spiram. That file simply enables SPIRAM.

Are you sure you want to enable 8MB of PSRAM, as opposed to 8MB of FLASH? According to expressif https://docs.espressif.com/projects/esp ... l-ram.html the "ESP32 has the ability to also use up to 4 MB of external SPI RAM memory." I think that means that 8MB of SPI RAMare out.

For 8 MB of flash, you would
- Select 8 MB as flash size in menuconfig
- Copy …/esp32/build/sdkconfig to …/esp32/boards/sdkconfig.base
- Change the last line of …/esp32/partitions.csv to

Code: Select all

vfs,      data, fat,     0x200000, 0x600000,
- In Makefile, set

Code: Select all

 BOARD ?= GENERIC_SPIRAM 
make, and make deploy

User avatar
RobH
Posts: 91
Joined: Fri Mar 23, 2018 3:37 pm
Location: Netherlands
Contact:

Re: Trouble with CMake and flash > 4 MB

Post by RobH » Sun Mar 07, 2021 9:19 am

@bertel

Thank you very much for your detailed 'instructions'! Unfortunately these seems not to work for me (could be my fault of course).
I don't know if it is relevant, but anyway:
- My board is a TTGO-T8 V1.7 with 4MB flash(chip 25Q32) and 8 MB PSRAM (chip ESP PSRAM64H)
- Commandline 'idf.py --version' reports: ESP-IDF v4.4-dev-4-g73db14240
When I change (only) the flash size to 8MB (leaving partitions.csv unchanged) , copy build/sdkconfig to boards/sdkconfig.base, run make and make deploy, the boot process of the ESP32 gets into a loop, with among others the message:

Code: Select all

E (556) spi_flash: Detected size(4096k) smaller than the size in the binary image header(8192k). Probe failed.
I have no idea why this happens. Has the 'Probe failed' todo with the '[*] Detect flash size when flashing bootloader' in menuconfig?
I haven't tried your suggestion for the partition change yet.

bertel
Posts: 27
Joined: Tue Feb 09, 2021 3:55 pm
Location: Tokyo

Re: Trouble with CMake and flash > 4 MB

Post by bertel » Wed Mar 10, 2021 10:57 am

RobH wrote:
Sun Mar 07, 2021 9:19 am

When I change (only) the flash size to 8MB (leaving partitions.csv unchanged) , copy build/sdkconfig to boards/sdkconfig.base, run make and make deploy, the boot process of the ESP32 gets into a loop,
.....
I haven't tried your suggestion for the partition change yet.
Well, you absolutely need to change partitions.csv as well. For 8MB, it should read something like this:

Code: Select all

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 0x180000,
vfs,      data, fat,     0x200000, 0x600000,
Don't deploy if make throws an error. It won't work.

wangshujun@tom.com
Posts: 61
Joined: Fri Feb 15, 2019 9:22 am

Re: Trouble with CMake and flash > 4 MB

Post by wangshujun@tom.com » Wed Mar 10, 2021 2:35 pm

It seems that your compilation is correct, but the flash model on the board is 4 megabytes. When writing firmware with larger configuration, this error prompt should appear

I encountered this error when modifying the 1.13 firmware

Post Reply