How to get the full capacity of an 8MB or 16MB flash chip?

All ESP32 boards running MicroPython.
Target audience: MicroPython users with an ESP32 board.
bertel
Posts: 27
Joined: Tue Feb 09, 2021 3:55 pm
Location: Tokyo

How to get the full capacity of an 8MB or 16MB flash chip?

Post by bertel » Tue Feb 09, 2021 4:11 pm

I could use some help with accessing the flash memory on my esp32 boards.

I have a bunch of factory expressif ESP32-DEVKITC-VE/VIE WROVER boardlets with 4 MB RAM and 8MB. I even could lay my hands on promising Chinese esp32 boards with 4 MB of RAM, 16 MB of flash and a CD slot thrown in. All in the hope of having more flash than the 1.89 MB that came with the DOIT boards I started with. I use them all with Micropython, because that’s what I know. However, I can’t get to the extra flash memory.

When I flash the boards with the latest GENERIC-SPIRAM firmware provided by https://micropython.org/download/esp32/, I get (nearly) the full 4 MB of RAM, but flash memory is reported as 1.89 MB, and not as close to 8 MB on the WROVER boards, or 16 MB on the latest Chinese imports. So, off to recompiling the firmware.

According to the recipes on the web, once the toolchain is set up, the job is as easy as adjusting the last line of partitions.csv and a make.
So, I make partitions.csv read like this for the 8MB flash chip:

Code: Select all

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 0x180000,
vfs,      data, fat,     0x200000, 0x600000,
I compile, flash, boot, and I get:

Code: Select all

rst:0x3 (SW_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:5048
load:0x40078000,len:11608
load:0x40080400,len:6120
entry 0x400806bc
;31mE (65) flash_parts: partition 3 invalid - offset 0x200000 size 0x600000 exceeds flash chip size 0x400000m
;31mE (66) boot: Failed to verify partition table
;31mE (69) boot: load partition table error!
So apparently, someone is thinking I only have a 4MB flash chip. Alright, lets humor whoever does the thinking and adjust the last line to:

Code: Select all

vfs,      data, fat,     0x200000, 0x400000,
Recompile, flash, and I get:

Code: Select all

partition 3 invalid - offset 0x200000 size 0x400000 exceeds flash chip size 0x400000
Really?
To make a long story (and many recompiles) short, the only successful last line in partitions.csv is

Code: Select all

vfs,      data, fat,     0x190000, 0x270000
It yields a puny 2.43 MB of flash memory. If I go to

Code: Select all

vfs,      data, fat,     0x190000, 0x280000
the board crashes again on reboot, saying that the requested 0x280000 exceed the (bogus) 0x400000 delivered by a flash chip rated at 8M. Of course, the results with the 16MB board are the same, only more disappointing.

Apparently, things are not as simple as changing the last line of partitions.csv.

Does anyone have an idea how to
- Recognize the full size of the flash chip?
- Hand the full amount of memory to the file system?

Thank you!
Last edited by bertel on Wed Feb 10, 2021 8:38 am, edited 1 time in total.

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: How to get the full capacity of an 8MB or 16MB flash chip?

Post by Roberthh » Tue Feb 09, 2021 5:52 pm

I recall having successfully made the changes to the partition map. I' ll look into it and tell later.

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: How to get the full capacity of an 8MB or 16MB flash chip?

Post by Roberthh » Tue Feb 09, 2021 7:18 pm

I just rebuilt that code, with exactly the same table, for a 8 MB device

Code: Select all

# Name,   Type, SubType, Offset,  Size, Flags
# Note: if you change the phy_init or app partition offset, make sure to change the offset in Kconfig.projbuild
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 0x180000,
vfs,      data, fat,     0x200000, 0x600000,
And it works. Not much help for you. You could try to erase flash first with make erase.
Just tell what the following call returns:

import esp
esp.flash_size()

If it is not the physical size of the flash chip, then there is a problem with the detection of it's size.

bertel
Posts: 27
Joined: Tue Feb 09, 2021 3:55 pm
Location: Tokyo

Re: How to get the full capacity of an 8MB or 16MB flash chip?

Post by bertel » Wed Feb 10, 2021 9:37 am

Hi, Robert.

Thank you for the quick reply. Here are the results of the tests.


"Stock image" means sp32spiram-idf4-20210206-unstable-v1.14-9-g9dedcf122.bin
"Last working image" means recompiled MicroPython v1.13-284-g4eaebc198-dirty (spiram) with last line of partitions.csv set to "vfs, data, fat, 0x190000, 0x270000" ( 0x200000, 0x270000 would crash - prior post corrected). Anything higher
than 270000 crashes on both parts.

All flashed with prior esptool erase_flash

"Free flash file space" means result of os.statvfs('//')[0]*os.statvfs('//')[3]/1048576


=======================================================================
Chinese 4 MB RAM, 16MB flash part, stock image:
esp.flash_size(): 4194304
Free flash file space: 1.988281

Expressif ESP32-WROVER-IE, 4MB RAM, 8 MB flash, stock image
esp.flash_size(): 4194304
Free flash file space: 1.988281

=======================================================================

Chinese 4 MB RAM, 16MB flash part, last working image
esp.flash_size(): 4194304
Free flash file space: 2.425781

Expressif ESP32-WROVER-IE, 4MB RAM, 8 MB flash, last working image
esp.flash_size(): 4194304
Free flash file space: 2.425781

=======================================================================

I think we can definitely state that the size of the flash chip is not detected correctly.
The flash chip on the 16MB part is a WINBOND 25Q128FVSQ, which has 16MB capacity.
The flash chip of the WROVER is under the metal can, but I guess we can take Expressif's word for it that it's in fact 8MBs

Tokyo grüßt NRW, Bertel

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: How to get the full capacity of an 8MB or 16MB flash chip?

Post by Roberthh » Wed Feb 10, 2021 10:35 am

It's an information taken from flash and somehow and somewhere written to it. I searched for that before and could not find it. The only thing I found is the verification step in the bootloader.
What you could try to set the 4th byte in firmware.bin either from 20h to 23h or from 20h to 30h for an 8 MB device. 20 to 24 or 20 to 40 for 16 MB. That seems to be the place where a flash size is configured. But that byte is identical for my images, so it's kind oi weird..

bertel
Posts: 27
Joined: Tue Feb 09, 2021 3:55 pm
Location: Tokyo

Re: How to get the full capacity of an 8MB or 16MB flash chip?

Post by bertel » Wed Feb 10, 2021 1:10 pm

Thank’s for the reply. O.K., I’ll break out the hex editor, and I will keep digging. My 4th byte is 20 also …..

In the meantime, I found https://github.com/loboris/MicroPython ... psRAM_LoBo. Using menuconfig, it is highly configurable (and likewise screw-uppable.) It comes with a bunch of promising features, including high-in-demand (😊) ssh (untested.) I could compile it successfully. It recognizes the 16MB chip. I couldn’t get it beyond 2.5 MB filespace though, probably due to the screw-uppable part (and also because it somehow picks up on an old partition table that won’t get erased no matter what.)

However, the last commit was 3 years ago, and the project seems abandoned. Also, it is yet another Micropython flavor that won’t come with modules we learned to love, for instance esp, and some we might love cause compile errs. I don’t think investing time into that dead-end project is worth it. However, the menu system, and a few modules might be worth a look.

Tokyo grüßt NRW, Bertel

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: How to get the full capacity of an 8MB or 16MB flash chip?

Post by Roberthh » Wed Feb 10, 2021 1:14 pm

Even the Loboris port cannot get across the problem with wrong-detected flash chips. It uses the same bootloader.

bertel
Posts: 27
Joined: Tue Feb 09, 2021 3:55 pm
Location: Tokyo

Re: How to get the full capacity of an 8MB or 16MB flash chip?

Post by bertel » Wed Feb 10, 2021 2:00 pm

Lobo's ./BUILD.sh flash, which appears to do the final partitioning, somehow seems to pick up on the correct flash size, but on boot, it is all forgotten, and the partition size is down to 2.5M .....

Code: Select all

=========================================
Flashing MicroPython firmware to ESP32...
=========================================
AR build/libnmea/liblibnmea.a
LD build/MicroPython.elf
esptool.py v2.5.0
Flashing binaries to serial port /dev/ttyUSB0 (app at offset 0x10000 )...
esptool.py v2.5.0
Serial port /dev/ttyUSB0
Connecting....
Chip is ESP32D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse
MAC: 10:52:1c:63:99:64
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
Configuring flash size...
Auto-detected Flash size: 16MB
Compressed 19488 bytes to 11580...
Wrote 19488 bytes (11580 compressed) at 0x00001000 in 0.2 seconds (effective 1026.6 kbit/s)...
Hash of data verified.
Compressed 144 bytes to 69...
Wrote 144 bytes (69 compressed) at 0x0000f000 in 0.0 seconds (effective 659.5 kbit/s)...
Hash of data verified.
Compressed 1490016 bytes to 862151...
Wrote 1490016 bytes (862151 compressed) at 0x00010000 in 13.7 seconds (effective 868.4 kbit/s)...
Hash of data verified.
Compressed 3072 bytes to 103...
Wrote 3072 bytes (103 compressed) at 0x00008000 in 0.0 seconds (effective 11422.8 kbit/s)...
Hash of data verified.

Leaving...
Good news: The stock image allows me to mount an SD card on a board that has one. With that, filesize can be up to 16G (it doesn't recognize bigger SDs)

bertel
Posts: 27
Joined: Tue Feb 09, 2021 3:55 pm
Location: Tokyo

Re: How to get the full capacity of an 8MB or 16MB flash chip?

Post by bertel » Wed Feb 10, 2021 3:51 pm

Yay!

Code: Select all

>>> import esp
>>> esp.flash_size()
16777216
Deep in the Makefile, there is a FLASH_SIZE setting, and when set to

Code: Select all

FLASH_SIZE ?= 16MB
the flash size is reflected properly. Apparently, there is no automatic detection. On top of it, the PYTHON2 setting of the Makefile was, well, python2. Python2 has long been banished from my system. For some reason, there was no compile error, I caught it only after I set verbose on. Now it's

Code: Select all

PYTHON2 ?= python3
and all is good. Thank you so much for the assist! Sorry for taking so much of your time, I'm not much of a C++ programmer.

B in T.

bertel
Posts: 27
Joined: Tue Feb 09, 2021 3:55 pm
Location: Tokyo

Re: How to get the full capacity of an 8MB or 16MB flash chip?

Post by bertel » Wed Feb 10, 2021 4:12 pm

Reconfirmed that there is no flash size checking. When I flashed an 8MB device with firmware compiled for 16M, the 8M part happily lied about having 16M of flash, and it didn't even flush.

Post Reply