ESP32 with 8MB Flash

[stanely]

I'm using a MakerFocus ESP32 Development Board Upgraded Version 8MB Flash. (I think it's actually a Heltec WiFi Kit, but don't know for sure.) I'm also using a pre-built MicroPython binary, (esp32-idf4-20200122-v1.12-68-g3032ae115.bin). It's one that's listed as "Firmware built with ESP-IDF v4.x, with support for BLE, but no LAN or PPP" for a generic ESP32.

Everything is working great and I'm very grateful to all the people here who made these tools available. Thank you!

I have a question about my available resources. When I check available free disk and RAM using this code (I think from stackoverflow):

Code: Select all

# Functions return disk free df()
# and memory free free()

import gc
import os

def df():
  s = os.statvfs('//')
  return ('{0} MB'.format((s[0]*s[3])/1048576))

def free(full=False):
  F = gc.mem_free()
  A = gc.mem_alloc()
  T = F+A
  P = '{0:.2f}%'.format(F/T*100)
  if not full: return P
  else : return ('Total:{0} Free:{1} ({2})'.format(T,F,P))

print('disk free: ', df())
print('memory free: ', free())

I get:

Code: Select all

disk free:  1.878906 MB
memory free:  65.72%

Using the disk os.ilistdir() command in REPL, I tallied up all of my files and came up with 72,016 bytes. The small amount of remaining disk makes me wonder if I'm getting any benefit from the 8MB upgraded flash? Should I be buying regular 4MB boards instead? They are significantly less expensive.

[tve]

I assume you need a different/modified partition table to expose the additional 4MB of flash...

[stanely]

I don't fully understand what you're saying, but I'm guessing you mean the existing pre-built MicroPython binaries only support 4MB. I get the gist that it would be possible to build an ESP32 binary with 8MB support. Is that right?

Is the size of available partition table something MicroPython could do on startup?

[russ_h]

As far as I know, it's set in the micropython/ports/esp32/partitions.csv file. You can change the last value of the vfs line to 0x600000. Recompile MicroPython then erase and re-flash your device.

Code: Select all

# Name,   Type, SubType, Offset,  Size, Flags
# Note: if you change the phy_init or app partition offset, make sure to change the offset in Kconfig.projbuild
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 0x180000,
vfs,      data, fat,     0x200000, 0x200000,

If you are not set up to recompile MicroPython on your own, my https://github.com/russhughes/TurtlePlotBot repo has a 1.11 firmware file I run on a 8mb Heltec device in https://github.com/russhughes/TurtlePlo ... icropython with brief instructions at https://penfold.owt.com/turtleplotbot/s ... 8mb-device

[stanely]

Thank you for your reply. No I am not set up to build MicroPython, so your offer for the binary is very much welcomed.

I think I mostly understand your reply, and have downloaded your turtlebot-8m binary. Because I already have MicroPython on my board, I don't have to erase my flash first, right? I've reflashed my MicroPython versions this way before.

From your instructions on penfold I'm also starting to get an idea how the ESP32 flash memory is laid out. MicroPython starts at 0x1000 and goes through 0x1FFFFF. That takes up the first 2MB. So the user FAT area starts at 0x200000. If I don't upload the turtleplotbot VFAT image, will MicroPython resize my disk to 4MB? Will my existing files be retained?

Thanks.

[stanely]

In the Espressif manual I found this:

SPI Flash Size
The SPI flash size is configured by writing a field in the software bootloader image header, flashed at offset 0x1000.

By default, the SPI flash size is detected by esptool.py when this bootloader is written to flash, and the header is updated with the correct size. Alternatively, it is possible to generate a fixed flash size by setting CONFIG_ESPTOOLPY_FLASHSIZE in project configuration.

If it is necessary to override the configured flash size at runtime, it is possible to set the chip_size member of the g_rom_flashchip structure. This size is used by esp_flash_* functions (in both software & ROM) to check the bounds.

Does this mean that esptool.py is supposed to figure out my flash size on boot but isn't? Is there an esptool switch that can set the FLASHSIZE?

[russ_h]

I think I mostly understand your reply, and have downloaded your turtlebot-8m binary. Because I already have MicroPython on my board, I don't have to erase my flash first, right? I've reflashed my MicroPython versions this way before.

From your instructions on penfold I'm also starting to get an idea how the ESP32 flash memory is laid out. MicroPython starts at 0x1000 and goes through 0x1FFFFF. That takes up the first 2MB. So the user FAT area starts at 0x200000. If I don't upload the turtleplotbot VFAT image, will MicroPython resize my disk to 4MB? Will my existing files be retained?

Thanks.

MicroPython won't resize the existing VFAT filesystem on it's own, you'll have to erase it or use something like this to reformat it and re-upload your files:

Code: Select all

import os
os.umount('/')
os.VfsFat.mkfs(bdev)
os.mount(bdev, '/')

When I ran into this issue I used the esptool to pull a copy of the existing smaller VFAT filesystem off the ESP32, then reformatted the partition to the new larger size. I used the esptool to pull another copy of the new larger but empty VFAT partition. I then mounted both partition images on a linux pc and copied the files from the old image to the new image. I finally re-flashed just the new larger VFAT image back on to the ESP32. Someone else may know of an easier method but this how I did it.

Does this mean that esptool.py is supposed to figure out my flash size on boot but isn't? Is there an esptool switch that can set the FLASHSIZE?

It probably does figure out the size, but that wont change the size of the partition table or the VFAT filesystem in the ESP32.

[stanely]

Woohooo! Thank you, @russ_h!!!

disk free: 3.878906 MB
memory free: 55.12%

Just ran upip to reload my libraries and copied all my files and everything works!

I really appreciate your clear and concise assistance.