What is maximum valid heap size

All ESP8266 boards running MicroPython.
Official boards are the Adafruit Huzzah and Feather boards.
Target audience: MicroPython users with an ESP8266 board.
Post Reply
manseekingknowledge
Posts: 61
Joined: Sun Oct 29, 2017 5:14 pm

What is maximum valid heap size

Post by manseekingknowledge » Fri Aug 02, 2019 7:41 am

main.c has this line:

Code: Select all

STATIC char heap[38 * 1024];
I've slowly been increasing this in an effort to process larger strings. I've seen some behavior I don't understand and I'm hoping someone can enlighten me.

When I have the ESP running in station mode I'm able to have the heap set to 47KB and things seem mostly stable. I do get a random reset which I asked about here, but I'm not sure if that is related or not.

When I have the ESP running as an AP I'm only able to have the heap set to 43KB before it becomes unstable. In this situation I don't even have any other code running. The only thing I've done is configure the AP. So this is confusing to me.

What is the max heap size that can be specified in main.c? Is disabling features in mpconfigport.h the best (only?) way to allow the heap size to be increased?

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: What is maximum valid heap size

Post by jimmo » Fri Aug 02, 2019 12:49 pm

I'm fairly sure the heap is fine tuned to match the available RAM of the target device, leaving enough space for stack, data, bss, etc. I'm less familiar with the non-STM32 ports (i.e. ESP*) which use an rtos.

But I suspect the issues you're seeing are memory corruption (stack and bss) by overrunning the heap.

Disabling features in mpconfigport won't likely help very much because features mostly take ROM (i.e. code space), and only use RAM when you actually call them from MicroPython. (There are a few exceptions to this, i.e. static buffers etc).

Unfortunately the ESP8266 just doesn't have that much RAM available for the heap.

manseekingknowledge
Posts: 61
Joined: Sun Oct 29, 2017 5:14 pm

Re: What is maximum valid heap size

Post by manseekingknowledge » Wed Aug 21, 2019 1:11 am

jimmo wrote:
Fri Aug 02, 2019 12:49 pm
I'm fairly sure the heap is fine tuned to match the available RAM of the target device, leaving enough space for stack, data, bss, etc. I'm less familiar with the non-STM32 ports (i.e. ESP*) which use an rtos.

But I suspect the issues you're seeing are memory corruption (stack and bss) by overrunning the heap.

Disabling features in mpconfigport won't likely help very much because features mostly take ROM (i.e. code space), and only use RAM when you actually call them from MicroPython. (There are a few exceptions to this, i.e. static buffers etc).

Unfortunately the ESP8266 just doesn't have that much RAM available for the heap.
How about the features listed in "Makefile"? Does disabling MICROPY_PY_USSL, MICROPY_SSL_AXTLS, MICROPY_PY_BTREE get any RAM back? If not then it seems like this thread and a few other places are spreading bad info.

A comment left by @pfalcon here makes it seem like turning those features off does nothing.

I've done my own testing, and things seem to be better (longer intervals in between errors), so it seems to me that maybe it is doing something, but I have on conclusive evidence and my changes have not been isolated.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: What is maximum valid heap size

Post by jimmo » Wed Aug 21, 2019 6:38 am

When you build the firmware image, it will tell you the size of the text, data, and bss segments. The text segment goes in flash, the data and bss go in RAM. (Well, the data segment goes in both).

data comes from any object with static lifetime that has an initialised value, e.g.:

Code: Select all

int foo = MODE_FOO;
bss comes from global unintialised data, e.g.

Code: Select all

char global_buffer[20];
The heap itself is in BSS. (38kiB by default).

So disabling any code that variables in data/bss will make more RAM available for the heap.

However, other than a few static buffers in some specific places, not much code in micropython uses global variables. Things tend to be allocated on the heap instead. So pfalcon is right that there is very little RAM to be gained by disabling functionality.

But let's do some science. Compiling ESP8266 firmware at master:

Code: Select all

   text	   data	    bss	    dec	    hex	filename
 586468	   1040	  66368	 653876	  9fa34	build/firmware.elf
Disabling btree:

Code: Select all

   text	   data	    bss	    dec	    hex	filename
 567244	   1024	  66368	 634636	  9af0c	build/firmware.elf
The disabling ussl & axtls

Code: Select all

   text	   data	    bss	    dec	    hex	filename
 539232	   1024	  66368	 606624	  941a0	build/firmware.elf
So you can see there are some significant reductions to the text segment (i.e. flash) due to the reduced code size. But no change to the data/bss.

The ESP8266 has 80 kiB of RAM total, so this leaves 15552 kiB for the stack. (i.e. not very much)

But, to look at this another way -- rather than guessing things to disable and removing them to see what happens, we can look at what's actually using bss and data.

Code: Select all

$ xtensa-lx106-elf-objdump -j .bss -j .data -t build/firmware.elf | sort -k5
3ffea890 g     O .bss	00000040 wDevCtrl
3ffee72c g     O .bss	00000074 chip6_phy_init_ctrl
3fff835c l     O .bss	00000090 pwm_single_toggle
3ffee698 g     O .bss	00000094 chip6_sleep_params
3ffedf20 g     O .bss	0000009c gScanStruct
3ffe9e60 g     O .bss	000000e8 pmc
3ffe90d4 l     O .bss	000000f0 arp_table
3ffe8940 g     O .bss	00000100 event_TaskQueue
3fff822c l     O .bss	00000100 stdin_ringbuf_array
3fff887c g     O .bss	0000018c emu_tasks
3fff8678 g     O .bss	000001d8 mp_state_ctx
3fff8001 l     O .bss	00000200 filebuf$5274
3fff8474 l     O .bss	00000200 LfnBuf
3ffe8c30 l     O .bss	00000460 dns_table
3ffed818 g     O .bss	000006b0 g_ic
3ffee7f8 l     O .bss	00009800 heap
Unfortunately most of this comes from internal ESP8266 stuff. g_ic comes from libnet80211.a, etc. LfnBuf comes from micropython's FAT filesystem, so you could experiment with disabling LFN support (for 1.1kiB).

manseekingknowledge
Posts: 61
Joined: Sun Oct 29, 2017 5:14 pm

Re: What is maximum valid heap size

Post by manseekingknowledge » Wed Aug 21, 2019 2:45 pm

Thanks for the excellent explanation. Is this info in some hidden document or have you just surmised all of this through experimentation and code examination?

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: What is maximum valid heap size

Post by jimmo » Wed Aug 21, 2019 11:08 pm

No worries, glad you found it useful! None of this is particularly MicroPytho- specific or even microcontroller-specific -- text/data/bss etc apply even for programs you run on your PC. The main difference is that when you have an OS you can request more "memory" (via mmap / sbrk) so you don't have to put your heap in bss.

manseekingknowledge
Posts: 61
Joined: Sun Oct 29, 2017 5:14 pm

Re: What is maximum valid heap size

Post by manseekingknowledge » Fri Aug 23, 2019 4:17 pm

For others like myself who are unfamiliar with the terms text, data, and bss as they pertain to stack, heap, and general memory layout, here is a good Wikipedia page explaining the relationships: https://en.m.wikipedia.org/wiki/Data_segment

manseekingknowledge
Posts: 61
Joined: Sun Oct 29, 2017 5:14 pm

Re: What is maximum valid heap size

Post by manseekingknowledge » Mon Aug 26, 2019 6:19 am

I played around disabling some features I didn't need in mpconfigport.h and found several that, when disabled, allowed me to reclaim 8 bytes of BSS space each and found that commenting out the dupterm initialization code in main.c allowed me to gain back 32 bytes of BSS space.

The big one to disable if you aren't using it though is MICRO_PY_WEBREPL. Disabling that let me reclaim 520 bytes of BSS space.
Last edited by manseekingknowledge on Mon Aug 26, 2019 3:13 pm, edited 2 times in total.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: What is maximum valid heap size

Post by jimmo » Mon Aug 26, 2019 7:22 am

Cool thanks for the update! I will investigate later why enabling the webrepl code makes a difference.

I've gotta ask though, and sorry because I don't want to take away from the excellent work you've done tracking this down, but for the time you've invested in this, it's probably worth upgrading to an ESP32 or Pyboard D!! You'll have so much more RAM!!

manseekingknowledge
Posts: 61
Joined: Sun Oct 29, 2017 5:14 pm

Re: What is maximum valid heap size

Post by manseekingknowledge » Mon Aug 26, 2019 1:51 pm

jimmo wrote:
Mon Aug 26, 2019 7:22 am
Cool thanks for the update! I will investigate later why enabling the webrepl code makes a difference.

I've gotta ask though, and sorry because I don't want to take away from the excellent work you've done tracking this down, but for the time you've invested in this, it's probably worth upgrading to an ESP32 or Pyboard D!! You'll have so much more RAM!!
Those boards are fine for one off projects, but are considerably more expensive when purchased at quantity.

Post Reply