Creating larger ROMs

All ESP8266 boards running MicroPython.
Official boards are the Adafruit Huzzah and Feather boards.
Target audience: MicroPython users with an ESP8266 board.
manseekingknowledge
Posts: 60
Joined: Sun Oct 29, 2017 5:14 pm

Creating larger ROMs

Post by manseekingknowledge » Sun Jul 07, 2019 8:03 pm

micropython commit 5c34c2ff7f0c44ec9e1d77059162584c6bd99c92

My ESP8266 has 4MB of flash, and though my ROM is just over 1MB in size, my ESP8266 will not boot when I load that ROM.

Here is some build output from a build that will not boot:

Code: Select all

   text	   data	    bss	    dec	    hex	filename
1141216	   1084	  68544	1210844	 1279dc	build/firmware.elf
Create build/firmware-combined.bin
esptool.py v2.6
Creating image for ESP8266...
('flash    ', 32944)
('padding  ', 3920)
('irom0text', 1109396)
('total    ', 1146260)
('md5      ', 'ce1d26c571d870e57d7b050dc66829cb')
And here is some build output from a build that will boot. This build is the same as the build above, but one of my frozen modules was removed to get the size down:

Code: Select all

   text	   data	    bss	    dec	    hex	filename
1022404	   1084	  68544	1092032	 10a9c0	build/firmware.elf
Create build/firmware-combined.bin
esptool.py v2.6
Creating image for ESP8266...
('flash    ', 32944)
('padding  ', 3920)
('irom0text', 990584)
('total    ', 1027448)
('md5      ', 'a52a78532b1989da99b1939371723057')
When I first got this irom0_0_seg error over a year ago...

Code: Select all

xtensa-lx106-elf-ld: build/firmware.elf section `.irom0.text' will not fit in region `irom0_0_seg'
xtensa-lx106-elf-ld: region `irom0_0_seg' overflowed by 190441 bytes
... I updated esp8266.ld per this issue to give me give some more room:

Code: Select all

/* GNU linker script for ESP8266 */

MEMORY
{
    dport0_0_seg : org = 0x3ff00000, len = 0x10
    dram0_0_seg  : org = 0x3ffe8000, len = 0x14000
    iram1_0_seg  : org = 0x40100000, len = 0x8000
    irom0_0_seg  : org = 0x40209000, len = 0xFFFFF /* https://github.com/micropython/micropython/issues/2700 */
}

/* define common sections and symbols */
INCLUDE esp8266_common.ld
It seems that I can make irom0_0_seg equal any sufficiently large value up to 0x3BFFFF to make the irom0_0_seg error go away. Anything larger than 0x3BFFFF prevents boot.py from being created. Regardless, even when the irom0_0_seg error is gone, the build will not boot.

Given that, I have a few questions:

1.) What is the largest ROM that can be built for an ESP8266 with a 4MB flash using micropython?
2.) Is there some place other than esp8266.ld where memory sizes should be adjusted?
3.) I'm currently storing pictures as frozen modules using data_to_py.py as described here. Is there a better way?
4.) Is there a way to build the ROM so that arbitrary files (like pictures) appear as siblings of boot.py when the ROM is flashed?

If I'm asking the wrong questions, please feel free to volunteer any helpful info.

User avatar
Roberthh
Posts: 1639
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: Creating larger ROMs

Post by Roberthh » Mon Jul 08, 2019 1:50 pm

As far as I remember from previous posts here, ESP8266 only executed code from the first 1 MByte of flash. One could wonder, why it is not possible to have frozen bytecode beyond that limit, because it is read as data.

manseekingknowledge
Posts: 60
Joined: Sun Oct 29, 2017 5:14 pm

Re: Creating larger ROMs

Post by manseekingknowledge » Mon Jul 08, 2019 2:19 pm

Roberthh wrote:
Mon Jul 08, 2019 1:50 pm
As far as I remember from previous posts here, ESP8266 only executed code from the first 1 MByte of flash. One could wonder, why it is not possible to have frozen bytecode beyond that limit, because it is read as data.
Are you lumping arbitrary files in as "code"? It would for sure be great to be able to have more frozen modules, but what I really need now is just to store some PNG files. I don't care if they are on the file system or if they are frozen modules, so long as I can access them in code that the ESP8266 is serving up.

When I turn a PNG file into byte code and then make it a frozen module, its size balloons nearly 4x. If I could store it on the file system that would be good because it wouldn't be adding to the size of my frozen modules, but I need it to be baked into the binary image that I flash to the ESP8266. I can't figure out how to do this.

User avatar
jimmo
Posts: 640
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia

Re: Creating larger ROMs

Post by jimmo » Mon Jul 08, 2019 2:37 pm

Roberthh wrote:
Mon Jul 08, 2019 1:50 pm
As far as I remember from previous posts here, ESP8266 only executed code from the first 1 MByte of flash. One could wonder, why it is not possible to have frozen bytecode beyond that limit, because it is read as data.
The ESP8266 memory maps the flash, but only has a 1MiB region to map it to. So this precludes frozen modules (even though they're just data, they still need to be memory mapped). (Actually frozen modules can have native code in them now too).
manseekingknowledge wrote:
Mon Jul 08, 2019 2:19 pm
but I need it to be baked into the binary image that I flash to the ESP8266. I can't figure out how to do this.
This can be done in theory -- you can build the FAT filesystem image and include that in the binary image at the correct offset.

You can get some hints here: viewtopic.php?f=2&t=6618&p=37834#p37834 and viewtopic.php?f=3&t=5877&p=33694#p33694

Sorry I haven't actually done this myself, so don't have more specific advice.

One thing you might want to try though is flashing the firmware to a device, then copying the files to the device filesystem, then reading back the entire flash contents? That'll give you a single firmware image that you can flash to other devices?

manseekingknowledge
Posts: 60
Joined: Sun Oct 29, 2017 5:14 pm

Re: Creating larger ROMs

Post by manseekingknowledge » Mon Jul 08, 2019 3:19 pm

jimmo wrote:
Mon Jul 08, 2019 2:37 pm
Roberthh wrote:
Mon Jul 08, 2019 1:50 pm
As far as I remember from previous posts here, ESP8266 only executed code from the first 1 MByte of flash. One could wonder, why it is not possible to have frozen bytecode beyond that limit, because it is read as data.
The ESP8266 memory maps the flash, but only has a 1MiB region to map it to. So this precludes frozen modules (even though they're just data, they still need to be memory mapped). (Actually frozen modules can have native code in them now too).
Did you mean to type "includes" instead of "precludes"? If I'm understanding this correctly, does this mean that most of the 4MB flash on an ESP8266 is inaccessible by a MicroPython build? Is there anyway to make that 1MiB region larger?
jimmo wrote:
Mon Jul 08, 2019 2:37 pm
One thing you might want to try though is flashing the firmware to a device, then copying the files to the device filesystem, then reading back the entire flash contents? That'll give you a single firmware image that you can flash to other devices?
I was thinking this as well, but I didn't want to mention it until I had tried it.

User avatar
Roberthh
Posts: 1639
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: Creating larger ROMs

Post by Roberthh » Mon Jul 08, 2019 3:30 pm

Any space beyond 1MB can be used e.g. for a file system.

manseekingknowledge
Posts: 60
Joined: Sun Oct 29, 2017 5:14 pm

Re: Creating larger ROMs

Post by manseekingknowledge » Mon Jul 08, 2019 4:00 pm

Thanks for the guidance guys. It is a shame that only 1MB of the 4MB flash can be used by MicroPython images. As I read on this subject, I see many others have had the same space constraint issues as I'm having because of this limitation. I would like to understand this limitation better. Is the limitation because of the hardware, some other technical thing, or just a feature that hasn't been developed yet? If it is the latter, it would be great if this feature could be developed.

Regardless, I'll try using the file system to store my pictures tonight and see what happens.

User avatar
jimmo
Posts: 640
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia

Re: Creating larger ROMs

Post by jimmo » Mon Jul 08, 2019 9:13 pm

manseekingknowledge wrote:
Mon Jul 08, 2019 4:00 pm
I would like to understand this limitation better. Is the limitation because of the hardware, some other technical thing, or just a feature that hasn't been developed yet?
This isn't a MicroPython limitation, it's part of the way the ESP8266 works.

The CPU only has a 1MiB region of its address space 0x40200000 to 0x40300000 where the contents of the spiflash is mapped. All other access to the spiflash has to be via explicit SPI reads (which is how the filesystem works).

In order to execute code, it must be memory mapped. Frozen modules work the same way, you need to be able ro create a pointer to their bytecode data.
manseekingknowledge wrote:
Mon Jul 08, 2019 3:19 pm
Did you mean to type "includes" instead of "precludes"?
I meant precludes, but what I actually meant to say was "also precludes frozen modules from being stores beyond the first 1MiB"


This may seem like an odd limitation, but it's worth remembering what this chip was originally designed for, I'm sure they thought "1 MiB of code should be enough for anyone!" :) (It's not even clear they intended people to write application code foe it!)

manseekingknowledge
Posts: 60
Joined: Sun Oct 29, 2017 5:14 pm

Re: Creating larger ROMs

Post by manseekingknowledge » Tue Jul 09, 2019 5:51 am

So what is the largest value that irom0_0_seg can be set to maximize the space for frozen modules? @pythoncoder set it to a value of 0xa7000 according to his post here. Based on the code below from esp8266_common.ld I would guess the max valid size for irom0_0_seg is 0xF7000. Is this correct? I guess I'm just confused because I don't understand why this value wouldn't be set to the max by default. Isn't it most common for people to want more space to freeze modules?

Code: Select all

/*
Showing my work...
_firmware_size = ORIGIN(irom0_0_seg) + LENGTH(irom0_0_seg) - 0x40200000;
_firmware_size = 0x40209000 + 0xF7000 - 0x40200000;
_firmware_size = 0x40300000 - 0x40200000;
_firmware_size = 0x100000
*/
_firmware_size = ORIGIN(irom0_0_seg) + LENGTH(irom0_0_seg) - 0x40200000;

User avatar
Roberthh
Posts: 1639
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: Creating larger ROMs

Post by Roberthh » Tue Jul 09, 2019 6:17 am

It was set to this value to retain space for a small file system on devices with 1MByte flash. Many ESP01 based modules have that size, as well as all modules with the esp8285 chip.

Post Reply