Native code generation, "esp.set_native_code_location()" and how to find spare space in the flash

All ESP8266 boards running MicroPython.
Official boards are the Adafruit Huzzah and Feather boards.
Target audience: MicroPython users with an ESP8266 board.
Post Reply
jcea
Posts: 27
Joined: Fri Mar 18, 2016 5:28 pm

Native code generation, "esp.set_native_code_location()" and how to find spare space in the flash

Post by jcea » Fri Mar 17, 2017 12:53 am

According to https://docs.micropython.org/en/latest/ ... e_location , I can generate native code to the flash. Choosing a block in the first megabyte of flash allows native execution without the limits of the <800 bytes default size in IRAM in my system.

Less than 800 bytes is very limiting, and after the code is fully debugged and the ESP8266 will be reset once per day or less, I am considering using "esp.set_native_code_location()" to save native code to the flash. Nice.

My question is... How can I find a spare block in the first megabyte of flash for use of this function?. It seems not trivial and without it this functions is not useful at all.

In a related note, how can I do extra space in IRAM for native compilation? Less than 800 bytes is quite limiting... Ideas?

Thanks a lot!.

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: Native code generation, "esp.set_native_code_location()" and how to find spare space in the flash

Post by Roberthh » Fri Mar 17, 2017 6:36 am

As far as I can tell it you have to do the following: In the script flashbdev.py, locate in esp9266/modules, you have to set RESERVED_SECS to a value different than 0. The start of the area available for code would then be given by esp.flash_user_start(). The size will be increased by multiples of the block size 4096. Since that effects the file system, you have to erase the device and recreate the file system to get the change effective. I never did that, just looked for the way to do it. In the commit log there is a discussion about it.
I also remember a discussion about a check being performed if a certain code is already in flash to avoid re-writing that code on every boot.

jcea
Posts: 27
Joined: Fri Mar 18, 2016 5:28 pm

Re: Native code generation, "esp.set_native_code_location()" and how to find spare space in the flash

Post by jcea » Fri Mar 17, 2017 4:21 pm

Robert, great answer. Very useful.

This should documented somewhere. Using Google I found this useful discussion https://github.com/micropython/micropython/issues/2735 describing the feature. Great. But this should be documented SOMEWHERE in the official docs.

"esp.flash_user_start()" is NOT documented in https://docs.micropython.org/en/latest/ ... y/esp.html either. Digging in micropython is quite frustrating, actually :-(.

I guess than changing "RESERVED_SECS" requires a custom firmware, isn't it?.

Thanks for your answer and your time, Robert. Very much appreciated.

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: Native code generation, "esp.set_native_code_location()" and how to find spare space in the flash

Post by Roberthh » Fri Mar 17, 2017 4:42 pm

I guess than changing "RESERVED_SECS" requires a custom firmware, isn't it?.
Yes, since that module is embedded into the flash. Making that is not too complicated, if you follow the instructions in micropython/esp8266/readme.md LITERALLY. I never tried that on a different system than Linux.
Yes, and I agree that the documentation does not always follow the code in a timely fashion.

jcea
Posts: 27
Joined: Fri Mar 18, 2016 5:28 pm

Re: Native code generation, "esp.set_native_code_location()" and how to find spare space in the flash

Post by jcea » Sat Mar 18, 2017 12:38 am

Roberthh, do you have a link to the thread about persistent flash storage of native code? I am not able to find anything relevant.

Thanks a lot.

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: Native code generation, "esp.set_native_code_location()" and how to find spare space in the flash

Post by Roberthh » Sat Mar 18, 2017 6:48 am

Hello @jcea,

look at these three links:

Code: Select all

https://github.com/micropython/micropython/commit/bae7798f1ef531c862421b37c0d5d462e34a6d2f
https://github.com/micropython/micropython/commit/c3f70c603efcb053810dbee8d5cc9f23ed62d01f
https://github.com/micropython/micropython/commit/e343bea6e708d69258cd3d771b35d482ae22626b
and the documentation of the esp module:

Code: Select all

http://docs.micropython.org/en/latest/esp8266/library/esp.html?highlight=esp

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: Native code generation, "esp.set_native_code_location()" and how to find spare space in the flash

Post by Roberthh » Sat Mar 18, 2017 8:08 am

Hello @jcea, I have prepared an image for the esp8266 with RESERVED_SECS set to 1, offering 4k space for native & viper code in flash. The image is here: https://github.com/robert-hh/Shared-Stuff
Flashing the image will destroy the previous file system. So save your file beforehand, erase the flash and then load this image. Embedded in the image is also my personal set of tools - ftp server, upysh (basic shell like commands) and editor. Unless loaded, they do not consume RAM.

jcea
Posts: 27
Joined: Fri Mar 18, 2016 5:28 pm

Re: Native code generation, "esp.set_native_code_location()" and how to find spare space in the flash

Post by jcea » Wed Mar 22, 2017 2:07 am

Thanks, Roberthh. That is valuable.

What would you think about storing metadata in the last area (before the SDK area) of the flash describing reservations and FAT layout?. In this way we could change sizes with a method call instead of recompiling and flashing a new image.

As I understand, flashing a new firmware destroys the FAT fs anyway, so no compatibility concerns. Better at the end because space under 1MB is special and valuable.

Do you think this idea has merit?.

Thanks.

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: Native code generation, "esp.set_native_code_location()" and how to find spare space in the flash

Post by Roberthh » Wed Mar 22, 2017 6:26 am

Hello @jcea, flashing a new firmware does normally NOT destroy the file system, if it's location and size is unchanged. I do not know which kind of metadata you want to place at the end. It can't be the native code, because that has to be in the lower 1 MB. You could store the FAT file system top-down. That should be an easy change in flashbdev.py. Storing other data there is equivalent to storing it into a file, so there is no advantage.

jcea
Posts: 27
Joined: Fri Mar 18, 2016 5:28 pm

Re: Native code generation, "esp.set_native_code_location()" and how to find spare space in the flash

Post by jcea » Wed Mar 22, 2017 11:20 am

I would store the size of the reserved area for native code, location and size of the fat FS, spare area (to directly read/delete flash for raw access), OTA area, etc. Kind of a partition table. I would like to have a fat filesystem, but for some applications a "raw" flash area is very convenient. I have a plan for a "log storage" to ease the wearing out of flash, avoiding "hot spots" like the fat sectors in a fat filesystem when hardware doesn't do remapping.

For example, a GPS logger could use simple and robust "data appended only" mode without the limitations and risks of FAT (for example, battery running out while you are updating FAT sectors is catastrophic). I don't require a complete filesystem for that application and, in fact, it is inconvenient to be forced to use one.

Also, I don't know the details of the flash used in the ESP8266 boards, but in most flash you can do several writes in a flash sector without erasing it if you are careful enough. For example, an erased sector is "all ones" and you can write zeros selectively without erasing first. That is, you just stores zeros, the ones are already there. Erasing is what wears out flash memory.

With a 4MB flash (my case), FAT starting <1MB is a waste, the area between the end of the firmware and 1MB should be available for another uses beside FAT. At least optionally and by software. Requiring a firmware recompilation is easy enough but pretty inconvenient. Even more if you are working in different projects, each requiring a custom firmware.

Being able to "reformat" those values by software instead of recompiling a complete firmware would be nice. Keeping the "formatting" between firmware upgrades would be another plus (and a reason to store it at the end of the flash, just below SDK area).

Post Reply