get more free RAM...

[jedie]

It will help to divide the code into "stable" and "volatile" modules. The stable parts would be deployed as frozen modules (hard to update), the volatile parts would be precompiled .mpy file (slightly easier to update).

I now have a solution that will do this "automatically"

Because now i have two kinds of OTA updates... I call it "hard" and "soft":

• The 'soft' OTA is my still existing pure micropython solution
• The 'hard' OTA is via yaota8266 bootloader that will replace the complete firmware

I update my existing solution, so it will only upload files to flash filesystem, if there change.
So it's possible to compile and freeze a new firmware and flash it. If i now start my "soft" OTA it will only upload missing files, like *.html and *.css etc.
If i then change one of my python files, it will upload it as compiled .mpy file.

This is possible, because i save the information about the frozen modules in a simple tuple list that looks like this:

Code: Select all

FROZEN_FILE_INFO = (
 ('http_send_file.py', 1121, '4f19032f29ef4dad0d72d761b710b24926c1ca3645ea02f9e17f0acf2a5d4acf'),
 ('http_set_dim_level.py', 1239, '431c02802acc39b92518f76b463c031e437406e06f9047f1c4490bf28392a61c'),
 ('reset.py', 931, '6e6584df3e04fab341c93970b931b6d4f045653c139bf19161a1af030ce683d3'),
...

It's the filename, file-size and sha256 from the file content... So i can detect if the file has been changed since the last flash...

[jedie]

Hey... If i understand this right: freezing modules safes RAM because the interpreter can execute the code directly, isn't it?

Pure py files must be compiled to .mpy bytecode and this must be stored into RAM, right?
And .mpy files must also be copy from flash to RAM to execute, right?

But why must compiled .mpy files copied from flash to RAM? Because the flash is marked as non executable memory or something like that?

[kevinkk525]

Yes
Yes
Yes

I don't know that for sure.

[jimmo]

Yes -- the interpreter needs the bytecode to be in randomly accessible memory (not specifically RAM, but just memory that it can access like a pointer to bytes). Which the firmware in ROM is (as you've seen with your hex dump tool).

Yes -- because MicroPython does not (currently) have a way to write bytecode to flash during execution, as all available flash is used for the filesystem. But maybe in the future.

Yes -- they must be copied to RAM because they're not really stored directly on flash, they're stored on the filesystem. Which means that they may be divided up into chunks and scattered all over the flash.
On a real computer, the MMU and virtual memory system can make this work -- you can pretend like stuff on a filesystem is actually just in a contiguous region of memory. But the sort of hardware that MicroPython runs on does not support this.

What you need is a filesystem where you can say "this entire file must be contiguous", which currently neither FAT or LittleFS support.

But additionally, the micropython filesystem might not be stored on a device that is not memory mappable either. As a simple example on the ESP8266, I think it's something like only hte first 1MiB is mapped. (Or something like that...) Or perhaps the filesystem is backed by a network connection, etc. Could be any arbitrary block device.

[jedie]

Interesting. Thanks for the information.

Is there a overview about the memory layout of ESP8266?

It would be cool to have something like the following for e.g. the HexViewer: viewtopic.php?f=16&t=7467 But with real addresses:

Screenshot_2020-01-04 Filesystem — ESP8266 Arduino Core 2 6 3-18-gde30762 documentation.png (9.81 KiB) Viewed 3905 times

This is from: https://arduino-esp8266.readthedocs.io/ ... ash-layout

EDIT: ESP32 ports has ports/esp32/partitions.csv and ports/esp32/partitions-2MiB.csv looks like:

Code: Select all

# Name,   Type, SubType, Offset,  Size, Flags
# Note: if you change the phy_init or app partition offset, make sure to change the offset in Kconfig.projbuild
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 0x180000,
vfs,      data, fat,     0x200000, 0x200000,

ESP8266 has e.g.: ports/esp8266/boards/esp8266.ld

Code: Select all

/* GNU linker script for ESP8266 */

MEMORY
{
    dport0_0_seg : org = 0x3ff00000, len = 0x10
    dram0_0_seg :  org = 0x3ffe8000, len = 0x14000
    iram1_0_seg :  org = 0x40100000, len = 0x8000
    irom0_0_seg :  org = 0x40209000, len = 0x8f000
}

/* define common sections and symbols */
INCLUDE boards/esp8266_common.ld

Are theses information available on the device, too?
FlashBdev / bdev instance contains some informations about flash start sector, sector size... and there is esp.flash_size() and esp.flash_user_start()

Maybe it's possible to puzzle all these information together and create a "memory layout table" ?