get more free RAM...

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
jedie
Posts: 247
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: get more free RAM...

Post by jedie » Wed Jan 01, 2020 11:38 pm

MostlyHarmless wrote:
Wed Dec 04, 2019 9:55 pm
It will help to divide the code into "stable" and "volatile" modules. The stable parts would be deployed as frozen modules (hard to update), the volatile parts would be precompiled .mpy file (slightly easier to update).
I now have a solution that will do this "automatically" :mrgreen:

Because now i have two kinds of OTA updates... I call it "hard" and "soft":
  • The 'soft' OTA is my still existing pure micropython solution
  • The 'hard' OTA is via yaota8266 bootloader that will replace the complete firmware
I update my existing solution, so it will only upload files to flash filesystem, if there change.
So it's possible to compile and freeze a new firmware and flash it. If i now start my "soft" OTA it will only upload missing files, like *.html and *.css etc.
If i then change one of my python files, it will upload it as compiled .mpy file.

This is possible, because i save the information about the frozen modules in a simple tuple list that looks like this:

Code: Select all

FROZEN_FILE_INFO = (
 ('http_send_file.py', 1121, '4f19032f29ef4dad0d72d761b710b24926c1ca3645ea02f9e17f0acf2a5d4acf'),
 ('http_set_dim_level.py', 1239, '431c02802acc39b92518f76b463c031e437406e06f9047f1c4490bf28392a61c'),
 ('reset.py', 931, '6e6584df3e04fab341c93970b931b6d4f045653c139bf19161a1af030ce683d3'),
...
It's the filename, file-size and sha256 from the file content... So i can detect if the file has been changed since the last flash...

jedie
Posts: 247
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: get more free RAM...

Post by jedie » Fri Jan 03, 2020 5:55 pm

Hey... If i understand this right: freezing modules safes RAM because the interpreter can execute the code directly, isn't it?

Pure py files must be compiled to .mpy bytecode and this must be stored into RAM, right?
And .mpy files must also be copy from flash to RAM to execute, right?

But why must compiled .mpy files copied from flash to RAM? Because the flash is marked as non executable memory or something like that?

kevinkk525
Posts: 755
Joined: Sat Feb 03, 2018 7:02 pm

Re: get more free RAM...

Post by kevinkk525 » Fri Jan 03, 2020 7:06 pm

Yes
Yes
Yes

I don't know that for sure.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

User avatar
jimmo
Posts: 1769
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: get more free RAM...

Post by jimmo » Fri Jan 03, 2020 11:33 pm

Yes -- the interpreter needs the bytecode to be in randomly accessible memory (not specifically RAM, but just memory that it can access like a pointer to bytes). Which the firmware in ROM is (as you've seen with your hex dump tool).

Yes -- because MicroPython does not (currently) have a way to write bytecode to flash during execution, as all available flash is used for the filesystem. But maybe in the future.

Yes -- they must be copied to RAM because they're not really stored directly on flash, they're stored on the filesystem. Which means that they may be divided up into chunks and scattered all over the flash.
On a real computer, the MMU and virtual memory system can make this work -- you can pretend like stuff on a filesystem is actually just in a contiguous region of memory. But the sort of hardware that MicroPython runs on does not support this.

What you need is a filesystem where you can say "this entire file must be contiguous", which currently neither FAT or LittleFS support.

But additionally, the micropython filesystem might not be stored on a device that is not memory mappable either. As a simple example on the ESP8266, I think it's something like only hte first 1MiB is mapped. (Or something like that...) Or perhaps the filesystem is backed by a network connection, etc. Could be any arbitrary block device.

jedie
Posts: 247
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: get more free RAM...

Post by jedie » Sat Jan 04, 2020 12:49 pm

Interesting. Thanks for the information.

Is there a overview about the memory layout of ESP8266?

It would be cool to have something like the following for e.g. the HexViewer: viewtopic.php?f=16&t=7467 But with real addresses:
Screenshot_2020-01-04 Filesystem — ESP8266 Arduino Core 2 6 3-18-gde30762 documentation.png
Screenshot_2020-01-04 Filesystem — ESP8266 Arduino Core 2 6 3-18-gde30762 documentation.png (9.81 KiB) Viewed 396 times
This is from: https://arduino-esp8266.readthedocs.io/ ... ash-layout

EDIT: ESP32 ports has ports/esp32/partitions.csv and ports/esp32/partitions-2MiB.csv looks like:

Code: Select all

# Name,   Type, SubType, Offset,  Size, Flags
# Note: if you change the phy_init or app partition offset, make sure to change the offset in Kconfig.projbuild
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 0x180000,
vfs,      data, fat,     0x200000, 0x200000,
ESP8266 has e.g.: ports/esp8266/boards/esp8266.ld

Code: Select all

/* GNU linker script for ESP8266 */

MEMORY
{
    dport0_0_seg : org = 0x3ff00000, len = 0x10
    dram0_0_seg :  org = 0x3ffe8000, len = 0x14000
    iram1_0_seg :  org = 0x40100000, len = 0x8000
    irom0_0_seg :  org = 0x40209000, len = 0x8f000
}

/* define common sections and symbols */
INCLUDE boards/esp8266_common.ld
Are theses information available on the device, too?
FlashBdev / bdev instance contains some informations about flash start sector, sector size... and there is esp.flash_size() and esp.flash_user_start()

Maybe it's possible to puzzle all these information together and create a "memory layout table" ?

Post Reply