compiling C modules into loadable shared objects

[OutoftheBOTS_]

I think the more interesting problem to solve is of distribution; how to ensure the right native module, built for the correct architecture, is made available...

I am wondering if it is possible to build into the .mpy file something that identifies whether it has port/arch pacific code. Then if an mpy file is called from an in correct port it can through an error.

[mattyt]

I am wondering if it is possible to build into the .mpy file something that identifies whether it has port/arch pacific code. Then if an mpy file is called from an in correct port it can through an error.

I believe that's possible - and I agree it should be done - but it shouldn't get to that point. Native modules should only be copied to a device if they contain appropriate native code. Flash is a fairly precious resource!

I'm thinking that a convention could work here; something like having a 'native' folder in the packaged/deployed module with all of the supported native modules inside it, laid out by the architecture. upip could then figure out which is appropriate and copy it.

Still early days for this though.

[jimmo]

I am wondering if it is possible to build into the .mpy file something that identifies whether it has port/arch pacific code. Then if an mpy file is called from an in correct port it can through an error.

This is already the case. There's a header in the .mpy file that identifies the native arch (if applicable), plus some other settings that affect the bytecode compatibility (map lookup caching, unicode support)

What are exactly Qstrings? Is it similar to interned-strings? Why is it specific to MicroPython? I thought CPython used the same approach.

Yup, that's right -- interned strings. Not specific to MicroPython, but not necessarily a thing you'd think about in the context of a shared library loader. MicroPython goes to some effort to make sure that QSTR data isn't duplicated in the .mpy file.

What are symbols from the main firmware? Don't they have an identifier?

An example would be something like your native code wanting to print something. It needs to know where the "print" function is. On a regular computer, yeah all the libraries have tables that tell the loader what symbols they need etc. MicroPython doesn't have quite the same luxury because of the "micro" part, but same concept is implemented.

What is memory map? Cannot we use the flash memory instead? I thought the flash memory was memory mapped.

Almost everything the microcontroller can do is presented as memory locations (even if they aren't really backed by RAM). So the memory map is where these things are in the address space. Actual RAM is in some part, internal flash in another, etc. Sometimes you can memory map external flash. And typically if you can memory map it, then you can execute code directly from it.

However, if there's a filesystem involved, then there might not be a simple mapping of "this address range is this block of code" because the filesystem might chunk things into smaller non-contiguous pieces. (i.e. the FAT filesystem does exactly this). On a big computer, the MMU and virtual memory can work around this.

Anyway, the summary is: if a .mpy is in the filesystem (internal flash, external memory mapped flash, external non-memory mapped flash) you cannot execute code directly out of the .mpy file. It first has to be copied into RAM.

In theory, you could reserve a small chunk of internal flash and use that as a scratch space for loading modules from the filesystem and then executing them. At the moment this feature isn't implemented, the main reason is that if you had spare flash you might as well just freeze your modules into the firmware image. (Or if it's native code, which is what this thread's about I guess, then it might as well just be part of the actual firmware). But that's not to say this wouldn't be a good feature, it just hasn't been a priority.

[sebi]

In theory, you could reserve a small chunk of internal flash...

Thank you @jimmo for those very clear explanations!
So freezing a module has a significant advantage today to reduce RAM consumption.

Is it really necessary to imagine reserving a small chunk of internal flash in advance? Wouldn't it be possible to imagine an os.freeze command that would reduce the size of the file system on the fly to liberate a small chunk of memory-mapped internal flash to move the mpy file to?

Up till now none of the depicted examples shown in the videos have addressed accessing port-specific features from within the C-code (like writing to a particular register). Will this be possible as well?

[v923z]

I am wondering if it is possible to build into the .mpy file something that identifies whether it has port/arch pacific code. Then if an mpy file is called from an in correct port it can through an error.

I believe that's possible - and I agree it should be done - but it shouldn't get to that point. Native modules should only be copied to a device if they contain appropriate native code. Flash is a fairly precious resource!

But has the code really got to run from flash? Can't it be fetched from any other storage location, SD card, or EEPROM, or SRAM? Does it really matter, how and where from the code gets into the RAM? If the location doesn't matter, then the flash is no longer precious, because it can arbitrarily be extended.

Here is another scenario: let us suppose I have a "motherboard" with a microcontroller and micropython on it, and this "motherboard" can accept hardware extension modules of various kinds. (Damien's D series of boards is along these lines, I believe). Now, the extension board could contain its own "driver", i.e., the micropython code required to run it. This would mean that the firmware on the motherboard wouldn't even have to know in advance that this or that board will be connected to it later, and it would still be able to handle the hardware.

I have already implemented such an architecture, but I had to store the code in the EEPROM in vanilla python. There are situations, when you don't want your code to be decipherable (an EEPROM hanging on an I2C line is especially easy to crack), so I would say that the option of distributing the binary code over multiple devices could definitely be an asset.

Cheers,
Zoltán

[OutoftheBOTS_]

In theory, you could reserve a small chunk of internal flash...

Thank you @jimmo for those very clear explanations!
So freezing a module has a significant advantage today to reduce RAM consumption.

Is it really necessary to imagine reserving a small chunk of internal flash in advance? Wouldn't it be possible to imagine an os.freeze command that would reduce the size of the file system on the fly to liberate a small chunk of memory-mapped internal flash to move the mpy file to?

Up till now none of the depicted examples shown in the videos have addressed accessing port-specific features from within the C-code (like writing to a particular register). Will this be possible as well?

Ok just checking that I understand everything correctly here. If the native code is located on the flash but outside the FAT then it doesn't need to be loaded in to RAM (heap) to be executed, as the flash can be memory mapped and the machine instructions can be executed directly from the flash??

Also so that I can check that I understand how MP sets up the Flash. It is my understanding it sets up a boot sector for the MP firmware (machine instructions) then left over flash is formatted with fat or spiffs for a file system accessible from MP. I do believe on the Lobo port for ESP32 he has an option of making the boot sector larger that way when you update the firmware with a larger binary file it doesn't corrupt the FAT.

Is it possible to create an option when building firmware u can opt to leave extra space in the boot sector for both later firmware updates but also the option of coping this compiled C code as native code to this space in the boot sector??

[jimmo]

So freezing a module has a significant advantage today to reduce RAM consumption.

Yep. This (freezing) is currently the only way to have the code (bytecode or native code) not take up RAM while executing. The internal flash is memory mapped and executable, so the bytecode can just be accessed as if it was in RAM, or the native code can literally be jumped to.

Is it really necessary to imagine reserving a small chunk of internal flash in advance? Wouldn't it be possible to imagine an os.freeze command that would reduce the size of the file system on the fly to liberate a small chunk of memory-mapped internal flash to move the mpy file to?

If I understand what you're saying, you're kind of suggesting that a "defrag" command could ensure that the bytes for a .mpy file are all contiguous on flash (even though they're in the filesystem). Yes! This would work, as long as the filesystem was itself in memory mapped storage (i.e. internal flash), or memory mapped external flash (for example the qspi flash on the pybd), or the first megabyte of external flash on an ESP8266. But that's the problem -- a micropython filesystem might not be on memory mappable flash. And the code that accesses the filesystem does so through the VFS layer, so there's a layer of abstraction here that makes it complicated. (For example, the filesystem might actually be over the network or over some external connection, like mprepl.py).

Up till now none of the depicted examples shown in the videos have addressed accessing port-specific features from within the C-code (like writing to a particular register). Will this be possible as well?

Yes. Writing to a particular register is "easy" because it's literally just writing to an absolute memory address. So if you're happy to go that route then that's fine. For the reasons described earlier though, this doesn't just mean you can (for example on ESP32) start using functions from the IDF (or whatever HAL you have for your board). Some thought required for this, but yes this is definitely a use case that is being thought about.

But has the code really got to run from flash? Can't it be fetched from any other storage location, SD card, or EEPROM, or SRAM? Does it really matter, how and where from the code gets into the RAM? If the location doesn't matter, then the flash is no longer precious, because it can arbitrarily be extended.

That's right - it doesn't _have_ to run from flash. What you're describing is exactly how it works today. the .mpy file (containing native or bytecode, or both) is loaded from wherever and copied into RAM and executed from there. This is a good feature that enables exactly the sort of use case that you're describing.

But RAM is precious and code can be quite large, so avoiding needing to put it in RAM is useful. I understand that in your case your might have a lot of code but only a tiny amount of it loaded at any time, so that's fine. (Sorry I feel like I kind of derailed the thread a bit with my comment about flash vs ram, but sounds like it was at least useful to discuss).

Ok just checking that I understand everything correctly here. If the native code is located on the flash but outside the FAT then it doesn't need to be loaded in to RAM (heap) to be executed, as the flash can be memory mapped and the machine instructions can be executed directly from the flash??

Yes. However there are (currently) only two ways to acheive this -- either the code is part of the actual firmware, or it's frozen into the firmware. MicroPython does not right now have a way to say "there's a .mpy file at 0xaddress". (Although that's pretty close to how the frozen support works)

And just to be really precise, "flash" in this case specifically means any memory mapped flash. So on PYBD for example, there are three flash areas:
- Internal flash -- obviously memory mapped. Half the firmware goes here.
- External QSPI flash 1 -- uses hardware qspi driver, memory mapped via the STM32F7's memory mapping & XIP support. The rest of the firmware goes here.
- External QSPI flash 2 -- uses a software block device, mounted into the VFS. The filesystem goes here.

Also so that I can check that I understand how MP sets up the Flash. It is my understanding it sets up a boot sector for the MP firmware (machine instructions) then left over flash is formatted with fat or spiffs for a file system accessible from MP. I do believe on the Lobo port for ESP32 he has an option of making the boot sector larger that way when you update the firmware with a larger binary file it doesn't corrupt the FAT.

There is some variance between boards here (see PYBD above) but yes. And what you said about the padding is true for all ports.

Again to be really precise, the boot sector isn't really a thing the same way it is on a PC. There's a boot loader (e.g. for firmware update), then the actual MicroPython firmware, then some padding, then space reserved for the filesystem. Some devices split this over multiple flash regions (internal & external) etc. Some devices have multiple levels of bootloader (e.g. pybd), some devices have bootloader in mask rom (i.e. not in user flash), etc.

Is it possible to create an option when building firmware u can opt to leave extra space in the boot sector for both later firmware updates but also the option of coping this compiled C code as native code to this space in the boot sector??

Yep! This is exactly what I was referring to when I said "In theory, you could reserve a small chunk of internal flash and use that as a scratch space".

[OutoftheBOTS_]

Is it possible to create an option when building firmware u can opt to leave extra space in the boot sector for both later firmware updates but also the option of coping this compiled C code as native code to this space in the boot sector??

Yep! This is exactly what I was referring to when I said "In theory, you could reserve a small chunk of internal flash and use that as a scratch space".

Thanks for your time to explain all this to us Jimmo

Considering MP is in it's infant stage and the direction of future potential hardware it is likely that most future boards will have a SD card that can hold a very large FAT file system and that the internal file system is mainly important to hold boot.py and main.py. It is my option that having the ability to add native code to the fash that is directly executable without needing RAM to hold the code is probably more valuable that a large internal FAT file system.

[sebi]

Indeed, thank you Jim for taking the time to give us all this precious information.

For the reasons described earlier though, this doesn't just mean you can (for example on ESP32) start using functions from the IDF (or whatever HAL you have for your board).

Sorry but I don't get what reasons described above you are referring to. Using HAL functions from within the C-code would be of great interest I believe.

It is my option that having the ability to add native code to the fash that is directly executable without needing RAM to hold the code is probably more valuable that a large internal FAT file system.

I have the same opinion.
One could use mprepl at the design stage if the internal file system is too small. And, if designing a datalogger, an external filesystem is to be considered.
The idea being to use esptool (or whatever DFU tool) to flash the core MicroPython firmware only. And to add (native and/or byte) code to the memory-mapped scratch space in a second step using MicroPython (rather than using the standard freezing procedure).

[OutoftheBOTS_]

The idea being to use esptool (or whatever DFU tool) to flash the core MicroPython firmware only. And to add (native and/or byte) code to the memory-mapped scratch space in a second step using MicroPython (rather than using the standard freezing procedure).

Sort of like a modified upip. This means anyone can all to their firmware based upon their own needs