Freezing / Compiling raw binary data in DFU

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
User avatar
TravisT
Posts: 54
Joined: Sun Feb 23, 2014 2:31 pm
Location: Portland, OR
Contact:

Freezing / Compiling raw binary data in DFU

Post by TravisT » Tue Jun 25, 2019 4:59 am

I am trying to store raw binary data efficiently but in a way that is easy to compile into my DFU for bootloading. The data can and will change frequently enough that it should be treated almost like application source code. It will hold font and bitmap information.

My current implementation is storing it in a module as a bytes object. This is to hopefully avoid too much RAM usage and make it compile just like any other micropython module. The hope is with minimal impact on RAM and being compiled in the DFU to help minimize Flash usage, I have a reasonable way to distribute binary information while making it almost as easy to modify as source.

I have a script that loads a binary file, then writes to a new ".py" file with the string representation of the bytes object, which is assigned to a variable name within the file. I then import this file (hoping since it is a bytes object it is not all thrown into RAM), load this into a external flash over SPI then not need it again until a restart or update happens.

One of my requirements is single file DFU deployment for the client, which is part of my motivation for this.

I am curious if there is a better way to do this, or am I on the right track.

Thanks for the time
_______________
Travis Travelstead

User avatar
TravisT
Posts: 54
Joined: Sun Feb 23, 2014 2:31 pm
Location: Portland, OR
Contact:

Re: Freezing / Compiling raw binary data in DFU

Post by TravisT » Tue Jun 25, 2019 5:33 am

When doing some testing with just cross compiled modules I do get the size significantly smaller, but the import does seem to live in RAM.

I have tried to "del" the module but I never could get that to work and free up memory.

Luckily I am using the F7 chip right now and have about 307k RAM free before running anything. But I can see that I would easily take up 100k with just imports of raw data.
_______________
Travis Travelstead

User avatar
jimmo
Posts: 630
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia

Re: Freezing / Compiling raw binary data in DFU

Post by jimmo » Tue Jun 25, 2019 6:32 am

Hi Travis,

It would be quite nice to have a feature that effectively automates what you've desribed (i.e. like frozen modules, have a directory of files that automatically become available in Python and backed by ROM).

But in your case, if I understand correctly, it's annoying that you're still going to end up with a copy in internal flash as part of your firmware and then external flash.

What format does the data look like in external SPI flash? Something that might be useful is that mboot (unlike the ST bootloader) can write to external (q)spi flash in addition to internal flash. So you can still keep your single .dfu file.

If the external spiflash has a filesystem on it, then viewtopic.php?f=3&t=6600 (especially the thread that dhylands links to) might be useful viewtopic.php?f=3&t=6600.

User avatar
jimmo
Posts: 630
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia

Re: Freezing / Compiling raw binary data in DFU

Post by jimmo » Tue Jun 25, 2019 6:34 am

jimmo wrote:
Tue Jun 25, 2019 6:32 am
Hi Travis,

It would be quite nice to have a feature that effectively automates what you've desribed (i.e. like frozen modules, have a directory of files that automatically become available as ROM-backed bytes() in Python).

But in your case, if I understand correctly, it's annoying that you're still going to end up with a copy in internal flash as part of your firmware and then external flash.

What format does the data look like in external SPI flash? Something that might be useful is that mboot (unlike the ST bootloader) can write to external (q)spi flash in addition to internal flash. So you can still keep your single .dfu file.

If the external spiflash has a filesystem on it, then viewtopic.php?f=3&t=6600 (especially the thread that dhylands links to) might be useful viewtopic.php?f=3&t=6600.

User avatar
TravisT
Posts: 54
Joined: Sun Feb 23, 2014 2:31 pm
Location: Portland, OR
Contact:

Re: Freezing / Compiling raw binary data in DFU

Post by TravisT » Tue Jun 25, 2019 7:00 am

Sorry I meant external RAM, this is what the chip that actually uses the bitmap and font data reads from. The micropython based device is really just providing assets and commands over SPI. I cannot easily get around writing to the external device.

My main issue is not so much taking up space in flash, it is the single use RAM allocation. At the moment I write the data to the external RAM, the imported binary stays in RAM on the Micropython device. There are ways I can get around this by writing, having something persist and then restarting and avoid writing after restart. But that is complex for something that could be simple.

So in an ideal world there would be a way to easily include binary files on the Micropython based device, that can be accessed by Micropython (similar to using the filesystem), and would only occupy RAM temporarily and GC could clean that up. Oh, and that can easily be compiled into a DFU for distribution.

Hope that clarifies some things.
_______________
Travis Travelstead

User avatar
TravisT
Posts: 54
Joined: Sun Feb 23, 2014 2:31 pm
Location: Portland, OR
Contact:

Re: Freezing / Compiling raw binary data in DFU

Post by TravisT » Sat Jun 29, 2019 7:31 pm

I am working now to utilize something that @pythoncoder did for his font utilities. I realize one thing I did wrong when evaluating is the difference between cross compiling and actually freezing the code into the DFU.

https://github.com/peterhinch/micropython-font-to-py

Hopefully this yield better results with RAM usage and solves my issue.
_______________
Travis Travelstead

User avatar
TravisT
Posts: 54
Joined: Sun Feb 23, 2014 2:31 pm
Location: Portland, OR
Contact:

Re: Freezing / Compiling raw binary data in DFU

Post by TravisT » Sun Jun 30, 2019 3:09 am

I was able to use the same method as Peter to minimally use RAM and read binary information from flash using bytes object in a python source file.

Similar I also have asset build scripts. It is an extra step with including assets in DFU build, but luckily assets do not change too often and I can use my normal development methods with the filesystem on the other files.
_______________
Travis Travelstead

User avatar
pythoncoder
Posts: 3597
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Freezing / Compiling raw binary data in DFU

Post by pythoncoder » Sun Jun 30, 2019 7:35 am

As you've doubtless figured out, the two key concepts in my font file format are to store the data in an immutable bytes object and to use a memoryview of that object to facilitate RAM-less slicing. I also wrote a generic version for similarly storing arbitrary binary data data_to_py but it seems to be little used.
Peter Hinch

User avatar
TravisT
Posts: 54
Joined: Sun Feb 23, 2014 2:31 pm
Location: Portland, OR
Contact:

Re: Freezing / Compiling raw binary data in DFU

Post by TravisT » Sun Jun 30, 2019 1:46 pm

You are correct, I was worried when cross compiling did not benefit me at all, and bytes objects alone did not help. Then I realized that of course it is optimization during when freezing and building the firmware. Your libraries also noted and confirmed this. One downside of the forums is that valuable knowledge can sometimes be hard to find depending on keywords and approach. In this case I did not find it the first time I was looking for options.

In my case the memoryview is not as much of a concern because I load the binary into another devices graphics RAM. But I am glad you reminded me of using memoryview if I as using local static information.

Your libraries are a wealth of both high level conceptual information as well as a practical implementation. It also hits something I have been wanting to work on but lack the time, which is design patterns specifically for Micropython. Coming from a just knowledgable enough and just ignorant enough perspective might be helpful.

I would also suggest that references to your code and high level concept be noted in the the "Maximizing Micropython Speed" and "Micropython on Microcontrollers" pages, as they seem to be the most common documentation to pop up in searches when needing to optimize or work around resource issues on a MCU.

https://docs.micropython.org/en/latest/ ... ython.html
https://docs.micropython.org/en/latest/ ... ained.html

Thanks again for your wonderful work and contribution to this project. Something I wish I had the time (at the moment time and money are related) and patience for.
_______________
Travis Travelstead

User avatar
pythoncoder
Posts: 3597
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Freezing / Compiling raw binary data in DFU

Post by pythoncoder » Tue Jul 02, 2019 6:32 am

TravisT wrote:
Sun Jun 30, 2019 1:46 pm
...
I would also suggest that references to your code and high level concept be noted in the the "Maximizing Micropython Speed" and "Micropython on Microcontrollers" pages, as they seem to be the most common documentation to pop up in searches when needing to optimize or work around resource issues on a MCU...
For understandable reasons, in the official docs @Damien generally avoids links to user contributed material. The wiki is a useful source of such references.
Peter Hinch

Post Reply