How to build non-python static resource files into binary image

All ESP8266 boards running MicroPython.
Official boards are the Adafruit Huzzah and Feather boards.
Target audience: MicroPython users with an ESP8266 board.
Post Reply
manseekingknowledge
Posts: 50
Joined: Sun Oct 29, 2017 5:14 pm

How to build non-python static resource files into binary image

Post by manseekingknowledge » Wed Jun 20, 2018 2:15 pm

I have my build script set up to freeze all my python modules and that works well, but I want to bake a JS file into my image. How do I do that?

User avatar
pythoncoder
Posts: 3544
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: How to build non-python static resource files into binary image

Post by pythoncoder » Thu Jun 21, 2018 8:43 am

I assume the obvious approach of storing the data in a file is unacceptable: is the aim to save RAM?

It's only possible to freeze Python source files. To freeze other objects you need a utility to convert them to Python source with a utility along the lines of font-to-py or data_to_py. If you can store the JS as a bytes object as per the font files then you should be able to access it with minimal RAM overhead.
Peter Hinch

manseekingknowledge
Posts: 50
Joined: Sun Oct 29, 2017 5:14 pm

Re: How to build non-python static resource files into binary image

Post by manseekingknowledge » Thu Jun 21, 2018 12:52 pm

I assume the obvious approach of storing the data in a file is unacceptable: is the aim to save RAM?
Thanks for your reply. I'm not sure I understand your first comment. What I really want is to have my JS file right next to my boot.py file after I flash the ESP8266 assuming the convenience gained isn't costing me any RAM or using excessive flash. I want to do this to save RAM and for convenience with RAM savings being most important. I don't want to manually transfer any files. I want the ESP8266 to be ready to go containing all the files I need it to have right after I get done flashing it.

Right now I base 64 encode the JS file and store it as a variable in a python file which is frozen. Do you expect data_to_py would be more efficient?

User avatar
pythoncoder
Posts: 3544
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: How to build non-python static resource files into binary image

Post by pythoncoder » Sat Jun 23, 2018 5:35 am

In terms of saving RAM you need to consider what happens when you access the data. If you have a large block of base64 encoded data and you decode it, the decoded data will be located in RAM. The way to store immutable data in flash is to have that data as a bytes object in a Python source file and to freeze that file. You have to be careful to avoid unintentional copying. In the case of font files where random access is required, I use a memoryview object to achieve access using slicing.

So (off the top of my head) I think you're best off storing the JS in your Python module as a bytes object, not base64 encoded. But you need to experiment: measure the RAM used by your code to ensure that you're never inadvertently copying the JS to RAM.

My aim in pointing out my utilities was to illustrate the techniques more than to advocate their use for your purpose. If you look at this Python font file you'll see an example.
Peter Hinch

manseekingknowledge
Posts: 50
Joined: Sun Oct 29, 2017 5:14 pm

Re: How to build non-python static resource files into binary image

Post by manseekingknowledge » Sat Jun 23, 2018 9:08 am

data_to_py.py works like a charm. I was around 95% RAM usage, but accessing the byte data from flash let me reclaim about 40% of that so now I'm around 55%. The image size is larger now than it was when I was using the base64 encoding, but that is to be expected. I had to adjust the segment sizes as you suggested here. Thanks!

Post Reply