How to eval bytecode from network?

Discussion about programs, libraries and tools that work with MicroPython. Mostly these are provided by a third party.
Target audience: All users and developers of MicroPython.
User avatar
libbkmz
Posts: 3
Joined: Thu Dec 12, 2019 5:07 pm

How to eval bytecode from network?

Post by libbkmz » Thu Dec 12, 2019 5:49 pm

Hi!

I'm thinking about some system that will serve your source code by some network socket (tcp/http/whatever...). The main idea is to have some boilerplate with the initialization network in main.py (client-side) on board and some code to connect to the server, which will compile every time your python source code to mpy and send it to the MCU.

The basic flow is like this:

1. MCU boots and connects to the wifi.
2. Gather the latest DateTime of the source files on flash
3. Send a request with lastest file or maybe with the map of latest fil update and the file names
4. The server selects files that are only updated and sends them to the client. Can by .py or .mpy
5. Client update files on flash and starts at the known entry point in those files.


The concept of overwriting every time file on board's flash is pretty simple, and nothing is challenging here. But, I'm worried too much about wearing the flash storage on my ESP8266. So, here are some questions:
1. Is it possible to do some 'eval' of externally downloaded bytecode? I've found only this presentation (https://micropython.org/resources/publi ... AU2018.pdf) of the architecture, and it seems like that the only way to run bytecode is to compile it from source? I want to be wrong...
2. Does reading bytecode from the flash is more memory efficient than loading all the bytecode in RAM? Or any other solutions?

So, I would like to do faster development, but I don't want to wear the Flash for that...


torwag
Posts: 220
Joined: Fri Dec 13, 2013 9:25 am

Re: How to eval bytecode from network?

Post by torwag » Fri Dec 13, 2019 12:27 pm

Hi,
people always fear this "wear out the flash" thing.
I challenge this. Unlike you have code which constantly writes to the flash (like writing log files), I would not see that a "pull an update at start" has any chance to wear out the flash within the usual lifetime of a system.
Even if the flash rewrite would be as low as 10.000 and it is usually from what I know much higher, you could write
for over one year 24/7 each hour a new version to the ROM.
Three times a day would last more than 9 years
If you opt out to refresh each day it would last 27 years.

If you now consider some sort of "smart" update only if a new version is available and take a usual update cycle of e.g. a few days or weeks, it would last easily over our lifetime.
I think esp. the last two examples are well above the lifetime of most applications.

Thus, I believe it is fine to simply write to the flash.

User avatar
libbkmz
Posts: 3
Joined: Thu Dec 12, 2019 5:07 pm

Re: How to eval bytecode from network?

Post by libbkmz » Fri Dec 13, 2019 12:33 pm

Yeah, Thank you for the link!
This sounds very similar to what I want. But I have another restriction. I don't want to wear the flash while developing. Because you need to change sources frequently and this will destroy the flash once on a while...

So, I'm curious - is there any way to make something similar to this?

Code: Select all

bytecode_buffer = get_bytecode()  # type: bytearray
eval(bytecode_buffer)

User avatar
libbkmz
Posts: 3
Joined: Thu Dec 12, 2019 5:07 pm

Re: How to eval bytecode from network?

Post by libbkmz » Fri Dec 13, 2019 12:50 pm

torwag wrote:
Fri Dec 13, 2019 12:27 pm
Hi,
people always fear this "wear out the flash" thing.
I have experience with Firmwares for the Flash drives with hundreds of gigs and terabytes of capacity. I know how the NAND is working and all this stuff. And that's why I'm trying to avoid the 'program' operation for the NAND array.

Yesterday I tried to make a particular flash friendly append-only log to external sd card, and as a result, I got 2 ruined sd cards. I have some issues trying to bring them up; I've got somehow corrupted CSD register with some strange data sometimes. So, during development, I have to use real hardware to debug the code. And at this stage, I have tons of changes in my source code. And I would like to avoid writing them to the flash.
Nevertheless, I'm anxious about this stuff. You never know will you write cause Program Failure or erase caused Erase Failure.
After the failure, a usually failed block is thrown away from the OP(OverProvision) area. Flash storage for the microcontroller is even smaller and has much more straightforward firmware inside it. And I hope to last my development sample last as long as much as possible.

If I can use my ESP8266 without any "flashing" to the NAND - I would be happy.


So, the main question is remains open, is there any possible way to do eval of the bytecode stored in bytearray?

Christian Walther
Posts: 169
Joined: Fri Aug 19, 2016 11:55 am

Re: How to eval bytecode from network?

Post by Christian Walther » Fri Dec 13, 2019 2:01 pm

If all else fails, there is a new example in the docs of how to make a filesystem in RAM: http://docs.micropython.org/en/latest/r ... ck-devices

jedie
Posts: 252
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: How to eval bytecode from network?

Post by jedie » Fri Dec 13, 2019 2:42 pm

libbkmz wrote:
Fri Dec 13, 2019 12:33 pm
I don't want to wear the flash while developing. Because you need to change sources frequently and this will destroy the flash once on a while...
My OTA stuff saves only new content. It checks the size + sha hash.

In practise: I test code with thonny: I open the "test" file and run it via F5... In this case it doesn't save to flash.

btw. littlefs is designed with flash in mind, and provides wear leveling over dynamic blocks. But currently it's not available on official ESP8266 builds, yet, see: https://github.com/micropython/micropython/issues/5387

jedie
Posts: 252
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: How to eval bytecode from network?

Post by jedie » Fri Dec 13, 2019 2:47 pm

Christian Walther wrote:
Fri Dec 13, 2019 2:01 pm
If all else fails, there is a new example in the docs of how to make a filesystem in RAM: http://docs.micropython.org/en/latest/r ... ck-devices
Especially with ESP8266 RAM is very limited and the bottleneck.

What I forgot: My OTA stuff will not directly overwrite existing files: It always created a new temp file, store the data, compare the hash again ans then the old file will be deleted and the temp file renamed.
IMHO a FAT filesystem will not directly frees the content of deleted files... So it's a kind of wear leveling... But will also result in fragmented filesystem :lol:

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: How to eval bytecode from network?

Post by pythoncoder » Sat Dec 14, 2019 7:43 am

jedie wrote:
Fri Dec 13, 2019 2:47 pm
...
IMHO a FAT filesystem will not directly frees the content of deleted files... So it's a kind of wear leveling...
The FAT filesystem constantly rewrites low numbered blocks: it is the antithesis of wear-levelling, I'm afraid. Hence the need for littlefs.
Peter Hinch
Index to my micropython libraries.

jedie
Posts: 252
Joined: Fri Jan 29, 2016 12:32 pm
Contact:

Re: How to eval bytecode from network?

Post by jedie » Sat Dec 14, 2019 8:49 am

Hm, yeah, I forgott the file allocation table, that always in at the start...

Hopefully littlefs is available soon...

Post Reply