Micro Python development wish list

[dhylands]

I think that the only way that this idea works securely, is if the builtin ROM does the checking and sends out the signature. Otherwise, its essentially trivial for someone to write some firmware which just copies a know good signature from firmware X and sends that out, even though it's firmware Y.

Since the firmware images that are flashed onto the pyboard originally are posted here: http://micropython.org/download/ somebody could verify that the board they received has a particular firmware image, by direct comparison. Given the current ROM, I think that this is the only way you can verify the firmware and have any confidence that it wasn't modified in some malicious manner.

[nelfata]

I would like to emphasize on the importance of eliminating heap fragmentation as Damien mentioned at the beginning of this post.
IMHO, this issue should be addressed as soon as possible as it is critical in running any serious application (critical or not).
I was starting to initially develop an application until I hit this problem and really hampered my progress and forced me to rewrite the code in C.
This issue will deter development of any serious application and without it I think once developers are informed about they most likely will choose another platform/environment unfortunately.

I really enjoyed working on MPY and I think the contributors are extremely knowledgeable and great to work/communicate with.

[pfalcon]

Yes I understand the part about using the function table - what I'm not grasping is how you load the code into the process?

Sorry, I don't see any complications here. The code is loaded in system-dependent manner. For example, for bare-metal, that would raw binary, with position-independent code. Within this code, an init function is called, passed a vtable pointer. Module stores it in global, yeah, and does any calls to core via it. In particular, init function will need to call mp_obj_new_module or something before returning. init function is looked up by name for systems which support concept of module symbols, init function will be looked up by name. For baremetal it can be start of binary module.

As mentioned above, there would be additional requirement that module can be compiled either statically (i.e. builtin into interpreter) or dynamically - with the same source (so all switching magic is done by macros).

Above, I gave a link to implementation of such mechanism for Squirrel for reference.

[Damien]

I would like to emphasize on the importance of eliminating heap fragmentation as Damien mentioned at the beginning of this post.
IMHO, this issue should be addressed as soon as possible as it is critical in running any serious application (critical or not).
I was starting to initially develop an application until I hit this problem and really hampered my progress and forced me to rewrite the code in C.

Without giving away any sensitive information, can you explain in more detail the situation which has problems with fragmentation? From memory it has something to do with allocating large buffers, correct?

We can eliminate fragmentation completely, but only at a very big expense. Maybe it can be done in a simpler way, but to do that I need to know the situations where fragmentation is a problem. Even in C land you can still suffer from fragmentation unless you write your code carefully (eg pre allocate all buffers). We might be able to apply this idea to uPy, and provide ways to pre allocate big objects, and have a scratch buffer for small ones.

[stijn]

Sorry, I don't see any complications here. The code is loaded in system-dependent manner.

Well, that was the complication: because of all the mentions of 'portable system-independent' I had the impression this was for loading as well
So, all clear now.

[nelfata]

Thank you Damien for the follow up.
As I mentioned earlier, the environment you and your team have provided so far has proven to be an excellent platform to be used for prototyping and even to be used for a real application. The main issues that I have found while developing my application (now relatively large) are:
1. Lack of debugger
2. Issues with file system and corruption
3. Issues with loading modules (memory allocation failures)
4. Issues with run time buffer allocation (memory allocation failures)

For issue #1, I guess it is what it is and can be dealt with, print statements are acceptable but slows down the development.
For issue #2, during the development the USB needs to be connected to the MPY board, but Windows occasionally attempts to access the SD Card, if the application tries to write to the SD Card, corruption occurs. Some of the script files are also damaged, or inaccessible, moving those files to lower level directories (mentioned in the forum) seems to alleviate the problem. Writing to files is still an issue. In my application I eventually disabled it.
For issue #3, trying to load some large scripts (10 to 20KB in size, 3 or 4 files, each of which importing scripts 1 to 2K in size) causes memory allocation errors while loading. Rearranging the order works sometimes, but modifying the files even with a print makes the error reappear.
I tried reducing the bloat for unused class methods and other things, but does not seem to help. I also tried to call gc.collect() in between imports, that helped but eventually hit the problem again.
For issue #4, I am trying to implement a web based application where the data exchange needs to handle about 600 bytes at most, and that causes occasional memory errors. The data needs to be scanned and manipulated to extract some information, typical messaging (web or network).
gc.collect() helps a bit, but not reliable.

In C land, i could easily pre-allocate buffers (typical in embedded systems) and manipulate them, I am not sure in MPY how you would do that based on the fact that the buffers are need to manipulated.

Note, that I have checked the available memory (gc.mem_free()) while loading modules, and the memory seems sufficient before loading the module. Assuming that function is accurate.

Any help would be greatly appreciated. Thank you.

[Damien]

1. Lack of debugger
2. Issues with file system and corruption
3. Issues with loading modules (memory allocation failures)
4. Issues with run time buffer allocation (memory allocation failures)

#1 & #2 are worthy topics to discuss, but I think #3 & #4 are more urgent at the moment so let's discuss them.

I have some ideas to help out here, which are not as drastic as rewriting the memory system to eliminate fragmentation.

1. Do you use the flash filesystem at all? If not then we can reclaim 64k RAM. It's not contiguous with the existing 128k, and can't be used for executing code, but it could still be used for other things and might help you to get just enough RAM for fragmentation to stop being an issue.

2. Can you spell out in more detail your loading process for #3? Eg do you have main.py importing 3 large modules, which then import small ones? (Ideally if you could send me your code I can test it myself...?)

3. It's possible that string interning is fragmenting memory a lot and leading to poor behaviour importing lots of scripts. I have plan to fix string interning that should help here. Alternatively, if you are able to build the firmware yourself, you can make a very simple but effective fix: predefine all your identifiers in C so they are put in ROM. To do this, simple add your identifiers (all the words used in your scripts) to stmhal/qstrdefsport.h. This should help reclaim a lot of memory. To see how much you currently use for interned strings, run pyb.info() and look at the qstr n_total_bytes= line.

4. Are you using preallocated bytearrays for your buffers in point #4? Or is that not possible? Can you explain in a bit more detail the processing you do with the 600 bytes?

5. With your processing in point #4, is the output of the processing a dynamic structure (like a list or dict), or is it a simple set of numbers, or a single string, or similar?

[nelfata]

1.
I currently use the internal flash to store configuration and calibration data. Now the data is being stored in a file and that is 1 to 2K in size, so the file system capacity could be reduced as you suggest.

2.
It is close to what you describe. I will try to wrap something up and send you to your personal email if you don't mind.

3.
I can build the firmware sure, I have added some modules but nothing modified other than that. So I can try your suggestion.

4 and 5.
I tried using bytearrays, but had some issues as byte arrays are immutable, on the other hand bytes are not and easier to manipulate. Perhaps I misused them, but pre-allocating them (as a class variable) did not do the effect I expected, I think it was due to the fact that manipulation to the buffer was needed and required additional allocations.
The concept is quite simple, the code expects a connection (done periodically) to update sensor data from a client (web browser), it is mainly acting as a webserver. The data being exchanged is a dictionary but converted to a byte stream with the JSON library (ujson). This byte stream is about 600 bytes. So this stream of bytes is sent to the client.
I had to send this data in blocks also, because passing the buffer and sending it to the lower layer module (wlan) seemed to work unreliably, but having block size of 256 or even 128K makes it at least stable.
In addition, I created another application to reduce the overhead on the board by dealing with the web/html processing on a PC which then relays the requests to simpler well defined messages. That worked until I introduced the long message streams (as mentioned about 600 bytes in size) and got similar behavior, the Json conversion resulted in memory allocation errors.
Of course I could have reduced the stream of bytes being sent, but I think the problem is still lurking. Overall 600 bytes is not a large chunk, or maybe I am mistaken).

In any case, I will try to see how I can prepare a subset of the code that fails so you can review it. Loading the modules is the biggest issue and easy to replicate.

I think after we resolve these issues, it would be good to advertise some guidelines so other developers will not fall into the same problems.

Thank you for the help.

[Damien]

Regarding points 4 and 5, let's not get too off topic here and instead discuss them over at http://forum.micropython.org/viewtopic.php?f=3&t=103

[thorwald]

The 2 ideas I have to eliminate fragmentation are: 1) a copying memory manager/garbage collector, that can move parts of the heap around so as to make contiguous regions available; 2)

Here's source code for a copying/compacting memory manager:

https://github.com/contiki-os/contiki/b ... lib/mmem.c