Project: run existing Python applications with MicroPython

[pfalcon]

This all started with a comment from Damien some time ago: http://forum.micropython.org/viewtopic. ... &t=49#p157 :

If Micro Python could run Django, even in some heavily stripped down form, then that would get people's attention. Imagine nginx + uPy

(see also previous message in that thread)

I'm still not sure if Damien was joking regarding Django . And that's definitely not how I intended to use MicroPython myself - while I care primarily about unix port, my original interest is to write unbloated code mostly from scratch. But I picked up Damien's call by following reasons: 1) if we implement Python, it should be able to run existing Python code; 2) specifically, making Python3 standard lib and testsuite would be important milestones.

So, I took Bottle and started to look into what it would take to make it run. That immediately recursed into stdlib, into things like http.server & http.client, whose dependencies I traced down to "leaf" modules like "copy". It took around 5 patches to uPy core to get "copy" to run. Exhausting. In 3 days, I was ready to make this post with conclusion: unrealistic to achieve in short time, long term goal. Well, instead of writing this post, I backlogged it, and wrote another patch.

So, while initially highly unrewarding, following this path uncovers many issues in uPy, and thus very good way to improve it. It's also grounded way to do so, rather than based on own speculations on what should be implemented and what not (if Python3 stdlib requires it, then well, we'll need to implement to be able to claim we're compliant Python implementation).


So, I finally open this thread to get together people interested to run existing apps on uPy, and to share experiencing trying to do so.

[pfalcon]

Just to clarify: this thread is not a place for requests like "Please make X run with MicroPython". This is the place for posts like "I consider running X on MicroPython to be so cool/interesting/useful, that I'm going to work on that".

[torwag]

Hi pfalcon,

Oh, oh, oh could you pleeaazzzeee make matplotlib and pygames running under micropython please!!!! ..... just kidding

I totally agree with your conclusion. It would be great if you could find the time to write down your findings and your finally established work flow for this particular example. Thus, other people get an idea how to start porting other libs and how to proceed. The dev wiki would be a good place.
A new page "How-to porting libs to micropython"?!
I for myself would be interested in porting pyusb which in turn would need a STM specific backend and most likely I will face similar challenges as you described.

Thanks for bringing this up

[pfalcon]

The problem of wiki pages is that they need to written, maintained, and even read. That's complicated, why not instead chat about it like this thread does .

Well, I still leisurely storm CPython http.client module to have it run with uPy. And I'm on the brink of being able to do that! Quite a lot needed to be changed in uPy for that, for example, more faithful object model implemented (__new__ method), them more faithful stream model - with proper separation between byte vs text streams.

But overall, it looks good, and amount of changes needed start to be pretty small. Here're examples of changes neded: https://github.com/micropython/micropyt ... 34ca495d25 , https://github.com/micropython/micropyt ... 39f4fada2d .

[pfalcon]

I also know that other people look into running existing code with uPy, like for example https://github.com/micropython/micropython/issues/602 suggests, but so far there're no details of those efforts.

[pfalcon]

Well, I still leisurely storm CPython http.client module to have it run with uPy.

Here's proof dumpshot that this is possible:

Code: Select all

$ /home/pfalcon/projects-3rdparty/micropython/unix/micropython -X heapsize=600000 test_client.py 
Notice: /home/pfalcon/.micropython/lib/email is imported as namespace package
Notice: /home/pfalcon/.micropython/lib/urllib is imported as namespace package
Address: ('localhost', 80)
Address infos: [(2, 1, 6, None, bytearray(b'\x02\x00\x00P\x7f\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00')), (2, 2, 17, None, bytearray(b'\x02\x00\x00P\x7f\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00')), (2, 3, 0, None, bytearray(b'\x02\x00\x00P\x7f\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00')), (2, 1, 6, None, bytearray(b'\x02\x00\x00P\x7f\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00')), (2, 2, 17, None, bytearray(b'\x02\x00\x00P\x7f\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00')), (2, 3, 0, None, bytearray(b'\x02\x00\x00P\x7f\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00'))]
<class 'NoneType'>
<HTTPResponse object at 0x46bef0>
b'<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL / was not found on this server.</p>\n<hr>\n<address>Apache/2.2.22 (Ubuntu) Server at localhost Port 80</address>\n</body></html>\n'

Of course, that doesn't mean that it really "works", just that one execution path within pretty dirty bloated code (after some patching) is now within MicroPython's abilities. Hurrah .

[pfalcon]

Example of changes which were required to run pyelftools library: https://github.com/pfalcon/pyelftools/c ... hon-compat

[cloudformdesign]

I'm going to port the msgpack library to micropython. msgpack is like json, but fast and small. Check out msgpack here: http://msgpack.org/

I think this library would be the best to start with as it is pure python and fairly simple (one file): https://github.com/vsergeev/u-msgpack-python

To do this I at least need io.BytesIO, check out my PR here: https://github.com/micropython/micropython-lib/pull/29

BAM: this was a pretty easy port indeed: https://github.com/vsergeev/u-msgpack-python/pull/14

[rankor]

Why Django and not Flask? Flask seems more suited.

[mbuesch]

I'm currently trying to get my soft-PLC running on µPy:
http://bues.ch/h/awlsim
So far the backend starts and runs the example code correctly. It still fails in the regression tests, though.

Why run this on µPy? Well, a PLC inherently is something that is supposed to run on a microcontroller. My current primary interpreter is pypy to do so, because I have plenty of memory (RaspberryPi and such). But it is nice to see that µPy uses even less memory than CPython to run awlsim. Nice. However the execution speed is slower than CPython (and has some weird jitter effect. May also come from 'time.perf_counter', which is used for measurement.). So it would be good to have some improvement in that area, too.

I'm currently running it on µPy/Unix, because awlsim (still) has a lot of Unix dependencies built in and it still uses quite some memory. I currently use a 16M heap. But that can probably be shrunk.