Micro Python development wish list

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
Damien
Site Admin
Posts: 622
Joined: Mon Dec 09, 2013 5:02 pm

Micro Python development wish list

Post by Damien » Thu Dec 18, 2014 12:23 pm

Micro Python is now a solid piece of software, with a heap of interest, and many different applications. There are still gradual improvements and bug fixes to make, but that's the case for all software :)

There are also some significant enhancements that can be made to the underlying code to make Micro Python even better. These ideas for enhancements come from feedback from using uPy in real-world applications, and can be split into 2 use-cases.

Improvements for uPy when used on a microcontroller:
  • Eliminate heap-memory fragmentation
    When running for an extended period of time (usually after allocating large buffers), the heap can become very fragmented, and although there is enough free memory for an allocation, the free memory is not contiguous (all in a row). Such a problem severely limits uPy's use in critical and long-running applications (eg a web server, a quadcopter). Code running on a microcontroller needs to be a lot more robust than desktop software running on a PC, and so special care must be taken to design a robust memory allocator.

    It's not clear how to improve uPy's current memory allocator so that fragmentation is reduced. Ultimately I would want to see a solution that completely eliminates fragmentation, so that programs are guaranteed to run indefinitely. Note that heap fragmentation is not because of the use of garbage collection: even with reference counting, or explicit free, you still have the same amount of fragmentation. The 2 ideas I have to eliminate fragmentation are: 1) a copying memory manager/garbage collector, that can move parts of the heap around so as to make contiguous regions available; 2) a fixed-block memory manager than allocates a single block size (eg 16 bytes), and if you want more you must chain multiple blocks (like a filesystem). Both of these ideas are difficult to implement, and will come with a speed and memory cost. But the cost will be worth it if we can get guaranteed memory allocation.
  • Ability to allocate memory on an interrupt
    Interrupts and callbacks play a large role in microcontrollers, especially to get real-time performance, and uPy should have good support for them. Currently you can run code on an interrupt, but you can't allocate heap memory during such an interrupt. The memory allocator can be made re-entrant so that memory can be allocated during an interrupt (steps have been made in this direction already, but more work and testing is needed). With this improvement you could write some very sophisticated control code that responded within a guaranteed time to an external event.
Improvements for embedded use (eg as replacement for Lua, or its Javascript equivalent: duktape, https://github.com/svaarala/duktape):
  • Eliminate need for any assembler in core
    Micro Python currently uses some assembler for exceptions (NLR code) and garbage collection. Removing this dependence, and making it pure C, would make embedding uPy much easier (and would make it possible to use Emscripten to compile uPy to Javascript to run it in the browser). NRL code can use setjmp/longjmp, but it would be nice to have the option to use, eg, native C++ exceptions, or something like that.
  • Eliminate all global state
    Embeddable languages (eg Lua) don't have any (much?) global state, and instead pass around a pointer to the interpreter state as the first argument to all functions. This keeps the state self-contained, and allows multiple instances of the language to run in the same process/thread. uPy doesn't need this feature for microcontroller use, but it would be very useful to have for embedded use. It's a straight-forward thing to implement, but touches all functions in the entire code base, and will increase code size, stack usage, and reduce speed.
I would really like to see all of these improvements implemented in Micro Python. It would be good if they could be made compile-time options, since they all impact on efficiency in some way, and are not all needed at one time (eg on a microcontroller, you don't care about having assembly, or global state; and in embedded, you don't care about interrupts).

Are there any ideas about how to implement the above? Are there any other significant enhancements that people would like to see?

blmorris
Posts: 348
Joined: Fri May 02, 2014 3:43 pm
Location: Massachusetts, USA

Re: Micro Python development wish list

Post by blmorris » Fri Dec 19, 2014 4:31 am

Top of my wish list would be I2S support, though I may be the only one asking for it. I have developed hardware (currently in production) on which an I2S module could be tested and put to use almost immediately - an audio amplifier which uses the STM32F405 to stream audio data from an SD card to a codec for playback. I had ambitiously hoped to be able to implement and contribute the I2S module myself, but it is a bit daunting for someone who wrote essentially zero lines of C code in the time between taking a class in '97 and learning about micropython a year ago.

We do have working code implementing I2S functionality (written by a colleague with some assistance from me.) However, it was developed using a much older version of the STM libraries, and apparently not easily portable to micropython. By now, micropython is a lot more advanced in most other regards than our firmware, and obviously scripting in Python is a lot more flexible than recompiling C code anytime we need to do a customization (which, of course, was the entire point of developing an audio amplifier with a microcontroller.)

If I can sell the idea further, this is also an opportunity to put micropython's stability in long-running applications to the test - our amplifiers often run for months at a time in commercial applications.

-Bryan

detour_vmx
Posts: 4
Joined: Fri Dec 19, 2014 3:15 am

Re: Micro Python development wish list

Post by detour_vmx » Sun Dec 21, 2014 8:53 am

My $.02:


Eliminate all global state:

I'd vote for this as the highest priority. Cooperative multi-tasking can
go a long way, but it gets dicey in programs with lots of math
computations involved and some hard real time events that need to be
serviced. Preemption is a solution that doesn't require yield in the compute functions.

Micropython is the most exciting project I've seen on the Internet. I'd love to be able to run multiple interpreters as independentt tasks in my little task multiplexer system.


Eliminate need for any assembler in core:

Micropython will probably be running on ARM processors for the near future, so porting some assembly to a different ARM core shouldn't be so difficult that it warrants a lot of effort at this time.


Memory allocation, aka black hole:

If you want a severe headache, this is the subject to explore.

Some comments;
1. Merging allocation for interrupt routines and task level is a sure fire way to generate bugs that may not be exposed for a very long time.
2. Interrupt routines should not be so complicated they need to dynamically allocate memory. Save the state and push the work up to task level.
3. Any interrupt routine that "must" have allocation should just use a pool of pre--allocated buffers.

That said, I do have a small malloc/free package that is being used in a fairly critical application. It was designed to be small, allocate fast, and not be real slow on free. I'm willing to share that if you want to try it out.

User avatar
kfricke
Posts: 342
Joined: Mon May 05, 2014 9:13 am
Location: Germany

Re: Micro Python development wish list

Post by kfricke » Sun Dec 21, 2014 2:12 pm

detour_vmx wrote:...
Memory allocation, aka black hole:
...
Some comments;
1. ...
2. Interrupt routines should not be so complicated they need to dynamically allocate memory. Save the state and push the work up to task level.
3. Any interrupt routine that "must" have allocation should just use a pool of pre--allocated buffers.
I'd sign this and part. The restriction is IMHO not a really big hurdle. It should maybe pointed out on a little more. Once one notices this "behaviour" you can already spot some really goode advice to cope with it easily.
This restriction forces people to write tight interrupt call back methods (ISRs), which like detour already did outline should be forced design goal.

To tackle most of the real problems with this topic, data structures like queues and stacks could be implemented or get specialized variants which do not allocate memory after initialization. Of course this is only posbible for primitive data types.

stijn
Posts: 366
Joined: Thu Apr 24, 2014 9:13 am

Re: Micro Python development wish list

Post by stijn » Mon Dec 22, 2014 1:04 pm

Interrupt routines should not be so complicated they need to dynamically allocate memory
Well that's what I've always learned as well. But it's not because it is the de-facto standard (mostly due to technicalities) that Micropython cannot step up and try to change that rule, within limits.


Wishlist for me would be all 4 of the above and dynamically loadable modules, not only because I happen to use them the whole time but also because I think they are beneficial (despite their disadvantages - mainly on the build side of things, but not negligable)
First of all the functionality: to get the same effect now, i.e. getting your own C code called from within scripts, you basically need to add code to the uPy source directly at build time. Which is messy and incovenient to maintain. The other option is modffi, but it's not as peformant, has a rather cumbersome syntax, has no support for classes and is currently only in the unix port and not exactly easy to get in other ports.
Second: often the question raises whether some feature should be in uPy (and possibly enabled/disabled via a #define), or simply not. There are no other options. So everytime there's a trade off between if it's needed/wanted be enough people/codesize/staying 'micro' etc. People end up with custom build configuration to enable/disable what they want. Or end up without a feature they want. Modules could give a third choice: extra functionality but without intervening with the core in any way, not even a define or makefile adjustment. They could even be developped in seperate repositories which keeps the main tree clean and free of external code.

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: Micro Python development wish list

Post by pfalcon » Thu Dec 25, 2014 11:36 pm

Improvements for embedded use (eg as replacement for Lua, or its Javascript equivalent: duktape, https://github.com/svaarala/duktape):
Live and learn. I never heard about duktape even though it appears to have some following, based on github stars.

On the other hand, I recently had a look at Tessel, which is JavaScript hack on top of Lua. It hardly seems to be too popular (again, based on github stars), but they have Contribution Guide: https://github.com/tessel/contribution-guide/ . I guess that's what we should have too. It doesn't have to be too technical, but should give people interested in contributing to MicroPython ideas of its approach and philosophy. With that in mind, I've written draft doc: https://github.com/micropython/micropyt ... Guidelines , which surely needs elaboration.

Working on docs in general should be very good idea too.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: Micro Python development wish list

Post by pfalcon » Thu Dec 25, 2014 11:50 pm

Comments to specific points presented above:
Eliminate heap-memory fragmentation
This should be a good aim to work on. But as you point, there will be trade-off with performance and memory overhead, so it would be nice to make memory allocators more pluggable/configurable first.
Ability to allocate memory on an interrupt
As other folks say, this is probably should not be a top-level goal, but something to gradually improve in the background.
Eliminate need for any assembler in core
It's actually already possible to build interpreter without any assembler code - that's how MIPS port is being built, for example. That doesn't support native code generation though, and I'm not sure that's achievable without asm or using even more clever hacks.
Eliminate all global state
That's good background aim too, with good progress already in https://github.com/micropython/micropython/pull/1017
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: Micro Python development wish list

Post by pfalcon » Thu Dec 25, 2014 11:56 pm

stijn wrote:Modules could give a third choice: extra functionality but without intervening with the core in any way, not even a define or makefile adjustment. They could even be developped in seperate repositories which keeps the main tree clean and free of external code.
I'm positive about supporting loadable C modules, the question is of course that someone who actually has need for them should start implementing them. But that of course should be done in portable, system-independent way. No relying on particular system's implementation of dynamic linking, etc. The biggest issue here will be defining a public API which external modules can use. It will be subset of complete MicroPython API, and thus it won't be the most perfomant and most featureful way to extend MicroPython - just wanted to set expectations right.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

Damien
Site Admin
Posts: 622
Joined: Mon Dec 09, 2013 5:02 pm

Re: Micro Python development wish list

Post by Damien » Sat Dec 27, 2014 11:49 am

Okay, let's add loadable C modules as a 5th item.

For me they would idealy be: 1) portable/system independent; 2) completely static and not require any linking.

There is actually already something we can use for this: the native emitter has a function table (see nativeglue.c) to "link" native code into the rest of the system. This table contains the minimal set of functions needed to interface with the uPy runtime. No macros, no magic numbers, just a table of pointers to functions. We could simply pass this table as the first argument to loadable C functions. This would pretty much satisfy both points above, as well as being very minimal (another good criteria to satisfy). Note we would also need to add to this table pointers to the True and False constants, but that's easy to do.

Damien
Site Admin
Posts: 622
Joined: Mon Dec 09, 2013 5:02 pm

Re: Micro Python development wish list

Post by Damien » Sat Dec 27, 2014 11:58 am

pfalcon wrote: Live and learn. I never heard about duktape even though it appears to have some following, based on github stars.
I never heard either until it appeard about a month ago on Hackernews, and I think that was when it got all its stars.
pfalcon wrote: On the other hand, I recently had a look at Tessel, which is JavaScript hack on top of Lua. It hardly seems to be too popular (again, based on github stars), but they have Contribution Guide: https://github.com/tessel/contribution-guide/
Wow, Tessel compile JS to Lua? Even though I've read through some of the Tessel code (for CC3000 driver hints), I never realised this.

Their contribution guide is quite nice with lots of info. I think what we need to start with is a philosophy, contribution guide, and roadmap.

Post Reply