MicroPython Forum (Archive)

Hi

I plan to embed micropython in a standalone application written in C that runs on
a Linux/ARM embedded platform. The purpose is to execute short computation
kernels written in Python, with a context (ie. input / output variables) that depends
on the C application. The C application set inputs variables, and get outputs after
the kernel execution.

To validate and benchmark the approach, I have written a simple testing environment:
https://github.com/texane/micropython_t ... nch/main.c

So far, it works quite well. Now, I am interested in optimizing the C operations associated
with the repeated execution of the Python computation kernel (ie. mostly loading and
storing variables).

Would you have any advice to optimize this kind of execution flow ?

For instance, I want to avoid using the garbage collection as much as possible ... To do
so, I try to minimize the mp_obj_new_xxx calls . But I see that there is no way to set an
object value. For instance, I had to add mp_obj_float_set, but it is not clean at all. Any
idea ?

Thanks a lot for any help,

You should probably look at uctypes.

This would allow you do define a memory region containing various types of data which might come from a c struct. The C code could pass a pointer to the struct to python and python could use an already initialized uctypes to extract the data.

But realistically, the time (on the C side) to load and store variables should be quite small in comparison to the time it takes for the python code to execute.

Or maybe I'm missing something? Perhaps you could provide a more concrete example.

Hi,

thanks for your answer, I will investigate uctypes.

The code I provided contain a matrix multiplication example, which is
quite concrete to me ...

The problem is not the time taken to load/store arguments and create
objects (mp_obj_xxx_t) associated with them. Rather, the problem is to
run the GC in order to collect the million of useless objects resulting from
this operation. After a few million iterations of the matrix multiplication code,
the heap gets full. At 1KHz, it takes a few hours reach this point.

You can also force a garbage collection. By calling gc.collect()

Performing any type of floating point operations will allocate intermediate floats. Even simple assignments like x= 1.5

Calling gc.collect on heap without many objects will be considerably faster than with a heap full of objects. So you may want to try calling gc.collect () every Nth iteration.

Thanks for the advice. I was also thinking about completely releasing the
heap and reallocating a fresh one. 1x free,malloc in the C program ...

fabien.lementec, embedding into existing applications is an area where MicroPython is a bit behind so far. People of course do that, and there were attempts to make steps to "officially" support that, but so far these attempts weren't fruitful (proposed patches from one party aren't liked by different parties, repeated). I can point to my "easy-embed" branch as a nice starting point (IMHO): https://github.com/pfalcon/micropython/ ... 7ec2b37325 . And actually, I'd love feedback whether it's helpful reference - again, my attempt to submit it upstream failed, but it's definitely clearer than previously proposed patches, so I'd like to collect votes to have it included after all, as this problem exists, and looking forward, there will be only more requests.

But I see that there is no way to set an
object value. For instance, I had to add mp_obj_float_set, but it is not clean at all. Any
idea ?

So, the way MicroPython is so small while offering so much builtin functionality is that we minimize any "abstraction" bloat. To set value of float, you just need to cast generic mp_obj_t to a specific float object and get/set value with -> C operator.

When that approach may stop to scale is when we have support for dynamically loaded native modules. And indeed, stabilizing API would be on critical path to let that work. So, feel free to watch https://github.com/micropython/micropython/pull/1627 .

Generally, all this is not yet supported because few people needed it, and even less actually looked into that. So, we need people to test what's available and provided feedback. It's crucial that people know what's already available, instead of trying to reinvent the wheel, because otherwise there will be only wasted effort and lack of progress.

Thanks, I will give a look at your fork and tell what I think.

I dont understand the casting method you suggest. Can you give
me an example of how to cast an integer and a float to set their
value ?

Thanks

Well, just have a look at objfloat.c, and you'll see a lot of it, e.g. how mp_obj_float_get() implemented.

Thanks, I will ... I thought it was not possible to do so. For instance, the mp_obj_uint_t
may be implemented using mpz if I remember well ... so casting does not make sense,
but I may have missed something at the time I looked for.