advices / code review for embedding in C application

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
Post Reply
fabien.lementec
Posts: 5
Joined: Wed Dec 02, 2015 11:54 am

advices / code review for embedding in C application

Post by fabien.lementec » Wed Dec 02, 2015 12:30 pm

Hi

I plan to embed micropython in a standalone application written in C that runs on
a Linux/ARM embedded platform. The purpose is to execute short computation
kernels written in Python, with a context (ie. input / output variables) that depends
on the C application. The C application set inputs variables, and get outputs after
the kernel execution.

To validate and benchmark the approach, I have written a simple testing environment:
https://github.com/texane/micropython_t ... nch/main.c

So far, it works quite well. Now, I am interested in optimizing the C operations associated
with the repeated execution of the Python computation kernel (ie. mostly loading and
storing variables).

Would you have any advice to optimize this kind of execution flow ?

For instance, I want to avoid using the garbage collection as much as possible ... To do
so, I try to minimize the mp_obj_new_xxx calls . But I see that there is no way to set an
object value. For instance, I had to add mp_obj_float_set, but it is not clean at all. Any
idea ?

Thanks a lot for any help,

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: advices / code review for embedding in C application

Post by dhylands » Wed Dec 02, 2015 4:43 pm

You should probably look at uctypes.

This would allow you do define a memory region containing various types of data which might come from a c struct. The C code could pass a pointer to the struct to python and python could use an already initialized uctypes to extract the data.

But realistically, the time (on the C side) to load and store variables should be quite small in comparison to the time it takes for the python code to execute.

Or maybe I'm missing something? Perhaps you could provide a more concrete example.

fabien.lementec
Posts: 5
Joined: Wed Dec 02, 2015 11:54 am

Re: advices / code review for embedding in C application

Post by fabien.lementec » Wed Dec 02, 2015 5:46 pm

Hi,

thanks for your answer, I will investigate uctypes.

The code I provided contain a matrix multiplication example, which is
quite concrete to me ...

The problem is not the time taken to load/store arguments and create
objects (mp_obj_xxx_t) associated with them. Rather, the problem is to
run the GC in order to collect the million of useless objects resulting from
this operation. After a few million iterations of the matrix multiplication code,
the heap gets full. At 1KHz, it takes a few hours reach this point.

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: advices / code review for embedding in C application

Post by dhylands » Wed Dec 02, 2015 7:33 pm

You can also force a garbage collection. By calling gc.collect()

Performing any type of floating point operations will allocate intermediate floats. Even simple assignments like x= 1.5

Calling gc.collect on heap without many objects will be considerably faster than with a heap full of objects. So you may want to try calling gc.collect () every Nth iteration.

fabien.lementec
Posts: 5
Joined: Wed Dec 02, 2015 11:54 am

Re: advices / code review for embedding in C application

Post by fabien.lementec » Wed Dec 02, 2015 8:39 pm

Thanks for the advice. I was also thinking about completely releasing the
heap and reallocating a fresh one. 1x free,malloc in the C program ...

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: advices / code review for embedding in C application

Post by pfalcon » Sat Dec 05, 2015 11:41 am

fabien.lementec, embedding into existing applications is an area where MicroPython is a bit behind so far. People of course do that, and there were attempts to make steps to "officially" support that, but so far these attempts weren't fruitful (proposed patches from one party aren't liked by different parties, repeated). I can point to my "easy-embed" branch as a nice starting point (IMHO): https://github.com/pfalcon/micropython/ ... 7ec2b37325 . And actually, I'd love feedback whether it's helpful reference - again, my attempt to submit it upstream failed, but it's definitely clearer than previously proposed patches, so I'd like to collect votes to have it included after all, as this problem exists, and looking forward, there will be only more requests.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: advices / code review for embedding in C application

Post by pfalcon » Sat Dec 05, 2015 11:54 am

But I see that there is no way to set an
object value. For instance, I had to add mp_obj_float_set, but it is not clean at all. Any
idea ?
So, the way MicroPython is so small while offering so much builtin functionality is that we minimize any "abstraction" bloat. To set value of float, you just need to cast generic mp_obj_t to a specific float object and get/set value with -> C operator.

When that approach may stop to scale is when we have support for dynamically loaded native modules. And indeed, stabilizing API would be on critical path to let that work. So, feel free to watch https://github.com/micropython/micropython/pull/1627 .

Generally, all this is not yet supported because few people needed it, and even less actually looked into that. So, we need people to test what's available and provided feedback. It's crucial that people know what's already available, instead of trying to reinvent the wheel, because otherwise there will be only wasted effort and lack of progress.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

fabien.lementec
Posts: 5
Joined: Wed Dec 02, 2015 11:54 am

Re: advices / code review for embedding in C application

Post by fabien.lementec » Sat Dec 05, 2015 2:29 pm

Thanks, I will give a look at your fork and tell what I think.

I dont understand the casting method you suggest. Can you give
me an example of how to cast an integer and a float to set their
value ?

Thanks

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: advices / code review for embedding in C application

Post by pfalcon » Sat Dec 05, 2015 4:42 pm

Well, just have a look at objfloat.c, and you'll see a lot of it, e.g. how mp_obj_float_get() implemented.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

fabien.lementec
Posts: 5
Joined: Wed Dec 02, 2015 11:54 am

Re: advices / code review for embedding in C application

Post by fabien.lementec » Sat Dec 05, 2015 5:09 pm

Thanks, I will ... I thought it was not possible to do so. For instance, the mp_obj_uint_t
may be implemented using mpz if I remember well ... so casting does not make sense,
but I may have missed something at the time I looked for.

Post Reply