compiling C modules into loadable shared objects

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
User avatar
jimmo
Posts: 1465
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: compiling C modules into loadable shared objects

Post by jimmo » Sun Sep 08, 2019 5:03 am

sebi wrote:
Sat Sep 07, 2019 3:28 pm
Sorry but I don't get what reasons described above you are referring to. Using HAL functions from within the C-code would be of great interest I believe.
I agree for sure, and it's definitely a feature worth working on!

This is about the point where I'm at the limits of my knowledge of the implementation of this feature and the IDF, but here's a simple example that highlights why this isn't "trivial".

Let's say there's a function "do_esp32_thing" in the IDF. You'd like to use it from both the main micropython firmware, but also from your loadable shared object. (Perhaps the function is something like "set_pin_mode"). The simple answer is that both the main firmware and the shared library include a copy of this piece of code (i.e. at link time, they both statically link in the relevant bits from the IDF object files). But now you have two copies of it on your ESP32. Or equivalently, what if you had two shared libraries that both use this IDF function?

It would be nice instead if the build system was clever enough to realise that this bit of code that you want to use in your shared library already exists in the micropython firmware. But where is it in the firmware? Can you rely on the address being consistent across builds? What if it's been inlined, etc.

These are all "solved problems", of course. Most of the time you don't have to think about this on Unix/Windows because the loader is much more sophisticated and you can afford to spend lots of bytes on symbol tables and stuff. And not to mention _lots_ of bytes on implementing a sophisticated loader. But this is MicroPython. (And TBH, even on Unix/Windows/etc, you do still end up thinking about this stuff pretty often)

This may seem like a contrived example, maybe having two copies of a function isn't too serious. But maybe a better example -- what if you have some ESP thing initialized in the micropython firmware, and the state is stored in some variable from IDF code somewhere. Now you use a totally different function in your shared library, but that function also uses that shared state. The compiler needs to know where that variable lives so it can generate correct code to access the variable in the right memory location. (Or the loader needs to be aware that it needs to fix it up at load time).

The way the native shared library feature (as demo'ed by Damien in those videos) works right now is that a small set of helper functions at known locations provide access to core micropython functionality. Also because the interface it's providing is based on sort of "Python-level" operations, it's simpler -- being a dynamic language, Python essentially provides a high-level loader like functionality anyway. (i.e. operations like "get the value of attribute 'foo' from this object"). Which is great if you're accessing Python functionality (i.e. most of MicroPython), but not so great for generic C APIs.

This isn't an issue for the QR code example from the videos because it (the QR library) is an entirely self-contained piece of code that has no other dependencies or shared state. Ok that's not actually true...it depends on (IIRC) strlen() so there's some handling for that.

OutoftheBOTS_
Posts: 732
Joined: Mon Nov 20, 2017 10:18 am

Re: compiling C modules into loadable shared objects

Post by OutoftheBOTS_ » Sun Sep 08, 2019 7:44 am

jimmo wrote:
Sun Sep 08, 2019 5:03 am
This isn't an issue for the QR code example from the videos because it (the QR library) is an entirely self-contained piece of code that has no other dependencies or shared state. Ok that's not actually true...it depends on (IIRC) strlen() so there's some handling for that.
OK I see there is 2 sort of cases of adding C code:
1. U just need to off load a very heavy computational part of your program to C e.g the QR code example
2. C code that needs access to internal parts of python functionality.

Although I can see the usefulness of case 2, I can also see the huge complexity involved considering the amount of different ports of MP and the list of ports is only going to grow.

Maybe on the short term and as a stepping stone it is possible to implement just the case 1 first. e.g if I need some computationally heavy task done I can write it in C then there is a tool that can add it to the flash without the need to recompile firmware.

stijn
Posts: 429
Joined: Thu Apr 24, 2014 9:13 am

Re: compiling C modules into loadable shared objects

Post by stijn » Sun Sep 08, 2019 10:58 am

With case 1 do you maen anything which access the public API, and/or can you give an example of what you consider case 2? Just asking because the public API of MicroPython (i.e. all functions which are non-static and accessible through it's headers) is pretty large in that it exposes pretty much everything needed to extend MicroPython. So not only the typcial mp_obj_newxxx etc which you always need for case 1 but also mp_module_register and allocation for example. And everything which isn't exposed - what I'd consider truly 'internal' but that's just because of the lack of another definition - usually makes little sense trying to access from the outside so isn't really anything you'd use in external C modules anyway.

OutoftheBOTS_
Posts: 732
Joined: Mon Nov 20, 2017 10:18 am

Re: compiling C modules into loadable shared objects

Post by OutoftheBOTS_ » Sun Sep 08, 2019 8:19 pm

OK trying not to get in over my head of understanding here.

My understanding is this.

case 1 is that I write a C function to do a heavy computational task maybe a numpy type task e.g fft. MP passes a buffer of data to that C function and the C function executes the fft and then returns a buffer with the output. From my understanding the C function that performs the fft doesn't need access to any of the MP modules (not sure how RAM allocation for the C functions works). I assume this compiled C function is stand alone machine instruction executed from flash that just needs ram for it's internal variables.

For me I love using python to create my projects as it is very fast and easy to develop in (also kiddie friendly, I have 3 kids). I find the big draw back for python is when you need fast heavy computational tasks. Having the ability to be able to write just this 1 small heavy computational part in C and compile that 1 function then add it to the firmware super easily would really addresses the biggest draw back of python.

User avatar
jimmo
Posts: 1465
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: compiling C modules into loadable shared objects

Post by jimmo » Sun Sep 08, 2019 10:23 pm

OutoftheBOTS_ wrote:
Sun Sep 08, 2019 8:19 pm
case 1 is that I write a C function to do a heavy computational task maybe a numpy type task e.g fft.
That's the case that was demoed in Damien's video from the Melbourne MicroPython Meetup. It's the same as generating the QR code.

OutoftheBOTS_
Posts: 732
Joined: Mon Nov 20, 2017 10:18 am

Re: compiling C modules into loadable shared objects

Post by OutoftheBOTS_ » Mon Sep 09, 2019 1:21 am

jimmo wrote:
Sun Sep 08, 2019 10:23 pm
OutoftheBOTS_ wrote:
Sun Sep 08, 2019 8:19 pm
case 1 is that I write a C function to do a heavy computational task maybe a numpy type task e.g fft.
That's the case that was demoed in Damien's video from the Melbourne MicroPython Meetup. It's the same as generating the QR code.
Again Jimmo I can't thank you enough for continuing to volunteering your time to answer our questions.

In Damien's video he wrapped it up in an .mpy file. Obviously it would be great to have a tool that can do this wrapping up into a .mpy but a tool that can add it to flash in the boot sector is even better.

I would image that for most ports of MP available flash is easier to come by than available RAM

User avatar
jimmo
Posts: 1465
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: compiling C modules into loadable shared objects

Post by jimmo » Mon Sep 09, 2019 3:10 am

OutoftheBOTS_ wrote:
Mon Sep 09, 2019 1:21 am
Again Jimmo I can't thank you enough for continuing to volunteering your time to answer our questions.
You're most welcome! :)
OutoftheBOTS_ wrote:
Mon Sep 09, 2019 1:21 am
In Damien's video he wrapped it up in an .mpy file. Obviously it would be great to have a tool that can do this wrapping up into a .mpy but a tool that can add it to flash in the boot sector is even better.
Yep, I know this is being considered, but I have no idea where it fits on the giant list of things that need to be done :)

I don't know any details but I imagine it would be feasible to make this completely transparent to the user -- i.e. put the .mpy on the regular filesystem using whatever tool (pyboard.py, ampy, rshell, etc), then in the board/port config define a scratch space, then the user just imports the module like normal, with a short delay the first time as it copies to flash, but subsequent loads are cached etc.

stijn
Posts: 429
Joined: Thu Apr 24, 2014 9:13 am

Re: compiling C modules into loadable shared objects

Post by stijn » Mon Sep 09, 2019 8:49 am

OutoftheBOTS_ wrote:
Sun Sep 08, 2019 8:19 pm
From my understanding the C function that performs the fft doesn't need access to any of the MP modules (not sure how RAM allocation for the C functions works).
Ok I get what you're saying. Such fft example indeed doesn't need much 'internal' things, but just returning a new buffer does mean having to call mp_new_xxx() which does mean access to the uPy memory allocation routines which possibly leads to garbage collection etc. So yeah most of it is 'standalone' but getting data out in this case does require caling into the MicroPython API.

v923z
Posts: 118
Joined: Mon Dec 28, 2015 6:19 pm

Re: compiling C modules into loadable shared objects

Post by v923z » Mon Sep 09, 2019 9:48 am

stijn wrote:
Mon Sep 09, 2019 8:49 am
OutoftheBOTS_ wrote:
Sun Sep 08, 2019 8:19 pm
From my understanding the C function that performs the fft doesn't need access to any of the MP modules (not sure how RAM allocation for the C functions works).
Ok I get what you're saying. Such fft example indeed doesn't need much 'internal' things, but just returning a new buffer does mean having to call mp_new_xxx() which does mean access to the uPy memory allocation routines which possibly leads to garbage collection etc. So yeah most of it is 'standalone' but getting data out in this case does require caling into the MicroPython API.
I think, what @OutoftheBOTS was trying to say would not require a call to mp_new_xxx(): you can write the results into the same buffer, or you can pass two buffers to the external module, one with the original data, and one for the processed data. You would create those in the interpreter. I believe, his idea was similar to what you do with pointers in C: you haven't got to return anything, if you can pass the pointer to the function.

stijn
Posts: 429
Joined: Thu Apr 24, 2014 9:13 am

Re: compiling C modules into loadable shared objects

Post by stijn » Mon Sep 09, 2019 10:14 am

Yes that's also possible, but then the calling code has to know exactly what the function is going to do and create a buffer of the correct size. Certainly doable, but more error prone and increases coupling of the code.

Post Reply