C module programming reference

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
v923z
Posts: 168
Joined: Mon Dec 28, 2015 6:19 pm

C module programming reference

Post by v923z » Sat Aug 31, 2019 5:10 pm

Hi all,

In the past couple of weeks, I have been working on a micropython math module similar to numpy (sneak peek under https://github.com/v923z/micropython-ulab), and I realised that a programming manual is sorely missing. I have written up a thorough reference of what I needed for ulab, and what I could learn by perusing the micropython code base.

At https://github.com/v923z/micropython-usermod/, as well as at https://micropython-usermod.readthedocs.io/en/latest/ you will find about a dozen simple examples of implementing various python features at the C level. Each feature is dissected, gutted, and explained at length. I hope that you'll find something useful.

Suggestions, corrections, admonitions and the like are more than welcome, either here, or on github.

Cheers,

Zoltán

bitrat
Posts: 41
Joined: Fri Jul 26, 2019 4:13 am

Re: C module programming reference

Post by bitrat » Sun Sep 01, 2019 2:44 am

Wow, this looks really great, thanks!!! I'm working through your examples in full!

..fyi (not a criticism) your index.rst isn't rendering. Also, I'm not that familiar with IPython etc, and you've left a space for a link here: If you don’t know what ipython magics are, you can read more at … In any case, in usermods_02.rst.

On generating docs from within jupyter... really? couldn't I just cd into a directory and type some command, then open index.html? you say make html but there's no makefile. from where should make be called. sorry, not criticism, just curious.'

UPDATE: frrrrrrpp... decided to have a look at jupyter via the github/ipython-in-depth. ....can we just have the condensed version please? First I was told I could use pip. So I do. Nek minnit, I MUST use conda, which isn't a tool I'm familiar with or want. Then, a warning that conda will install a huge amount of data... (I do embedded for one simple reason. It's small...)

Sorry about all these complaints. I really appreciate your work and it's great, but the buy in to read about it is a bit too much... Here's my perspective: programming is explaining to humans what we want the machine to do. If that weren't the case, we'd all be writing in machine code, or solder. On that basis, the most important thing to do is to explain our programming tools to new users. Jupyter is a magnificent tool for that, but not if nobody is bothered to install it.

Your intro to adding usermods is better than anything else I've found. I'll leave my comments below as is, but mostly now redundant. One thing tho, I added CFLAGS_EXTRA=-DMODULE_EXAMPLE_ENABLED=1 and also added the definition to mpconfigport.mk, but it wasn't until I added it to mpconfigport.h that things worked..

Your library gets me to thinking... it can do the color space conversion so, if you add DCT, it could be the basis of a jpeg/mpeg codec...

For myself, I'm just getting to grips with my first C extension to MicroPython and it's been frustrating having to work through errors in the instructions at http://docs.micropython.org/en/latest/d ... dules.html ... for example, no micropython.mk in the source tree (now py/py.mk I assume. No wait, that's the interpreter source... How the hell do I add a user module...? Do I just have to drop the source in extmod? Ok, I'll try that... mutter mutter...).

UPDATE: getting there, but missing some definition. compiling ports/unix, some precompile string pasting macro not happening...

Code: Select all

In file included from ../../usercmods/example/example.c:2:0:
../../usercmods/example/example.c:24:50: error: ‘MP_QSTR_example’ undeclared here (not in a function); did you mean ‘MP_QSTR_enable’?
     { MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_example) },
.
.
.

make: *** [build/example/example.o] Error 1
My make command is:

Code: Select all

make USER_C_MODULES=../../usercmods  CFLAGS_EXTRA=-DMODULE_EXAMPLE_ENABLED=1 V=1 all
I also tried adding MODULE_EXAMPLE_ENABLED = 1 to mpconfigport.mk

SOLVED!

added #define MODULE_EXAMPLE_ENABLED (1) to mpconfigport.mk (as per instructions...).

from MicroPython external C modules — MicroPython 1.11 documentation: Note that the exact method depends on the port as they have different structures. If not done correctly it will compile but importing will fail to find the module.

If this is the case, perhaps the author of the port should include an example...

So, in summary ....it would be great to have a minimal example C user module in the master branch, that newbies could hack on and get up to speed. The real learning curve here is not how to write C code, but how to mesh new files into the make. That step could be trivial it the instructions were accurate or if there were a working example. Instead, well, it took 2 hours, and .... reading Makefiles is one of my pet hates.

:-)

When I know what I'm doing, I may add an example and update the docs. My final target is ESP32, so I'll add that too.
Last edited by bitrat on Sun Sep 01, 2019 9:16 am, edited 1 time in total.

uCTRL
Posts: 47
Joined: Fri Oct 12, 2018 11:50 pm

Re: C module programming reference

Post by uCTRL » Sun Sep 01, 2019 9:09 am

So, in summary ....it would be great to have a minimal example C user module in the master branch, that newbies could hack on and get up to speed. The real learning curve here is not how to write C code, but how to mesh new files into the make. That step could be trivial it the instructions were accurate or if there were a working example.
+1.

User avatar
mattyt
Posts: 410
Joined: Mon Jan 23, 2017 6:39 am

Re: C module programming reference

Post by mattyt » Sun Sep 01, 2019 9:23 am

There is the MicroPython external C modules documentation...that's about as minimal as you can get. :)

And yes, v923z, this looks great! Thanks for your efforts, it's one of the lesser-known aspects of MicroPython development.

v923z
Posts: 168
Joined: Mon Dec 28, 2015 6:19 pm

Re: C module programming reference

Post by v923z » Sun Sep 01, 2019 5:30 pm

bitrat wrote:
Sun Sep 01, 2019 2:44 am
Wow, this looks really great, thanks!!! I'm working through your examples in full!

..fyi (not a criticism) your index.rst isn't rendering. Also, I'm not that familiar with IPython etc, and you've left a space for a link here: If you don’t know what ipython magics are, you can read more at … In any case, in usermods_02.rst.
Indeed. Thanks for pointing that out, I will update it.
bitrat wrote:
Sun Sep 01, 2019 2:44 am
On generating docs from within jupyter... really? couldn't I just cd into a directory and type some command, then open index.html? you say make html but there's no makefile. from where should make be called. sorry, not criticism, just curious.'
One thing that I didn't elaborate on was setting up sphinx. The reason for that is that I didn't want to include all the sphinx-generated files in the git commit. But you are absolutely right, I should've explained what is going on. I stand corrected.

As for jupyter, I mentioned why I used that: on the one hand, I had to write the documentation, on the other, I had to compile the code (which was part of the documentation, because I that is what I was discussing), then I had to test the code on the unix port, and sometimes I even had to test python code in CPython. I believe, jupyter is the only reasonable way of achieving all this. I understand that you can always open python, and micropython on the command line, and you can also run make for the documentation on the command line, but I felt that that was driving me crazy.
bitrat wrote:
Sun Sep 01, 2019 2:44 am
UPDATE: frrrrrrpp... decided to have a look at jupyter via the github/ipython-in-depth. ....can we just have the condensed version please? First I was told I could use pip. So I do. Nek minnit, I MUST use conda, which isn't a tool I'm familiar with or want. Then, a warning that conda will install a huge amount of data... (I do embedded for one simple reason. It's small...)
I think, the simplest, one-click way of installing ipython is anaconda: https://www.anaconda.com/distribution/ . I have never had difficulties with it, and I don't use any extra packages.
bitrat wrote:
Sun Sep 01, 2019 2:44 am
Sorry about all these complaints. I really appreciate your work and it's great, but the buy in to read about it is a bit too much... Here's my perspective: programming is explaining to humans what we want the machine to do. If that weren't the case, we'd all be writing in machine code, or solder. On that basis, the most important thing to do is to explain our programming tools to new users. Jupyter is a magnificent tool for that, but not if nobody is bothered to install it.
The thing is, you don't need to install anything. You can clone the repository, and you simply run make. You don't need jupyter. I generated the code and the documentation in jupyter, but you need only the c, and make files, which are included in the under
https://github.com/v923z/micropython-us ... r/snippets
bitrat wrote:
Sun Sep 01, 2019 2:44 am
Your intro to adding usermods is better than anything else I've found. I'll leave my comments below as is, but mostly now redundant. One thing tho, I added CFLAGS_EXTRA=-DMODULE_EXAMPLE_ENABLED=1 and also added the definition to mpconfigport.mk, but it wasn't until I added it to mpconfigport.h that things worked..
I will look into that, thanks!
bitrat wrote:
Sun Sep 01, 2019 2:44 am
Your library gets me to thinking... it can do the color space conversion so, if you add DCT, it could be the basis of a jpeg/mpeg codec...
I don't quite know what exactly it should do, but I am more than happy to include it, if you either explain it, or even better, present an implementation.
bitrat wrote:
Sun Sep 01, 2019 2:44 am
So, in summary ....it would be great to have a minimal example C user module in the master branch, that newbies could hack on and get up to speed. The real learning curve here is not how to write C code, but how to mesh new files into the make. That step could be trivial it the instructions were accurate or if there were a working example. Instead, well, it took 2 hours, and .... reading Makefiles is one of my pet hates.
As I said earlier, just clone the repository, add the

Code: Select all

#define MODULE_SIMPLEFUNCTION_ENABLED (1)
#define MODULE_SIMPLECLASS_ENABLED (1)
#define MODULE_SPECIALCLASS_ENABLED (1)
#define MODULE_KEYWORDFUNCTION_ENABLED (1)
#define MODULE_CONSUMEITERABLE_ENABLED (1)
#define MODULE_VECTOR_ENABLED (1)
#define MODULE_RETURNITERABLE_ENABLED (1)
#define MODULE_PROFILING_ENABLED (1)
#define MODULE_MAKEITERABLE_ENABLED (1)
#define MODULE_SUBSCRIPTITERABLE_ENABLED (1)
#define MODULE_SLICEITERABLE_ENABLED (1)
#define MODULE_VARARG_ENABLED (1)
to your mpconfigport.h, and run make in the port's root folder. All my jupyter does is this. (Beyond holding the documentation, the code testing, and everything else, that is.)

I hope this helps, and many thanks for the feedback!

Cheers,
Zoltán
Last edited by v923z on Sun Sep 01, 2019 5:37 pm, edited 1 time in total.

v923z
Posts: 168
Joined: Mon Dec 28, 2015 6:19 pm

Re: C module programming reference

Post by v923z » Sun Sep 01, 2019 5:35 pm

uCTRL wrote:
Sun Sep 01, 2019 9:09 am
So, in summary ....it would be great to have a minimal example C user module in the master branch, that newbies could hack on and get up to speed. The real learning curve here is not how to write C code, but how to mesh new files into the make. That step could be trivial it the instructions were accurate or if there were a working example.
+1.
As @mattyt pointed out, the minimal example is under https://docs.micropython.org/en/latest/ ... dules.html. I should also mention https://micropython-dev-docs.readthedocs.io/en/latest/ , which I found useful.

Best,

Zoltán

bitrat
Posts: 41
Joined: Fri Jul 26, 2019 4:13 am

Re: C module programming reference

Post by bitrat » Sun Sep 01, 2019 8:05 pm

Thanks all, and I apologise for my griping. In my defense, I thought a record of my struggles from a newbie's pov might be useful.

============ Anaconda diversion..
Zoltán, I'll install Anaconda, although your examples are so excellent and self explanatory, I'll probably just build them (...and rtfm later, lol).

UPDATE:
Cannot see Anaconda menu shortcuts after installation on Windows
After installing on Windows, in the Windows Start menu I cannot see Anaconda prompt, Anaconda Cloud and Navigator shortcuts.
Cause
This may be caused by the way Windows updates the Start menu, or by having multiple versions of Python installed, where they are interfering with one another. Existing Python installations, installations of Python modules in global locations, or libraries that have the same names as Anaconda libraries can all prevent Anaconda from working properly.
Solution
If start menu shortcuts are missing, Microsoft recommends rebooting your computer or restarting Windows Explorer.
If that doesn’t work, clear $PYTHONPATH and re-install Anaconda. Other potential solutions are covered in the “Conflicts with system state” section of this blog post.
... translation: "have you tried restarting windows?" A) replace my python installation. really? B) shame it didn't work :D

...last time I tried one of these notebooks (I believe it was SageMath) I downloaded a VM. It was brilliant! Installed Node-RED the same way. also brilliant. I expect there's something now with containers like Docker, but probably only on Linux. For the record, I use virtualenv for everything, but was blindsided by Anaconda informing me half way thru the install that it would replace my python installation... I don't think this is acceptable. Maybe if I reboot, but not keen, given the number of terminals I have open.

I think things might have been easier on Linux. (I'm using WSL on Windows 10 for this... Maybe I'll try it from scratch on my own machine, as I'd like to see Anaconda working, as it looks like fun :-) )

... was expecting Navigator but have Anaconda Prompt (miniconda3) so what now? Is that it?

============ end of diversion


As I said earlier, I was confused by the instructions for adding external modules to make in MicroPython external C modules where they say:
micropython.mk contains the Makefile fragment for this module.
I expected to find micropython.mk in the repo somewhere. I've now figured all this out and it makes more sense.

I think a cookbook approach would be good here (ie: step 1, step 2, step 3), aligned with a working example in the source tree, perhaps just for the unix port. Maybe it should also be in tests, so it's maintained.

The docs at Welcome to MicroPython Development Documentation are also really useful. Perhaps a link could be added to the MicroPython external C modules page?

Finally, and this is probably worthy of a separate post ...and maybe not entirely a MicroPython question at all:

In the development docs, at Adding a Module, there are some comments on memory layout:
The second file you will need to add to is esp8266.ld, which is the map of memory used by the compiler. You have to add it to the list of files to be put in the .irom0.text section, so that your code goes into the instruction read-only memory (iROM). If you fail to do that, the compiler will try to put it in the instruction random-access memory (iRAM),
This seems quite important, as I don't want my text in my ram, but there is very little detail, and nothing about esp32, which is my target. The ports/esp32 directory only seems to contain .ld files autogenerated by the espressif esp-idf. Can anybody suggest some introductory docs relating to all this?

UPDATE: It's ok, I'm figuring out how to read the map file :-) One thing I've discovered is that while global data appears in the map file (application.map, under COMMON), static data (char[N]) doesn't. Does that data get space allocated when the module is imported? Where? I've found it's declaration duplicated in the file ports/esp32/build-GENERIC/genhdr/qstr.i.last Does that mean it's being interned as a qstr? Why would that be, when it's writable? (disclaimer: it's a while since I've needed to worry about available RAM.. also, I have yet to read up on QSTRs..)

Also, the differences between the two sets of instructions (Adding a Module and adding a MicroPython external C module) raises the question: what is the difference? Are these effectively the same or fundamentally different? Does 'external module' mean a dynamic 'import' and is the other module statically linked to the interpreter?

Thanks everybody. I hope this isn't hijacking your op, Zoltán. It does seem mostly relevant, but I can start another thread if you'd prefer.

Cheers,
Bitrat

v923z
Posts: 168
Joined: Mon Dec 28, 2015 6:19 pm

Re: C module programming reference

Post by v923z » Mon Sep 02, 2019 5:26 pm

uCTRL wrote:
Sun Sep 01, 2019 9:09 am
So, in summary ....it would be great to have a minimal example C user module in the master branch, that newbies could hack on and get up to speed. The real learning curve here is not how to write C code, but how to mesh new files into the make. That step could be trivial it the instructions were accurate or if there were a working example.
+1.
I forgot to mention yesterday that in my reference, each module is a minimal C example that you can start to hack right away. I stressed this point also in the introduction, and this is something I find very important: each example is stripped to the bare minimum. I am also annoyed, when people try to explain something with a piece of code that contains five totally unrelated concepts.

I hope this clarifies the issue,

Zoltán

v923z
Posts: 168
Joined: Mon Dec 28, 2015 6:19 pm

Re: C module programming reference

Post by v923z » Mon Sep 02, 2019 5:29 pm

bitrat wrote:
Sun Sep 01, 2019 8:05 pm

Finally, and this is probably worthy of a separate post ...and maybe not entirely a MicroPython question at all:

In the development docs, at Adding a Module, there are some comments on memory layout:
The second file you will need to add to is esp8266.ld, which is the map of memory used by the compiler. You have to add it to the list of files to be put in the .irom0.text section, so that your code goes into the instruction read-only memory (iROM). If you fail to do that, the compiler will try to put it in the instruction random-access memory (iRAM),
This seems quite important, as I don't want my text in my ram, but there is very little detail, and nothing about esp32, which is my target. The ports/esp32 directory only seems to contain .ld files autogenerated by the espressif esp-idf. Can anybody suggest some introductory docs relating to all this?

UPDATE: It's ok, I'm figuring out how to read the map file :-) One thing I've discovered is that while global data appears in the map file (application.map, under COMMON), static data (char[N]) doesn't. Does that data get space allocated when the module is imported? Where? I've found it's declaration duplicated in the file ports/esp32/build-GENERIC/genhdr/qstr.i.last Does that mean it's being interned as a qstr? Why would that be, when it's writable? (disclaimer: it's a while since I've needed to worry about available RAM.. also, I have yet to read up on QSTRs..)

Also, the differences between the two sets of instructions (Adding a Module and adding a MicroPython external C module) raises the question: what is the difference? Are these effectively the same or fundamentally different? Does 'external module' mean a dynamic 'import' and is the other module statically linked to the interpreter?

Thanks everybody. I hope this isn't hijacking your op, Zoltán. It does seem mostly relevant, but I can start another thread if you'd prefer.

Cheers,
Bitrat
It is OK, but you might get more relevant responses, if you open a new thread with a more descriptive title ;)

Cheers,
Zoltán

v923z
Posts: 168
Joined: Mon Dec 28, 2015 6:19 pm

Re: C module programming reference

Post by v923z » Mon Sep 02, 2019 5:31 pm

Hi all,

I have tried to fix the issues that Bitrat pointed out in his first post. You can find the new version of the docs under https://micropython-usermod.readthedocs.io/en/latest/, or https://github.com/v923z/micropython-usermod/.
All code segments are left unchanged.

Zoltán

Post Reply