Combo of a C and python module?

[stepansnigirev]

I have couple questions about C modules.

1. I am writing a module that contains a highly optimized C code and python wrapper around it.
Is it possible to extend a python module with C submodule?

At the moment I am using two separate modules in global scope - I have a C module named with an underscore (_mod), and in the mod.py I imports everything from _mod and adds a few functions on top.

I don't like that _mod remains in global namespace - I have it appearing in help('modules') even though I am not using it anywhere. Is there a way to remove it from global scope and put into python module?

2. In one of my pure-python modules I need access to a large chunk of constant data - a word dictionary.
At the moment I have it hardcoded in the python file, but this means that if the file is imported all this data is loaded into the memory.
What I need instead is to keep it in flash and read small parts of it when necessary.

I could just put a file in flash, but I want this file to be included in the build and read-only.
Is there a way to freeze data / binary files in the build?

At the moment I am thinking about a C module that just has static const array with the data, but maybe there is a better way.

[jimmo]

1. Can you clarify what you mean by "global scope". If your python wrapper (foo.py) imports foo_mod (or from foo_mod import *), then the stuff from foo_mod will be scoped to the foo module.

I'm not aware of a way to make built-in modules not appear in help('modules'). Maybe worth considering sending a PR to remove any underscore-prefixed names.

2. This sounds like what the module freezing process is for (especially if you already have it in a Python file). If you freeze foo.py into your build, then any constant data (i.e. strings) in foo.py will be stored in ROM. These two threads might be useful https://github.com/micropython/micropython/issues/5394 and viewtopic.php?f=2&t=7555

[stepansnigirev]

What I am trying to achieve is:

mod/__init__.py - a python module, can be imported as "import mod"
submod.c - a user module, should be a submodule of mod, can be imported as "from mod import submod"

At the moment I can only get it like this:

mod/__init__.py - frozen in the firmware, can be imported as "import mod"
_submod.c - compiled as user module, can be imported as "import _submod"
mod/submod.py - frozen in the firmware with a single line "from _submod import *"

It almost what I want, except that _submod is still visible in the help('modules'), but if I understood correctly at the moment I can't remove it from there...

I think it would be nice to have a way to hide modules from help('modules'), but _thread is using underscore prefix...
Maybe a field in the module structure? Or hide _thread as well?

[deshipu]

You can allegedly do that in the newest version, by generating a .mpy module following: http://docs.micropython.org/en/latest/d ... atmod.html — look at the example at https://github.com/micropython/micropyt ... /features2

However, I haven't managed to get a non-trivial module to compile yet, and there is zero support on this, so good luck.

[jimmo]

At the moment there's no way for build-in modules to be packages (or subpackages), only top-level modules. See https://github.com/micropython/micropython/pull/4731

But yes, .mpy files can do this, so deshipu is right that the new native .mpy feature could help.

However, I haven't managed to get a non-trivial module to compile yet, and there is zero support on this, so good luck.

I've seen you say this in three separate places now... do you have a link to a github issue or something with more details? There's a lot of moving parts, no surprise that there might be some issues.

Edit: found viewtopic.php?f=3&t=7456&p=42590#p42590
I missed that thread over the break, will take a look.

Maybe a field in the module structure? Or hide _thread as well?

Hiding "_thread" might be OK?

A field in the module structure might be too expensive, unless you can find an unused bit somewhere.

[stijn]

Hiding "_thread" might be OK?

If it's configurable and off by default, otherwise: please no. Having those modules visible and available for use can be really helpful for debugging plus in the end they're just modules like all others. Treating them consistently is an advantage and the path of the least confusion and resistance. While on the other hand I'm not sure if the perceived problem of them being visible is really a problem (i.e. is it anything more than just a cosmetic annoyance)?

[jimmo]

Makes sense, thanks for the explanation!