Frustrations with missing libraries?

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
KJM
Posts: 158
Joined: Sun Nov 18, 2018 10:53 pm
Location: Sydney AU

Re: Frustrations with missing libraries?

Post by KJM » Fri Aug 05, 2022 7:40 am

I'd like to try lora in upython on an esp32 but I need (u)crypto & upip can't seem to quite pull it off

Code: Select all

>>> upip.install('crypto')
Installing to: /lib/
Error installing 'crypto': , packages may be partially installed
I'm wondering if I need a different flavour of upython? All the online examples of upython lora seem to be based on lemariva? Anybody know a way to get crypto on vanilla upython?

User avatar
scruss
Posts: 360
Joined: Sat Aug 12, 2017 2:27 pm
Location: Toronto, Canada
Contact:

Re: Frustrations with missing libraries?

Post by scruss » Fri Aug 05, 2022 3:23 pm

Check with help('modules') in the REPL, but ucryptolib is a builtin module on the ESP32 I have here. Check that you have the most recent version.

This is really a different question to the OP's, so maybe start a new thread in the ESP32 boards section.

User avatar
aivarannamaa
Posts: 171
Joined: Fri Sep 22, 2017 3:19 pm
Location: Estonia
Contact:

Re: Frustrations with missing libraries?

Post by aivarannamaa » Sat Aug 06, 2022 8:27 am

Jibun no kage wrote:
Mon Aug 01, 2022 8:48 pm
I have to say, that the one frustrating thing with MicroPython is what appears to be no documentation now how to manage libraries. Thonny does it one way, others say just copy files to the root file system of the device, etc., etc, etc.
I've had similar feelings when developing MicroPython and CircuitPython package management for Thonny. For CPython there are just PyPI and pip. For MicroPython there is PyPI with stripped (not pip-compatible) sdists for upip and there is micropython.org/pi (the result of a quarrel if I'm not mistaken). For CircuitPython there are zip-bundles with their own system for expressing dependencies (there are also PyPI packages, which are kind of usable but as they are meant for CPython, they have this huge blinka dependency).

It's quite messy, but I don't think it has to be like this. For example -- I'm told that some library developers don't publish sdists for MicroPython, because adding upip-compatibility is tedious. They could forget about stripping sdists if micropython.org deployed a proxy index for upip, which does the necessary stripping on the fly.

scruss wrote:
Wed Aug 03, 2022 11:13 pm
... the most common MicroPython board has a little over 30 K free. Consequently, many MicroPython packages don't have space for clever dependency management: you basically have to install packages until an import stops failing.
Dependency management doesn't have to be a runtime issue ie. requiring RAM (it isn't even in CPython). As far as I can see, upip (and external package management tools) could easily store some metadata in lib directory next to the modules and make good use of it, as we have used to expect in CPython world. That's why I created https://pypi.org/project/pipkin/, successor to my earlier minipip. Thonny 4.0 is using pipkin instead of minipip.

For frozen modules there could be a mechanism for making their metadata explicit (eg. by mounting a read-only directory containing the metadata of corresponding packages, or by providing another mechanism for querying it).

Jibun no kage wrote:
Thu Aug 04, 2022 4:15 pm
I am suggesting to Thonny development team, they should add a filter feature to the search to allow for narrowed search, for example only list packages that are explicitly CPython or MicroPython compliant for example, there has to be something in PyPi that allows for this idea no? If not, maybe that is part of the problem?
In fact, there are PEP-s for describing how to mark package compatibility ("Programming Language :: Python :: Implementation :: MicroPython") and/or publish different wheels for different targets. For example, if MicroPython community decided that it is willing to publish separately marked wheels for MicroPython, then PyPI and pip would be already there -- for example you could use `pip download <package name>` to get all packages and their dependencies downloaded and extracted into a specified folder (see https://github.com/aivarannamaa/packaging_experiment)

I'm not sure the community is willing to change MicroPython packaging more similar to CPython, though. When I introduced pipkin and hoped to spark discussion, I got no replies neither in MicroPython forum (viewtopic.php?f=15&t=11783) nor in CircuitPython forum (https://forums.adafruit.com/viewtopic.php?f=60&t=189621). In fact, this thread is the first I've met, which complains about the current situation.

There is https://github.com/micropython/micropython/pull/8914 which is supposed to be "the first step towards updating MicroPython's package management and deployment system in a way that combines freezing, on-device-install and host-based-install", but as little as I understand it, it seems to move MicroPython package management even farther from PyPI and existing Python packaging standards. (It would be great if someone explained the bigger picture behind this PR.)

We'll see what happens with MicroPython packaging, but at least in Thonny 4.0, I'm going to promote PyPI and existing standards (but also support micropython.org/pi and stripped sdists).
Aivar Annamaa
https://thonny.org

User avatar
scruss
Posts: 360
Joined: Sat Aug 12, 2017 2:27 pm
Location: Toronto, Canada
Contact:

Re: Frustrations with missing libraries?

Post by scruss » Sat Aug 06, 2022 7:35 pm

aivarannamaa wrote:
Sat Aug 06, 2022 8:27 am
I'm not sure the community is willing to change MicroPython packaging more similar to CPython, though. When I introduced pipkin and hoped to spark discussion, I got no replies neither in MicroPython forum (viewtopic.php?f=15&t=11783) nor in CircuitPython forum (https://forums.adafruit.com/viewtopic.php?f=60&t=189621). In fact, this thread is the first I've met, which complains about the current situation.
Sorry that no-one picked up on your announcement, Aivar. I'm fairly sure (and the minimal response here to the RFC: Move forums to GitHub Discussions thread may confirm it) that the MicroPython team don't look at the forum for project guidance.

Personally, the project name brings back a nightmare-fuel television programme from my early childhood, Pipkins … ☺

Jibun no kage
Posts: 144
Joined: Mon Jul 25, 2022 9:45 pm

Re: Frustrations with missing libraries?

Post by Jibun no kage » Thu Aug 11, 2022 2:09 am

The forum is where discussion and debate is often the most logical place to start, but it should not be isolated or siloed. GitHub is more concrete and specific (usually) so I see the point, but if solution developers (i.e. MicroPython) are not engaged in the forum, it begs the question, what is the point of either the forum or Github issues, features options? Are you suggesting that having to separate 'groups' is functional or of benefit? I serious doubt that is the intent. I am in the forum, you are in GitHub... and never to the two shall meet? Lacks common sense, no?

Node Red development is one example where they ask... post a given question or issue in one scope or the other, but this is because the developers of Node Red ARE VERY ACTIVE in both the official forum and on GitHub. So they want to avoid duplication... not avoid participation. Oh, and, Tasmota has another similar mindset, where they ask post to Discord, or GitHub or forum, not multiple locations... to avoid duplication.

stijn
Posts: 735
Joined: Thu Apr 24, 2014 9:13 am

Re: Frustrations with missing libraries?

Post by stijn » Thu Aug 11, 2022 9:52 am

Jibun no kage wrote:
Thu Aug 11, 2022 2:09 am
if solution developers (i.e. MicroPython) are not engaged in the forum
That is not really the case though: most of the people having contributed to micropython on Github also have and still do here. But this is all about volunteering so one cannot expect every single question to be seen/answered. Moreover even if literally none of the contributers would ever engage here, it would still make sense to have a forum, for the users, because some users know more than others so the former can help the latter.
it begs the question, what is the point of either the forum or Github issues, features options?
Simply said the forum is for general Q&A, mostly by users. Github is for developers, reporting bugs, feature requests, ... And to make things complicated of course there can be overlap :)

User avatar
scruss
Posts: 360
Joined: Sat Aug 12, 2017 2:27 pm
Location: Toronto, Canada
Contact:

Re: Frustrations with missing libraries?

Post by scruss » Thu Aug 11, 2022 2:19 pm

Jibun no kage wrote:
Thu Aug 11, 2022 2:09 am
Are you suggesting that having to separate 'groups' is functional or of benefit?
No, I'm saying that the MicroPython team is considering closing this forum and moving to Github Discussions.

User avatar
aivarannamaa
Posts: 171
Joined: Fri Sep 22, 2017 3:19 pm
Location: Estonia
Contact:

Re: Frustrations with missing libraries?

Post by aivarannamaa » Sat Aug 13, 2022 7:45 am

The readers of this thread may find the discussion under https://github.com/micropython/micropython/pull/8914 interesting.
Aivar Annamaa
https://thonny.org

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Frustrations with missing libraries?

Post by jimmo » Sat Aug 13, 2022 2:12 pm

aivarannamaa wrote:
Sat Aug 06, 2022 8:27 am
I'm not sure the community is willing to change MicroPython packaging more similar to CPython, though. When I introduced pipkin and hoped to spark discussion, I got no replies neither in MicroPython forum (viewtopic.php?f=15&t=11783) nor in CircuitPython forum (https://forums.adafruit.com/viewtopic.php?f=60&t=189621). In fact, this thread is the first I've met, which complains about the current situation.
Sorry I missed this -- I took some time off the forum around then because we had a baby (a few days before that post..!)

I found out about pipkin relatively recently when jepler reached out to you about circup integration in Thonny etc.
scruss wrote:
Sat Aug 06, 2022 7:35 pm
Sorry that no-one picked up on your announcement, Aivar. I'm fairly sure (and the minimal response here to the RFC: Move forums to GitHub Discussions thread may confirm it) that the MicroPython team don't look at the forum for project guidance.
I spend a lot of time on the forum and use the experience in the forum to feed back to the rest of the MicroPython team. Unfortunately I'm just one person and (as above) sometimes I take time off.

I'm still keen to do the move to GitHub Discussions very much for this reason.


Anyway, I've spent a large chunk of time over the past couple of weeks on exactly this issue raised in this thread (i.e. the state of micropython-lib and package management in general). You can follow along at https://github.com/micropython/micropython/pull/8914 and https://github.com/micropython/micropython-lib/pull/506 (and also related: https://github.com/micropython/micropython/issues/9018)

(I actually started this because I was updating the docs about how to manage packages and decided that it was too broken to document, so started fixing it).

For anything in micropython-lib, simply installing the package by name, either on-device with `upip.install("foo")` or via an external tool like mpremote with `mpremote connect /dev/ttyACM0 install foo` will "just work" and take care of all dependencies. I am also working on extending this to non-micropython-lib packages too.

In summary: totally understand the frustration, we definitely need to fix it, and I'm working on it.

Post Reply