Jibun no kage wrote: ↑Mon Aug 01, 2022 8:48 pm
I have to say, that the one frustrating thing with MicroPython is what appears to be no documentation now how to manage libraries. Thonny does it one way, others say just copy files to the root file system of the device, etc., etc, etc.
I've had similar feelings when developing MicroPython and CircuitPython package management for Thonny. For CPython there are just PyPI and pip. For MicroPython there is PyPI with stripped (not pip-compatible) sdists for upip and there is micropython.org/pi (the result of a quarrel if I'm not mistaken). For CircuitPython there are zip-bundles with their own system for expressing dependencies (there are also PyPI packages, which are kind of usable but as they are meant for CPython, they have this huge blinka dependency).
It's quite messy, but I don't think it has to be like this. For example -- I'm told that some library developers don't publish sdists for MicroPython, because adding upip-compatibility is tedious. They could forget about stripping sdists if micropython.org deployed a proxy index for upip, which does the necessary stripping on the fly.
scruss wrote: ↑Wed Aug 03, 2022 11:13 pm
... the most common MicroPython board has a little over 30 K free. Consequently, many MicroPython packages don't have space for clever dependency management: you basically have to install packages until an import stops failing.
Dependency management doesn't have to be a runtime issue ie. requiring RAM (it isn't even in CPython). As far as I can see, upip (and external package management tools) could easily store some metadata in lib directory next to the modules and make good use of it, as we have used to expect in CPython world. That's why I created
https://pypi.org/project/pipkin/, successor to my earlier minipip. Thonny 4.0 is using pipkin instead of minipip.
For frozen modules there could be a mechanism for making their metadata explicit (eg. by mounting a read-only directory containing the metadata of corresponding packages, or by providing another mechanism for querying it).
Jibun no kage wrote: ↑Thu Aug 04, 2022 4:15 pm
I am suggesting to Thonny development team, they should add a filter feature to the search to allow for narrowed search, for example only list packages that are explicitly CPython or MicroPython compliant for example, there has to be something in PyPi that allows for this idea no? If not, maybe that is part of the problem?
In fact, there are PEP-s for describing how to mark package compatibility ("Programming Language :: Python :: Implementation :: MicroPython") and/or publish different wheels for different targets. For example, if MicroPython community decided that it is willing to publish separately marked wheels for MicroPython, then PyPI and pip would be already there -- for example you could use `pip download <package name>` to get all packages and their dependencies downloaded and extracted into a specified folder (see
https://github.com/aivarannamaa/packaging_experiment)
I'm not sure the community is willing to change MicroPython packaging more similar to CPython, though. When I introduced pipkin and hoped to spark discussion, I got no replies neither in MicroPython forum (
viewtopic.php?f=15&t=11783) nor in CircuitPython forum (
https://forums.adafruit.com/viewtopic.php?f=60&t=189621). In fact, this thread is the first I've met, which complains about the current situation.
There is
https://github.com/micropython/micropython/pull/8914 which is supposed to be "the first step towards updating MicroPython's package management and deployment system in a way that combines freezing, on-device-install and host-based-install", but as little as I understand it, it seems to move MicroPython package management even farther from PyPI and existing Python packaging standards. (It would be great if someone explained the bigger picture behind this PR.)
We'll see what happens with MicroPython packaging, but at least in Thonny 4.0, I'm going to promote PyPI and existing standards (but also support micropython.org/pi and stripped sdists).