Frustrations with missing libraries?

[Jibun no kage]

Frustrations with missing libraries? I need get the 'copy' module, need to do a deep copy on a dictionary, so I find it under manage packages in thonny 'micropython-copy' I install it, now I need the 'types' module? So I find 'micropython-types' I install it... now I get...

1: python setup.py egg_info Check the logs for full command output.
minipip returned with error code 1

Which means the 'types' module is not really installed. What the heck do I do now?

I have to say, that the one frustrating thing with MicroPython is what appears to be no documentation now how to manage libraries. Thonny does it one way, others say just copy files to the root file system of the device, etc., etc, etc.

Very confusing and frustrating. Apparently there is a group of standard libraries on github, but how the heck do I know if I need those or something else? To fix the error above?

[OlivierLenoir]

Thanks to share your code and the error message to understand the situation.
For library, you can download and copy them on your device or use upip, documentation is here.

[Jibun no kage]

This not a code issue, per se, but development environment issue. You want a code example...

import uasyncio as asyncio

Error... no import for...

If I try to find it via Thonny, I get every variant listed BUT the actual uasyncio for MicroPython. I get 10 different packages but none are actually part of MicroPython, so to get that correct one, I have to copy the py file to the file system of the device, say Pico or ESP from GitHub.

That does not make a lot of sense from a user friendly perspecitve. Never mind the 100s of packages that say 'MicroPython package' but are ports CPython... which fail to install. If there is a port for CPython, where is the original MicroPython package? The CPython port is listed but not the original that it is based on? What the heck?

In GitHub, because Thonny apparently can't see it or find it. This is basic issue is not specific to Thonny, PyCraft has its own issues as well finding MicroPython packages.

Never mind packages that are written but never added to any repository, so you are left searching GitHub again, and copying source files, which may have many dependencies.

As for upip, I can't get it to work, because after it is installed, and I try use it, I get error saying ussl is missing. What the heck? I think you can see the my point, right? Where is ussl, not visible to Thonny, back to GitHub... but where on GitHub.... is the question.

I am not stranger to Python, but the fact that there is apparently not a user friendly method to add to the base MicroPython image, which almost every project requires of course, to the great variance of project needs, is not great situation.

As a counter point, Arduino IDE you rarely have to search for anything, as long as you add the parent URL for a tree of sketches, examples, and libraries, everything just works, is easy to find right in the IDE.

[scruss]

Yes, it's not straightforward. It's probably more of an issue for you because you have some familiarity of Python. I find it helpful to forget as much about full-fat CPython when working with MicroPython: assume you have no memory, no file system and no network, and work up from there.

You're working with a Raspberry Pi Pico W, right? In MicroPython terms, it's huge! About 150 K free memory after loading the network: the most common MicroPython board has a little over 30 K free. Consequently, many MicroPython packages don't have space for clever dependency management: you basically have to install packages until an import stops failing.

Different builds have vastly different builtin libraries. help('modules') on a recent Pico W nightly image has an impressive list:

Code: Select all

__main__          gc                uasyncio/lock     upip
_boot             lwip              uasyncio/stream   upip_utarfile
_boot_fat         math              ubinascii         urandom
_onewire          micropython       ucollections      ure
_rp2              neopixel          ucryptolib        urequests
_thread           network           uctypes           uselect
_uasyncio         onewire           uerrno            usocket
_webrepl          rp2               uhashlib          ussl
builtins          uarray            uheapq            ustruct
cmath             uasyncio/__init__ uio               usys
dht               uasyncio/core     ujson             utime
ds18x20           uasyncio/event    umachine          uwebsocket
framebuf          uasyncio/funcs    uos               uzlib
Plus any modules on the filesystem

The general advice about finding library modules seems to be to keep a pull of micropython / micropython-lib handy and install what you need in your board's /lib folder. micropython-lib's been cleaned up a lot recently (it used to be full of code that would only run on certain ancient forks, but with no information as to what it was supposed to work on) but it's still not perfect. The MicroPython dev team is very small and underpaid.

If you're needing libraries for specific devices, Awesome MicroPython is your best resource. There are also many working-but-abandoned libraries for device interfacing on github under the adafruit/adafruit-micropython-* headings. I'm not sure if many of those are in Awesome MicroPython.

Thonny is more likely to try to install a CPython library unless you check every candidate's information link

A couple of things, though, make me think there might be something wrong with your setup:

1. import uasyncio as asyncio works as expected on my Raspberry Pi Pico W running v1.19.1 (2022-07-29)
2. upip complaints about SSL for me too, but downloads the packages as expected

It's rather unfair to compare Arduino - a system that's been around for nearly 20 years and has millions of users - with MicroPython. You might lead a charmed life with Arduino: for me, finding the right core package, hoping a library will work, and then making sure the compiler will link and upload is a big issue.

[pythoncoder]

This not a code issue, per se, but development environment issue. You want a code example...

import uasyncio as asyncio

In any remotely recent build of MicroPython uasyncio is buit in. No installation necessary. If you're running a build where this isn't the case I strongly recommend you upgrade.

[Jibun no kage]

@pythoncoder, Nightly build should have it right? I think I loaded the stable build which was like June this year. Will snag the nightly and try it.

@scruss, you raise valid points, but if no one expresses where maybe (IMHO) things could be improved? I realize some of my frustration is applicable to Thonny, but the entire 'environment' has a pretty painful learning curve in general. Don't even get me started on uPyCraft, it still requires DLLs that are over 11 years old. That said, I have reset the environment (got to love virtual machines), and walking through the setup again, I agree something is odd, because the dependencies should be pulled in.

I am suggesting to Thonny development team, they should add a filter feature to the search to allow for narrowed search, for example only list packages that are explicitly CPython or MicroPython compliant for example, there has to be something in PyPi that allows for this idea no? If not, maybe that is part of the problem?

If Arduino (finally) got it right, why not compare everything to the 'defacto' standard? Not sure I agree with you on that single point. After all, even Node, with npm has compatibility meta data, and JavaScript can be even crazier than other environments! If you have ever tried to resolve conflicts with npm you know what I mean.

[Jibun no kage]

@pythoncoder, I pulled the latest nightly build for ESP8266 1M devices, flashed the device, then did the following:

>>> import machine
>>> import network
>>> import uasyncio
ImportError: no module named 'asyncio'

I believe you said uasyncio was now included in the latest firmware builds? Did i do something completely wrong?

Image i downloaded and flashed was 'esp8266-1m-20220802-unstable-v1.19.1-229-g579f33050.bin'.

>>> import os
>>> os.uname()
(sysname='esp8266', nodename='esp8266', release='2.2.0-dev(9422289)', version='v1.19.1-229-g579f33050 on 2022-08-02', machine='ESP module (1M) with ESP8266')

Code: Select all

__main__          gc                ucollections      urandom
_boot             inisetup          ucryptolib        ure
_onewire          lwip              uctypes           uselect
_webrepl          math              uerrno            usocket
apa102            micropython       uhashlib          ussl
btree             neopixel          uheapq            ustruct
builtins          network           uio               usys
dht               ntptime           ujson             utime
ds18x20           onewire           umachine          uwebsocket
esp               port_diag         uos               uzlib
flashbdev         uarray            upip              webrepl
framebuf          ubinascii         upip_utarfile     webrepl_setup

So I guess I pulled the wrong build?

Also, just as a test...

>>> theInterface=network.WLAN(network.STA_IF)
>>> theInterface.active(True)
>>> theInterface.active()
>>> theInterface.active()
True
>>> theInterface.isconnected()
False
>>> theInterface.connect('Dachshund Digital_optout_nomap','D@chshund Digit@1')
>>> theInterface.isconnected()
True
>>> import upip
>>> upip.install("micropython-uasyncio")
>>> upip.install("micropython-uasyncio")
Installing to: /lib/
Warning: micropython.org SSL certificate is not validated
Installing micropython-uasyncio 3.0 from https://micropython.org/pi/uasyncio/uasyncio-3.0.tar.gz
>>> import uasyncio
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: incompatible .mpy file

Now, I have no clue how the mpy is wrong, everything is ESP8266 as far as I know, maybe not?

[Jibun no kage]

Ok, I found I think the right bin file...

Code: Select all

__main__          math              ucollections      ure
_boot             micropython       ucryptolib        urequests
_onewire          neopixel          uctypes           urllib/urequest
_uasyncio         network           uerrno            uselect
_webrepl          ntptime           uhashlib          usocket
apa102            onewire           uheapq            ussl
btree             port_diag         uio               ustruct
builtins          ssd1306           ujson             usys
dht               uarray            umachine          utime
ds18x20           uasyncio/__init__ umqtt/robust      uwebsocket
esp               uasyncio/core     umqtt/simple      uzlib
flashbdev         uasyncio/event    uos               webrepl
framebuf          uasyncio/funcs    upip              webrepl_setup
gc                uasyncio/lock     upip_utarfile
inisetup          uasyncio/stream   upysh
lwip              ubinascii         urandom

However, I think I need to learn how to compile my own bin image, for what I want to do. This should be interesting!

[scruss]

I believe you said uasyncio was now included in the latest firmware builds? Did i do something completely wrong?

Image i downloaded and flashed was 'esp8266-1m-20220802-unstable-v1.19.1-229-g579f33050.bin'.

The esp8266-1m images don't include uasyncio because they don't have enough flash storage. ESP-01, ESP01s and ESP8265 boards are best kept away from MicroPython. The first ever ESP8266 board I got in 2015 or so had 4 MB of flash, so there's no point in suffering with tiny chips in 2022.

I am suggesting to Thonny development team, they should add a filter feature to the search to allow for narrowed search, for example only list packages that are explicitly CPython or MicroPython compliant for example, there has to be something in PyPi that allows for this idea no? If not, maybe that is part of the problem?

Thonny development team = 1 person called Aivar. There doesn't seem to be a reliable way to filter on PyPI. Thonny also supports regular old CPython and CircuitPython, so their messy, overlapping packages have to be allowed too.

The ways you can make things change in MicroPython are:

1. raise issues in the relevant github repo: "suggesting" here does nothing
2. if you know how, develop and test a solution then make a pull request
3. sponsor

If Arduino (finally) got it right, why not compare everything to the 'defacto' standard?

But Arduino has got it horribly wrong. Their compile time has become ridiculous, even on very powerful hardware. Their IDE is slow, and their ever-present Create Agent (required to develop on some of the newest boards) is currently sucking 200 MB RAM and roughly 5 W on my development machine just idling.

If you're running boards apart from the wheezy 20-year-old ATMega chips, you run into libraries that won't work with your hardware, and multiple competing toolchains. For example, for ESP8266, STM32 and RP2040, I know of at least two competing Arduino toolchains for each platform. These toolchains don't work with all libraries. How do you find out? Well, does you code compile? Does your code run? That's all!

Arduino have also moved all of their docs (the "Find in Reference" feature in the IDE is now effectively useless) so old tutorials/libraries point to missing features. And don't get me started on the Library Manager, which relies on library maintainers updating a file manually so that Library Manager can find the new release. Which they often forget to do, so your libraries rot with age.

For me, compared to Arduino, MicroPython is a breath of fresh air.

[Jibun no kage]

My reference is more to the IDE, than say the technology, and I guess my experience is different than yours. However. As much I as once liked Arduino, its ease of use, it is the least used of all the micro controllers I have worked on. Maybe that is a sad but true explanation? Still enjoy the fact that it just works, and yet, I ways think of Arduino last for projects. Maybe it is like the old debate over Macintosh versus PC compatible, I have owned a few Macs over the years, consider them (or once) far better design, but what are most of the systems I have built, owned, etc., PCs. Maybe that is just ironic?

Maybe the most telling observation... i am typing this in a MicroPython forum, not a C based Arduino forum? Did someone say VHS vs. BetaMax? Sorry, Just could resist making another ironic comparison.

To get from point A to B, I think I am just going to learn how to roll-my-own firmware images. Not that standard builds are not valid, but if roll my own I get exactly what I need.

For example, I am trying to get MicroPython working on 1Mb ESP01, and yes I know it is a difficult task, but that challenge is enjoyable. I have several original Picos I need to add WiFi to, and I have a several ESP01s. So... and I hate the idea of using the AT command set on the ESP01s. I already have a simple 'bridge' solution working for the most part so that ESP01 (C code) receives and sends MQTT, and drops same onto the Pico UART, not pretty but ugly. Now if I could just get a slim version of MicroPython working on the ESP01... well, that is the idea anyway!