Frustrations with missing libraries?
-
- Posts: 144
- Joined: Mon Jul 25, 2022 9:45 pm
Frustrations with missing libraries?
Frustrations with missing libraries? I need get the 'copy' module, need to do a deep copy on a dictionary, so I find it under manage packages in thonny 'micropython-copy' I install it, now I need the 'types' module? So I find 'micropython-types' I install it... now I get...
1: python setup.py egg_info Check the logs for full command output.
minipip returned with error code 1
Which means the 'types' module is not really installed. What the heck do I do now?
I have to say, that the one frustrating thing with MicroPython is what appears to be no documentation now how to manage libraries. Thonny does it one way, others say just copy files to the root file system of the device, etc., etc, etc.
Very confusing and frustrating. Apparently there is a group of standard libraries on github, but how the heck do I know if I need those or something else? To fix the error above?
1: python setup.py egg_info Check the logs for full command output.
minipip returned with error code 1
Which means the 'types' module is not really installed. What the heck do I do now?
I have to say, that the one frustrating thing with MicroPython is what appears to be no documentation now how to manage libraries. Thonny does it one way, others say just copy files to the root file system of the device, etc., etc, etc.
Very confusing and frustrating. Apparently there is a group of standard libraries on github, but how the heck do I know if I need those or something else? To fix the error above?
- OlivierLenoir
- Posts: 126
- Joined: Fri Dec 13, 2019 7:10 pm
- Location: Picardie, FR
Re: Frustrations with missing libraries?
Thanks to share your code and the error message to understand the situation.
For library, you can download and copy them on your device or use upip, documentation is here.
For library, you can download and copy them on your device or use upip, documentation is here.
Olivier Lenoir
https://gitlab.com/olivierlenoir
https://gitlab.com/olivierlenoir
-
- Posts: 144
- Joined: Mon Jul 25, 2022 9:45 pm
Re: Frustrations with missing libraries?
This not a code issue, per se, but development environment issue. You want a code example...
import uasyncio as asyncio
Error... no import for...
If I try to find it via Thonny, I get every variant listed BUT the actual uasyncio for MicroPython. I get 10 different packages but none are actually part of MicroPython, so to get that correct one, I have to copy the py file to the file system of the device, say Pico or ESP from GitHub.
That does not make a lot of sense from a user friendly perspecitve. Never mind the 100s of packages that say 'MicroPython package' but are ports CPython... which fail to install. If there is a port for CPython, where is the original MicroPython package? The CPython port is listed but not the original that it is based on? What the heck?
In GitHub, because Thonny apparently can't see it or find it. This is basic issue is not specific to Thonny, PyCraft has its own issues as well finding MicroPython packages.
Never mind packages that are written but never added to any repository, so you are left searching GitHub again, and copying source files, which may have many dependencies.
As for upip, I can't get it to work, because after it is installed, and I try use it, I get error saying ussl is missing. What the heck? I think you can see the my point, right? Where is ussl, not visible to Thonny, back to GitHub... but where on GitHub.... is the question.
I am not stranger to Python, but the fact that there is apparently not a user friendly method to add to the base MicroPython image, which almost every project requires of course, to the great variance of project needs, is not great situation.
As a counter point, Arduino IDE you rarely have to search for anything, as long as you add the parent URL for a tree of sketches, examples, and libraries, everything just works, is easy to find right in the IDE.
import uasyncio as asyncio
Error... no import for...
If I try to find it via Thonny, I get every variant listed BUT the actual uasyncio for MicroPython. I get 10 different packages but none are actually part of MicroPython, so to get that correct one, I have to copy the py file to the file system of the device, say Pico or ESP from GitHub.
That does not make a lot of sense from a user friendly perspecitve. Never mind the 100s of packages that say 'MicroPython package' but are ports CPython... which fail to install. If there is a port for CPython, where is the original MicroPython package? The CPython port is listed but not the original that it is based on? What the heck?
In GitHub, because Thonny apparently can't see it or find it. This is basic issue is not specific to Thonny, PyCraft has its own issues as well finding MicroPython packages.
Never mind packages that are written but never added to any repository, so you are left searching GitHub again, and copying source files, which may have many dependencies.
As for upip, I can't get it to work, because after it is installed, and I try use it, I get error saying ussl is missing. What the heck? I think you can see the my point, right? Where is ussl, not visible to Thonny, back to GitHub... but where on GitHub.... is the question.
I am not stranger to Python, but the fact that there is apparently not a user friendly method to add to the base MicroPython image, which almost every project requires of course, to the great variance of project needs, is not great situation.
As a counter point, Arduino IDE you rarely have to search for anything, as long as you add the parent URL for a tree of sketches, examples, and libraries, everything just works, is easy to find right in the IDE.
Re: Frustrations with missing libraries?
Yes, it's not straightforward. It's probably more of an issue for you because you have some familiarity of Python. I find it helpful to forget as much about full-fat CPython when working with MicroPython: assume you have no memory, no file system and no network, and work up from there.
You're working with a Raspberry Pi Pico W, right? In MicroPython terms, it's huge! About 150 K free memory after loading the network: the most common MicroPython board has a little over 30 K free. Consequently, many MicroPython packages don't have space for clever dependency management: you basically have to install packages until an import stops failing.
Different builds have vastly different builtin libraries. help('modules') on a recent Pico W nightly image has an impressive list:
The general advice about finding library modules seems to be to keep a pull of micropython / micropython-lib handy and install what you need in your board's /lib folder. micropython-lib's been cleaned up a lot recently (it used to be full of code that would only run on certain ancient forks, but with no information as to what it was supposed to work on) but it's still not perfect. The MicroPython dev team is very small and underpaid.
If you're needing libraries for specific devices, Awesome MicroPython is your best resource. There are also many working-but-abandoned libraries for device interfacing on github under the adafruit/adafruit-micropython-* headings. I'm not sure if many of those are in Awesome MicroPython.
Thonny is more likely to try to install a CPython library unless you check every candidate's information link
A couple of things, though, make me think there might be something wrong with your setup:
You're working with a Raspberry Pi Pico W, right? In MicroPython terms, it's huge! About 150 K free memory after loading the network: the most common MicroPython board has a little over 30 K free. Consequently, many MicroPython packages don't have space for clever dependency management: you basically have to install packages until an import stops failing.
Different builds have vastly different builtin libraries. help('modules') on a recent Pico W nightly image has an impressive list:
Code: Select all
__main__ gc uasyncio/lock upip
_boot lwip uasyncio/stream upip_utarfile
_boot_fat math ubinascii urandom
_onewire micropython ucollections ure
_rp2 neopixel ucryptolib urequests
_thread network uctypes uselect
_uasyncio onewire uerrno usocket
_webrepl rp2 uhashlib ussl
builtins uarray uheapq ustruct
cmath uasyncio/__init__ uio usys
dht uasyncio/core ujson utime
ds18x20 uasyncio/event umachine uwebsocket
framebuf uasyncio/funcs uos uzlib
Plus any modules on the filesystem
If you're needing libraries for specific devices, Awesome MicroPython is your best resource. There are also many working-but-abandoned libraries for device interfacing on github under the adafruit/adafruit-micropython-* headings. I'm not sure if many of those are in Awesome MicroPython.
Thonny is more likely to try to install a CPython library unless you check every candidate's information link
A couple of things, though, make me think there might be something wrong with your setup:
- import uasyncio as asyncio works as expected on my Raspberry Pi Pico W running v1.19.1 (2022-07-29)
- upip complaints about SSL for me too, but downloads the packages as expected
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: Frustrations with missing libraries?
In any remotely recent build of MicroPython uasyncio is buit in. No installation necessary. If you're running a build where this isn't the case I strongly recommend you upgrade.Jibun no kage wrote: ↑Wed Aug 03, 2022 10:20 pmThis not a code issue, per se, but development environment issue. You want a code example...
import uasyncio as asyncio
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
-
- Posts: 144
- Joined: Mon Jul 25, 2022 9:45 pm
Re: Frustrations with missing libraries?
@pythoncoder, Nightly build should have it right? I think I loaded the stable build which was like June this year. Will snag the nightly and try it.
@scruss, you raise valid points, but if no one expresses where maybe (IMHO) things could be improved? I realize some of my frustration is applicable to Thonny, but the entire 'environment' has a pretty painful learning curve in general. Don't even get me started on uPyCraft, it still requires DLLs that are over 11 years old. That said, I have reset the environment (got to love virtual machines), and walking through the setup again, I agree something is odd, because the dependencies should be pulled in.
I am suggesting to Thonny development team, they should add a filter feature to the search to allow for narrowed search, for example only list packages that are explicitly CPython or MicroPython compliant for example, there has to be something in PyPi that allows for this idea no? If not, maybe that is part of the problem?
If Arduino (finally) got it right, why not compare everything to the 'defacto' standard? Not sure I agree with you on that single point. After all, even Node, with npm has compatibility meta data, and JavaScript can be even crazier than other environments!
If you have ever tried to resolve conflicts with npm you know what I mean.
@scruss, you raise valid points, but if no one expresses where maybe (IMHO) things could be improved? I realize some of my frustration is applicable to Thonny, but the entire 'environment' has a pretty painful learning curve in general. Don't even get me started on uPyCraft, it still requires DLLs that are over 11 years old. That said, I have reset the environment (got to love virtual machines), and walking through the setup again, I agree something is odd, because the dependencies should be pulled in.
I am suggesting to Thonny development team, they should add a filter feature to the search to allow for narrowed search, for example only list packages that are explicitly CPython or MicroPython compliant for example, there has to be something in PyPi that allows for this idea no? If not, maybe that is part of the problem?
If Arduino (finally) got it right, why not compare everything to the 'defacto' standard? Not sure I agree with you on that single point. After all, even Node, with npm has compatibility meta data, and JavaScript can be even crazier than other environments!
![Smile :)](./images/smilies/icon_e_smile.gif)
-
- Posts: 144
- Joined: Mon Jul 25, 2022 9:45 pm
Re: Frustrations with missing libraries?
@pythoncoder, I pulled the latest nightly build for ESP8266 1M devices, flashed the device, then did the following:
>>> import machine
>>> import network
>>> import uasyncio
ImportError: no module named 'asyncio'
I believe you said uasyncio was now included in the latest firmware builds? Did i do something completely wrong?
Image i downloaded and flashed was 'esp8266-1m-20220802-unstable-v1.19.1-229-g579f33050.bin'.
>>> import os
>>> os.uname()
(sysname='esp8266', nodename='esp8266', release='2.2.0-dev(9422289)', version='v1.19.1-229-g579f33050 on 2022-08-02', machine='ESP module (1M) with ESP8266')
So I guess I pulled the wrong build?
Also, just as a test...
>>> theInterface=network.WLAN(network.STA_IF)
>>> theInterface.active(True)
>>> theInterface.active()
>>> theInterface.active()
True
>>> theInterface.isconnected()
False
>>> theInterface.connect('Dachshund Digital_optout_nomap','D@chshund Digit@1')
>>> theInterface.isconnected()
True
>>> import upip
>>> upip.install("micropython-uasyncio")
>>> upip.install("micropython-uasyncio")
Installing to: /lib/
Warning: micropython.org SSL certificate is not validated
Installing micropython-uasyncio 3.0 from https://micropython.org/pi/uasyncio/uasyncio-3.0.tar.gz
>>> import uasyncio
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: incompatible .mpy file
Now, I have no clue how the mpy is wrong, everything is ESP8266 as far as I know, maybe not?
>>> import machine
>>> import network
>>> import uasyncio
ImportError: no module named 'asyncio'
I believe you said uasyncio was now included in the latest firmware builds? Did i do something completely wrong?
Image i downloaded and flashed was 'esp8266-1m-20220802-unstable-v1.19.1-229-g579f33050.bin'.
>>> import os
>>> os.uname()
(sysname='esp8266', nodename='esp8266', release='2.2.0-dev(9422289)', version='v1.19.1-229-g579f33050 on 2022-08-02', machine='ESP module (1M) with ESP8266')
Code: Select all
__main__ gc ucollections urandom
_boot inisetup ucryptolib ure
_onewire lwip uctypes uselect
_webrepl math uerrno usocket
apa102 micropython uhashlib ussl
btree neopixel uheapq ustruct
builtins network uio usys
dht ntptime ujson utime
ds18x20 onewire umachine uwebsocket
esp port_diag uos uzlib
flashbdev uarray upip webrepl
framebuf ubinascii upip_utarfile webrepl_setup
Also, just as a test...
>>> theInterface=network.WLAN(network.STA_IF)
>>> theInterface.active(True)
>>> theInterface.active()
>>> theInterface.active()
True
>>> theInterface.isconnected()
False
>>> theInterface.connect('Dachshund Digital_optout_nomap','D@chshund Digit@1')
>>> theInterface.isconnected()
True
>>> import upip
>>> upip.install("micropython-uasyncio")
>>> upip.install("micropython-uasyncio")
Installing to: /lib/
Warning: micropython.org SSL certificate is not validated
Installing micropython-uasyncio 3.0 from https://micropython.org/pi/uasyncio/uasyncio-3.0.tar.gz
>>> import uasyncio
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: incompatible .mpy file
Now, I have no clue how the mpy is wrong, everything is ESP8266 as far as I know, maybe not?
-
- Posts: 144
- Joined: Mon Jul 25, 2022 9:45 pm
Re: Frustrations with missing libraries?
Ok, I found I think the right bin file...
However, I think I need to learn how to compile my own bin image, for what I want to do. This should be interesting!
Code: Select all
__main__ math ucollections ure
_boot micropython ucryptolib urequests
_onewire neopixel uctypes urllib/urequest
_uasyncio network uerrno uselect
_webrepl ntptime uhashlib usocket
apa102 onewire uheapq ussl
btree port_diag uio ustruct
builtins ssd1306 ujson usys
dht uarray umachine utime
ds18x20 uasyncio/__init__ umqtt/robust uwebsocket
esp uasyncio/core umqtt/simple uzlib
flashbdev uasyncio/event uos webrepl
framebuf uasyncio/funcs upip webrepl_setup
gc uasyncio/lock upip_utarfile
inisetup uasyncio/stream upysh
lwip ubinascii urandom
Re: Frustrations with missing libraries?
The esp8266-1m images don't include uasyncio because they don't have enough flash storage. ESP-01, ESP01s and ESP8265 boards are best kept away from MicroPython. The first ever ESP8266 board I got in 2015 or so had 4 MB of flash, so there's no point in suffering with tiny chips in 2022.Jibun no kage wrote: ↑Thu Aug 04, 2022 9:43 pmI believe you said uasyncio was now included in the latest firmware builds? Did i do something completely wrong?
Image i downloaded and flashed was 'esp8266-1m-20220802-unstable-v1.19.1-229-g579f33050.bin'.
Thonny development team = 1 person called Aivar. There doesn't seem to be a reliable way to filter on PyPI. Thonny also supports regular old CPython and CircuitPython, so their messy, overlapping packages have to be allowed too.Jibun no kage wrote: ↑Thu Aug 04, 2022 4:15 pmI am suggesting to Thonny development team, they should add a filter feature to the search to allow for narrowed search, for example only list packages that are explicitly CPython or MicroPython compliant for example, there has to be something in PyPi that allows for this idea no? If not, maybe that is part of the problem?
The ways you can make things change in MicroPython are:
- raise issues in the relevant github repo: "suggesting" here does nothing
- if you know how, develop and test a solution then make a pull request
- sponsor
But Arduino has got it horribly wrong. Their compile time has become ridiculous, even on very powerful hardware. Their IDE is slow, and their ever-present Create Agent (required to develop on some of the newest boards) is currently sucking 200 MB RAM and roughly 5 W on my development machine just idling.If Arduino (finally) got it right, why not compare everything to the 'defacto' standard?
If you're running boards apart from the wheezy 20-year-old ATMega chips, you run into libraries that won't work with your hardware, and multiple competing toolchains. For example, for ESP8266, STM32 and RP2040, I know of at least two competing Arduino toolchains for each platform. These toolchains don't work with all libraries. How do you find out? Well, does you code compile? Does your code run? That's all!
Arduino have also moved all of their docs (the "Find in Reference" feature in the IDE is now effectively useless) so old tutorials/libraries point to missing features. And don't get me started on the Library Manager, which relies on library maintainers updating a file manually so that Library Manager can find the new release. Which they often forget to do, so your libraries rot with age.
For me, compared to Arduino, MicroPython is a breath of fresh air.
-
- Posts: 144
- Joined: Mon Jul 25, 2022 9:45 pm
Re: Frustrations with missing libraries?
My reference is more to the IDE, than say the technology, and I guess my experience is different than yours. However. As much I as once liked Arduino, its ease of use, it is the least used of all the micro controllers I have worked on. Maybe that is a sad but true explanation? Still enjoy the fact that it just works, and yet, I ways think of Arduino last for projects. Maybe it is like the old debate over Macintosh versus PC compatible, I have owned a few Macs over the years, consider them (or once) far better design, but what are most of the systems I have built, owned, etc., PCs. Maybe that is just ironic?
Maybe the most telling observation... i am typing this in a MicroPython forum, not a C based Arduino forum? Did someone say VHS vs. BetaMax? Sorry, Just could resist making another ironic comparison.
To get from point A to B, I think I am just going to learn how to roll-my-own firmware images. Not that standard builds are not valid, but if roll my own I get exactly what I need.
For example, I am trying to get MicroPython working on 1Mb ESP01, and yes I know it is a difficult task, but that challenge is enjoyable. I have several original Picos I need to add WiFi to, and I have a several ESP01s. So... and I hate the idea of using the AT command set on the ESP01s. I already have a simple 'bridge' solution working for the most part so that ESP01 (C code) receives and sends MQTT, and drops same onto the Pico UART, not pretty but ugly. Now if I could just get a slim version of MicroPython working on the ESP01... well, that is the idea anyway!![Smile :)](./images/smilies/icon_e_smile.gif)
Maybe the most telling observation... i am typing this in a MicroPython forum, not a C based Arduino forum? Did someone say VHS vs. BetaMax? Sorry, Just could resist making another ironic comparison.
To get from point A to B, I think I am just going to learn how to roll-my-own firmware images. Not that standard builds are not valid, but if roll my own I get exactly what I need.
For example, I am trying to get MicroPython working on 1Mb ESP01, and yes I know it is a difficult task, but that challenge is enjoyable. I have several original Picos I need to add WiFi to, and I have a several ESP01s. So... and I hate the idea of using the AT command set on the ESP01s. I already have a simple 'bridge' solution working for the most part so that ESP01 (C code) receives and sends MQTT, and drops same onto the Pico UART, not pretty but ugly. Now if I could just get a slim version of MicroPython working on the ESP01... well, that is the idea anyway!
![Smile :)](./images/smilies/icon_e_smile.gif)