Next stretch goal - upip running native on esp8266

All ESP8266 boards running MicroPython.
Official boards are the Adafruit Huzzah and Feather boards.
Target audience: MicroPython users with an ESP8266 board.
Post Reply
pfalcon
Posts: 1056
Joined: Fri Feb 28, 2014 2:05 pm

Next stretch goal - upip running native on esp8266

Post by pfalcon » Thu Oct 20, 2016 9:15 pm

The next stretch goal results we'd like to present to our Kickstarter backers and esp8266 community in general is upip package manager running natively on a ESP8266 module (see http://forum.micropython.org/viewtopic.php?f=15&t=524 if you didn't hear about upip, you've missed a lot). So, you could prepare a dependency modules snapshot on a desktop machine using unix port of MicroPython and copy over files to your module, and fairly speaking, that remains the recommended way to deploy "bigger" applications. But that's a bit too many steps for quick hacking and enjoying capabilities of your tiny wireless device. So, now you can install modules directly on a device!

Now about caveats: porting upip was integration of many features we've developed previously (and also features developed specifically to make upip porting possible): network sockets, HTTP support, SSL support, JSON parsing, etc. And doing it all together puts quite a strain on modest ESP8266 resources. So, while it definitely works for simple cases, it requires completely "clean" memory (as in: after reset) and would likely fail for more complex cases (like packages with recursive dependencies). That's why we talk about "soft launch": there are issues, but we'd like interested people to be able to play with it and share their experiences. We'll keep working on optimizing it for future releases (as well as look for ways to get more heap memory).

That's one side of the issues. Another is - what packages are available and how many of them are suitable for esp8266. Major source of package is micropython-lib project (see http://forum.micropython.org/viewtopic.php?f=15&t=70 if you didn't hear about it, you miss a lot). There're more than 150 packages in micropython-lib, but good share of them are just placeholders (i.e. yet need to be ported), and large share are intended for bigger environments, like unix port of MicroPython. But even a dozen of package (like functools, itertools) should be a good start. And this all also should be a hint for developers of MicroPython modules - now there's no excuse to not publish them on PyPI (but please make sure they're named to avoid name clashing with micropython-lib modules, to not confuse users).

Ok, so how to use it? Use 1.8.5 release (upip is not available in daily development builds). A transcript below shows how to install a pystone benchmark module optimized for low-memory devices and run it. Try it on a freshly reset module. If it fails with a memory error, retry it, sometimes it helps ;-).

Code: Select all

>>> import upip
>>> upip.install("micropython-pystone_lowmem")
Installing to: /lib/
Warning: pypi.python.org SSL certificate is not validated
Installing micropython-pystone_lowmem 3.4.2-3 from https://pypi.python.org/packages/8c/70/98379ddfea62d55973653fecb7bd58d7372a3e3b5510844f23260e90fbc0/micropython-pystone_lowmem-3.4.2-3.tar.gz
>>> import pystone_lowmem
>>> pystone_lowmem.main()
Pystone(1.2) time for 500 passes = 2333ms
This machine benchmarks at 214 pystones/second
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib

User avatar
deshipu
Posts: 1331
Joined: Thu May 28, 2015 5:54 pm

Re: Next stretch goal - upip running native on esp8266 (soft launch)

Post by deshipu » Fri Oct 21, 2016 7:40 am

Great work!

On my side, I asked for trove classifiers for MicroPython to be added to PyPi, so that we can easily mark packages as compatible with MicroPython. The relevant thread is at https://mail.python.org/pipermail/distu ... 29755.html. Please chime in if you have your own suggestions.

pfalcon
Posts: 1056
Joined: Fri Feb 28, 2014 2:05 pm

Re: Next stretch goal - upip running native on esp8266 (soft launch)

Post by pfalcon » Tue Nov 15, 2016 7:44 pm

As a next step on this route, upip is now included even in the development builds (whereas majority of other modules are bundled only with release builds, to keep dailies lean and mean).
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib

User avatar
deshipu
Posts: 1331
Joined: Thu May 28, 2015 5:54 pm

Re: Next stretch goal - upip running native on esp8266 (soft launch)

Post by deshipu » Mon Feb 13, 2017 9:43 pm

deshipu wrote:Great work!

On my side, I asked for trove classifiers for MicroPython to be added to PyPi, so that we can easily mark packages as compatible with MicroPython. The relevant thread is at https://mail.python.org/pipermail/distu ... 29755.html. Please chime in if you have your own suggestions.
After I got no answer for some time, I decided to ask again, and this time it was suggested that I create an issue for it. So I created https://github.com/pypa/warehouse/issues/1744

pfalcon
Posts: 1056
Joined: Fri Feb 28, 2014 2:05 pm

Re: Next stretch goal - upip running native on esp8266

Post by pfalcon » Sat May 06, 2017 9:11 am

I'm happy to report that upip goes out of "soft launch" and into "production" mode - over this time, there were few optimizations and errors handling improvements, so now it should perform much better than before. Note that if you run it after other scripts, you may still need to reset the board first - unfortunately, memory on ESP8266 is plain scarce, while real-world data objects required to install modules from PyPI are big, so that's it. Ideas for optimizing that further are at https://github.com/micropython/micropython/issues/3072 , but they're not low-hanging any longer at all.

So, upip 1.2 with these improvements are now available in daily ESP8266 at the usual place. Feel free to test it and report any possible regressions, ahead of MicroPython 1.9 release which will bring it to everyone.
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib

User avatar
deshipu
Posts: 1331
Joined: Thu May 28, 2015 5:54 pm

Re: Next stretch goal - upip running native on esp8266

Post by deshipu » Sat May 06, 2017 11:50 am

My request for a MicrtoPython trove classifier also went through, so we can now tag the packages on PyPi with Programming Language :: Python :: Implementation :: MicroPython.

Post Reply