upip - Self-hosted package manager

Discussion about programs, libraries and tools that work with MicroPython. Mostly these are provided by a third party.
Target audience: All users and developers of MicroPython.
pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: upip - Self-hosted package manager

Post by pfalcon » Thu Oct 15, 2015 4:19 pm

So, Dave essentially answered that question, I just would like to mention few corner-case points, which may be good to know:

MICROPYPATH is of course designed to work much like, and serve the same purpose as CPYthon's PYTHONPATH. But:

1. Unlike PYTHONPATH, which just augments CPython's search path (CPython will still look at some hardcoded locations even if you set it), MICROPYPATH specifies complete and only search path for MicroPython. The obvious reason is that MicroPython simply doesn't have any "default" library, any modules besides built into executable are external and should be installed separately. That choice also fits well with "micro" design and with the idea that uPy is fully relocatable and self-contained (you can have multiple versions of uPy with different version of modules running on the same system, and due to way MICROPYPATH works, it's very easy to avoid "version hell").

2. MicroPython doesn't support namespace packages in full fledge. If there're namespace packages, the should be installed in one top-level directory (even though uPy supports searching multiple dirs).


And the final point - upip should respect MICROPYPATH, but currently it doesn't. This is bug and would be first thing I fix in upip. (UPDATE: This was fixed of course.) In the meantime, as Dave suggests, you can use -p switch to set installation path explicitly.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: upip - Self-hosted package manager

Post by dhylands » Thu Oct 15, 2015 4:40 pm

pfalcon wrote:And the final point - upip should respect MICROPYPATH, but currently it doesn't. This is bug and would be first thing I fix in upip. In the meantime, as Dave suggests, you can use -p switch to set installation path explicitly.
Just to clarify a detail, presumably this would pick the first path from MICROPYPATH and use that as the default installation directory, still overridable by using the -p option?

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: upip - Self-hosted package manager

Post by pfalcon » Thu Oct 15, 2015 4:45 pm

dhylands wrote:@pfalcon - so assuming that I've installed some modules (like micropython-os) what's the procedure for keeping them up-to-date?

Right now it looks like I'd have to keep track of all of the various modules installed and then reinstall them.
Ok, to answer this question, let's consider how CPython lets keep you modules like "os" up to date. Turns out, it doesn't really offer any automagic way to do that. There're no automatic upgrades to standard library. You have to wait for next Python release, then install it to get updates. If there will be security issue, they'll haste a new point release, but it's still up to you to install it. Of course, if you run modern distro with auto-updates, your distro may automatically update package in case of security issue, if you allow to do it such things behind your back. The point, it's not CPython's job, but user's or distro's.

The above was about CPython stdlib modules, like "os". Now let's look how it works for a particular Python application. So, you have an application which has some dependencies. These dependences are listed in standard (by convention) file requirements.txt. Then you run "pip install -i requirements.txt", and voila, you app is ready to be run. Then you run it happily. Then when you feel like upgrading to new version of dependences, you run the same command again - it will pick up whatever new version of modules are available.

So, the main point is that upgrades are driven by user, not by the systems. And the reason is obvious - an upgrade may break what was worked previously, so user should be doing it from their side, in good faith and ready to handle consequences.

So, how all the above applies to MicroPython? uPy doesn't have any builtin stdlib, so you don't need to wait for next release to get new version of stdlib module, already an improvement over CPython ;-). And upip of course supports -i switch and requirements.txt, and due to the way MICROPYPATH works, it's very easily to prepare a self-contained application snapshot, which won't be adversely affected by any module updates. In CPython, you need to use virtualenv for that, in uPy, it essentially comes for free.


So, the only question left is how to update "default" stdlib (one which you use for interactive sessions and experimenting, not to deploy apps). Well, currently you have an option to prepare a (long) requirements.txt and run upip against it from time to time. But the idea I had for a long time is to create micropython-all meta-package which would allow to install entire stdlib in one go, then allow to keep it up to date easily. It's not there yet because: I hope you already saw that upip isn't too smart and downloads a module every time even if it's already installed. That's because upip currently doesn't store package metadata, so can't know which version local package is. Once that it's resolved, there will be almost complete bliss ;-).
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: upip - Self-hosted package manager

Post by pfalcon » Thu Oct 15, 2015 4:49 pm

dhylands wrote:
pfalcon wrote:And the final point - upip should respect MICROPYPATH, but currently it doesn't. This is bug and would be first thing I fix in upip. In the meantime, as Dave suggests, you can use -p switch to set installation path explicitly.
Just to clarify a detail, presumably this would pick the first path from MICROPYPATH and use that as the default installation directory, still overridable by using the -p option?
Exactly, that always was the idea, I'm actually surprised I managed to forget to implement it ;-). Hopefully installing to first dir in the list is pretty obvious and non-surprising, and installing everything to a single dir is what required to not face problems with namespace packages (if somebody ever creates them for upy of course ;-) ).
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

User avatar
platforma
Posts: 258
Joined: Thu May 28, 2015 5:08 pm
Location: Japan

Re: upip - Self-hosted package manager

Post by platforma » Fri Oct 16, 2015 9:25 am

That's grand. Thanks for your answers!

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: upip - Self-hosted package manager

Post by pfalcon » Sat Oct 17, 2015 5:19 pm

Ok, MICROPYPATH environment var is now handled properly by upip: https://github.com/micropython/micropyt ... f452c36868
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: upip - Self-hosted package manager

Post by pfalcon » Sat Oct 17, 2015 5:21 pm

pfalcon wrote:But the idea I had for a long time is to create micropython-all meta-package which would allow to install entire stdlib in one go, then allow to keep it up to date easily.

Well, nothing precludes starting to work on this right away: I submitted https://github.com/micropython/micropyt ... /issues/47 to make the naming of this meta-package right. Comments welcome.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

doudz
Posts: 2
Joined: Fri Mar 25, 2016 10:26 am

Re: upip - Self-hosted package manager

Post by doudz » Fri Mar 25, 2016 10:32 am

Hi !

I tried upip and it works like a charm, but I'm not sure I can use it to add libs in order to add them in my ESP8266.
So the question is : I would like to add a lib to my ESP8266 firmware (micropython-socket for example), could I use upip ? (and how ?) or should I put it in the "scripts" sub directory ?

User avatar
kfricke
Posts: 342
Joined: Mon May 05, 2014 9:13 am
Location: Germany

Re: upip - Self-hosted package manager

Post by kfricke » Fri Mar 25, 2016 5:44 pm

(...just trying to keep pressure from the main developers while answering stuff I do not really know the truth about...)

The upip support of the ESP port is one of the later stretch goals and of course let one think you can do what you did describe.

But there have not been any signs of this on the Github commit logs. At the moment the first parts of the Websockets seem to land in Github repository.
There are still some features present in the higher-tier backer only firmware, relating to flash-based FAT filesystem, block devices and stuff, which seem to not have been refactored yet and thus not found their way into the Github repository. These should be requirements before a functional upip port makes sense.

So I would suggest to stay patient and wait for thees features to arrive in order.

As far as I think to know pfalcon, he will post here immediately when this little baby of him is ready for us to be used!

Post Reply