upip - Self-hosted package manager

Discussion about programs, libraries and tools that work with MicroPython. Mostly these are provided by a third party.
Target audience: All users and developers of MicroPython.
pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

upip - Self-hosted package manager

Post by pfalcon » Sat Jan 31, 2015 4:03 pm

As the number of modules written for MicroPython grows, it would be nice to install them directly on a target running MicroPython, including smaller ones, like PyBoard - instead of current situation where package manager is a wrapper around standard CPython's "pip", so running it requires full CPython3, so packages have to be "cross-installed" on a host.

I've been working towards that direction in the background for some time, and guess it's worth to have a specific aim now. I've submitted https://github.com/micropython/micropyt ... /issues/19 to track that. Any (focused) ideas and help are welcome.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

Turbinenreiter
Posts: 288
Joined: Sun May 04, 2014 8:54 am

Re: Self-hosted package manager

Post by Turbinenreiter » Sat Jan 31, 2015 4:27 pm

Is your initial prototyping available somewhere?

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: Self-hosted package manager

Post by pfalcon » Sat Jan 31, 2015 6:24 pm

As usual, writing something more or less serious in uPy leads to bunch of missing dependencies and bugs. So, currently I'm implementing/fixing/polishing initial dirty hacks I have. Here's recent example, non-dummy gzip module: https://github.com/micropython/micropyt ... 22c7e58764 . To implement it, had to add support for raw DEFLATE decoding to uzlib, and turned out the module had issues with decompressing big data blocks. All those issues are fixed now. More to do.

So, I hope to spool all the prerequisites required for current prototype in the near time, and elaborate the prototype itself too.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

Turbinenreiter
Posts: 288
Joined: Sun May 04, 2014 8:54 am

Re: Self-hosted package manager

Post by Turbinenreiter » Sat Jan 31, 2015 7:02 pm

I think it would be worth knowing which prerequisites are missing. Kinda hard to help when I don't know what to hack on ;)

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: Self-hosted package manager

Post by pfalcon » Sun Feb 01, 2015 5:42 pm

As ticket quoted above mentioned, on critical path to complete self-hostedness is SSL support. There's a separate ticket for that: https://github.com/micropython/micropython/issues/952 . Other issue without solving which self-hosted package manager won't work on small-memory targets is stream support for uzlib - currently, uncompressed size is limited by the amount of continuous free memory block.

Past that, what package manager does is downloading tarballs (tar.gz) from HTTPS, and uncompessing them. The latter part is already handled (modulo memory issues above) by micropython-lib:
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: Self-hosted package manager

Post by pfalcon » Mon Feb 16, 2015 9:49 pm

A basic working implementation is now available in https://github.com/pfalcon/micropython- ... aster/upip . Help with testing is welcome.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: Self-hosted package manager

Post by pfalcon » Sun Apr 05, 2015 12:51 pm

So far I wasn't able to get back to this and thoroughly test it myself, so would like to extend request for help with testing.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: Self-hosted package manager

Post by pfalcon » Sun May 03, 2015 9:32 pm

There unfortunately wasn't any feedback during this time. I finally sat down to make sure there're fruits of all this effort. It is able to install https://github.com/pfalcon/notes-pico , which is pretty good case of dependency handling. Then with some cleanups, upip 0.5 is released: https://pypi.python.org/pypi/micropython-upip/0.5 . I'll be preparing/submitting changes to make upip the default/recommended package manager for micropython.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

stijn
Posts: 735
Joined: Thu Apr 24, 2014 9:13 am

Re: Self-hosted package manager

Post by stijn » Mon May 04, 2015 7:39 am

Can you lay out all the steps needed to test this?

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: Self-hosted package manager

Post by pfalcon » Mon May 04, 2015 10:37 am

Yes:

1. Build unix version of micropython, make sure it's available in PATH.
2. Bootstrap upip using script from: https://github.com/micropython/micropython/pull/1219
3. Clone https://github.com/pfalcon/notes-pico
4. Use "make lib" on it to install dependencies using upip
5. Use "make run" to start the webapp.
6. Following printed link, try to add/delete some notes.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

Post Reply