MicroPython package (module!) manager ideas

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

MicroPython package (module!) manager ideas

Post by pfalcon » Mon Mar 31, 2014 11:01 pm

I'm starting to actually consider may previous idea of (re)using existing Python tools and infrastructure for distributing and installing modules/packages.

Basic requirement thus:
  • Do not reinvent or reimplement the wheel, reuse existing "big Python" stuff.
But "big Python" is bloat, so some specific requirements:
  • Should support packaging and installing modules, not just packages.
  • Should be able to install without using .pth files and other funky novelties.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: MicroPython package (module!) manager ideas

Post by pfalcon » Mon Mar 31, 2014 11:09 pm

Note that I'm not really familiar with internals of Python packaging framework(s), so below is just quick&dirty half-blind attempts.

So, setuptools is used to package python modules/packages. Ahem, it's all centered around packages, even setup()'s argument is called "packages". Intuitive value of "." leads to:

Code: Select all

WARNING: '.' not a valid package name; please use only.-separated package names in setup.py
But stupid thing does the right thing - it packages module without any intermediate dirs. Still, the messages suggest this behavior is not compliant to anything.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: MicroPython package (module!) manager ideas

Post by pfalcon » Mon Mar 31, 2014 11:30 pm

Ok, by running "python setup.py sdist" we built "simple-0.1.tar.gz" which can be uploaded to PyPi.

Now let's try to install it. Of course, just trying to install it into virtualenv puts it into a subdir and adds easy-install.pth bloat. "python setup.py install --help" shows gazillion of path-related options, but using most of them leads to:

Code: Select all

$ python setup.py install --prefix ~/tmp/inst --no-compile
WARNING: '.' not a valid package name; please use only.-separated package names in setup.py
running install
Checking .pth file support in /home/pfalcon/tmp/inst/lib/python3.4/site-packages/
/home/pfalcon/tmp/.venv/bin/python -E -c pass
TEST FAILED: /home/pfalcon/tmp/inst/lib/python3.4/site-packages/ does NOT support .pth files
error: bad install directory or PYTHONPATH

You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from.  The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    /home/pfalcon/tmp/inst/lib/python3.4/site-packages/

and your PYTHONPATH environment variable currently contains:

    ''

Here are some of your options for correcting the problem:

* You can choose a different installation directory, i.e., one that is
  on PYTHONPATH or supports .pth files

* You can add the installation directory to the PYTHONPATH environment
  variable.  (It must then also be on PYTHONPATH whenever you run
  Python and want to use the package(s) you are installing.)

* You can set up the installation directory to support ".pth" files by
  using one of the approaches described here:

  https://pythonhosted.org/setuptools/easy_install.html#custom-installation-locations

Please make the appropriate changes for your system and try again.

Surely, they miss the most obvious suggestion - how to disable usage of flameing .pth files.

Finally, success with --root option - ugly .pth is gone! A bit of trial and error, and following gives pretty goofd layout:

Code: Select all

python setup.py install --install-base . \
    --install-purelib lib \
    --install-platlib lib \
    --install-scripts . \
    --install-headers . \
    --install-data . \
    --root ~/tmp/inst --no-compile
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: MicroPython package (module!) manager ideas

Post by pfalcon » Mon Mar 31, 2014 11:36 pm

The last mile is getting rid of those .egg-info dirs (just think - they could install all that metadata under single directory well-destined, but nope, need to intersperse it with the real source files).

Turns it actually easy:

Code: Select all

python setup.py install_lib \
    --no-compile \
    --install-dir ~/tmp/inst
So, "install" action is actually "install_lib", followed by other "install_*", followed by "install_egg_info".
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: MicroPython package (module!) manager ideas

Post by pfalcon » Mon Mar 31, 2014 11:41 pm

So, I consider these results reassuring. setuptools can be told to do the right thing. Now the question is how to make pip call setuptools in the right way. Fortunately, there's reassuring info for that - it's possible to have user-level config for setuptools: http://stackoverflow.com/questions/3560 ... tutils-cfg , maybe it even has an envvar to pass one in.

As a risk management, we can patch pip.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: MicroPython package (module!) manager ideas

Post by pfalcon » Mon Mar 31, 2014 11:45 pm

Folks who know Python packaging stuff are welcome to comment! As a final touch, bedtime story: Python Packaging: Hate, hate, hate everywhere.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: MicroPython package (module!) manager ideas

Post by pfalcon » Tue Apr 01, 2014 7:49 pm

Ok, found an official way to package a module. The recipe is simple actually: don't go to setuptools and other bloat whose tutorials force usage of packages, distutils from stdlib is enough: https://docs.python.org/3.3/distutils/i ... le-example
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: MicroPython package (module!) manager ideas

Post by pfalcon » Tue Apr 01, 2014 7:53 pm

Another news is that I got pip to work as needed. It's a bit borderline, I had to exploit pip internal behavior details to work around some issues, but woohoo - it will be possible to install uPy modules using pip! (I hope it's clear that the talk is about preparing on a host complete set of lib files required to run a particular project - there's no talk about running pip on a device).
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

stijn
Posts: 735
Joined: Thu Apr 24, 2014 9:13 am

Re: MicroPython package (module!) manager ideas

Post by stijn » Wed Apr 30, 2014 6:33 pm

Would this also work with so-called non-pure modules, i.e. containing C code? I think the standard distutils can in such case invoke the compiler etc.
This maybe requires that the core uPython functionality resides in a dynamic library so it can be reused by the external modules (and current REPL's main would also use the core library)?

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: MicroPython package (module!) manager ideas

Post by pfalcon » Wed Apr 30, 2014 7:50 pm

There's no support for building C modules outside uPy tree, no dynamic native modules support, and were no people interested to work on that (including me, who are much more interested in uPy under "unix" than on bare metal).
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

Post Reply