Page 1 of 2

MicroPython package (module!) manager ideas

Posted: Mon Mar 31, 2014 11:01 pm
by pfalcon
I'm starting to actually consider may previous idea of (re)using existing Python tools and infrastructure for distributing and installing modules/packages.

Basic requirement thus:
  • Do not reinvent or reimplement the wheel, reuse existing "big Python" stuff.
But "big Python" is bloat, so some specific requirements:
  • Should support packaging and installing modules, not just packages.
  • Should be able to install without using .pth files and other funky novelties.

Re: MicroPython package (module!) manager ideas

Posted: Mon Mar 31, 2014 11:09 pm
by pfalcon
Note that I'm not really familiar with internals of Python packaging framework(s), so below is just quick&dirty half-blind attempts.

So, setuptools is used to package python modules/packages. Ahem, it's all centered around packages, even setup()'s argument is called "packages". Intuitive value of "." leads to:

Code: Select all

WARNING: '.' not a valid package name; please use only.-separated package names in setup.py
But stupid thing does the right thing - it packages module without any intermediate dirs. Still, the messages suggest this behavior is not compliant to anything.

Re: MicroPython package (module!) manager ideas

Posted: Mon Mar 31, 2014 11:30 pm
by pfalcon
Ok, by running "python setup.py sdist" we built "simple-0.1.tar.gz" which can be uploaded to PyPi.

Now let's try to install it. Of course, just trying to install it into virtualenv puts it into a subdir and adds easy-install.pth bloat. "python setup.py install --help" shows gazillion of path-related options, but using most of them leads to:

Code: Select all

$ python setup.py install --prefix ~/tmp/inst --no-compile
WARNING: '.' not a valid package name; please use only.-separated package names in setup.py
running install
Checking .pth file support in /home/pfalcon/tmp/inst/lib/python3.4/site-packages/
/home/pfalcon/tmp/.venv/bin/python -E -c pass
TEST FAILED: /home/pfalcon/tmp/inst/lib/python3.4/site-packages/ does NOT support .pth files
error: bad install directory or PYTHONPATH

You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from.  The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    /home/pfalcon/tmp/inst/lib/python3.4/site-packages/

and your PYTHONPATH environment variable currently contains:

    ''

Here are some of your options for correcting the problem:

* You can choose a different installation directory, i.e., one that is
  on PYTHONPATH or supports .pth files

* You can add the installation directory to the PYTHONPATH environment
  variable.  (It must then also be on PYTHONPATH whenever you run
  Python and want to use the package(s) you are installing.)

* You can set up the installation directory to support ".pth" files by
  using one of the approaches described here:

  https://pythonhosted.org/setuptools/easy_install.html#custom-installation-locations

Please make the appropriate changes for your system and try again.

Surely, they miss the most obvious suggestion - how to disable usage of flameing .pth files.

Finally, success with --root option - ugly .pth is gone! A bit of trial and error, and following gives pretty goofd layout:

Code: Select all

python setup.py install --install-base . \
    --install-purelib lib \
    --install-platlib lib \
    --install-scripts . \
    --install-headers . \
    --install-data . \
    --root ~/tmp/inst --no-compile

Re: MicroPython package (module!) manager ideas

Posted: Mon Mar 31, 2014 11:36 pm
by pfalcon
The last mile is getting rid of those .egg-info dirs (just think - they could install all that metadata under single directory well-destined, but nope, need to intersperse it with the real source files).

Turns it actually easy:

Code: Select all

python setup.py install_lib \
    --no-compile \
    --install-dir ~/tmp/inst
So, "install" action is actually "install_lib", followed by other "install_*", followed by "install_egg_info".

Re: MicroPython package (module!) manager ideas

Posted: Mon Mar 31, 2014 11:41 pm
by pfalcon
So, I consider these results reassuring. setuptools can be told to do the right thing. Now the question is how to make pip call setuptools in the right way. Fortunately, there's reassuring info for that - it's possible to have user-level config for setuptools: http://stackoverflow.com/questions/3560 ... tutils-cfg , maybe it even has an envvar to pass one in.

As a risk management, we can patch pip.

Re: MicroPython package (module!) manager ideas

Posted: Mon Mar 31, 2014 11:45 pm
by pfalcon
Folks who know Python packaging stuff are welcome to comment! As a final touch, bedtime story: Python Packaging: Hate, hate, hate everywhere.

Re: MicroPython package (module!) manager ideas

Posted: Tue Apr 01, 2014 7:49 pm
by pfalcon
Ok, found an official way to package a module. The recipe is simple actually: don't go to setuptools and other bloat whose tutorials force usage of packages, distutils from stdlib is enough: https://docs.python.org/3.3/distutils/i ... le-example

Re: MicroPython package (module!) manager ideas

Posted: Tue Apr 01, 2014 7:53 pm
by pfalcon
Another news is that I got pip to work as needed. It's a bit borderline, I had to exploit pip internal behavior details to work around some issues, but woohoo - it will be possible to install uPy modules using pip! (I hope it's clear that the talk is about preparing on a host complete set of lib files required to run a particular project - there's no talk about running pip on a device).

Re: MicroPython package (module!) manager ideas

Posted: Wed Apr 30, 2014 6:33 pm
by stijn
Would this also work with so-called non-pure modules, i.e. containing C code? I think the standard distutils can in such case invoke the compiler etc.
This maybe requires that the core uPython functionality resides in a dynamic library so it can be reused by the external modules (and current REPL's main would also use the core library)?

Re: MicroPython package (module!) manager ideas

Posted: Wed Apr 30, 2014 7:50 pm
by pfalcon
There's no support for building C modules outside uPy tree, no dynamic native modules support, and were no people interested to work on that (including me, who are much more interested in uPy under "unix" than on bare metal).