Wanted: Travis CI integration

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
pfalcon
Posts: 1135
Joined: Fri Feb 28, 2014 2:05 pm

Re: Wanted: Travis CI integration

Post by pfalcon » Thu Apr 17, 2014 8:01 pm

(If we want to go conditional, we first need to structure and order tests - like have a basic core to run first, to make sure that for example conditional works as expected).
Well, @dpgeorge just did something like that: https://github.com/micropython/micropyt ... 952b45581c
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1135
Joined: Fri Feb 28, 2014 2:05 pm

Re: Wanted: Travis CI integration

Post by pfalcon » Thu Apr 17, 2014 8:20 pm

dhylands, Good thinking, 70% matches my thoughts.

But let's start with focusing on how to build multiple configurations in the most efficient way. The obvious one would be to avoid rebuilding files not changed between configurations. I guess, you, as the author of dependency tracking code in makefiles, might have some ideas about that. Because I don't see a way to make individual files to depend on individual options, without going to some funky tricks. For example, I may imagine following scheme: a file which has #if for MICROPY_FOO, should include file config/micropy_foo.h, which has that sole option defined. Then, by setting values in such individual config files, we achieve the desired effect. But again, I don't think that's practical.

Then, my hope is on ccache. I guess if options would be passed on commandline (e.g. make CC="gcc -DMICROPY_FOO=1"), it should work (TBC). If options set will be put in a config header file, then probably won't work (because all preprocessed files will be different, unless (and only possibly) if number of lines in such config header file is kept constant).

So, any ideas on these matters and on build speedup in general? Thanks!
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

User avatar
dhylands
Posts: 3057
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Wanted: Travis CI integration

Post by dhylands » Thu Apr 17, 2014 9:52 pm

As for making individual files depend on individual options, the kernel plays a neat game.

The .config contains the entire configuration, and include/generated/autoconf.h contains the resulting configuration in C source form.

include/config contains a directory tree with an empty file for each config option.

The kernel build then removes the dependency on linux/autoconf.h and for each CONFIG option used in the source, it adds a dependency on the zero-length file.

If you update the config options, then it touches the .h file in include/config for each CONFIG option which was changed.

So when you change a config option, only the affected files get rebuilt.

pfalcon
Posts: 1135
Joined: Fri Feb 28, 2014 2:05 pm

Re: Wanted: Travis CI integration

Post by pfalcon » Thu Apr 17, 2014 9:58 pm

Ok, so what's your suggestion? Adopt Kconfig? Replicate this core config handling? Or do you think ccache would be easier solution still?
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

User avatar
dhylands
Posts: 3057
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Wanted: Travis CI integration

Post by dhylands » Thu Apr 17, 2014 10:08 pm

So we don't necessarily need to adopt Kconfig, just use the same ideas.

We could extract the config options and put them all into .mk files. Then generate a .h file and the empty header files (seems like a fairly straight forward python script) and write a script to fixup the dependencies (just by searching the source for MICROPY_).

ccache also seems like a reasonable approach (and you could use it regardless of what else you decide to do as far as dependencies go).
Its obvioulsy much less effort to use ccache, and I think its beneficial to use ccache regardless of anything else we decide to do.

pfalcon
Posts: 1135
Joined: Fri Feb 28, 2014 2:05 pm

Re: Wanted: Travis CI integration

Post by pfalcon » Thu Apr 17, 2014 10:20 pm

Unfortunately, ccache doesn't work the I'd expect it to:

(with empty .ccache dir)

Code: Select all

$ make clean
$ time make CC="ccache gcc"
real	0m15.636s
$ time make CC="ccache gcc -DMICROPY_CPYTHON_COMPAT=0"
real	0m14.484s
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

User avatar
dhylands
Posts: 3057
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Wanted: Travis CI integration

Post by dhylands » Thu Apr 17, 2014 10:35 pm

Since you're modifying the command line for every source file, I would expect that to recompile each and every file, thereby giving no speedup.

And yeah - modifying mpconfigport.h will cause almost all of the files to get rebuilt.

The big advantage of ccache is:

Code: Select all

$ ccache -C
$ make clean
$ time make CC="ccache gcc"
real	0m9.385s
$ make clean
$ time make CC="ccache gcc"
real	0m2.611s
So if we combine ccache with finer granularity on the dependencies then you could set some options, build, clean, modify options build and the second build will reuse objects from the first (when possible).

You could also just do: modify config, build, modify config, build (without the clean) and then you wouldn't need to use ccache.

pfalcon
Posts: 1135
Joined: Fri Feb 28, 2014 2:05 pm

Re: Wanted: Travis CI integration

Post by pfalcon » Thu Apr 17, 2014 10:40 pm

Ok, so I personally don't feel like working on config stuff currently, because I'm not sure about which way to go and specific details, let's see if someone will have enough vision/drive for that.
You could also probably set things up so that it cycles through combinations so that it doesn't actually run all of the combinations on all of the checkins. This gives you the coverage, but doesn't take as many resources. (take the travis build number modulo some number of configurations to come up with a configuration to test this time around).
My thinking either. But I wasn't sure how to achieve fair cycling thru configs. Using build number for that seems like neat idea. @lurch, you of us probably read most Travis CI docs, does it expose current build # as envvar?
And you could have something special where you actually do a build which runs all of the combinations
My thinking too. We could have separate branch with suitable .travis.yaml and push to it from time to time.

And we definitely should start with building default unix configuration, and then minimal (by criteria of executable size) one.

I'll look into these last ideas.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1135
Joined: Fri Feb 28, 2014 2:05 pm

Re: Wanted: Travis CI integration

Post by pfalcon » Thu Apr 17, 2014 10:44 pm

Since you're modifying the command line for every source file, I would expect that to recompile each and every file, thereby giving no speedup.
I would expect it to be picky about stuff on cmdline which forces recompile - -D in itself doesn't affect generated code, it should be used in #if*, and that should be caught when checking preprocessor output. Or so I'd think.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

User avatar
lurch
Posts: 11
Joined: Sat Apr 12, 2014 2:21 pm
Location: Cambridge, UK
Contact:

Re: Wanted: Travis CI integration

Post by lurch » Sat Apr 19, 2014 12:02 am

IMHO whichever solution gets decided upon, a useful first step would be a machine-readable 'meta-file' listing the names of the various config options, and the legal values each option can take. (and maybe it would be useful for this meta-file to also specify if the config option applies at the global, port or board-specific level, if that makes sense?)

Post Reply