MicroPython Forum (Archive)

Here's a hint for folks who are not yet up to speed to contribute code, but who'd want to help with development: it would be nice to set up some continuous integration for projects, and canonical one for github is Travis CI. Googling and browsing thru some github projects will give more details.

Can you either explain some of your terminology or link us to definitions of your terminology? Here's my attempt at parsing what you wrote:

Here's a hint for folks who are not yet up to speed to contribute code, but who'd want to help with development: it would be nice to set up some continuous integration for micropython and all of it's children, and canonical one for github is Travis CI. Googling and browsing thru some github projects will give more details.

Hey SwimDud0614,

So here's a bit of background on Travis: http://en.wikipedia.org/wiki/Travis_CI

Basically, what it does is on every checkin of code (some people call this a push or a merge, or a pull request) into github, then it will kick off a Travis run. The Travis run is essentially a bunch of tests to verify that what's been comitted isn't breaking anything.

At work we have it setup so that just creating a pull request causes a travis run to go against the pull request. Then you can know that the PR won't break things when it gets merged.

I think this is our live test results from work (I don't work on this area of the code much)
https://travis-ci.org/mozilla-b2g/gaia (shows whats building right now)
https://travis-ci.org/mozilla-b2g/gaia/builds (shows what has finished building)

green means that the tests passed, red means something failed, and IIRC yellow is in progress.

Travis and github can be integrated. For example, if you look at this:
https://github.com/mozilla-b2g/gaia/pull/17321
you'll see a red X beside the git hash. It you hover your mouse over it then it will say that it failed (and there is also a Failed notification beside the merge icon).

Here's a success example:
https://github.com/mozilla-b2g/gaia/pull/17314

The mbedmicro repository also has integration with Travis setup:
https://github.com/mbedmicro/mbed/pull/220 (you can see the green checkmarks here)

Lets suppose I wanted to commit a change to micropython, then the process should go something like this:
- I do my changes locally and then create a pull request (PR)
- Creating the PR should kick off some travis tests which go and verify that I didn't break anything
- Once the travis tests are green (and Damien is otherwise happy with the PR), then Damien would go ahead and merge it.
- This would probably kick off another set of tests
- When tests start failing, then someone needs to figure out if the test si bad or if the commited code is bad.
If the commited code is bad, then it generally gets backed out
If the tests are bad, then the tests need to be fixed.

A non-Travis example of continuous integration (this is what I actually use on a day-to-day basis):
https://tbpl.mozilla.org/?tree=B2g-Inbound

B is for builds. If you scroll down a bit, you'll start to see other letters and numbers beside the B. Those are test suites (each one is hundreds or perhaps even thousands of tests). You'll see that we run builds and tests on a variety of different platforms, with both Debug and non-Debug builds. Things with a star beside them are failures that have a known reason for failing.

If a commit gets an unknown failure, then it typically gets backed out and the developer has to investigate what went wrong, fix it, and resubmit. At work we also use integration and master branches. The link I posted above was for an integration branch. Periodically, (once or twice a day) when the integration branch is green, then it gets merged to the master branch, and the whole process is repeated on the master branch.

When things start failing really badly (lots of red), then the trees will sometimes be closed for further checkins until it can be brought back to a green state.

By running tests on every checkin, it makes it much easier to figure out which particular checking caused a problem and then back it out. We have hundreds of commits per day, so without this type of automation, things would be broken most of the time, whereas the way things are now, the stuff on master is actually fairly stable.

If you still have questions, please feel free to ask away.

And although I may know a bunch about continuous integration and how it works, I've never actually set something like this up.

Thank you for the thorough explanation! I'm surprised I've never heard of anything like this at my job or from anyone else. We of course have plenty of tests that get run often, but I don't recall hearing of automated tests.

I've also heard Continuous Integration referred to as smoke tests.

Just been skimming through http://docs.travis-ci.com/user/ci-environment/ - it says

To set up the system for your build, you can use the sudo command to install packages, to change configuration, create users, and so on.

- do we definitely know if we'll be able to install and use the arm-none-eabi-gcc compiler?
In terms of architecture it only says

Ubuntu 12.04 LTS Server Edition 64 bit.

so I guess we'd only be able to run the tests against the x64 unix port, and not the x86 and ARM unix versions too (but of course we'd still be able to check they at least cross-compile successfully).

Note that this comment doesn't mean I'm volunteering for this task though!

Thanks to the effort and contributions from several people. Travis CI is now integrated and fully working
https://travis-ci.org/micropython/micropython

Great work! So, let's think where to go from here.

We already have handful of configuration options and want to have more, per https://github.com/micropython/micropython/issues/35 . The only way to sustain such level of configurability is to have it tested. 10 independent boolean options means 1024 various configuration to test, so doing it brute-force way won't work.

Generally, Travis CI is courtesy service, and while we surely not that most resource-consuming project there and hopefully have some headroom to increase build time, we definitely should not abuse the hospitality and make sure we have it well optimized if we want to build multiple configs. Increased build time will of course work against us either: it's cool to have a pull request tested in 2 minutes and be open for merging, instead of waiting half an hour.

So, I'd like to invite to discuss idea how to move forward with our test coverage.

I think testing with all options (that make sense) enabled, and those same options disabled would buy you a huge amount of coverage.

Then we'll need a way to adapt the tests based on the options. For example, if you turn off floating point support, then you don't want to be testing '{:e}'.format(1.0)

The typical problem that I've seen is that it compiles the default way, but when some option is toggled then things don't compile.

Most of the options *should* be mutually exclusive.

The typical problem that I've seen is that it compiles the default way, but when some option is toggled then things don't compile.

Exactly. So I don't even want to bring question of conditional test for starters. (If we want to go conditional, we first need to structure and order tests - like have a basic core to run first, to make sure that for example conditional works as expected). Summing up, initial aim is just ensure buildability with various config options.

Most of the options *should* be mutually exclusive.

Yeah, most (or many) should be. But it's not that simple. We have non-bool options (e.g. long int impl), and ultimately there is/will be context dependency, for example, if some type is completely disabled, there's no point to disable its methods individually.

If we went with a kbuild-like configuration, at least you can specify dependencies between options.
The kernel has a similar problem (try compiling for the all of the variations of config options).
The kernel has: allyesconfig, allnoconfig, defconfig.

Realisitcally, we really only need to test each set of options for the boards that people are actually using. Testing for other combinations is probably nice, but probably not required.

Float has 3 variations and int has 3 variations. These seem like some fairly core fundamental options, and I would be inclined to take a default config, test by trying the 3 float options, and then test by trying the 3 int options.

You could also probably set things up so that it cycles through combinations so that it doesn't actually run all of the combinations on all of the checkins. This gives you the coverage, but doesn't take as many resources. And you could have something special where you actually do a build which runs all of the combinations (perhaps before a major release or something) (take the travis build number modulo some number of configurations to come up with a configuration to test this time around).