MICROPY_GIT_TAG and deterministic builds

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Post Reply
MindJuice
Posts: 18
Joined: Tue Apr 14, 2020 6:09 am

MICROPY_GIT_TAG and deterministic builds

Post by MindJuice » Thu May 13, 2021 2:15 am

I'm using MP and trying to get reproducible builds.

The only diff in my builds between two different repos with the same source code (but different commit histories) comes from 7 bytes in 2 different places in the final binary.

I was able to track it down to the `MICROPY_GIT_TAG` and `MICROPY_GIT_HASH` being different.

These just appear to be used for informational purposes only in the REPL, and I don't think I need them anywhere since my shipping product never exposes the REPL.

Any downside I'm missing from hardcoding these in `py/makeversionhdr.py`?

Thanks.

MindJuice
Posts: 18
Joined: Tue Apr 14, 2020 6:09 am

Re: MICROPY_GIT_TAG and deterministic builds

Post by MindJuice » Thu May 13, 2021 2:19 am

Same for `MICROPY_BUILD_DATE`.

stijn
Posts: 735
Joined: Thu Apr 24, 2014 9:13 am

Re: MICROPY_GIT_TAG and deterministic builds

Post by stijn » Thu May 13, 2021 6:12 am

For the date you're supposed to set SOURCE_DATE_EPOCH see https://github.com/micropython/micropython/pull/6789.

For the other fields: apart from the hardcoding itself which is hard to maintain (could be fixed by either building from within a git tree or modifying makeversionhdr to force it to use the vesrion from the cods) there aren't side effects for MicroPython itself.

MindJuice
Posts: 18
Joined: Tue Apr 14, 2020 6:09 am

Re: MICROPY_GIT_TAG and deterministic builds

Post by MindJuice » Sat May 15, 2021 1:24 am

Thanks for the info!

MindJuice
Posts: 18
Joined: Tue Apr 14, 2020 6:09 am

Re: MICROPY_GIT_TAG and deterministic builds

Post by MindJuice » Fri Aug 06, 2021 7:11 pm

To follow up on the deterministic builds topic, we ran into one more area of non-determinism in the MicroPython build.

Previously I was the only one building the code, but now there are two of us working on it, and we were getting different binaries when building identical source trees.

The issue turned out to be that when make processes the $(wildcard) function, the results are not guaranteed to be the same between computers. The list of files matching $(wildcard $(BOARD_DIR)/*.c) on my colleague's computer was in a different order than on my computer. As a result, the files listed in the SRC_C variable in make were not in the same order.

When SRC_C was passed to the compiler (and eventually the linker), the filenames were in a different order. You could see the same code was produced, but the addresses in the map file and final binary were different.

The fix we made is below:

Code: Select all

--- a/ports/stm32/Makefile
+++ b/ports/stm32/Makefile
@@ -302,7 +302,7 @@ SRC_C = \
        servo.c \
        dac.c \
        adc.c \
-       $(wildcard $(BOARD_DIR)/*.c)
+       $(sort $(wildcard $(BOARD_DIR)/*.c))
 
 SRC_O = \
        $(STARTUP_FILE) \
It took us several hours to track down exactly why this was happening, so hopefully this saves someone else some time.

Post Reply