Build Secrets

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
User avatar
devnull
Posts: 452
Joined: Sat Jan 07, 2017 1:52 am
Location: Singapore / Cornwall
Contact:

Build Secrets

Post by devnull » Mon Dec 23, 2019 9:52 am

Starting this thread to share information about build procedures, overrides, trick & hints & tips.

Apparently, you can override port/board configurations by creating a folder and including the path in the make command:

Code: Select all

make BOARD=myboard BOARD_DIR=~/src/projects/myboard
I just tested this, I have 2 files in my custom folder:

Code: Select all

GNUmakefile
mpconfigboard.mk
mpconfigboard.mk

Code: Select all

# Enable/disable extra modules
MICROPY_PY_LWIP ?= 1
# wiznet5k module for ethernet support; valid values are:
#   0    : no Wiznet support
#   5200 : support for W5200 module
#   5500 : support for W5500 module
MICROPY_PY_WIZNET5K ?= 5500
# cc3k module for wifi support
MICROPY_PY_CC3K ?= 0
GNUmakefile

Code: Select all

include Makefile
CFLAGS += -Wno-error
This results in a build failure and error:

Code: Select all

make: *** No rule to make target 'build-PYBD_SF2/genhdr/pins.h', needed by 'build-PYBD_SF2/py/mpstate.o'.  Stop.
So it seems that the build does not read the original file and replace any values with your values, but instead only reads your files.

I don't think this is very user friendly, as it means that you have to first copy the original file, and then modify it which will lead to problems when the original file is modified at source, or am I not doing this right ??

User avatar
jimmo
Posts: 1633
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Build Secrets

Post by jimmo » Mon Dec 23, 2019 10:41 am

GNUmakefile should be in the directory where you run make (i.e. ports/stm32). It's unrelated to the BOARD / BOARD_DIR feature.

Your board directory needs to have at least:
- mpconfigboard.mk
- mpconfigboard.h
- pins.csv
- stm32fyxx_hal_conf.h

Then you invoke it with

Code: Select all

make BOARD=myboard BOARD_DIR=~/src/projects/myboard
(Note that you're specifying the board there twice, both as the final directory of the BOARD_DIR, and directly as the BOARD).

The "build-PYBD_SF2" in the error message is surprising? Are you invoking it with maybe BOARD_DIR=my/board/dir but with BOARD still set to PYBD_SF2?

But that actual error is telling you that you don't have pins.csv I think.

I would start with just copying exactly the contents of ports/stm32/boards/SOMETHING. But then you can (for example) make your versions of mpconfigboard.h and mpconfigboard.mk include/import the version out of the main repo. Then add your overrides before/after the import.

This feature is really designed for when the board config is standalone though, inheriting from an existing board config isn't really what it's meant for (but it does work).

Also following on from what you said in the github issue -- you should be specifying the FROZEN_MANIFEST in mpconfigboard.mk, no need to put anything other than BOARD_DIR and BOARD on the make command line. (And you could put that in GNUmakefile too).

User avatar
devnull
Posts: 452
Joined: Sat Jan 07, 2017 1:52 am
Location: Singapore / Cornwall
Contact:

Re: Build Secrets

Post by devnull » Mon Dec 23, 2019 10:51 am

Jimmo - Thanks, but that's not the behaviour I was hoping for ;)

What would be really useful, is if you don't need to source the original files at all and instead, if a file exists in the custom folder, then the build process will first read the original git source file, and then override that file with any settings it finds in your custom file.

Having to duplicate the source files and modify it is quite 'inconvenient' and also leads to problems whereby the source file is modified at some time after you clone it, meaning that every time you do an update you have to re-clone and re-modify all the custom files !

That is exactly the problem that caused my recent build failure, I coped mpconfigport.mk many months ago and then modified it and copied it to the ports folder before build.

At that time this file did not contain

Code: Select all

MICROPY_VFS_FAT ?= 1
which must only have been added recently.

To be honest, I would not use the current design as it means more rather than less work for me, I will have to stick to my bash scripts to copy files, create symbolic links and append to the source files / folders :)

User avatar
jimmo
Posts: 1633
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Build Secrets

Post by jimmo » Mon Dec 23, 2019 11:25 am

Maybe I'm misunderstanding what you're trying to do. I guess you don't really have a custom board so much as a custom build configuration for an existing board?

i.e. if your scenario is "pyb1.1 with extra features enabled" that's a bit different to "pyb1.1 but with a different PCB".

But if all you want is to have a few different build presents (i.e. that only override makefile variables), then there's a much easier alternatives to using board configs...

In the same way that you use GNUmakefile (or lowercase "makefile") to set defaults for your local environment, you can just specify any arbitrary makefile to use.

So for example, I could make ~/src/projects/mything/mything.mk , where "mything" is a pybv11 with LWIP and no FAT FS and a custom manifest.py (which is in the same dir as mything.mk):

Code: Select all

MICROPY_VFS_FAT = 0
MICROPY_PY_LWIP = 1
BOARD = PYBV11
FROZEN_MANIFEST = $(dir $(realpath $(firstword $(MAKEFILE_LIST))))/manifest.py

include Makefile
then from ports/stm32:

Code: Select all

make -f ~/src/projects/mything/mything.mk

User avatar
devnull
Posts: 452
Joined: Sat Jan 07, 2017 1:52 am
Location: Singapore / Cornwall
Contact:

Re: Build Secrets

Post by devnull » Mon Dec 23, 2019 11:38 am

Maybe I'm misunderstanding what you're trying to do. I guess you don't really have a custom board so much as a custom build configuration for an existing board?
Yes, sorry that's exactly what I am looking to do, I did not make that very clear.

Let me test out what you have proposed, so it does not matter what the filename is as long as it is sourced using -f in the make command ?
Last edited by devnull on Mon Dec 23, 2019 11:59 am, edited 1 time in total.

User avatar
jimmo
Posts: 1633
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Build Secrets

Post by jimmo » Mon Dec 23, 2019 11:58 am

devnull wrote:
Mon Dec 23, 2019 11:38 am
so it does not matter what the filename is as long as it is sourced using -f in the make command ?
Yes... It does not matter. Using ".mk" is a convention, but you can call it whatever you like.

User avatar
devnull
Posts: 452
Joined: Sat Jan 07, 2017 1:52 am
Location: Singapore / Cornwall
Contact:

Re: Build Secrets

Post by devnull » Mon Dec 23, 2019 12:20 pm

Sadly it does not work for me:

custom.mk

Code: Select all

MICROPY_PY_LWIP ?= 1

# wiznet5k module for ethernet support; valid values are:
#   0    : no Wiznet support
#   5200 : support for W5200 module
#   5500 : support for W5500 module
MICROPY_PY_WIZNET5K ?= 5500

# cc3k module for wifi support
MICROPY_PY_CC3K ?= 0
Results in:

Code: Select all

make -j8 FROZEN_MANIFEST=/nfs/qnap/dev/uPython/bash/../build/micropython/ports/stm32/frozen.py -f /nfs/qnap/dev/uPython/bash/../build/micropython/ports/stm32/custom.mk MICROPY_VFS_FAT=1 BOARD=PYBD_SF6 CFLAGS_EXTRA='-DMICROPY_PY_THREAD=1'
make: *** No targets.  Stop.

User avatar
jimmo
Posts: 1633
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Build Secrets

Post by jimmo » Mon Dec 23, 2019 12:56 pm

Need "include Makefile" at the end.

Also the whole idea is to move everything from the command line into the .mk file.

User avatar
devnull
Posts: 452
Joined: Sat Jan 07, 2017 1:52 am
Location: Singapore / Cornwall
Contact:

Re: Build Secrets

Post by devnull » Mon Dec 23, 2019 10:51 pm

@jimmo - Great, this is working, so now my custom.mk file replaces a symbolic-linked GNUmakefile and my modified mpcconfigport.mk file without having to modify any source files and means that I can remove some bash scripts that copied and linked files before building.

nice :D

custom.mk

Code: Select all


## LWIP NETWORKING
MICROPY_PY_LWIP ?= 1

# WIZNET SUPPORT (0,5200,5500)
MICROPY_PY_WIZNET5K ?= 5500

# CC3K FOR WIFI
MICROPY_PY_CC3K ?= 0

#MAKE FILE
CFLAGS += -Wno-error

## FILE SYSTEM
#MICROPY_VFS_FAT=1
MICROPY_VFS_LFS2=1

## THREADING
CFLAGS_EXTRA='-DMICROPY_PY_THREAD=1'

## REQUIRED
include Makefile

User avatar
devnull
Posts: 452
Joined: Sat Jan 07, 2017 1:52 am
Location: Singapore / Cornwall
Contact:

Re: Build Secrets - Frozen Manifest

Post by devnull » Tue Dec 24, 2019 3:02 am

If you have a custom manifest file, In order to include the files in the port/modules folder (which are the default frozen files) you need to include the modules folder in your manifest file:

Code: Select all

freeze('$(PORT_DIR)/modules')
OR

Code: Select all

include('$(PORT_DIR)/boards/manifest.py')
freeze("modules")
If you don't then things might be broken, such as on the esp32 you will be unable to access the file system

Post Reply