Organizing Custom Builds, Recommended Practices?

[thestumbler]

After several years using MicroPython on various platforms, I'm not sure what the best practices are to organize various custom builds. The customization I've encountered boils down to three categories:

1. COTS board with MP support, just tweak some #defines
2. Same as (1) plus pre-compiled modules
3. COTS board without MP support, but similar to an existing board / processor port
4. My own board, but designed to be similar to an existing board / processor port
5. COTS board, vendor provided MP library patches and modified MP standard PICO definition

Do most folks just clone the entire MP repo once per project?
Or like I was trying to do, keep one version of the MP repo?

What's the tipping point where a pull request should be made to add a new board vs keeping it local? When I built my own MP board for an experiment, it was obvious nobody would need it. But when I see a vendor with a somewhat well-known board distributing MP support as diff patches instead of putting it to MP proper via a pull request, I wonder what the threshold is for this.

[hippy]

Do most folks just clone the entire MP repo once per project?
Or like I was trying to do, keep one version of the MP repo?

I have also wondered what others do.

I don't have as much experience as you do, have only ventured into having my own cloned and tweaked build of the official 'rp2' port for the Raspberry Pi Pico and RP2040; a few #define changes, some tweaked files and some added ones, some changes to the CMake build itself -- I created a new port within './ports', alongside 'rp2' and the others.

But even with something as simple and straightforward as that, keeping that working as MicroPython has evolved has been something of a nightmare with builds breaking and changes being incompatible with what I have. It has often taken me hours to get the build working again, mostly in figuring out exactly why it no longer builds, what the problem is, rather than fixing the issue.

I have been tempted to create a separate MicroPython branch just for my port but I anticipate seeing many of the same breakages should I ever update MicroPython within that. There is no real advantage I can see if one does want to keep up with the latest MicroPython.

If one is happy with a MicroPython frozen in time, I would say a separate tree may be better because that will be a fixed and unchanging entity. Otherwise I don't think there's much difference. And putting something under './ports' of a single MicroPython is the easiest and logical option.

Plus, while keeping up to date regularly may mean breakages, I believe it's better to just have 'shifting sand' than suddenly finding there's been an earthquake later which has shaken everything up in one hit, which will likely present a bigger challenge to solve.

For my project, which I hope I can one day share with others, putting it under './ports' so others can use it with what they already have, working with the latest MicroPython, makes that the best option for me.

[karfas]

Do most folks just clone the entire MP repo once per project?

Either you want to build micropython again and again (with the observed unstable effects on build and function) or you want to build something with micropython.
"Projects" very seldom require the latest, best (and maybe the most buggy) micropython version.

I have one local micropython clone as well as my micropython clone on github (as far as I know, this is an requirement to create pull requests).

PRs and upy development happen in the local clone (in separate git branches, of course).

Further, I have a few (2-3) uPy versions checked out as "git worktree".
There are the sources of known-to-work uPy versions + *my* patches (also known to work)

My projects I prefer to code in python.
A "project" gets it's own directory structure with
- board definition
- build directory
- build + flash + install scripts
- my .py files, external dependencies
- a reference (via an environment) to one of the (for me!) stable, checked-out worktrees.
For project development, I avoid touching the micropython sources.
In this way, a build remains local to the project, is most likely repeatable and has no side effects to other projects.

But when I see a vendor ... distributing MP support as diff patches

Maintaining a PR does require knowledge+time. In micropython, PRs get merged months, sometimes years after creation and get old pretty fast. I have three PRs open. Two of them would require a rebase+manual merge each month or so. No wonder they are most of the time in a "unmergeable" state.

[thestumbler]

Gee, I am super late to the thread, after starting it well over a year ago. Somehow I missed these replies.

I thing the answer (for me) is twofold. For quick prototype projects, I would prefer the "shifting sands" approach and keep my local MP repository up-to-date. With just one computer, I think there is no issue -- I keep my own ports as normal in the MP directory structure, and when I pull new MP updates, no worries (unless an update adds a file / folder with the same name as one of my custom projects). But alas, I work from multiple machines in different cities and different OSes. Probably making a custom "fork" for myself would be best, said fork would conceptually track constant MP updates and include my own changes. This way I can pull the whole shebang onto any computer of the moment.

But for a long-term project / product, I agree it is best to freeze your MP version. That could be simply a one-time clone of the MP repo in your project's directory with the .git folder removed so it won't be recognized as a submodule. Or alternatively make it a real submodule with a fixed commit hash.

I've been lucky so far that my projects haven't been impacted by a MP update. Quite the opposite ... there have been a couple of cases where getting the update helped me because of new features (I don't remember the exact details, but it was related to me trying to re-use a few classes from desktop C Python in MP as well).

[thestumbler]

Gee, just days later after posting the above, I encountered a case such as you described for the first time. A couple of years ago, I made a custom board definition for an STM32H750 board available locally here in South Korea. I got it working great in MP, and used it briefly for testing in another project. I wanted to use that board again this week ( for a C++ project, not MP ), but first I decided to dust off the MP build and make sure all the hardware and ports were still working. The original MP stored in the board still worked, but I erased the flash for some other tests. Went to rebuild the custom MP, and no joy. I spent about 3 hours trying to solve it, an no joy. The problem seems to be that the method(s) for specifying QSPI flash changed in the meantime ( my custom build was Aug 2021 ).

I was able to finally build it again, but only after checking out the version of MP that was current in Aug 2021 (v1.16), updating (downdating?) the submodules. Note that I had to switch to Linux to even make this work, apparently something in MacOS changed that keeps one from successfully building mpy-cross anymore.

To be fair, during those 3 hours of debugging, I realized that I didn't exactly know what I was doing with regards to configuring the QSPI flash in the config files. I must have copy/pasted from another example back then. I see that updates since v1.16 have improved the external flash configuration specifications, but I couldn't find a clear explanation. Since this board is just a quick test, rolling back to 1.16 is good enough for now.

-Chris