Reducing the Micropython Firmware

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
cyrano1960
Posts: 39
Joined: Fri Mar 29, 2019 7:08 pm

Reducing the Micropython Firmware

Post by cyrano1960 » Fri Jul 29, 2022 4:47 pm

Hello, I am wondering how to exclude classes from the Micropython Firmware to get a smaller footprint for the ESP32 port. For me, it is more of a way to dive deeper into Micropython. But it seems to be very hard to find out the build sequence. I am not so familiar with cmake and there are a number of different makefiles. I guess that the CMakeLists.txt in ports/esp32/main must be one of the first files that go into action and here some source files are explicitly listed. So I have commented out line 43, which includes the dht files. But after this, I will run into a linker error. So instead of going on with my try and error (mostly error) strategy, I want to ask you:

1. Is there a general description of how to remove individual modules from the firmware?
2. Is there a description, in which order the individual Makefiles/CMakeLists.txt are called?

I would really appreciate any tips and assistance. Thanks in advance!
Cyrano

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Reducing the Micropython Firmware

Post by jimmo » Wed Aug 17, 2022 12:05 pm

cyrano1960 wrote:
Fri Jul 29, 2022 4:47 pm
So I have commented out line 43, which includes the dht files. But after this, I will run into a linker error.
The reason for this particular linker error is that the dht_readinto function (i.e. `esp.dht_readinto`) references this driver.
So you'll also need to remove that from modesp.c
cyrano1960 wrote:
Fri Jul 29, 2022 4:47 pm
1. Is there a general description of how to remove individual modules from the firmware?
2. Is there a description, in which order the individual Makefiles/CMakeLists.txt are called?
In general (with the unfortunate exception of the DHT driver), the idea is that you don't need to edit the CMake / make config, and instead control which features are enabled via the configuration flags in mpconfigport.h

So for example, similar to the DHT example, to disable support for onewire, you can set MICROPY_PY_ONEWIRE to 0.

Even though the .c files will still be compiled, they won't contribute anything to the final firmware because the entire file is #if'ed by that flag. e.g. see https://github.com/micropython/micropyt ... wire.c#L33

One limitation right now is that any Python code frozen into the binary will not be affected by these flags, so you will need to modify manifest.py to remove any additional Python files you don't want (e.g. dht.py, onewire.py).

Jibun no kage
Posts: 144
Joined: Mon Jul 25, 2022 9:45 pm

Re: Reducing the Micropython Firmware

Post by Jibun no kage » Sun Aug 21, 2022 7:48 pm

Is this not documented in detail somewhere? What happens if something crashes or a disaster happens? And you have to create the build environment for the nightly builds, for example? In enterprise design, we had to have everything documented, so someone could walk in, read the documentation and recreate whatever was lost, with little or no experience with the previous environment. So as a best practice such documentation must exist somewhere right? The biggest issue I keep tripping over is how badly out of date every example I have found thus far is.

My personal goal, I want to create my own firmware image, to remove a lot of stuff I don't need. But all I can find via Google is very bad, dated, useless examples of how to compile the various ports. Once I know how to create firmware images that are specific to defined goals, I would be more than happy to document in detail how such customization is done, help others do same, etc. But I am really hitting a WALL here.

I designed, and wrote applicable documentation for Fortune 10 firms, secure banking environments, designing world wide enterprise deployment methods for 100,000s of servers, etc. So if someone needs to improve or establish complete documentation, I would be more than happy to contribute to such, but I have learn how to do it first, before I can even hope to improve the situation, if such in fact needs to be done.

Please tell me, this is not a case of, we don't want people to know how do it... can't believe that is true, right?

How about this to start:

For ESP8266 port, what version of the ESP SDK do I need right now?
Where do i find it?
How to I install it, so the git clone of micropython will see same when I 'make' the ESP8266 port?

If I could just get the current port for ESP8266 to compile as is, that would be a start. I know I have the basics in place, because the very simple Unix port compiles. But Unix port does not require any external SDK, like the Pico or ESP ports right?

I could point to 10 different given Google examples that all are very wrong, or very dated thus wrong, that fail to answer the above questions today. They may have years ago, but not anymore.

Jibun no kage
Posts: 144
Joined: Mon Jul 25, 2022 9:45 pm

Re: Reducing the Micropython Firmware

Post by Jibun no kage » Sun Aug 21, 2022 8:16 pm

Illustrative example... following the current directions at ...

1) Install requirements and dependencies
2) Debian specific requirements
3) Git ESP Open SDK
4) Compile SDK/Tool-Chain

# cd /root

# apt-get install make unrar-free autoconf automake libtool gcc g++ gperf flex bison texinfo gawk ncurses-dev libexpat-dev python3-dev python3 python3-serial sed git unzip bash help2man wget bzip2
# apt-get install libtool-bin


# git clone --recursive https://github.com/pfalcon/esp-open-sdk.git
# cd esp-open-sdk

// Typical... Tried this one first...
# make

Hit this issue...
checking for absolute path to patch... /usr/bin/patch
checking for bash >= 3.1... no
configure: error: could not find bash >= 3.1
make[1]: *** [../Makefile:149: _ct-ng] Error 1
make[1]: Leaving directory '/root/esp-open-sdk/crosstool-NG'
make: *** [Makefile:145: crosstool-NG/ct-ng] Error 2

The current installed version of BASH is 5.1...

# bash --version
bash --version
GNU bash, version 5.1.4(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

How do I overcome this issue?


The documentation makes no comment on this issue. Seems like it would be very common issue. Maybe the documentation worked on Debian 10 or older? Not Debian 11?

Jibun no kage
Posts: 144
Joined: Mon Jul 25, 2022 9:45 pm

Re: Reducing the Micropython Firmware

Post by Jibun no kage » Sun Aug 21, 2022 8:34 pm

Apparently, there is a known issue that was qualified in 2019? But nothing was done to fix it apparently? Really?

I tried the following https://github.com/pfalcon/esp-open-sdk/issues/380#top

Then hit the following error...
make: Leaving directory '/root/esp-open-sdk/crosstool-NG'
make[1]: Leaving directory '/root/esp-open-sdk/crosstool-NG'
cp -f 1000-mforce-l32.patch crosstool-NG/local-patches/gcc/4.8.5/
cp: cannot create regular file 'crosstool-NG/local-patches/gcc/4.8.5/': No such file or directory
make: *** [Makefile:130: crosstool-NG/.built] Error 1

So I tried the following https://github.com/pfalcon/esp-open-sdk/issues/383

And wonder of wonders, issue 383 suggests issue 380 as fix. WHAT THE HECK? All this was known in 2019, but the documentation was never updated?

Jibun no kage
Posts: 144
Joined: Mon Jul 25, 2022 9:45 pm

Re: Reducing the Micropython Firmware

Post by Jibun no kage » Sun Aug 21, 2022 9:39 pm

So lets try again... This source on git might be better https://github.com/ChrisMacGregor/esp-o ... untu-21.10...

However, its first step, is wrong, the documentation is wrong, it directs the user to the ORIGINAL git source, which we know is already badly broken and can't work as tested above, previously. Moreover, neither example makes the point that there are know issues with cross-tool-NG under root. This is a major issue, omission that will drive users nuts. So the real initial steps should be...

# apt install sudo
# adduser crosstoolng
# usermod -aG sudo crosstoolng
# login crosstoolng

$ pwd
/home/crosstoolng

$ sudo apt-get install make unrar-free autoconf automake libtool gcc g++ gperf flex bison texinfo gawk ncurses-dev libexpat-dev python3-dev python3 python3-serial sed git unzip bash help2man wget bzip2
$ sudo apt-get install libtool-bin

NOTE: Corrected the references to python as now python3, so that no assumptions are made or accepted. This is another issue with the dated nature of the original git repo documentation and same bad assumption inherited by the forked instance. Again, appears that NO ONE is updating anything.

Using the correct git close source, for this variant git clone --recursive https://github.co ... en-sdk.git. I was able to at least get the make command to work for most of the tool chain generation.

$ git clone --recursive https://github.com/ChrisMacGregor/esp-open-sdk.git
$ cd esp-open-sdk
$ make

Since I am generating the tool-chain in a virtual machine it does take a while, but after about 20 minutes of compiling we have yet another error! Why am I not surprised. But, the good news, I guess is that the error below is because the make file is assuming there is a 'python' alias for the installed version of python, but that may not exist. Again, with the retirement of python2, and python3 now applicable, something was lost.

[INFO ] Installing cross-gdb
[ERROR] configure: error: python is missing or unusable
[ERROR] gmake[4]: *** [Makefile:8725: configure-gdb] Error 1
[ERROR] gmake[3]: *** [Makefile:845: all] Error 2
[ERROR]
[ERROR] >>
[ERROR] >> Build failed in step 'Installing cross-gdb'
[ERROR] >> called in step '(top-level)'
[ERROR] >>
[ERROR] >> Error happened in: CT_DoExecLog[scripts/functions@257]
[ERROR] >> called from: do_debug_gdb_build[scripts/build/debug/300-gdb.sh@120]
[ERROR] >> called from: do_debug[scripts/build/debug.sh@35]
[ERROR] >> called from: main[scripts/crosstool-NG.sh@646]
[ERROR] >>
[ERROR] >> For more info on this error, look at the file: 'build.log'
[ERROR] >> There is a list of known issues, some with workarounds, in:
[ERROR] >> 'share/doc/crosstool-ng/crosstool-ng-1.22.0-60-g37b07f6f-dirty/B - Known issues.txt'
[ERROR]
[ERROR] (elapsed: 17:42.18)
[17:42] / gmake[2]: *** [ct-ng:152: build] Error 2
gmake[2]: Leaving directory '/home/crosstoolng/esp-open-sdk/crosstool-NG'
make[1]: *** [../Makefile:142: _toolchain] Error 2
make[1]: Leaving directory '/home/crosstoolng/esp-open-sdk/crosstool-NG'
make: *** [Makefile:134: crosstool-NG/.built] Error 2

We need to find where python alias was broken or lost...

crosstoolng@psycho:~/esp-open-sdk$ python
-bash: python: command not found
crosstoolng@psycho:~/esp-open-sdk$ type python
-bash: type: python: not found
crosstoolng@psycho:~/esp-open-sdk$ type python3
python3 is /usr/bin/python3
crosstoolng@psycho:~/esp-open-sdk$ command -v python
crosstoolng@psycho:~/esp-open-sdk$ type -a python
-bash: type: python: not found

$ sudo apt-get install python3
[sudo] password for crosstoolng:
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
python3 is already the newest version (3.9.2-3).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

$ which python
$ python3 --version
Python 3.9.2

$ ls -l /usr/bin/python*
lrwxrwxrwx 1 root root 9 Apr 5 2021 /usr/bin/python3 -> python3.9
-rwxr-xr-x 1 root root 5479736 Feb 28 2021 /usr/bin/python3.9
lrwxrwxrwx 1 root root 33 Feb 28 2021 /usr/bin/python3.9-config -> x86_64-linux-gnu-python3.9-config
lrwxrwxrwx 1 root root 16 Apr 5 2021 /usr/bin/python3-config -> python3.9-config

$ sudo ln -s /usr/bin/python3.3 /usr/bin/python

$ ls -l /usr/bin/python*
lrwxrwxrwx 1 root root 18 Aug 21 14:34 /usr/bin/python -> /usr/bin/python3.3
lrwxrwxrwx 1 root root 9 Apr 5 2021 /usr/bin/python3 -> python3.9
-rwxr-xr-x 1 root root 5479736 Feb 28 2021 /usr/bin/python3.9
lrwxrwxrwx 1 root root 33 Feb 28 2021 /usr/bin/python3.9-config -> x86_64-linux-gnu-python3.9-config
lrwxrwxrwx 1 root root 16 Apr 5 2021 /usr/bin/python3-config -> python3.9-config

Now we have the python alias, soft link back as the make file expects.

Will try to run the tool-chain make again...

$ make

Jibun no kage
Posts: 144
Joined: Mon Jul 25, 2022 9:45 pm

Re: Reducing the Micropython Firmware

Post by Jibun no kage » Sun Aug 21, 2022 9:55 pm

Well, fixing the python alias did not resolve the make issue.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Reducing the Micropython Firmware

Post by jimmo » Mon Aug 22, 2022 12:48 am

Jibun no kage wrote:
Sun Aug 21, 2022 7:48 pm
Is this not documented in detail somewhere?
Your reply is about two things:
- Documentation about configuring the build of micropython
- It's hard to compile (especially esp8266).

For the second port, we strongly recommend using docker for this. That's why it's the first thing in the README and marked as recommended. It works well.

Unfortunately the ESP8266 was never designed to be used for this purpose and setting up the SDK (such as it is) is a bit of a process.

Our continuous integration builds from source (without docker) and it uses a pre-built SDK (provided by jepler from Adafruit). We should update the readme to describe this instead. https://github.com/micropython/micropyt ... ci.sh#L154

Unfortunately (of course) we cannot fix what appears in Google other than what's in our github repo and docs.micropython.org. If you can point me to examples from there (other than switching the README over to use jepler's toolchain rather than pfalcon's) we can definitely fix them. I've also written to authors of third-party documentation with corrections on several occasions over the years, with varying level of success.
Jibun no kage wrote:
Sun Aug 21, 2022 7:48 pm
Please tell me, this is not a case of, we don't want people to know how do it... can't believe that is true, right?
MicroPython is a very small team, and sadly on a _very_ long list of priorities, ESP8266 is not currently near the top. We definitely welcome contributions to improve the documentation though!

As far as configuring the build, our top priority is to make the default build as sensible as possible, and then where possible we provide alternative board definitions with other "general purpose" configurations. Unfortunately doing anything more custom will require a bit more manual effort understanding the build process. Again we'd love to document this more!

Jibun no kage
Posts: 144
Joined: Mon Jul 25, 2022 9:45 pm

Re: Reducing the Micropython Firmware

Post by Jibun no kage » Mon Aug 22, 2022 1:17 am

True, I am going off the reservation with the ESP8266 image, that is the key reason why I am doing all this myself. The surprise, was the dated information and at times conflicting information, that has been tripping me up, again and again.

I am not against using docker, given I engineered various virtualization environments since like 2001 or 2002 when even VMware was pretty much unknown but to a few of us. :) And, there are 40 odd ESP SDK oriented docker images? And the latest and greatest maybe not even be what I need, given ESP32 is new generation, since I am working on ESP8266 at the moment.

Moreover, docker hides details, lets you skip key points of understanding. This is fine when you just need something to work, but it does not foster core knowledge development, if you see my perspective. It is a lot like Game consoles vs. Game PCs. When you just want to play a game, a console makes sense, but when you want to get 60 FPS at UltraHD, completely transparent reflex action, you need a water cooled, monster processor and GPU setup, for the same game title on the console, no less. And the endless tinkering on a Game PC develops knowledge and understanding that a Game console user never learns (not that they likely care).

I am tracking the issues I am finding, even as I resolve pitfalls. I plan to write up everything I find and document things so someone with absolutely the minimum experience has a current source of information. I plan to maintain it as well, so it will likely be a reference source on GitHub.

For years before I retired from 30+ year IT career, I was known for always publishing two document types... The first was always WHAT you do, and WHY you do it. The second was HOW you do it. At first the culture at the time, pushed back, and give me stress for the extra work, all the operational teams wanted was HOW, HOW, HOW. Then, was support costs increased and knowledge decreased, people started asking me if I still wrote the WHAT and WHY documents, which I always did. It did not take long for this concept to spread, I know for a fact that my approach has popped up all over the world, I have seen documents that clearly were inspired by my methodology years ago, at the time, at VMware, RedHat, EMC, HP (Compaq), Dell, and a few others, in the Financial bubble, I can't mention, but if I could, you would recognize. I say the above, to let you know I am serious about this effort.

The help I have received and continue to get in this forum is wonderful, even when at times there have been a few frustrating issues. Now if I can use get my custom ESP01 firmware finished!

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Reducing the Micropython Firmware

Post by jimmo » Mon Aug 22, 2022 1:23 am

Jibun no kage wrote:
Mon Aug 22, 2022 1:17 am
I am not against using docker, given I engineered various virtualization environments since like 2001 or 2002 when even VMware was pretty much unknown but to a few of us. :) And, there are 40 odd ESP SDK oriented docker images? And the latest and greatest maybe not even be what I need, given ESP32 is new generation, since I am working on ESP8266 at the moment.
There's one docker image that we recommend for compiling esp8266.

I have sent a PR to update the docs. https://github.com/micropython/micropython/pull/9080

Post Reply