How do i get the full 16MB on the file system? ESP32

[jimmo]

I think the only dependency of "deploy" is esptool, which you can install very easily (i.e. with pip or os package, but no need for idf or sdk).

So you can run make inside the container to build, then regular make deploy outside to deploy. The great thing about the way docker works is that the build artifacts (especially the final .bin file) end up in the regular place.

You can also share ttyUSB0 with the container with an extra flag.

[misaalanshori]

for 16mb the size is changed to 0xE00000, but what about 4mb or 8mb? and how did you get these values?

[Roberthh]

For 4 MB it is 0x200000 (which is the actual default)
For 8 MB is is 0x600000

Since the file system partition starts at 2MB (= 0x200000), the length of the file partition is Flash-Size - 2 MB

[MostlyHarmless]

@jimmo Yes, there was an environment variable set for ESPTOOL. After clearing that, it worked. And yes, deploy does not work. The container does not have a /dev/ttyUSB0 in view. So this approach is of limited use for me.

Hi Robert,

New member here (first post, actually). I was looking if someone had already done some docker stuff and like always, someone did. So I'm not going to post what I came up with because the Dockerfile discussed here is a better approach. Let me just try to show you how you might want to integrate it into your environment.

That Dockerfile specifies the esp-idf commit to use in the line

Code: Select all

ARG ESPIDF_SUPHASH=310beae373446ceb9a4ad9b36b5428d7fdf2705f

Change that to

Code: Select all

ARG ESPIDF_SUPHASH=6ccb4cf5b7d1fdddb8c2492f9cbc926abaf230df

for current micropython:HEAD

After that build the docker image with

Code: Select all

docker build -t esp32-idf .

The "-t esp32-idf" gives the image a title, so it can be specified in a subsequent "docker run" command by title, not by some hash. And if a previous one exists with that title, it will be overwritten. This "docker build" command has to be repeated every time you need to change the ESPIDF_SUPHASH. I can build something that will adjust that checkout on the fly, but for a first round let's just go with it as it is.

Next set up an alias to easily execute that docker image:

Code: Select all

alias esp32='docker run --rm -t -v $HOME:$HOME -w $PWD --device=/dev/ttyUSB0 esp32-idf'

This line ultimately belongs into your ~/.bashrc file. Right at the bottom it says "# User specific aliases and functions". Now we are ready to go. There now is an alias 'esp32' that switches into the docker environment to compile esp32 stuff.

Code: Select all

(venv36) [wieck@jupiter ~]$ cd docker/esp32/micropython
(venv36) [wieck@jupiter micropython]$ esp32 make -C mpy-cross
Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
make: Entering directory `/home/wieck/docker/esp32/micropython/mpy-cross'
mkdir -p build/genhdr
GEN build/genhdr/mpversion.h
GEN build/genhdr/moduledefs.h
GEN build/genhdr/qstr.i.last
GEN build/genhdr/qstr.split
GEN build/genhdr/qstrdefs.collected.h
QSTR updated
GEN build/genhdr/qstrdefs.generated.h
mkdir -p build/py/
CC ../py/mpstate.c
CC ../py/nlr.c
CC ../py/nlrx86.c
CC ../py/nlrx64.c
CC ../py/nlrthumb.c
CC ../py/nlrpowerpc.c
...
C ../py/showbc.c
CC ../py/repl.c
CC ../py/smallint.c
CC ../py/frozenmod.c
CC main.c
CC gccollect.c
LINK mpy-cross
   text	   data	    bss	    dec	    hex	filename
 269799	    800	    872	 271471	  4246f	mpy-cross
make: Leaving directory `/home/wieck/docker/esp32/micropython/mpy-cross'

More importantly:

Code: Select all

(venv36) [wieck@jupiter micropython]$ esp32 make -C ports/esp32 -j4
make: Entering directory '/home/wieck/docker/esp32/micropython/ports/esp32'
Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
Building with ESP IDF v3
mkdir -p build-GENERIC/bootloader//esp/components/bootloader/subproject/main/
mkdir -p build-GENERIC/bootloader//esp/components/bootloader_support/src/
mkdir -p build-GENERIC/bootloader//esp/components/log/
mkdir -p build-GENERIC/bootloader//esp/components/micro-ecc/micro-ecc/
mkdir -p build-GENERIC/bootloader//esp/components/soc/esp32/
mkdir -p build-GENERIC/bootloader//esp/components/soc/src/
mkdir -p build-GENERIC/bootloader//esp/components/spi_flash/
Create build-GENERIC/partitions.bin
mkdir -p build-GENERIC/genhdr
mkdir -p build-GENERIC/build-GENERIC/
...
CC ../../lib/utils/sys_stdio_mphal.c
CC ../../drivers/bus/softspi.c
CC ../../drivers/dht/dht.c
CC build-GENERIC/frozen_content.c
LINK build-GENERIC/application.elf
   text    data     bss     dec     hex filename
 927620  240772   29580 1197972  124794 build-GENERIC/application.elf
Create build-GENERIC/application.bin
esptool.py v2.8-dev
Create build-GENERIC/firmware.bin
bootloader     20720
partitions      3072
application  1168528
total        1234064
make: Leaving directory '/home/wieck/docker/esp32/micropython/ports/esp32'

Looking good so far ... so let's flash that (the alias enabled the docker container to access /dev/ttyUSB0):

Code: Select all

(venv36) [wieck@jupiter micropython]$ esp32 make -C ports/esp32 erase
make: Entering directory '/home/wieck/docker/esp32/micropython/ports/esp32'
Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
Building with ESP IDF v3
Erasing flash
esptool.py v2.8-dev
Serial port /dev/ttyUSB0
Connecting........__
Chip is ESP32D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 3c:71:bf:62:eb:98
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Erasing flash (this may take a while)...
Chip erase completed successfully in 8.7s
Hard resetting via RTS pin...
make: Leaving directory '/home/wieck/docker/esp32/micropython/ports/esp32'

Code: Select all

(venv36) [wieck@jupiter micropython]$ esp32 make -C ports/esp32 deploy
make: Entering directory '/home/wieck/docker/esp32/micropython/ports/esp32'
Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
Building with ESP IDF v3
Writing build-GENERIC/firmware.bin to the board
esptool.py v2.8-dev
Serial port /dev/ttyUSB0
Connecting........__
Chip is ESP32D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 3c:71:bf:62:eb:98
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Auto-detected Flash size: 4MB
Compressed 1229968 bytes to 778705...
Writing at 0x000bd000... (100 %)
Wrote 1229968 bytes (778705 compressed) at 0x00001000 in 18.1 seconds (effective 544.9 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...
make: Leaving directory '/home/wieck/docker/esp32/micropython/ports/esp32'

There is one tiny gotcha with all of this. The "esp32" alias will not work at all if there is nothing connected to /dev/ttyUSB0. So one must have a module attached to USB to even dry compile things. I certainly can live with that. While hacking micropython I always have a module attached.

Best Regards, Jan

[Roberthh]

Thanks, Jan. Docker is surely on my interest list. But for the moment the plain installation on my Linux system works well for me. And if I need another OS, I use virtual machines. The trial I had before worked well, after all, but not that convincing.

[MostlyHarmless]

Docker is surely on my interest list.

Highly recommended.

I am geared towards RedHat based systems, so CentOS is my home default. Usually nothing "fancy" works on that out of the box and for some reasons all the stuff related to my hobbies (FreeCAD, Prusa-Slic3r, KiCAD, ...) won't install easy on CentOS. They like Debian and Ubuntu. And don't even get me started on when those tools have conflicting library version requirements ...

Wrapping each of those tools into Docker containers has made my life a lot easier.

However, this is getting off topic here and I think there would be interest in starting a separate "Using Docker to compile Micropython" thread.


Regards, Jan

[pidou46]

Thanks MostlyHarmless for your explanation to ease the build of micropython firmware.

Yes, I think it definetly worth a specific topic.

[pythoncoder]

I'm trying to compile a native module for ESP32. I can compile and run it on Unix and STM by changing to the directory and issuing make. However I get no joy with docker. I've tried following the instructions of @Jimmo and also @MostlyHarmless and am getting similar results. I am completely new to docker and am doubtless doing something very stupid.

Here is a successful run with armv7m specified

Code: Select all

[adminpete@capybara]: /mnt/qnap2/data/Projects/MicroPython/micropython/examples/natmod/pete
$ make
make: Warning: File 'Makefile' has modification time 2 s in the future
CC lcopy.c
LINK build/lcopy.o
arch:         EM_ARM
text size:    224
bss size:     0
GOT entries:  2
GEN lcopy.mpy
make: warning: Clock skew detected. Your build may be incomplete.

(The warnings are because the code is on a server).

Here is what happens if I change the arch to xtensawin and run docker:

Code: Select all

[adminpete@capybara]: /mnt/qnap2/data/Projects/MicroPython/micropython/examples/natmod/pete
$ docker run --rm -v $HOME:$HOME -u $UID -w $PWD new-esp32 make
make: *** No targets specified and no makefile found.  Stop.

I have tried everything I can think of to point it at the Makefile but nothing works.

Docker appears to be properly installed:

Code: Select all

Client:
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.13.8
 Git commit:        afacb8b7f0
 Built:             Wed Oct 14 19:43:43 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       afacb8b7f0
  Built:            Wed Oct 14 16:41:21 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu2
  GitCommit:        
 runc:
  Version:          spec: 1.0.1-dev
  GitCommit:        
 docker-init:
  Version:          0.18.0
  GitCommit:        

[mattyt]

Hi Peter,

I'd try manually expanding out the environment variables and, in particular, check that $PWD points to the appropriate location inside the container.

My guess is that the location that the volume has been mounted inside the container is different to the working directory location specified.

The way the "-v f1:f2" works is that it will mount the host folder f1 inside the container at location f2. You've also specified the working director with "-w $PWD" - the working directory is a location inside the container that will be the current working directory for the container when it's started. In your case you're executing make inside the working directory - you need to ensure that's where you expect it to be (ie the working directory should contain a Makefile).

A good way to test a docker configuration is to "step inside" it manually. Add "-ti" to the arguments and run bash instead of make:

Code: Select all

$ docker run -ti --rm -v $HOME:$HOME -u $UID -w $PWD new-esp32 bash

Then you're running bash inside the container. The directory you start in is the working directory.

Hope that helps!

[pythoncoder]

...
A good way to test a docker configuration is to "step inside" it manually. Add "-ti" to the arguments and run bash instead of make:

Code: Select all

$ docker run -ti --rm -v $HOME:$HOME -u $UID -w $PWD new-esp32 bash

Then you're running bash inside the container. The directory you start in is the working directory...

Thank you! Magic. Now I can see what's going on with the directories.

Now to get it to build...