How do i get the full 16MB on the file system? ESP32

All ESP32 boards running MicroPython.
Target audience: MicroPython users with an ESP32 board.
User avatar
jimmo
Posts: 1465
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: How do i get the full 16MB on the file system? ESP32

Post by jimmo » Sun Oct 27, 2019 7:59 am

I think the only dependency of "deploy" is esptool, which you can install very easily (i.e. with pip or os package, but no need for idf or sdk).

So you can run make inside the container to build, then regular make deploy outside to deploy. The great thing about the way docker works is that the build artifacts (especially the final .bin file) end up in the regular place.

You can also share ttyUSB0 with the container with an extra flag.

misaalanshori
Posts: 24
Joined: Sat Jun 22, 2019 6:07 am

Re: How do i get the full 16MB on the file system? ESP32

Post by misaalanshori » Mon Oct 28, 2019 8:53 am

for 16mb the size is changed to 0xE00000, but what about 4mb or 8mb? and how did you get these values?

User avatar
Roberthh
Posts: 1894
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: How do i get the full 16MB on the file system? ESP32

Post by Roberthh » Mon Oct 28, 2019 9:35 am

For 4 MB it is 0x200000 (which is the actual default)
For 8 MB is is 0x600000

Since the file system partition starts at 2MB (= 0x200000), the length of the file partition is Flash-Size - 2 MB

User avatar
MostlyHarmless
Posts: 147
Joined: Thu Nov 21, 2019 6:25 pm
Location: Pennsylvania, USA

Re: How do i get the full 16MB on the file system? ESP32

Post by MostlyHarmless » Thu Nov 21, 2019 8:51 pm

Roberthh wrote:
Sun Oct 27, 2019 6:07 am
@jimmo Yes, there was an environment variable set for ESPTOOL. After clearing that, it worked. And yes, deploy does not work. The container does not have a /dev/ttyUSB0 in view. So this approach is of limited use for me.
Hi Robert,

New member here (first post, actually). I was looking if someone had already done some docker stuff and like always, someone did. So I'm not going to post what I came up with because the Dockerfile discussed here is a better approach. Let me just try to show you how you might want to integrate it into your environment.

That Dockerfile specifies the esp-idf commit to use in the line

Code: Select all

ARG ESPIDF_SUPHASH=310beae373446ceb9a4ad9b36b5428d7fdf2705f
Change that to

Code: Select all

ARG ESPIDF_SUPHASH=6ccb4cf5b7d1fdddb8c2492f9cbc926abaf230df
for current micropython:HEAD

After that build the docker image with

Code: Select all

docker build -t esp32-idf .
The "-t esp32-idf" gives the image a title, so it can be specified in a subsequent "docker run" command by title, not by some hash. And if a previous one exists with that title, it will be overwritten. This "docker build" command has to be repeated every time you need to change the ESPIDF_SUPHASH. I can build something that will adjust that checkout on the fly, but for a first round let's just go with it as it is.

Next set up an alias to easily execute that docker image:

Code: Select all

alias esp32='docker run --rm -t -v $HOME:$HOME -w $PWD --device=/dev/ttyUSB0 esp32-idf'
This line ultimately belongs into your ~/.bashrc file. Right at the bottom it says "# User specific aliases and functions". Now we are ready to go. There now is an alias 'esp32' that switches into the docker environment to compile esp32 stuff.

Code: Select all

(venv36) [wieck@jupiter ~]$ cd docker/esp32/micropython
(venv36) [wieck@jupiter micropython]$ esp32 make -C mpy-cross
Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
make: Entering directory `/home/wieck/docker/esp32/micropython/mpy-cross'
mkdir -p build/genhdr
GEN build/genhdr/mpversion.h
GEN build/genhdr/moduledefs.h
GEN build/genhdr/qstr.i.last
GEN build/genhdr/qstr.split
GEN build/genhdr/qstrdefs.collected.h
QSTR updated
GEN build/genhdr/qstrdefs.generated.h
mkdir -p build/py/
CC ../py/mpstate.c
CC ../py/nlr.c
CC ../py/nlrx86.c
CC ../py/nlrx64.c
CC ../py/nlrthumb.c
CC ../py/nlrpowerpc.c
...
C ../py/showbc.c
CC ../py/repl.c
CC ../py/smallint.c
CC ../py/frozenmod.c
CC main.c
CC gccollect.c
LINK mpy-cross
   text	   data	    bss	    dec	    hex	filename
 269799	    800	    872	 271471	  4246f	mpy-cross
make: Leaving directory `/home/wieck/docker/esp32/micropython/mpy-cross'
More importantly:

Code: Select all

(venv36) [wieck@jupiter micropython]$ esp32 make -C ports/esp32 -j4
make: Entering directory '/home/wieck/docker/esp32/micropython/ports/esp32'
Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
Building with ESP IDF v3
mkdir -p build-GENERIC/bootloader//esp/components/bootloader/subproject/main/
mkdir -p build-GENERIC/bootloader//esp/components/bootloader_support/src/
mkdir -p build-GENERIC/bootloader//esp/components/log/
mkdir -p build-GENERIC/bootloader//esp/components/micro-ecc/micro-ecc/
mkdir -p build-GENERIC/bootloader//esp/components/soc/esp32/
mkdir -p build-GENERIC/bootloader//esp/components/soc/src/
mkdir -p build-GENERIC/bootloader//esp/components/spi_flash/
Create build-GENERIC/partitions.bin
mkdir -p build-GENERIC/genhdr
mkdir -p build-GENERIC/build-GENERIC/
...
CC ../../lib/utils/sys_stdio_mphal.c
CC ../../drivers/bus/softspi.c
CC ../../drivers/dht/dht.c
CC build-GENERIC/frozen_content.c
LINK build-GENERIC/application.elf
   text    data     bss     dec     hex filename
 927620  240772   29580 1197972  124794 build-GENERIC/application.elf
Create build-GENERIC/application.bin
esptool.py v2.8-dev
Create build-GENERIC/firmware.bin
bootloader     20720
partitions      3072
application  1168528
total        1234064
make: Leaving directory '/home/wieck/docker/esp32/micropython/ports/esp32'
Looking good so far ... so let's flash that (the alias enabled the docker container to access /dev/ttyUSB0):

Code: Select all

(venv36) [wieck@jupiter micropython]$ esp32 make -C ports/esp32 erase
make: Entering directory '/home/wieck/docker/esp32/micropython/ports/esp32'
Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
Building with ESP IDF v3
Erasing flash
esptool.py v2.8-dev
Serial port /dev/ttyUSB0
Connecting........__
Chip is ESP32D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 3c:71:bf:62:eb:98
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Erasing flash (this may take a while)...
Chip erase completed successfully in 8.7s
Hard resetting via RTS pin...
make: Leaving directory '/home/wieck/docker/esp32/micropython/ports/esp32'

Code: Select all

(venv36) [wieck@jupiter micropython]$ esp32 make -C ports/esp32 deploy
make: Entering directory '/home/wieck/docker/esp32/micropython/ports/esp32'
Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
Building with ESP IDF v3
Writing build-GENERIC/firmware.bin to the board
esptool.py v2.8-dev
Serial port /dev/ttyUSB0
Connecting........__
Chip is ESP32D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 3c:71:bf:62:eb:98
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Auto-detected Flash size: 4MB
Compressed 1229968 bytes to 778705...
Writing at 0x000bd000... (100 %)
Wrote 1229968 bytes (778705 compressed) at 0x00001000 in 18.1 seconds (effective 544.9 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...
make: Leaving directory '/home/wieck/docker/esp32/micropython/ports/esp32'
There is one tiny gotcha with all of this. The "esp32" alias will not work at all if there is nothing connected to /dev/ttyUSB0. So one must have a module attached to USB to even dry compile things. I certainly can live with that. While hacking micropython I always have a module attached.

Best Regards, Jan

User avatar
Roberthh
Posts: 1894
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: How do i get the full 16MB on the file system? ESP32

Post by Roberthh » Fri Nov 22, 2019 7:14 am

Thanks, Jan. Docker is surely on my interest list. But for the moment the plain installation on my Linux system works well for me. And if I need another OS, I use virtual machines. The trial I had before worked well, after all, but not that convincing.

User avatar
MostlyHarmless
Posts: 147
Joined: Thu Nov 21, 2019 6:25 pm
Location: Pennsylvania, USA

Re: How do i get the full 16MB on the file system? ESP32

Post by MostlyHarmless » Fri Nov 22, 2019 10:50 pm

Roberthh wrote:
Fri Nov 22, 2019 7:14 am
Docker is surely on my interest list.
Highly recommended.

I am geared towards RedHat based systems, so CentOS is my home default. Usually nothing "fancy" works on that out of the box and for some reasons all the stuff related to my hobbies (FreeCAD, Prusa-Slic3r, KiCAD, ...) won't install easy on CentOS. They like Debian and Ubuntu. And don't even get me started on when those tools have conflicting library version requirements ...

Wrapping each of those tools into Docker containers has made my life a lot easier.

However, this is getting off topic here and I think there would be interest in starting a separate "Using Docker to compile Micropython" thread.


Regards, Jan

pidou46
Posts: 68
Joined: Sat May 28, 2016 7:01 pm

Re: How do i get the full 16MB on the file system? ESP32

Post by pidou46 » Mon Nov 25, 2019 5:02 pm

Thanks MostlyHarmless for your explanation to ease the build of micropython firmware.

Yes, I think it definetly worth a specific topic.
nodemcu V2 (amica)
micropython firmware Daily build 05/31/2016

Post Reply