Creating larger ROMs

All ESP8266 boards running MicroPython.
Official boards are the Adafruit Huzzah and Feather boards.
Target audience: MicroPython users with an ESP8266 board.
manseekingknowledge
Posts: 61
Joined: Sun Oct 29, 2017 5:14 pm

Re: Creating larger ROMs

Post by manseekingknowledge » Tue Jul 09, 2019 7:25 am

I've been trying to flash a pre-made FAT file system to my ESP8266 all night unsuccessfully. I create the file system as follows:

1.) Determined the max size of the FAT file system by using info from flashbdev.py and some esp module calls:

Code: Select all

esp.flash_user_start()  = 1048576  # Exactly 0x100000.  This is what esp.flash_user_start() returns when irom0_0_seg = 0xF7000
esp.flash_size()        = 4194304
SEC_SIZE                = 4096
RESERVED_SECS           = 1
START_SEC               = 257
blocks                  = 762

SEC_SIZE * blocks       = 3121152  # Exactly 0x2FA000. This is the max FAT file system size
2.) Used DD to create an image of the correct size:

Code: Select all

dd if=/dev/zero of=fat.fs count=762 bs=4096
3.) Used mkfs.fat to format the image. I tried each of the four command below independently and none of them changed the end result:

Code: Select all

mkfs.fat fat.fs
mkfs.vfat fat.fs
mkfs.fat -F 12 -R 1 -S 4096 fat.fs
mkfs.vfat -F 12 -R 1 -S 4096 fat.fs
4.) Mounted the image, copied boot.py to it and an additional file, and unmounted the image:

Code: Select all

sudo mount fat.fs micropython_fat_fs
sudo cp boot.py micropython_fat_fs
sudo cp favicon_package/favicon.ico micropython_fat_fs
sudo umount micropython_fat_fs
5.) Made a build and flashed it WITHOUT trying to add the FAT file system created above. I used repl to test the build to make sure it worked as expected and it did. Here is the build output:

Code: Select all

   text	   data	    bss	    dec	    hex	filename
 990572	   1084	  68544	1060200	 102d68	build/firmware.elf
Create build/firmware-combined.bin
esptool.py v2.6
Creating image for ESP8266...
('flash    ', 32944)
('padding  ', 3920)
('irom0text', 958752)
('total    ', 995616)
('md5      ', 'c1c4a736e051a9bda72e7538fdc29968')
6.) Re-flashed with the command below in an attempt to try and flash my firmware and FAT file system at the same time. The flash was successful:

Code: Select all

sudo esptool.py --port /dev/ttyUSB0 --baud 921600 write_flash --erase-all 0x0 "build/firmware-combined.bin" 0x100000 ~/Desktop/fat.fs
7.) Attempted to use repl to test the build, but repl didn't work.

Not sure what is causing the failure. I think all the sizes are correct. Maybe I have something wrong with the exact formatting of the file system? There are a lot of FAT options to choose from :(

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Creating larger ROMs

Post by jimmo » Tue Jul 09, 2019 1:13 pm

I gave this a quick test and had some success, hopefully you can build on it further.

FWIW - I'm running the official build of 1.11

Code: Select all

>>> esp.flash_user_start()
622592
>>> esp.flash_size()
4194304
First I read back the flash from the device

Code: Select all

$ esptool --port /dev/ttyUSB0 --baud 460800 read_flash 0 $((1024*1024*4)) esp.flash
Then verified that the filesystem was at the right place:

Code: Select all

$ hexdump -C esp.flash
...
*
00099000  eb fe 90 4d 53 44 4f 53  35 2e 30 00 10 01 01 00  |...MSDOS5.0.....|
00099010  01 00 02 62 03 f8 01 00  3f 00 ff 00 00 00 00 00  |...b....?.......|
...
Then extracted that to an image (I grabbed the bootsector first and used `file` to check how many sectors and bytes/sectors it was)

Code: Select all

dd if=esp.flash of=esp.fs bs=4096 count=866 skip=$((626688/4096))
Then I mounted esp.fs, added some files, then flashed it back:

Code: Select all

esptool --port /dev/ttyUSB0 --baud 460800 write_flash --flash_size=detect 626688 esp.fs
I also tried this using a fat12 image generated with mkfs.fat, but no luck (the device appears to think the bootsector is zeros). I'm unsure why, but haven't investigated further.

manseekingknowledge
Posts: 61
Joined: Sun Oct 29, 2017 5:14 pm

Re: Creating larger ROMs

Post by manseekingknowledge » Tue Jul 09, 2019 3:31 pm

Just to confirm, you are saying the method you described worked? If so, I see one of my errors by looking at your hexdump. The hexdump shows the file system starting at 0x99000 (626688), not your esp.flash_user_start() value of 622592 which is where I was expecting it to start in my previous post. It is obvious now that I was ignoring a key piece of FlashBdev which set the start sector:

Code: Select all

START_SEC = esp.flash_user_start() // SEC_SIZE + RESERVED_SECS
Thanks for helping me unlock this. I'll start round #2 this evening!
jimmo wrote:
Tue Jul 09, 2019 1:13 pm
I also tried this using a fat12 image generated with mkfs.fat, but no luck (the device appears to think the bootsector is zeros). I'm unsure why, but haven't investigated further.
If you could let me know what the exact mkfs command you used was that might be helpful. I would expect the generated file system to not have a boot sector. Do you agree?

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Creating larger ROMs

Post by jimmo » Tue Jul 09, 2019 11:48 pm

manseekingknowledge wrote:
Tue Jul 09, 2019 3:31 pm
Just to confirm, you are saying the method you described worked?
Yup, I was able to add files to the filesystem and then after flashing it back, i could see those files on the EAP8266.
manseekingknowledge wrote:
Tue Jul 09, 2019 3:31 pm
If you could let me know what the exact mkfs command you used was that might be helpful. I would expect the generated file system to not have a boot sector. Do you agree?
Just `mkfs.fat fat.fs` on a 512kiB image -- all the other parameters seemed like they matched the default. The result was a fat12 filesystem. All fat volumes have a bootsector (even if they're not intended to be bootable), it's where the configuration is stored (i.e. cluster size, offset to the FAT, etc).

manseekingknowledge
Posts: 61
Joined: Sun Oct 29, 2017 5:14 pm

Re: Creating larger ROMs

Post by manseekingknowledge » Wed Jul 10, 2019 4:09 am

jimmo wrote:
Tue Jul 09, 2019 11:48 pm
manseekingknowledge wrote:
Tue Jul 09, 2019 3:31 pm
If you could let me know what the exact mkfs command you used was that might be helpful. I would expect the generated file system to not have a boot sector. Do you agree?
Just `mkfs.fat fat.fs` on a 512kiB image -- all the other parameters seemed like they matched the default. The result was a fat12 filesystem. All fat volumes have a bootsector (even if they're not intended to be bootable), it's where the configuration is stored (i.e. cluster size, offset to the FAT, etc).
Now that I've corrected my start segment value, two of the mkfs commands I tried previously will result in a FAT file system that can be read by the ESP. So no need to dump the flash to get a compatible FAT file system:

Code: Select all

mkfs.fat -F 12 -R 1 -S 4096 fat.fs
mkfs.vfat -F 12 -R 1 -S 4096 fat.fs
Now that we've got the hard stuff sorted out I'll work on updating makeimg.py so it can do the following:

1.) Look in some pre-determined directory for files to add to the file system.
2.) Do the math to figure out the file system size and starting location.
3.) Combine the firmware image and the file system image into a single flashable file.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Creating larger ROMs

Post by jimmo » Wed Jul 10, 2019 5:44 am

Woohoo! That's awesome :)

manseekingknowledge
Posts: 61
Joined: Sun Oct 29, 2017 5:14 pm

Re: Creating larger ROMs

Post by manseekingknowledge » Sat Jul 13, 2019 4:43 pm

Below are some updates that can be made to makeimg.py to allow automatic creation of a file system with files baked into firmware-combined.bin. Just create a directory named micropython/ports/esp8266/fs_files and put the files you want on your ESP8266 into that directory. If the the fs_files directory exists then a file system will be created, if it does not exist, then the build will take place as it has done in the past. Since initsetup.py typically creates the file system and puts boot.py on it, you will need to make sure fs_files has a copy of boot.py in it unless you are booting some other way. Also, you will need to set the flash_size variable equal to the size of your ESP8266's flash module. The code below assumes a 4MB flash.

Code: Select all

import sys
import struct
import hashlib
import os

SEGS_MAX_SIZE = 0x9000

assert len(sys.argv) == 4

md5 = hashlib.md5()

with open(sys.argv[3], 'wb') as fout:

    with open(sys.argv[1], 'rb') as f:
        data_flash = f.read()
        fout.write(data_flash)
        # First 4 bytes include flash size, etc. which may be changed
        # by esptool.py, etc.
        md5.update(data_flash[4:])
        print('flash    ', len(data_flash))

    with open(sys.argv[2], 'rb') as f:
        data_rom = f.read()

    pad = b'\xff' * (SEGS_MAX_SIZE - len(data_flash))
    assert len(pad) >= 4
    fout.write(pad[:-4])
    md5.update(pad[:-4])
    len_data = struct.pack("I", SEGS_MAX_SIZE + len(data_rom))
    fout.write(len_data)
    md5.update(len_data)
    print('padding  ', len(pad))

    fout.write(data_rom)
    md5.update(data_rom)
    print('irom0text', len(data_rom))

    fout.write(md5.digest())

    print('total    ', SEGS_MAX_SIZE + len(data_rom))
    print('md5      ', md5.hexdigest())

if os.path.isdir('fs_files'):
    import distutils.dir_util
    import re
    import subprocess
    import tempfile

    # Build file system
    print('')
    print('Building file system (build/fat.fs)')

    # Read ELF file to determine firmware size
    cmd = 'readelf -a ' + 'build/firmware.elf'
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
    out, error = proc.communicate()

    pattern = re.compile(' *\\d+: (\\d+).*_firmware_size')  # Example (without quotes): "  3453: 00100000     0 NOTYPE  GLOBAL DEFAULT  ABS _firmware_size"
    match = re.findall(pattern, out)
    if match:
        flash_user_start = int(match[0], 16)  # The same value that esp.flash_user_start() returns. When irom0_0_seg = 1011712 (0xF7000) which is the max value for an ESP8266 with a 4MB flash, this will be 1048576 (0x100000).

    else:
        raise Exception('Unable to determine firmware size!')

    #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    #
    # Set flash_size to the size of your target flash. This will be the same value returned by esp.flash_size().
    #
    # For an ESP8266 with a 4MB flash, this will be 4194304 (0x400000)
    # For an ESP8266 with a 2MB flash, this will be 2097152 (0x200000)
    # For an ESP8266 with a 1MB flash, this will be 1048576 (0x100000)
    # For an ESP8266 with a 512KB flash, this will be 524288 (0x80000)
    #
    #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    flash_size = 0x400000

    sec_size = 4096  # Needs to match SEC_SIZE from flashbdev.py
    reserved_secs = 1  # Needs to match RESERVED_SECS from flashbdev.py
    start_sec = flash_user_start // sec_size + reserved_secs  # Same formula used to determine START_SEC in flashbdev.py
    blocks = (flash_size - 20480) // sec_size - start_sec  # Same formula used to determine the value passed to FlashBdev in flashbdev.py
    fs_start = start_sec * sec_size  # Starting location of the file system

    # Delete the the old file system if it exists
    try:
        os.remove('build/fat.fs')

    except Exception:
        pass

    # Use DD to create zeroed out file of the correct size for the file system
    cmd = 'dd if=/dev/zero of=build/fat.fs count=' + str(blocks) + ' bs=' + str(sec_size)
    subprocess.call(cmd, shell=True)

    # Use mkfs to convert the previously created file into a FAT file system
    cmd = 'mkfs.fat -F 12 -R ' + str(reserved_secs) + ' -S ' + str(sec_size) + ' build/fat.fs'
    subprocess.call(cmd, shell=True)

    # Add files to file system
    print('')
    print('Adding files to file system (build/fat.fs)')

    # Get the mount directory name
    tmp_dir = tempfile.gettempdir()
    mount_dir = os.path.join(tmp_dir, 'fs_files')

    # Delete the the mount directory
    try:
        distutils.dir_util.remove_tree(mount_dir)

    except Exception:
        pass

    # Make the mount direcotry
    os.makedirs(mount_dir)

    # Mount the file system
    cmd = 'mount build/fat.fs ' + mount_dir
    subprocess.call(cmd, shell=True)

    try:
        # Copy all the files to the files to the file system
        distutils.dir_util.copy_tree('fs_files', mount_dir)

    except Exception as e:
        # If you hit this the files you are trying to add are probably too big
        raise e

    finally:
        # Unmount the file system
        cmd = 'umount ' + mount_dir
        subprocess.call(cmd, shell=True)

    # Delete the the mount directory
    try:
        distutils.dir_util.remove_tree(mount_dir)

    except Exception:
        pass

    # Calculate padding needed between fw and fs
    fout_name = sys.argv[3]
    fout_size = os.path.getsize(fout_name)
    pad = b'\xff' * (fs_start - fout_size)

    # Append the file system to the bin file
    print('')
    print('Appending file system (build/fat.fs) to ' + fout_name)

    with open(fout_name, 'ab') as fout:
        if pad:
            fout.write(pad)

        with open('build/fat.fs', 'rb') as fat_fs:
            fout.write(fat_fs.read())

    # Give warning message if no boot.py is found in the files copied to the file system
    if not os.path.isfile('fs_files/boot.py'):
        print('Warning: Your build has a file system without a boot.py file. This will prevent your ESP8266 from booting unless you have done something to change that.')

pthwebdev
Posts: 8
Joined: Tue Jul 02, 2019 1:49 pm

Re: Creating larger ROMs

Post by pthwebdev » Sat Jul 20, 2019 9:48 am

First, great job that you can create an image that has a filesystem in it.

A note, however. In my own firmware, I use the concept of resources, comparable to what classic MacOS and Win32 applications had. I am in the process of incorporating that into MicroPython. So I had to see if the assumption was correct that the segment could not exceed 0x8F000. That would severely limit what I had planned. I never ran into such problems with my own firmware, but I needed to double check.

First I created a binary out of all the resources (data, text, html, images, the lot). I have a resource compiler for that. I included a few additional jpg's (37K+51K+250K) to make sure that together it was rather large. Usually, a whole web interface can be done in under 150K.

I tested this on a Wemos D1 mini with 4MB flash
BEWARE: this excercise will corrupt any filesytem you might have on the device, so make sure there are no important/unique files stored in the flash. With the larger image, you will still have a filesystem. It just is a bit smaller. And empty.

For comparison, this is a regular build (it has several frozen modules added):
size of bin file: 620240 (606k)

output from esp.info()

Code: Select all

    _text_start=40100000
    _text_end=40107938
    _irom0_text_start=40209000
    _irom0_text_end=402976c0
    _data_start=3ffe8000
    _data_end=3ffe8400
    _rodata_start=3ffe8400
    _rodata_end=3ffe8728
    _bss_start=3ffe8728
    _bss_end=3fff8a28
    _heap_start=3fff8a28
    _heap_end=3fffc000
This tells me
size of _irom0_text =
0x402976c0 - 0x40209000 = 0x8E6C0 = 583360 = 569kB

Now the enlarged build:
size of bin file: 1099204 (1073kB, a little over 1MB)

output from esp.info()

Code: Select all

    _text_start=40100000
    _text_end=4010798c
    _irom0_text_start=40209000
    _irom0_text_end=4030c5b4
    _data_start=3ffe8000
    _data_end=3ffe8400
    _rodata_start=3ffe8400
    _rodata_end=3ffe8728
    _bss_start=3ffe8728
    _bss_end=3fff8a28
    _heap_start=3fff8a28
    _heap_end=3fffc000
size of _irom0_text =
0x4030c5b4 - 0x40209000 = 0x1035B4 = 1062324 = 1MB + 13kB + 436B

For this to work, I had to change file esp8266.ld (I simple changed 0x8F000 to 0x18F000) and esp8266_common.ld

Conclusion: the image can exceed 1M and the rom can exceed 0x8F000

I am not sure if there is another limit I may run into, but MicroPython does run correctly with the larger ROM.

I still need to complete what I had planned, but I feel confident that the assumption about the limit seems to be mistaken.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Creating larger ROMs

Post by jimmo » Sat Jul 20, 2019 11:50 am

pthwebdev wrote:
Sat Jul 20, 2019 9:48 am
I am not sure if there is another limit I may run into, but MicroPython does run correctly with the larger ROM.

I still need to complete what I had planned, but I feel confident that the assumption about the limit seems to be mistaken.
The assumption was based on the memory map I looked at that only lists 0x40200000-0x402FFFFF as the mapped region. Maybe this is outdated?

For what it's worth though, the MicroPython code is likely all in the beginning of the segment. Can you verify that you can read your resource data past 0x40300000 ? You can quickly test this using machine.mem8. https://github.com/esp8266/esp8266-wiki/wiki/Memory-Map suggests that you'll get back patterns of 0x00 and 0x80. But hopefully this is incorrect or out of date.

Changing the linker config to allow irom0_text to be larger than 1MiB (or any other change to the .ld files) will happily let the linker put the data in spiflash wherever you like -- the flashing process doesn't care.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Creating larger ROMs

Post by jimmo » Sat Jul 20, 2019 11:58 am

jimmo wrote:
Sat Jul 20, 2019 11:50 am
pthwebdev wrote:
Sat Jul 20, 2019 9:48 am
I am not sure if there is another limit I may run into, but MicroPython does run correctly with the larger ROM.

I still need to complete what I had planned, but I feel confident that the assumption about the limit seems to be mistaken.
The assumption was based on the memory map I looked at that only lists 0x40200000-0x402FFFFF as the mapped region. Maybe this is outdated? Changing the linker config to allow irom0_text to be larger than 1MiB (or any other change to the .ld files) will happily let the linker put the data in spiflash wherever you like -- the flashing process doesn't care.

For what it's worth though, the MicroPython code is likely all in the beginning of the segment. Can you verify that you can read your resource data past 0x40300000 ? You can quickly test this using machine.mem8. https://github.com/esp8266/esp8266-wiki/wiki/Memory-Map suggests that you'll get back patterns of 0x00 and 0x80. But hopefully this is incorrect or out of date.

Post Reply