Creating larger ROMs

All ESP8266 boards running MicroPython.
Official boards are the Adafruit Huzzah and Feather boards.
Target audience: MicroPython users with an ESP8266 board.
pthwebdev
Posts: 8
Joined: Tue Jul 02, 2019 1:49 pm

Re: Creating larger ROMs

Post by pthwebdev » Sat Jul 20, 2019 8:35 pm

OK, that was a lot of work. This is my first external module, so there were quite a lot of things I had to find out.

The bad news: 0x4030000 is indeed the limit when reading flash as regular memory. The pattern is not exactly as in the documentation, but the data is, read after read, dump after dump, not stable beyond the limit. Probably some random noise.

The good news is: porting the code does read the resources, even beyond 0x4030000.

The code is still somewhat rough around the edges and needs some work (e.g. an iterator, see if it works with the webserver). I did not have time to paint it or to build it to scale.

The complete image is 2973032 and has 2352680 bytes of resources. This is exaggerated because there are many copies of the same jpg.

I am happy with the result.

manseekingknowledge
Posts: 60
Joined: Sun Oct 29, 2017 5:14 pm

Re: Creating larger ROMs

Post by manseekingknowledge » Fri Oct 04, 2019 7:02 am

manseekingknowledge wrote:
Sat Jul 13, 2019 4:43 pm
Below are some updates that can be made to makeimg.py to allow automatic creation of a file system with files baked into firmware-combined.bin. Just create a directory named micropython/ports/esp8266/fs_files and put the files you want on your ESP8266 into that directory. If the the fs_files directory exists then a file system will be created, if it does not exist, then the build will take place as it has done in the past. Since initsetup.py typically creates the file system and puts boot.py on it, you will need to make sure fs_files has a copy of boot.py in it unless you are booting some other way. Also, you will need to set the flash_size variable equal to the size of your ESP8266's flash module. The code below assumes a 4MB flash.

Code: Select all

import sys
import struct
import hashlib
import os

SEGS_MAX_SIZE = 0x9000

assert len(sys.argv) == 4

md5 = hashlib.md5()

with open(sys.argv[3], 'wb') as fout:

    with open(sys.argv[1], 'rb') as f:
        data_flash = f.read()
        fout.write(data_flash)
        # First 4 bytes include flash size, etc. which may be changed
        # by esptool.py, etc.
        md5.update(data_flash[4:])
        print('flash    ', len(data_flash))

    with open(sys.argv[2], 'rb') as f:
        data_rom = f.read()

    pad = b'\xff' * (SEGS_MAX_SIZE - len(data_flash))
    assert len(pad) >= 4
    fout.write(pad[:-4])
    md5.update(pad[:-4])
    len_data = struct.pack("I", SEGS_MAX_SIZE + len(data_rom))
    fout.write(len_data)
    md5.update(len_data)
    print('padding  ', len(pad))

    fout.write(data_rom)
    md5.update(data_rom)
    print('irom0text', len(data_rom))

    fout.write(md5.digest())

    print('total    ', SEGS_MAX_SIZE + len(data_rom))
    print('md5      ', md5.hexdigest())

if os.path.isdir('fs_files'):
    import distutils.dir_util
    import re
    import subprocess
    import tempfile

    # Build file system
    print('')
    print('Building file system (build/fat.fs)')

    # Read ELF file to determine firmware size
    cmd = 'readelf -a ' + 'build/firmware.elf'
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
    out, error = proc.communicate()

    pattern = re.compile(' *\\d+: (\\d+).*_firmware_size')  # Example (without quotes): "  3453: 00100000     0 NOTYPE  GLOBAL DEFAULT  ABS _firmware_size"
    match = re.findall(pattern, out)
    if match:
        flash_user_start = int(match[0], 16)  # The same value that esp.flash_user_start() returns. When irom0_0_seg = 1011712 (0xF7000) which is the max value for an ESP8266 with a 4MB flash, this will be 1048576 (0x100000).

    else:
        raise Exception('Unable to determine firmware size!')

    #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    #
    # Set flash_size to the size of your target flash. This will be the same value returned by esp.flash_size().
    #
    # For an ESP8266 with a 4MB flash, this will be 4194304 (0x400000)
    # For an ESP8266 with a 2MB flash, this will be 2097152 (0x200000)
    # For an ESP8266 with a 1MB flash, this will be 1048576 (0x100000)
    # For an ESP8266 with a 512KB flash, this will be 524288 (0x80000)
    #
    #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    flash_size = 0x400000

    sec_size = 4096  # Needs to match SEC_SIZE from flashbdev.py
    reserved_secs = 1  # Needs to match RESERVED_SECS from flashbdev.py
    start_sec = flash_user_start // sec_size + reserved_secs  # Same formula used to determine START_SEC in flashbdev.py
    blocks = (flash_size - 20480) // sec_size - start_sec  # Same formula used to determine the value passed to FlashBdev in flashbdev.py
    fs_start = start_sec * sec_size  # Starting location of the file system

    # Delete the the old file system if it exists
    try:
        os.remove('build/fat.fs')

    except Exception:
        pass

    # Use DD to create zeroed out file of the correct size for the file system
    cmd = 'dd if=/dev/zero of=build/fat.fs count=' + str(blocks) + ' bs=' + str(sec_size)
    subprocess.call(cmd, shell=True)

    # Use mkfs to convert the previously created file into a FAT file system
    cmd = 'mkfs.fat -F 12 -R ' + str(reserved_secs) + ' -S ' + str(sec_size) + ' build/fat.fs'
    subprocess.call(cmd, shell=True)

    # Add files to file system
    print('')
    print('Adding files to file system (build/fat.fs)')

    # Get the mount directory name
    tmp_dir = tempfile.gettempdir()
    mount_dir = os.path.join(tmp_dir, 'fs_files')

    # Delete the the mount directory
    try:
        distutils.dir_util.remove_tree(mount_dir)

    except Exception:
        pass

    # Make the mount direcotry
    os.makedirs(mount_dir)

    # Mount the file system
    cmd = 'mount build/fat.fs ' + mount_dir
    subprocess.call(cmd, shell=True)

    try:
        # Copy all the files to the files to the file system
        distutils.dir_util.copy_tree('fs_files', mount_dir)

    except Exception as e:
        # If you hit this the files you are trying to add are probably too big
        raise e

    finally:
        # Unmount the file system
        cmd = 'umount ' + mount_dir
        subprocess.call(cmd, shell=True)

    # Delete the the mount directory
    try:
        distutils.dir_util.remove_tree(mount_dir)

    except Exception:
        pass

    # Calculate padding needed between fw and fs
    fout_name = sys.argv[3]
    fout_size = os.path.getsize(fout_name)
    pad = b'\xff' * (fs_start - fout_size)

    # Append the file system to the bin file
    print('')
    print('Appending file system (build/fat.fs) to ' + fout_name)

    with open(fout_name, 'ab') as fout:
        if pad:
            fout.write(pad)

        with open('build/fat.fs', 'rb') as fat_fs:
            fout.write(fat_fs.read())

    # Give warning message if no boot.py is found in the files copied to the file system
    if not os.path.isfile('fs_files/boot.py'):
        print('Warning: Your build has a file system without a boot.py file. This will prevent your ESP8266 from booting unless you have done something to change that.')
Here are some improvements to the changes I made to makeimg.py. The code below uses mcopy (part of the mtools Linux package) instead of mount for adding files to the file system image. This change was necessary to make the script work with WSL 1.0 which has a sub-par/incomplete implementation of mount. Maybe WSL 2.0 which is slated to be released next year will have a better implementation of mount.

Code: Select all

import sys
import struct
import hashlib
import os

SEGS_MAX_SIZE = 0x9000

assert len(sys.argv) == 4

md5 = hashlib.md5()

with open(sys.argv[3], 'wb') as fout:

    with open(sys.argv[1], 'rb') as f:
        data_flash = f.read()
        fout.write(data_flash)
        # First 4 bytes include flash size, etc. which may be changed
        # by esptool.py, etc.
        md5.update(data_flash[4:])
        print('flash    ', len(data_flash))

    with open(sys.argv[2], 'rb') as f:
        data_rom = f.read()

    pad = b'\xff' * (SEGS_MAX_SIZE - len(data_flash))
    assert len(pad) >= 4
    fout.write(pad[:-4])
    md5.update(pad[:-4])
    len_data = struct.pack("I", SEGS_MAX_SIZE + len(data_rom))
    fout.write(len_data)
    md5.update(len_data)
    print('padding  ', len(pad))

    fout.write(data_rom)
    md5.update(data_rom)
    print('irom0text', len(data_rom))

    fout.write(md5.digest())

    print('total    ', SEGS_MAX_SIZE + len(data_rom))
    print('md5      ', md5.hexdigest())

fs_files = 'fs_files'  # Location of files to copy to file system
fat_fs = os.path.join('build', 'fat.fs')  # Location of file system
if os.path.isdir(fs_files):
    import apt
    import re
    import subprocess

    apt_cache = apt.Cache()
    if 'mtools' not in apt_cache or not apt_cache['mtools'].is_installed:
        print('The command "mcopy" (a part of "mtools") is required to build the file system image.  To install type "sudo apt-get install mtools".')
        sys.exit(1)

    print('Building file system image "{fat_fs}"'.format(fat_fs=fat_fs))

    # Read ELF file to determine firmware size
    cmd = 'readelf -a build/firmware.elf'
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
    out, error = proc.communicate()

    pattern = re.compile(' *\\d+: (\\d+).*_firmware_size')  # Example (without quotes): "  3453: 00100000     0 NOTYPE  GLOBAL DEFAULT  ABS _firmware_size"
    match = re.findall(pattern, out.decode())
    if match:
        flash_user_start = int(match[0], 16)  # The same value that esp.flash_user_start() returns. When irom0_0_seg = 1011712 (0xF7000) which is the max value for an ESP8266 with a 4MB flash, this will be 1048576 (0x100000).

    else:
        raise Exception('Unable to determine firmware size!')

    #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    #
    # Set flash_size to the size of your target flash. This will be the same value returned by esp.flash_size().
    #
    # For an ESP8266 with a 4MB flash, this will be 4194304 (0x400000)
    # For an ESP8266 with a 2MB flash, this will be 2097152 (0x200000)
    # For an ESP8266 with a 1MB flash, this will be 1048576 (0x100000)
    # For an ESP8266 with a 512KB flash, this will be 524288 (0x80000)
    #
    #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    flash_size = 0x400000

    sec_size = 4096  # Needs to match SEC_SIZE from flashbdev.py
    reserved_secs = 1  # Needs to match RESERVED_SECS from flashbdev.py
    start_sec = flash_user_start // sec_size + reserved_secs  # Same formula used to determine START_SEC in flashbdev.py
    blocks = (flash_size - 20480) // sec_size - start_sec  # Same formula used to determine the value passed to FlashBdev in flashbdev.py
    fs_start = start_sec * sec_size  # Starting location of the file system

    # Delete the the old file system if it exists
    if os.path.isfile(fat_fs):
        try:
            os.remove(fat_fs)

        except Exception:
            print('Failed to delete existing file system image "{fat_fs}"'.format(fat_fs=fat_fs))
            sys.exit(1)

    # Use DD to create zeroed out file of the correct size for the file system
    cmd = 'dd if=/dev/zero of={fat_fs} count={blocks} bs={sec_size}'.format(fat_fs=fat_fs, blocks=str(blocks), sec_size=str(sec_size))
    if subprocess.call(cmd, shell=True) != 0:
        print('Failed to create file system image "{fat_fs}" with command "{cmd}"'.format(fat_fs=fat_fs, cmd=cmd))
        sys.exit(1)

    # Use mkfs to convert the previously created file into a FAT file system
    cmd = 'mkfs.fat -F 12 -R {reserved_secs} -S {sec_size} {fat_fs}'.format(reserved_secs=reserved_secs, sec_size=str(sec_size), fat_fs=fat_fs)
    if subprocess.call(cmd, shell=True) != 0:
        print('Failed to convert file system image "{fat_fs}" to FAT file system with command "{cmd}"'.format(fat_fs=fat_fs, cmd=cmd))
        sys.exit(1)

    # Add files to file system
    print('Adding files to file system image "{fat_fs}"'.format(fat_fs=fat_fs))

    for root, dirs, files in os.walk(fs_files):
        target_dir = os.path.relpath(root, fs_files)

        if root != fs_files:
            cmd = 'mmd -i {fat_fs} ::{target_dir}'.format(fat_fs=fat_fs, target_dir=target_dir)
            if subprocess.call(cmd, shell=True) != 0:
                print('Failed to make directory in file system image "{fat_fs}" with command "{cmd}"'.format(fat_fs=fat_fs, cmd=cmd))
                sys.exit(1)

        for file in files:
            source_file = os.path.join(root, file)

            if root == fs_files:
                target_file = file

            else:
                target_file = os.path.join(target_dir, file)

            cmd = 'mcopy -i {fat_fs} {source_file} ::{target_file}'.format(fat_fs=fat_fs, source_file=source_file, target_file=target_file)
            if subprocess.call(cmd, shell=True) != 0:
                print('Failed to copy file to file system image "{fat_fs}" with command "{cmd}"'.format(fat_fs=fat_fs, cmd=cmd))
                sys.exit(1)

    # Calculate padding needed between fw and fs
    fout_name = sys.argv[3]
    fout_size = os.path.getsize(fout_name)
    pad = b'\xff' * (fs_start - fout_size)

    # Append the file system to the bin file
    print('Appending file system image "{fat_fs}" to "{fout_name}"'.format(fat_fs=fat_fs, fout_name=fout_name))

    with open(fout_name, 'ab') as fout:
        if pad:
            fout.write(pad)

        with open(fat_fs, 'rb') as fat_fs_hndl:
            fout.write(fat_fs_hndl.read())

    # Give warning message if no boot.py is found in the files copied to the file system
    if not os.path.isfile(os.path.join(fs_files, 'boot.py')):
        print('Warning: Your build has a file system without a boot.py file. This will prevent your ESP8266 from booting unless you have done something to change that.')

Post Reply