OK, that was a lot of work. This is my first external module, so there were quite a lot of things I had to find out.
The bad news: 0x4030000 is indeed the limit when reading flash as regular memory. The pattern is not exactly as in the documentation, but the data is, read after read, dump after dump, not stable beyond the limit. Probably some random noise.
The good news is: porting the code does read the resources, even beyond 0x4030000.
The code is still somewhat rough around the edges and needs some work (e.g. an iterator, see if it works with the webserver). I did not have time to paint it or to build it to scale.
The complete image is 2973032 and has 2352680 bytes of resources. This is exaggerated because there are many copies of the same jpg.
I am happy with the result.
Creating larger ROMs
-
- Posts: 61
- Joined: Sun Oct 29, 2017 5:14 pm
Re: Creating larger ROMs
Here are some improvements to the changes I made to makeimg.py. The code below uses mcopy (part of the mtools Linux package) instead of mount for adding files to the file system image. This change was necessary to make the script work with WSL 1.0 which has a sub-par/incomplete implementation of mount. Maybe WSL 2.0 which is slated to be released next year will have a better implementation of mount.manseekingknowledge wrote: ↑Sat Jul 13, 2019 4:43 pmBelow are some updates that can be made to makeimg.py to allow automatic creation of a file system with files baked into firmware-combined.bin. Just create a directory named micropython/ports/esp8266/fs_files and put the files you want on your ESP8266 into that directory. If the the fs_files directory exists then a file system will be created, if it does not exist, then the build will take place as it has done in the past. Since initsetup.py typically creates the file system and puts boot.py on it, you will need to make sure fs_files has a copy of boot.py in it unless you are booting some other way. Also, you will need to set the flash_size variable equal to the size of your ESP8266's flash module. The code below assumes a 4MB flash.
Code: Select all
import sys import struct import hashlib import os SEGS_MAX_SIZE = 0x9000 assert len(sys.argv) == 4 md5 = hashlib.md5() with open(sys.argv[3], 'wb') as fout: with open(sys.argv[1], 'rb') as f: data_flash = f.read() fout.write(data_flash) # First 4 bytes include flash size, etc. which may be changed # by esptool.py, etc. md5.update(data_flash[4:]) print('flash ', len(data_flash)) with open(sys.argv[2], 'rb') as f: data_rom = f.read() pad = b'\xff' * (SEGS_MAX_SIZE - len(data_flash)) assert len(pad) >= 4 fout.write(pad[:-4]) md5.update(pad[:-4]) len_data = struct.pack("I", SEGS_MAX_SIZE + len(data_rom)) fout.write(len_data) md5.update(len_data) print('padding ', len(pad)) fout.write(data_rom) md5.update(data_rom) print('irom0text', len(data_rom)) fout.write(md5.digest()) print('total ', SEGS_MAX_SIZE + len(data_rom)) print('md5 ', md5.hexdigest()) if os.path.isdir('fs_files'): import distutils.dir_util import re import subprocess import tempfile # Build file system print('') print('Building file system (build/fat.fs)') # Read ELF file to determine firmware size cmd = 'readelf -a ' + 'build/firmware.elf' proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True) out, error = proc.communicate() pattern = re.compile(' *\\d+: (\\d+).*_firmware_size') # Example (without quotes): " 3453: 00100000 0 NOTYPE GLOBAL DEFAULT ABS _firmware_size" match = re.findall(pattern, out) if match: flash_user_start = int(match[0], 16) # The same value that esp.flash_user_start() returns. When irom0_0_seg = 1011712 (0xF7000) which is the max value for an ESP8266 with a 4MB flash, this will be 1048576 (0x100000). else: raise Exception('Unable to determine firmware size!') #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! # # Set flash_size to the size of your target flash. This will be the same value returned by esp.flash_size(). # # For an ESP8266 with a 4MB flash, this will be 4194304 (0x400000) # For an ESP8266 with a 2MB flash, this will be 2097152 (0x200000) # For an ESP8266 with a 1MB flash, this will be 1048576 (0x100000) # For an ESP8266 with a 512KB flash, this will be 524288 (0x80000) # #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! flash_size = 0x400000 sec_size = 4096 # Needs to match SEC_SIZE from flashbdev.py reserved_secs = 1 # Needs to match RESERVED_SECS from flashbdev.py start_sec = flash_user_start // sec_size + reserved_secs # Same formula used to determine START_SEC in flashbdev.py blocks = (flash_size - 20480) // sec_size - start_sec # Same formula used to determine the value passed to FlashBdev in flashbdev.py fs_start = start_sec * sec_size # Starting location of the file system # Delete the the old file system if it exists try: os.remove('build/fat.fs') except Exception: pass # Use DD to create zeroed out file of the correct size for the file system cmd = 'dd if=/dev/zero of=build/fat.fs count=' + str(blocks) + ' bs=' + str(sec_size) subprocess.call(cmd, shell=True) # Use mkfs to convert the previously created file into a FAT file system cmd = 'mkfs.fat -F 12 -R ' + str(reserved_secs) + ' -S ' + str(sec_size) + ' build/fat.fs' subprocess.call(cmd, shell=True) # Add files to file system print('') print('Adding files to file system (build/fat.fs)') # Get the mount directory name tmp_dir = tempfile.gettempdir() mount_dir = os.path.join(tmp_dir, 'fs_files') # Delete the the mount directory try: distutils.dir_util.remove_tree(mount_dir) except Exception: pass # Make the mount direcotry os.makedirs(mount_dir) # Mount the file system cmd = 'mount build/fat.fs ' + mount_dir subprocess.call(cmd, shell=True) try: # Copy all the files to the files to the file system distutils.dir_util.copy_tree('fs_files', mount_dir) except Exception as e: # If you hit this the files you are trying to add are probably too big raise e finally: # Unmount the file system cmd = 'umount ' + mount_dir subprocess.call(cmd, shell=True) # Delete the the mount directory try: distutils.dir_util.remove_tree(mount_dir) except Exception: pass # Calculate padding needed between fw and fs fout_name = sys.argv[3] fout_size = os.path.getsize(fout_name) pad = b'\xff' * (fs_start - fout_size) # Append the file system to the bin file print('') print('Appending file system (build/fat.fs) to ' + fout_name) with open(fout_name, 'ab') as fout: if pad: fout.write(pad) with open('build/fat.fs', 'rb') as fat_fs: fout.write(fat_fs.read()) # Give warning message if no boot.py is found in the files copied to the file system if not os.path.isfile('fs_files/boot.py'): print('Warning: Your build has a file system without a boot.py file. This will prevent your ESP8266 from booting unless you have done something to change that.')
Code: Select all
import sys
import struct
import hashlib
import os
SEGS_MAX_SIZE = 0x9000
assert len(sys.argv) == 4
md5 = hashlib.md5()
with open(sys.argv[3], 'wb') as fout:
with open(sys.argv[1], 'rb') as f:
data_flash = f.read()
fout.write(data_flash)
# First 4 bytes include flash size, etc. which may be changed
# by esptool.py, etc.
md5.update(data_flash[4:])
print('flash ', len(data_flash))
with open(sys.argv[2], 'rb') as f:
data_rom = f.read()
pad = b'\xff' * (SEGS_MAX_SIZE - len(data_flash))
assert len(pad) >= 4
fout.write(pad[:-4])
md5.update(pad[:-4])
len_data = struct.pack("I", SEGS_MAX_SIZE + len(data_rom))
fout.write(len_data)
md5.update(len_data)
print('padding ', len(pad))
fout.write(data_rom)
md5.update(data_rom)
print('irom0text', len(data_rom))
fout.write(md5.digest())
print('total ', SEGS_MAX_SIZE + len(data_rom))
print('md5 ', md5.hexdigest())
fs_files = 'fs_files' # Location of files to copy to file system
fat_fs = os.path.join('build', 'fat.fs') # Location of file system
if os.path.isdir(fs_files):
import apt
import re
import subprocess
apt_cache = apt.Cache()
if 'mtools' not in apt_cache or not apt_cache['mtools'].is_installed:
print('The command "mcopy" (a part of "mtools") is required to build the file system image. To install type "sudo apt-get install mtools".')
sys.exit(1)
print('Building file system image "{fat_fs}"'.format(fat_fs=fat_fs))
# Read ELF file to determine firmware size
cmd = 'readelf -a build/firmware.elf'
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
out, error = proc.communicate()
pattern = re.compile(' *\\d+: (\\d+).*_firmware_size') # Example (without quotes): " 3453: 00100000 0 NOTYPE GLOBAL DEFAULT ABS _firmware_size"
match = re.findall(pattern, out.decode())
if match:
flash_user_start = int(match[0], 16) # The same value that esp.flash_user_start() returns. When irom0_0_seg = 1011712 (0xF7000) which is the max value for an ESP8266 with a 4MB flash, this will be 1048576 (0x100000).
else:
raise Exception('Unable to determine firmware size!')
#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
#
# Set flash_size to the size of your target flash. This will be the same value returned by esp.flash_size().
#
# For an ESP8266 with a 4MB flash, this will be 4194304 (0x400000)
# For an ESP8266 with a 2MB flash, this will be 2097152 (0x200000)
# For an ESP8266 with a 1MB flash, this will be 1048576 (0x100000)
# For an ESP8266 with a 512KB flash, this will be 524288 (0x80000)
#
#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
flash_size = 0x400000
sec_size = 4096 # Needs to match SEC_SIZE from flashbdev.py
reserved_secs = 1 # Needs to match RESERVED_SECS from flashbdev.py
start_sec = flash_user_start // sec_size + reserved_secs # Same formula used to determine START_SEC in flashbdev.py
blocks = (flash_size - 20480) // sec_size - start_sec # Same formula used to determine the value passed to FlashBdev in flashbdev.py
fs_start = start_sec * sec_size # Starting location of the file system
# Delete the the old file system if it exists
if os.path.isfile(fat_fs):
try:
os.remove(fat_fs)
except Exception:
print('Failed to delete existing file system image "{fat_fs}"'.format(fat_fs=fat_fs))
sys.exit(1)
# Use DD to create zeroed out file of the correct size for the file system
cmd = 'dd if=/dev/zero of={fat_fs} count={blocks} bs={sec_size}'.format(fat_fs=fat_fs, blocks=str(blocks), sec_size=str(sec_size))
if subprocess.call(cmd, shell=True) != 0:
print('Failed to create file system image "{fat_fs}" with command "{cmd}"'.format(fat_fs=fat_fs, cmd=cmd))
sys.exit(1)
# Use mkfs to convert the previously created file into a FAT file system
cmd = 'mkfs.fat -F 12 -R {reserved_secs} -S {sec_size} {fat_fs}'.format(reserved_secs=reserved_secs, sec_size=str(sec_size), fat_fs=fat_fs)
if subprocess.call(cmd, shell=True) != 0:
print('Failed to convert file system image "{fat_fs}" to FAT file system with command "{cmd}"'.format(fat_fs=fat_fs, cmd=cmd))
sys.exit(1)
# Add files to file system
print('Adding files to file system image "{fat_fs}"'.format(fat_fs=fat_fs))
for root, dirs, files in os.walk(fs_files):
target_dir = os.path.relpath(root, fs_files)
if root != fs_files:
cmd = 'mmd -i {fat_fs} ::{target_dir}'.format(fat_fs=fat_fs, target_dir=target_dir)
if subprocess.call(cmd, shell=True) != 0:
print('Failed to make directory in file system image "{fat_fs}" with command "{cmd}"'.format(fat_fs=fat_fs, cmd=cmd))
sys.exit(1)
for file in files:
source_file = os.path.join(root, file)
if root == fs_files:
target_file = file
else:
target_file = os.path.join(target_dir, file)
cmd = 'mcopy -i {fat_fs} {source_file} ::{target_file}'.format(fat_fs=fat_fs, source_file=source_file, target_file=target_file)
if subprocess.call(cmd, shell=True) != 0:
print('Failed to copy file to file system image "{fat_fs}" with command "{cmd}"'.format(fat_fs=fat_fs, cmd=cmd))
sys.exit(1)
# Calculate padding needed between fw and fs
fout_name = sys.argv[3]
fout_size = os.path.getsize(fout_name)
pad = b'\xff' * (fs_start - fout_size)
# Append the file system to the bin file
print('Appending file system image "{fat_fs}" to "{fout_name}"'.format(fat_fs=fat_fs, fout_name=fout_name))
with open(fout_name, 'ab') as fout:
if pad:
fout.write(pad)
with open(fat_fs, 'rb') as fat_fs_hndl:
fout.write(fat_fs_hndl.read())
# Give warning message if no boot.py is found in the files copied to the file system
if not os.path.isfile(os.path.join(fs_files, 'boot.py')):
print('Warning: Your build has a file system without a boot.py file. This will prevent your ESP8266 from booting unless you have done something to change that.')
Re: Creating larger ROMs
Flash a premade filesystem to flash sounds interesting. (See also: viewtopic.php?f=2&t=7555 )
Has anybody tried to make this with LittleFS instead of FAT ?
Has anybody tried to make this with LittleFS instead of FAT ?
Re: Creating larger ROMs
The same would be possible with littlefs using the littlefs FUSE driver.