Synchronizing Modules Recursively from Desktop to Board

[cefn]

Hi all

I'm facing the need to synchronize a filesystem to an ESP8266 module running Micropython with locally-edited files changing as development proceeds.

Previously I've just overwritten things selectively, but I would like to support learners synchronizing from their own edited files.

I want to avoid them...

• potentially forgetting to synchronize something that has changed (if you don't sync everything every time)
• waiting ages while everything syncs (if you sync everything every time)
• nuking a board's flash by writing it too much (if every edit leads to a full flash+upload)

For this reason I was thinking of a minimal shim around os.walk() which operates a bit like rsync - checking file hashes between the desktop and the board before bothering to upload via ampy utils. Most files will only be read, and the only files to be written will be ones which have changed

Are there any utilities I've missed which do this job? Something which can be invoked like...

sync.py --port /dev/ttyUSB0 fromBaseDir toBaseDir

If there isn't anything, I might consider writing a minimal routine around the assumption that

Code: Select all

import hashlib

works on the desktop and

Code: Select all

import uhashlib

works on the module. Does that make sense? Is there a better way?

[dhylands]

rshell (which needs to be invoked with --buffer-size=30 and maybe --ascii, to work with the esp8266) has an rsync command which will synchronize two directories: https://github.com/dhylands/rshell#rsync

[cefn]

Thanks @dhylands!

How is it that the file write times are reliable on the board?

I had assumed I couldn't use such a convention since the board time is not aligned with desktop time. Do I have to make sure it has the right time on the board via NTP? Alternatively does the write operation actually transfer the accurate times from the desktop file stats? If so, I can't see where this happens.

[dhylands]

When rshell connects to the board it sets the time on the board to match the time on the desktop.

[cefn]

For anyone who is curious how to calculate SHA1 of a file from a Micropython board, the following generated consistent results with the desktop tool sha1sum...

Code: Select all

import uhashlib

def calculateSha1(filepath):
    try:
        f = open(filepath, 'rb')
        hasher = uhashlib.sha1()
        bufferLength = 128
        buffer = bytearray(bufferLength)
        readLength = None
        while readLength != 0:
            readLength = f.readinto(buffer)
            if readLength == bufferLength:
                hasher.update(buffer)
            else:
                hasher.update(memoryview(buffer)[:readLength])
        return hasher.digest()
    finally:
        f.close()

def calculateSha1String(filepath):
    return "".join(map("{:02x}".format, calculateSha1(filepath)))

resultString = calculateSha1String('color.py')

Produced resultString ...
072ae30d260cedefdd6c86438038e59d8ab341c1

While running sha1sum color.py on the desktop...

$ sha1sum color.py
072ae30d260cedefdd6c86438038e59d8ab341c1 color.py

However, note the string join in calculateSha1String() is likely to fragment the memory so is only there for debugging. It's best to compare the result of calculateSha1() directly with the bytes of a known SHA1.

[pythoncoder]

You can avoid the string join (and the ingenious use of map/format) by using ubinascii.hexlify:

Code: Select all

>>> ubinascii.hexlify(calculateSha1('rats.py'))
b'53e9dbba64ac30746f8c1e6cb477fe7a5fcb02d5'
>>>