Synchronizing Modules Recursively from Desktop to Board

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Post Reply
cefn
Posts: 205
Joined: Tue Aug 09, 2016 10:58 am

Synchronizing Modules Recursively from Desktop to Board

Post by cefn » Fri Apr 20, 2018 3:00 pm

Hi all

I'm facing the need to synchronize a filesystem to an ESP8266 module running Micropython with locally-edited files changing as development proceeds.

Previously I've just overwritten things selectively, but I would like to support learners synchronizing from their own edited files.

I want to avoid them...
  • potentially forgetting to synchronize something that has changed (if you don't sync everything every time)
  • waiting ages while everything syncs (if you sync everything every time)
  • nuking a board's flash by writing it too much (if every edit leads to a full flash+upload)
For this reason I was thinking of a minimal shim around os.walk() which operates a bit like rsync - checking file hashes between the desktop and the board before bothering to upload via ampy utils. Most files will only be read, and the only files to be written will be ones which have changed

Are there any utilities I've missed which do this job? Something which can be invoked like...

sync.py --port /dev/ttyUSB0 fromBaseDir toBaseDir

If there isn't anything, I might consider writing a minimal routine around the assumption that

Code: Select all

import hashlib
works on the desktop and

Code: Select all

import uhashlib
works on the module. Does that make sense? Is there a better way?

User avatar
dhylands
Posts: 2858
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Synchronizing Modules Recursively from Desktop to Board

Post by dhylands » Fri Apr 20, 2018 3:04 pm

rshell (which needs to be invoked with --buffer-size=30 and maybe --ascii, to work with the esp8266) has an rsync command which will synchronize two directories: https://github.com/dhylands/rshell#rsync

cefn
Posts: 205
Joined: Tue Aug 09, 2016 10:58 am

Re: Synchronizing Modules Recursively from Desktop to Board

Post by cefn » Fri Apr 20, 2018 3:41 pm

Thanks @dhylands!

How is it that the file write times are reliable on the board?

I had assumed I couldn't use such a convention since the board time is not aligned with desktop time. Do I have to make sure it has the right time on the board via NTP? Alternatively does the write operation actually transfer the accurate times from the desktop file stats? If so, I can't see where this happens.

User avatar
dhylands
Posts: 2858
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Synchronizing Modules Recursively from Desktop to Board

Post by dhylands » Fri Apr 20, 2018 4:04 pm

When rshell connects to the board it sets the time on the board to match the time on the desktop.

cefn
Posts: 205
Joined: Tue Aug 09, 2016 10:58 am

Re: Synchronizing Modules Recursively from Desktop to Board

Post by cefn » Fri Apr 20, 2018 7:54 pm

For anyone who is curious how to calculate SHA1 of a file from a Micropython board, the following generated consistent results with the desktop tool sha1sum...

Code: Select all

import uhashlib

def calculateSha1(filepath):
    try:
        f = open(filepath, 'rb')
        hasher = uhashlib.sha1()
        bufferLength = 128
        buffer = bytearray(bufferLength)
        readLength = None
        while readLength != 0:
            readLength = f.readinto(buffer)
            if readLength == bufferLength:
                hasher.update(buffer)
            else:
                hasher.update(memoryview(buffer)[:readLength])
        return hasher.digest()
    finally:
        f.close()

def calculateSha1String(filepath):
    return "".join(map("{:02x}".format, calculateSha1(filepath)))

resultString = calculateSha1String('color.py')

Produced resultString ...
072ae30d260cedefdd6c86438038e59d8ab341c1

While running sha1sum color.py on the desktop...

$ sha1sum color.py
072ae30d260cedefdd6c86438038e59d8ab341c1 color.py

However, note the string join in calculateSha1String() is likely to fragment the memory so is only there for debugging. It's best to compare the result of calculateSha1() directly with the bytes of a known SHA1.

User avatar
pythoncoder
Posts: 2991
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Synchronizing Modules Recursively from Desktop to Board

Post by pythoncoder » Sat Apr 21, 2018 8:39 am

You can avoid the string join (and the ingenious use of map/format) by using ubinascii.hexlify:

Code: Select all

>>> ubinascii.hexlify(calculateSha1('rats.py'))
b'53e9dbba64ac30746f8c1e6cb477fe7a5fcb02d5'
>>>
Peter Hinch

Post Reply