autoftp: Super-fast Micropython Development

Discussion about programs, libraries and tools that work with MicroPython. Mostly these are provided by a third party.
Target audience: All users and developers of MicroPython.
jdts
Posts: 36
Joined: Mon Dec 23, 2019 2:55 pm

autoftp: Super-fast Micropython Development

Post by jdts » Fri Mar 12, 2021 9:55 pm

Microcontroller development can be boring. I was recently reminded of this while hacking on an Espressif C-based firmware, with its 1-2min compile/link/upload/reboot times. I'd literally forget what it was I was intending to test while waiting for all the PlatformIO stuff to complete.

One of the true joys of MicroPython for microcontroller development is the devel cycle is so fast. Have an idea? Try it out. Yet for large multi-file projects, the standard methods of uploading files many of us use (tools like rshell), were for me still too slow. Partly this is due to lousy serial port speeds, but partly due to the repetitive Exit REPL, cp file over, return to REPL, re-initialize workflow.

For network connected MicroPython devices, there is a better way: run a small background FTP server on the device, and use a tool that auto-sends any files that change in your project directory. With this my save/upload/rerun time drops below a second, even when hacking on fairly large python files.

I call it autoftp. Take a look if interested, or just watch this short video of autoftp in action.

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: autoftp: Super-fast Micropython Development

Post by kevinkk525 » Sat Mar 13, 2021 11:53 am

This looks great!
When developing on esp32 I always use a ftpserver and then run a script to sync all files over wifi when I'm done with the changes. But this would make it even easier. No manual sync anymore (also my script copies every file, not just the changed ones, which can take a while for a project with >100 files). Might just need to adapt it to automatically create an mpy from changed files and upload that one because my project is too big to use uncompiled .py files :D
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

jdts
Posts: 36
Joined: Mon Dec 23, 2019 2:55 pm

Re: autoftp: Super-fast Micropython Development

Post by jdts » Sat Mar 13, 2021 6:25 pm

Glad you like it. How about a command line option like:

-s | --process: '*.py, mpy_script'

Then autoftp -p '*.mpy' -s '*.py, mpy_script' would notice changes in *.py files, and run mpy_script file.py on them instead of uploading. The script would presumably make/modify the matching .mpy files, then autoftp would notice these *.mpy file(s) appear, and transfer them up.

jdts
Posts: 36
Joined: Mon Dec 23, 2019 2:55 pm

Re: autoftp: Super-fast Micropython Development

Post by jdts » Sat Mar 13, 2021 6:45 pm

BTW you can upload only changed files in a script using a tool like ncftpput: ncftpput -vRz host.local / *.py lib. It skips all files which haven't changed (using timestamp/size to determine). This is fine, but still takes several seconds to login, check files, etc. autoftp is 10x faster. Good way to "pre-seed" a device from a project directory though.

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: autoftp: Super-fast Micropython Development

Post by kevinkk525 » Sat Mar 13, 2021 8:45 pm

jdts wrote:
Sat Mar 13, 2021 6:25 pm
Glad you like it. How about a command line option like:

-s | --process: '*.py, mpy_script'
That would be a great option. Automatically process the file and upload the result.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

jdts
Posts: 36
Joined: Mon Dec 23, 2019 2:55 pm

Re: autoftp: Super-fast Micropython Development

Post by jdts » Sat Mar 13, 2021 9:11 pm

OK, I added a -s|--process option. You can pass as many -s options as you need (one 'pattern, script' per). The script is given the file pathname as its first and only arg. To avoid endless loops, if the "source" files for your script should be uploaded as well, your script shouldn't update these files themselves, or endless process/upload loops will commence. Let me know how this works for you!

Rather than uploading the "result" of a script, I expect you'll just match the file type it produces, and that new/updated file will be noticed and uploaded. Seems to work with my simple testing. So your call would look something like:

% autoftp host.local -p '*.mpy' -s '*.py, mpy-cross'

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: autoftp: Super-fast Micropython Development

Post by kevinkk525 » Sat Mar 13, 2021 9:38 pm

jdts wrote:
Sat Mar 13, 2021 9:11 pm
Rather than uploading the "result" of a script, I expect you'll just match the file type it produces, and that new/updated file will be noticed and uploaded.
That would require having all project files also as .mpy either in the same directory or a separate one. Imho that wouldn't be practical and the direct upload of the processing result would be favorable in my opinion.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

jdts
Posts: 36
Joined: Mon Dec 23, 2019 2:55 pm

Re: autoftp: Super-fast Micropython Development

Post by jdts » Sun Mar 14, 2021 12:59 am

Since a script could in principle create any number of upload-able matching files in various directories, how could you know where to place the files (and what to call them?). Imagine a language normalization script that produces various constants files in different languages, for example. I could perhaps add another flag that lets you specify file patterns to delete locally after upload. So the sequence would be:
  • notices a changed .py file, runs cross-compiler
  • notices a new .mpy file, uploads
  • sees that .mpy files are to be discarded after upload, delete the local file
I suspect your FTP script must do something like this?

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: autoftp: Super-fast Micropython Development

Post by kevinkk525 » Sun Mar 14, 2021 6:04 am

If the script option is a pre-processor, I'd assume it preprocesses the changed .py script and then uploads the result. The result file would just be a temporary files.
You are right that it's possible a script modifies multiple files or creates multiple new files but I wouldn't call that a pre-processor, it's a full-blown automation script.
Guess the question is which way you want to support and which way makes more sense in your environment. Without having tried it, when passing the file location to mpy-cross to create a .mpy file, that file would be created in the same directory as the original .py file, so my project directory would be cluttered with .mpy files.

My script does it like this:
* copy all .py files into a temporary location
* create .mpy of all files
* delete all .py in the temporary location
* upload all remaining (.mpy) files
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

jdts
Posts: 36
Joined: Mon Dec 23, 2019 2:55 pm

Re: autoftp: Super-fast Micropython Development

Post by jdts » Sun Mar 14, 2021 2:23 pm

when passing the file location to mpy-cross to create a .mpy file, that file would be created in the same directory as the original .py file, so my project directory would be cluttered with .mpy files.
Not if I add a -k (update: added as -k or --updelete):

autoftp host.local -p '*.mpy' -s '*.py, mpy-cross' -k '*.mpy'

would create all the .mpy files _in place_, upload them, and delete them locally. No temp directory needed. Keeping the logic fully constrained to "matching files", and the script invocation simple ("script filepath"), will keep autoftp from turning into a full-blown build tool! Does this satisfy your use case?

BTW, I guess I still don't understand how your "pre-processor" script idea would work: the script would need to tell auto-ftp what to call the file on the remote end (even if you limit scripts to processing single files "in place"). And then it needs to deliver the fully pre-processed content of the file, maybe to stdout. So a robust interface between script & autoftp would be needed.

PS: Give it a try.

Post Reply