rshell: rsync chockes on binary files

[ttmetro]

It appears that rsync has problems with binary files:

Code: Select all

rsync -v -m lib /flash/lib
Adding /flash/lib/README.md
Adding /flash/lib/mpu9x50
Adding /flash/lib/mpu9x50/vector3d.md
Adding /flash/lib/mpu9x50/vector3d.py
Adding /flash/lib/mpu9x50/README.md
Adding /flash/lib/mpu9x50/classes_imu.png
False
>OKTraceback (most recent call last):
  File "<stdin>", line 1
SyntaxError: invalid syntax

Removing the binary file appears to solve the problem, but since I am on a Mac, binary files (.DS_Store) are generated automatically by the OS and removing them is an endless battle. It also would be nice to be able to upload some binary files (e.g. for a webpage).

Note: I am using loboris' firmware (stating this in case the issue is tied to that).

[dhylands]

Which board are you using?

Are you using a USB link or a serial link?

Can you send me a copy of the binary file which is causing you troubles?

Having rshell ignore .DS_Store files sounds like a reasonable thing to do. To the best of my knowledge, rshell does support binary files and I've used it to transfer binary files in the past. In fact, one of the test suite files is a binary file with all 256 characters present.

[ttmetro]

Which board are you using?

Huzzah32 with SiLabs USB chip

Are you using a USB link or a serial link?

I run rshell with

Code: Select all

rshell -p /dev/cu.SLAB_USBtoUART

Can you send me a copy of the binary file which is causing you troubles?

E.g. classes_imu.png, attached. It seems that the binary file gets copied but after that rshell does not copy the remaining files and won't return to the prompt. Getting out with ^C and restarting the rsync picks up from where it got stuck.

Having rshell ignore .DS_Store files sounds like a reasonable thing to do.

.git files would be another good candidate to ignore.

Thanks,
Bernhard

[pythoncoder]

@ttmetro How are you invoking rshell? Some platforms object to certain sequences of bytes so rshell has a -a option to force ASCII transfers. Note this behaviour is an issue with the target firmware and not with rshell.

A second question purely to satisfy my own curiosity. Why do you want to copy the MPU9x50 driver's UML diagram to the target?

[ttmetro]

@pythoncoder SUCCESS: the -a flag indeed fixes the problem. This may be specific to the ESP32 or Loboris' firmware which I am using.

As for your 2nd question: My idea was not to "touch" your library (i.e. just use what I downloaded). Before I was using FileZilla which has a filter to tell it what to synchronize and I was using that to restrict the uploads. Now that I am switching to rshell I am deleting what I don't want to be uploaded to the microcontroller and run a script that deletes the .DS_Store and .git files before invoking rshell. Would be nice to add a filter to the rsync command in rshell, maybe I'll get to that some day.

Unrelated ...

Many thanks for all the great stuff you and other folks contributing to Micropython are doing. I will be using it in a new class I'll be teaching in spring at UC Berkeley. Micropython is a great alternative to C (Arduino) and I hope I can expose a much larger audience to IoT.

One issue I am not yet sure about is real time control. One project is a robot that can balance on two wheels. Quite easy to do in C, but with Micropython it's challenging to get a 100Hz update rate on the ESP32 I am using. Even on better hardware (pyboard or other ARM processors) gc is a problem. Perhaps doing everything with integers is a solution, but introduces new complexity on its own. I suppose solutions will be found - the advantages of using managed languages (over C) are just too compelling. What's your opinion?

[pythoncoder]

My experience of MicroPython real time control is limited to the Pyboard. Tests on the ESP8266 and ESP32 indicate that interrupt latency is a problem: values of ~600μs have been observed and that may not be the worst case. On the Pyboard it's more like 10μs if my memory serves me well. The Espressif chips have an underlying RTOS which services the WiFi link. This means that the performance of realtime applications (e.g. using uasyncio) is hard to quantify accurately.

On the Pyboard MicroPython runs on baremetal. The maintainers have put in considerable effort to make MicroPython and key libraries like uasyncio deterministic, using allocation as little as possible. Inevitably writing realtime applications requires some MicroPython-specific expertise - notably in techniques for minimising allocation. This is discussed here http://docs.micropython.org/en/latest/p ... ained.html.

In practice allocation will occur so garbage collection is required. In RT applications it is worth explicitly calling this in the main loop, so it runs before it has much to do. This reduces GC time from (potentially) several ms to ~1ms.

The bottom line is that, on a Pyboard or similar, a 100Hz update is (depending on application) definitely feasible. As an example I built a balancing robot where the loop time is constrained to run at 200Hz. There is a lengthy discussion on balancing robots here viewtopic.php?f=5&t=658.

[ttmetro]

@pythoncoder Thank you very much for this most encouraging reply! Apparently with the right hardware, real time performance (at the level I need it now) has already been demonstrated!

The ESP32, despite it's advantages (the huge interest e.g. by this community and low price) has some shortcomings (aside from timing, e.g. ADC performance is very poor). But without doubt these issues will be addressed in revisions or other solutions will come to market. I use this solution now since (1) I need wifi (2) low cost.

Incidentally, are you aware of available solutions for wifi with the pyboard? Apparently there is one based on the CC3000 but that chip is a bit dated. Also, I am surprised that the CC3200 was essentially abandoned in favor of the ESP32. The latter has of course better raw performance, but presumably specs like irq delay and adc offset are better?

Concerning bare metal ... on "loboris" python, user code runs on a separate thread.

Code: Select all

import _thread
thread.list()

lists a single thread (MainThread). Perhaps it behaves like single threaded "bare metal" if I don't start other threads?

I want to run some tests, but ran into a problem with the Timer.

[pythoncoder]

For ESP32 IRQ latency see viewtopic.php?f=18&t=2904&p=18828&hilit ... ncy#p18828.

I don't know of another generic MicroPython/WiFi bridge. I devised a solution for the MQTT protocol only, using an ESP8266 to perform the interface. Note that this solution is not fast - I wouldn't use it to control a quadcopter https://github.com/peterhinch/micropython-mqtt.git

[ttmetro]

For ESP32 IRQ latency see viewtopic.php?f=18&t=2904&p=18828&hilit ... ncy#p18828.

I saw that, especially Damien's comment. Looks like espressif has still much to learn (not to speak about A/D converters ...). Your MQTT solution has the potential advantage of offloading mqtt timing to an auxiliary cpu. Sockets would be more general, but I did not understand the existing implementations when trying to see if they could be adapted easily.

[pythoncoder]

A way of using an ESP8266 (or ESP32) to bring nonblocking sockets to a Pyboardwould be a great project, and I have considered it. I'm not sure I have the necessary skills to undertake it.