Optimal speed when writing to MicroPython REPL

[aivarannamaa]

As far as I know, all tools communicating with MicroPython REPL limit writing speed (eg. by writing commands in blocks and sleeping a bit between blocks) in order not to overflow some (circular?) buffers.

I added similar blocks and delays to Thonny IDE (256-byte blocks and 0.01 second delays between blocks, as in pyboard.py). Still, a Thonny user was able to receive a SyntaxError with code that looks fine and works on my machine (https://github.com/thonny/thonny/issues/1336). From posts discussing rshell, I learned that sometimes it's useful to use even smaller blocks when writing to the device.

In order to learn more about this, I made a test-program using PySerial, which is supposed to tell me whether a block of given size goes through intact or not:

Code: Select all

import serial

PORT = "/dev/ttyACM0"
LENGTH_OF_STRING_LITERAL = 1200

# The output of following code tells me whether the code was received intact or not
code = """
s = "%s"
print()
print(len(s), set(s))
""" % ("*" * LENGTH_OF_STRING_LITERAL)

s = serial.Serial(PORT, 115200)

print("Interrupting and requesting raw promt...")
s.write(b"\x03")
s.write(b"\x03")
s.write(b"\x01")


print(s.read_until(b"exit\r\n>"))
print("Got raw prompt. Executing code...")

for i in range(20):
    s.write(code.encode("UTF-8"))
    s.flush()
    s.write(b"\x04")
    print(s.read_until(b"OK"))

    # wait until completion
    print(s.read_until(b">"))

while True:
    print(s.read(1).decode(), end="")

I tested with different code sizes and to my surprise, all devices I tested with, always received the code intact up to the size where the device started giving memory errors (around 1800 with Pyboard). I tested on Linux, macOS and Windows 10 with Pyboard, ESP 32, ESP 2866, W600-Pico, BBC micro:bit, Adafruit Trinket M0 -- all of them received at least 1000 bytes-blocks without errors. I never saw a SyntaxError or any other indication to a buffer overrun (the reported number of characters in the string literal was always correct and none of the asterisk characters was changed to something else).

The only way I could make the device receive broken code, was by omitting the line marked with "# wait until completion" -- it looks like I was able overwhelm the device only when it was busy executing some code.

My questions:

* Can you see any mistakes in my experiment?
* Can you suggest another experiment or another device for reproducing idle raw-REPL receiving broken input?
* Where are these input buffers that need chunked input and delays between chunks? On the device? Inside USB driver?
* Would you recommend using paste-mode for inputting long commands to the REPL? It looks like the echo provided by this mode could be used as flow control (by reading the echo of previous line before writing next line). Do you see any problems with this?

[dhylands]

The issue with rshell is that sometimes you’re writing to the filesystem on the device and that can cause the Rx buffer on the device to overflow, since interrupts (and/or code execution) can wind up getting paused for indeterminate amounts of time.

[aivarannamaa]

The issue with rshell is that sometimes you’re writing to the filesystem on the device and that can cause the Rx buffer on the device to overflow, since interrupts (and/or code execution) can wind up getting paused for indeterminate amounts of time.

By writing to the filesystem, do you mean sending Python code which is writing a bytes literal to a file? Why is this case different from sending other kinds of Python code (from the viewpoint of Rx buffer)?

Or do you mean that uncontrolled emptying of the file buffer may introduce unexpected delays? In Thonny I'm complementing each write command with a flush and it seems to solve this issue.

[dhylands]

I just mean taking the data received and writing to a file. rshell doesn't differentiate between python code or any other file content.

Some UARTS have a HW FIFO (which will work even though interrupts are disabled) and I think all ports can also have an Rx buffer (which requires hardware interrupts to be enabled).

It looks like rshell calls os.sync() (when it exists). Looking through the code, this takes different code paths than calling flush() so perhaps I'll have it do both.

The ESP32 (or any of the ports which have an underlying OS) can introduce additional latency (and/or periods with interrupts disabled) which can mess up serial buffers.

Early on this was a big problem on the STM32 since it has no UART FIFO, so if interrupts got disabled for more than a character time you could lose characters.

On devices with USB directly connected to the MCU, you can typically transfer 1024 bytes with no issues. On devices where you're interacting with a real UART, than you can get different behaviours (depends on UART FIFO length or whether a FIFO even exists). The NUCLEO (stm32) devices with no USB connector (so the REPL goes through a UART rather than USB) seem to be fairly sensitive.

Things are definitely better now than when rshell was first written (5-1/2 years ago).

[aivarannamaa]

Thank you, Dave!

From your answer I take that there are enough variables to warrant such delay based flow control. And today I ran my experiment again with more iterations and I finally saw what I was looking for -- when I repeatedly sent ca 1830 bytes to Pyboard in single bursts, then sometimes the received input indeed was broken.

It looks like rshell calls os.sync() (when it exists)

What does `os.sync()` actually do? Documentation only says "Sync all filesystems". In CPython there is `os.fsync` which applies to a single file. Does MicroPython's `os.sync()` do the same for all open files?

[aivarannamaa]

Would you recommend using paste-mode for inputting long commands to the REPL? It looks like the echo provided by this mode could be used as flow control (by reading the echo of previous line before writing next line). Do you see any problems with this?

I was worrying that reading the echo would hurt the performance when compared to the raw mode, but my initial experiments suggest that it's not bad at all.

I sent a 2000-character script to a Pyboard 100 times in a row using different techniques (all giving zero errors):

• via raw REPL by splitting the script into 1024-byte blocks and waiting 0.01 seconds between blocks (ie. 0.01 second waiting time per iteration). Total time: 11 seconds.
• via raw REPL, but with 256-byte blocks (ie. 0.07 seconds waiting time per iteration). Total time: 17 seconds.
• via paste mode with 1024-byte blocks and reading the echo of previous block before writing the next. Total time: 12 seconds.
• via paste mode, but with 256-byte blocks. Total time: 12 seconds.

From this the paste mode approach looks really promising and I'm considering using it in Thonny instead of raw-mode (the back-end for Unix port already uses this, as it doesn't have raw-mode). I would lose the ability to distinguish between exceptions and normal output, but I would gain a reliable flow control mechanism.

Can anyone spot a possible pitfall here?