MicroPython Forum (Archive)

Posted: **Tue Jun 08, 2021 9:46 pm**

Background: I want to use the ST MCU on the pyboard to flash a SPI ROM, which will receive the data dynamically from the host PC. I don't want to use the mapped drive. I would like to use only the REPL to accomplish this.

I am able to transfer commands/source to the REPL using the pyboard.py, no problem. I can execute code and receive the results in the form of string output. I am using the VCP over USB.

I have a binary file on the PC. How can I read from it and transfer it over to the Pyboard as quickly as possible? I intend to send over one page (256 bytes) of data at a time, since the page programming takes some time to finish, I can use that time to let the REPL buffer the the next page of data. (I think that I may have to recompile MicroPython to increase the REPL buffer size to make this work.)

Here is the problem. I think that I must convert binary data to strings before I can send them over to the REPL. Even in raw REPL mode, it is expecting strings. That means that an example single-byte binary value that I fetch from the file like 0xff becomes something like b'\xff', right? Or perhaps just '\xff'? My single-byte payload just became 6 or 7 bytes, since each ASCII character is 1 byte. Am I missing something?

How can I send binary data as binary data to the REPL?

Posted: **Wed Jun 09, 2021 10:03 am**

You could use binascii.hexlify (and unhexlify on the device) -- this will "only" double each byte.

Or you could do what rshell does -- execute a loop on the device which reads from sys.stdin.buffer (https://github.com/dhylands/rshell/blob ... n.py#L1072).

Posted: **Wed Jun 09, 2021 10:37 pm**

If you're going to transfer binary data then you want to make sure that you call:

Code: Select all

import micropython

micropython.kbd_intr(-1)

Otherwise binary 0x03 characters will be interpreted as Control-C and interrupt your program.

Once you've completed your transfer you can call

Code: Select all

micropython.kbd_intr(0x03) to re-enable the Control-C behaviour.

Posted: **Fri Jun 11, 2021 3:46 pm**

Wouldn't the REPL still receive all data as strings, including newline? How do I suppress the REPL temporarily to ignore the incoming data and instead readirect it *somewhere* that I can access with my SPI write function?

The only way I know to do this is just call a spi_write(b'\xff') function to transfer 8 bits of data as ASCII, which is then converted to binary in the spi_write() function before being written to the SPI device. But that is not ideal because it inflates the binary data size when sending as ASCII.

What I think I want is:
>>> spi_write() <-- Read by the REPL and enters a binary recv mode
... Recv binary data and exit binary mode
>>> <-- Next REPL prompt after binary data transferred

Posted: **Fri Jun 11, 2021 4:11 pm**

REPL is only active when no Python program is running. A python script can act like REPL, but for that it has to read from sys.stdin. So ,pur code has to read from sys.stdin.buffer and write that using SPI. There is no automatic redirect of sys.stdin to spi.write.

Posted: **Fri Jun 11, 2021 8:11 pm**

OK, I got it running. Here is the code on the pyboard side:

Code: Select all

import micropython
import sys

class RECV_TEST():
    def __init__(self):
        self.bufsize = 256
        self.read_buffer = bytearray(self.bufsize)
        return
    
def recv_test(self):
    # Disable ctrl-c
    micropython.kbd_intr(-1)

    bytes_read = sys.stdin.buffer.readinto(self.read_buffer, self.bufsize)

    print("Read %d bytes!" % bytes_read)

    # Restore ctrl-c
    micropython.kbd_intr(3)
    return

Here is changes to pyboard.py to transfer the binary data:

Code: Select all

def exec_raw(self, command, timeout=10, data_consumer=None, binary_data=None):
    self.exec_raw_no_follow(command)
    if binary_data:
        print("Sending binary data...")
        self.serial.write(binary_data)
    return self.follow(timeout, data_consumer)

One problem: This works all the way up to 255 bytes. I have tested 8, 32, 128, 255. They all work. The moment I use 256 as the transfer size, the host times out after 10 seconds. I open a terminal an press return, I get the output that 256 bytes were transferred. I can then examine the read_buffer in the object and I see that /r from my keypress becomes the 256th byte in the array on the pyboard. It sort of says "NO!" to that 256th byte but then allows it later. Seems like the UART buffers are too small but the default should be 512 bytes for USB.

Why is it hitting a wall at exactly 256 bytes?

Posted: **Sat Jun 12, 2021 1:03 am**

What can happen is that with some MCU configurations, especially those using USB-to-serial interfaces is that the serial buffer on the pyboard side fills up and the next character that comes in gets dropped.

rshell uses flow control. So a buffer gets sent, then the sender waits for an ACK character to come back from the pyboard before sending the send buffer's worth.

Which board are you using?

Posted: **Tue Jun 15, 2021 7:09 am**

I am using a custom developed board, not the actual official pyboard, and I learned that the buffer was indeed set to 256 in my MicroPython firmware build. Using a small delay was intermittently failing, so I had to keep increasing it.

I modified the code to use the ACK, so that the board sends the ACK just before it listens for data, with no delays needed:

Code: Select all

    def exec_raw(self, command, timeout=10, data_consumer=None, binary_data=None):
        self.exec_raw_no_follow(command)
        if binary_data:
            data = self.read_until(1, b'\x06', timeout=timeout)
            if not data.endswith(b'\x06'):
                raise PyboardError('timeout waiting for ACK to receive read data')
            print("Sending binary data...")
            self.serial.write(binary_data)
        return self.follow(timeout, data_consumer)

The function running on the board side sends the ACK byte via sys.stdout.write('\x06') just before the read of stdin.

This works without the delay reliably. However, if I put a one second delay in the code on the board side after the ACK (but before the stdin read) in an attempt to fill the buffer exactly (completely), I lose the last byte again, and it is repeatable. I can't explain that, as the first 255 bytes are as expected with no corruption. I guess I will increase the buffer size in the firmware for some padding. It just makes me nervous. Why would it drop a byte when exactly full?

If I only send 255 bytes, it works all day long. It needs a byte of overhead for some reason.

Posted: **Tue Jun 15, 2021 6:57 pm**

Yeah - the algorithm that is currently used requires an empty byte to work properly. It only maintains a get and put pointer, so if the get and put pointers are equal you can't tell if the buffer is empty or full.

So rather than add extra memory to keep a flag to keep track of it, it requires one byte of overhead from the buffer so that if the get pointer equals the put pointer it knows that the buffer is empty.

Practically speaking this means that the a buffer size of 256 bytes can only buffer 255 bytes of data.

MicroPython Forum (Archive)

How to send binary data to the REPL?

How to send binary data to the REPL?

Re: How to send binary data to the REPL?

Re: How to send binary data to the REPL?

Re: How to send binary data to the REPL?

Re: How to send binary data to the REPL?

Re: How to send binary data to the REPL?

Re: How to send binary data to the REPL?

Re: How to send binary data to the REPL?

Re: How to send binary data to the REPL?