uart.read() is giving accumulated data

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
tangerino
Posts: 17
Joined: Sun Jul 25, 2021 8:34 am

Re: uart.read() is giving accumulated data

Post by tangerino » Wed Aug 25, 2021 10:57 am

I've been using serial ports for more than few decades.
My point is: read() only returns anything when the buffer has 120 bytes.
Before, nothing coes back.
In my case, I'm using a extremely simple protocol, 8 bytes long, 110bps, half-duplex.
I have to read byte by byte and parse the frame on the fly and sincronize.

Code: Select all

def test_rx():
    uart = UART(2, baudrate=110, timeout_char=1)
    print(uart)
    frame = []
    while True:
        buffer = uart.read(1)
        if buffer:
            for byte in buffer:
                frame.append(byte)
            if len(frame) == 8:
                uart.write("U")
                line = ""
                for byte in frame:
                    line += "{} ".format(str(byte))
                print(line)
                frame = []
I saw this in a book other day and the '120' calls my attention.
I can see in the MP source code that flow control is off.

We start by populating a uart_config_t structure instance. This provides the core
settings for a UART we want to use. An example might be:

Code: Select all

uart_config_t myUartConfig;
myUartConfig.baud_rate = 115200;
myUartConfig.data_bits = UART_DATA_8_BITS;
myUartConfig.parity    = UART_PARITY_DISABLE;
myUartConfig.stop_bits = UART_STOP_BITS_1;
myUartConfig.flow_ctrl = UART_HW_FLOWCTRL_DISABLE;
myUartConfig.rx_flow_ctrl_thresh = 120;[/i]

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: uart.read() is giving accumulated data

Post by Roberthh » Wed Aug 25, 2021 12:18 pm

Which board and port do you use? Generally, uart.read returns all bytes that are present at the time of calling. You may set a timeout, but the default is 0. Some ports may behave differently.

User avatar
francis
Posts: 28
Joined: Sat Aug 14, 2021 8:14 am

Re: uart.read() is giving accumulated data

Post by francis » Wed Aug 25, 2021 2:26 pm

I've only ever used default uarts in micopython, but have built uarts in VHDL, and used them everywhere.

A uart is a packet streamer. A packet is 10 bits. A start bit, 8 payload, stop bit.
packets can be sent back to back so "hello, world" on a 10MBaud modem takes 10us
sending a packet with each character every 1us

The receiver might be dma, or interrupt handler and it normally streams characters to a buffer to send to the process occasionally
Occasions are buffer full, timeout, and just because.

You are living in the process. There are normally two kinds of read:
  • read timeout
  • read poll
read timeout hangs for a while then returns as much as it can.
read poll returns immediately with nothing if there was nothing there.
alternatively poll tells you True or False if 1 or more character available for the process.
Sometimes uarts can be a bit weird like sending only if flushed, or buffer big enough, or character reception could trigger "send if possible"

But you have scope traces so it must be a receive side issue, not transmit side

You could bit bash a 9600 baud modem by interrupting on each rx pin change. An interrupt will come potentially every 104us
State machine and 2 timers watch for start and stop bits, and collect payload.

A 115200 baud will have interrupts every 8us if 0x55 is the transmit character, and at least 2 interrupts 72us apart for start and stop

tangerino
Posts: 17
Joined: Sun Jul 25, 2021 8:34 am

Re: uart.read() is giving accumulated data

Post by tangerino » Wed Sep 01, 2021 3:59 pm

Very frustrated with forum support.
Many places I see the "buffered serial read" problem (came in chunks of 120 bytes).
I'm giving up to use MP :(

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: uart.read() is giving accumulated data

Post by Roberthh » Wed Sep 01, 2021 4:19 pm

You did not tell which board and firmware version you use. Besides that I cannot confirm your observation. uart.read() generally returns what actually is in the receive buffer.

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: uart.read() is giving accumulated data

Post by dhylands » Wed Sep 01, 2021 6:49 pm

After working with hundreds of serial devices for several decades (since the mid-80's) I've come to the conclusion that the only sane way to write a receiver for serial data is to write something that processes 1 character at a time.

I always try to do buffered reads to get as much data as is available, but I always process it one character at a time. You never know when you're going to get multiple records in your buffer (especially when the sender has timeouts and resends data) or partial records in your buffer. It seems that you never get reliable processing when you try to process more than one character, because one day, the conditions will line up such that you won't get all of the data you expected (or you'll get noise) and then you're basically having to drop back to processing it one character at a time anyways.

This is independent of the language being used.

tangerino
Posts: 17
Joined: Sun Jul 25, 2021 8:34 am

Re: uart.read() is giving accumulated data

Post by tangerino » Sun Oct 24, 2021 8:43 am

Found a solution for this
Ive made a PR (hard to be accepted) with:

Code: Select all

    // configure UART interrupts
    if (args[ARG_rxfifo_full_thresh].u_int > 0 ||
        args[ARG_rx_timeout_thresh].u_int > 0 ||
        args[ARG_txfifo_empty_intr_thresh].u_int > 0) {
        uart_intr_config_t uart_intr = {
                .intr_enable_mask = UART_INTR_CONFIG_FLAG,
                .rxfifo_full_thresh = args[ARG_rxfifo_full_thresh].u_int,
                .rx_timeout_thresh = args[ARG_rx_timeout_thresh].u_int,
                .txfifo_empty_intr_thresh = args[ARG_txfifo_empty_intr_thresh].u_int
        };
        uart_intr_config(self->uart_num, &uart_intr);
    }
In my case (low baud rate 110bps and small frame size 8 bytes) I can now read byte by byte if I set the serial as:

Code: Select all

self.uart = UART(uart_number, baudrate=baudrate, rx=rx, tx=tx, timeout=timeout, rxfifo_full_thresh=1)

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: uart.read() is giving accumulated data

Post by Roberthh » Sun Oct 24, 2021 8:52 am

uart.read(1) should return a single byte as well.

tangerino
Posts: 17
Joined: Sun Jul 25, 2021 8:34 am

Re: uart.read() is giving accumulated data

Post by tangerino » Mon Oct 25, 2021 11:40 pm

This pending PR solves this problem

Code: Select all

https://github.com/micropython/micropython/pull/7883
My case is a low baud rate (110) and small frame size (9 bytes)
Code attached

I'm using like this

Code: Select all

UART = UART(UART1, baudrate=110, rxfifo_full_thresh=1, timeout=150)
Attachments
machine_uart.c.zip
(4.97 KiB) Downloaded 106 times

Post Reply