uart.read() is giving accumulated data

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Duality
Posts: 16
Joined: Tue Jun 09, 2020 2:43 pm

uart.read() is giving accumulated data

Post by Duality » Wed Jan 06, 2021 12:33 pm

Hello All,

I am using micropython on esp32. I have got a query regarding uart.read() function. Here is a basic code for a uart read:

Code: Select all

from machine import UART, Pin
import utime, uselect
u2 = UART(2, 9600, stop=1, tx=23,rx=22, txbuf=512, rxbuf=512, timeout_char=2)
count=0

while True:
    poll = uselect.poll()
    poll.register(u2, uselect.POLLIN)
    poll.poll()
    print('query: ', u2.read(), ' count: ', count, ' buffer: ', u2.any())
    utime.sleep_ms(100)
    count+=1
I am sending 8 bytes every second on these pins (checked with an oscilloscope). I do not get data each second. In fact, I get data of 120 bytes after 15-16 seconds.

Here is the REPL log:
4.JPG
4.JPG (70.19 KiB) Viewed 4958 times
Please help.

Regards

Duality
Posts: 16
Joined: Tue Jun 09, 2020 2:43 pm

Re: uart.read() is giving accumulated data

Post by Duality » Thu Jan 07, 2021 5:09 am

Hi All,

This is the updated problem statement. I am doing a loopback in this case (in earlier scenario, I was using a different device which was sending the UART data).

Code: Select all

from machine import UART
import utime, uselect
u1 = UART(1, 9600, tx=17,rx=16, parity=None, stop=1, )
u2 = UART(2, 9600, tx=23,rx=22)

#wait = ((8*8)/9600)

while True:
    u1.write('hello113')
    
    # if(u2.any()!=0):
    # poll = uselect.poll()
    # poll.register(u2, uselect.POLLIN)
    # print(poll.poll())
    print(u2.any(), u2.read())
    #utime.sleep_us(10000)
The REPL shows this repeatedly:
None

Code: Select all

0 None
0 None
0 None
0 None
0 None
0 None
0 None
120 b'hello113hello113hello113hello113hello113hello113hello113hello113hello113hello113hello113hello113hello113hello113hello113'
0 None
0 None
0 None
0 None
0 None
0 None
The buffer sizes are 512 bytes. Also, I see that the HW FIFO size is 128 bytes. I am only writing 8 bytes, hence, there is no overflow. Also, I am reading it as many times as I write. whenever the uart RX FIFO gets any bytes, the function should output it. But, I am only seeing the read output 120 bytes together.

When I put utime.sleep() with wait time above a particular value, then the read function works as expected.
Am I missing something here? The data is not lost, means whatever u2 is receiving, it is getting stored somewhere (may be HW FIFO). But why the uart.read() is not outputting, if the data is stored somewhere ?

Regards

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: uart.read() is giving accumulated data

Post by pythoncoder » Thu Jan 07, 2021 2:46 pm

From the docs, if .read() is called without an arg
read as much data as possible. It may return sooner if a timeout is reached.
This seems to be exactly what is happening: it's returning with a timeout until it's ready to empty its buffer.

Where possible it's best to send a newline (b'\n') and use .readline(). Alternatively if messages are of fixed length, use .read(length) - however this approach can make synchronisation difficult if the transmitting device starts before the receiver has been set up.

It's also worth looking at uasyncio's StreamReader and StreamWriter objects.
Peter Hinch
Index to my micropython libraries.

Duality
Posts: 16
Joined: Tue Jun 09, 2020 2:43 pm

Re: uart.read() is giving accumulated data

Post by Duality » Fri Jan 08, 2021 6:16 am

Hi Peter,

Thank you for your suggestions,

I put a timeout in uart definition like:

Code: Select all

u2 = UART(2, 9600, tx=23,rx=22, timeout=20)
This has resolved the issue :) . Below 20 ms (critical timeout), it is behaving like previously (accumulated data of 120 bytes). Also if I increase the baud rate to 19200, the code works as expected till I reduce the timeout to half, i.e. 10 ms. Why is this critical timeout exist? Any timeout value above 0, should give at least few characters as output of uart.read(). Right?

I am not familiar with uasyncio module :| , but will try it out.

Regards

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: uart.read() is giving accumulated data

Post by pythoncoder » Fri Jan 08, 2021 8:55 am

In my opinion the no-argument read() is virtually useless precisely because you don't know how much data you're going to get: the quantity depends on the exact timing of the transmitting device relative to your receiving task. The way to think about communications is in terms of messages: you want to receive a complete message. In general messages should either be fixed length or delimited with a specific character, typically b'\n'.

These observations are common to almost all communications devices, whether UARTs, sockets or radios. It is necessary to figure out how to synchronise the receiver to a remote transmitter whose timing is usually not under your control.

The only way I could envisage using the no-argument read() is if multiple reads were concatenated into a large buffer which was then subsequently parsed to produce individual messages. This would only make sense if an individual message might be longer than the maximum amount of data returned by read. Despite having programmed UARTs for 45 years I don't recall ever actually doing this.

This link illustrates the uasyncio approach.
Peter Hinch
Index to my micropython libraries.

njepsen
Posts: 11
Joined: Tue Jun 22, 2021 4:08 am
Location: New Zealand.

Re: uart.read() is giving accumulated data

Post by njepsen » Sun Jul 25, 2021 4:37 am

I'm running into a similar issue. I'm reading a response from a POST request that looks like this:
b'HTTP/1.1 200 OK\r\nDate: Sun, 25 Jul 2021 04:19:15 GMT\r\nServer: \r\nContent-Length: 45\r\nConnection: close\r\nContent-Type: text/html\r\n\r\nlocaltimeis25/07/2021, 16:19:15,1627186755$$\n\r\nNO CARRIER\r\n'
And I only want the local date time to set my RTC to local time. In this case 25/07/2021,16:19:15, UTC

After i send the POST to the server I run:

Code: Select all

 # end of TCP POST is here
 
 timeout = time.ticks_add(time.ticks_ms(),20000)  #20 sec timeout            
 sleep(1)                    # allow a little response time       
        while  uart.any() <100 and ticks_diff(timeout,ticks_ms()) >0:     #loop until uart says 100 somethings
                  print (ticks_diff(timeout,ticks_ms()))    # this to  12 - 16 seconds usually.
                 sleep(0.5)
        
        r = uart.read()       # read everything
        print ('uart said',r,'   length=',len(r))         
        r = r.decode('utf-8')
        #go and process the RTC
This works but I don't like it because I'm not really sure how it works. 99% of the time the response is read correctly but because the msg length varies from 190 - 200 odd, I cant set the length accurately. I really need to look for r.find('CARRIER') in the response but that also is clumsy.
If I set "while uart.any() <10 " for example, I get the full server response 90% of the time; I think the code relies on the fact that the final uart.read() is 0.5sec after the 'while' statement finishes. Again - skin of the teeth stuff. Incidently - the response from the server takes 12 - 16 seconds.

tangerino
Posts: 17
Joined: Sun Jul 25, 2021 8:34 am

Re: uart.read() is giving accumulated data

Post by tangerino » Sat Aug 21, 2021 7:51 am

I'm facing the same problem (120 bytes buffered somewhere).
Does anyone manage to fix this behavior?
Thanks in advance

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: uart.read() is giving accumulated data

Post by Roberthh » Sat Aug 21, 2021 8:25 am

as @pythoncoder told, there is no general "fix". uart.read() returns whatever is in the receive buffer at that time, and in general UART data is a raw byte stream. If you need it to be structured in any form, you have to parse that stream yourself. State machines are very common for that task. The only small support for s structured stream is readline() which returns data up to a \n character or until a timeout expires.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: uart.read() is giving accumulated data

Post by pythoncoder » Sat Aug 21, 2021 9:39 am

To further clarify: this problem is fundamental to the way that UARTs work and is not a MicroPython issue. UARTs present some interesting challenges.

A UART receives a sequence of bytes. These arrive at times determined by the transmitter. The receiving UART (or its device driver) buffers these characters. If, at some arbitrary point in time, you read the contents of that buffer, you will get N characters where N is the number that the transmitter has sent in the time between the last read and the present moment. Unless there is some specific mechanism that synchronises the transmitter to the receiver, this number will be fairly arbitrary.

Making sense of this arbitrary number of characters is a challenge. The easy way is to design the transmitter code so that it interposes a delimiter at the end of each message (typically b'\n'). This only works if you can guarantee that the message itself cannot include the delimiter. The receiving process then concatenates successive batches of received characters and uses the delimiters to split them into individual messages. These can then be interpreted.

A factor that can complicate this further is the case where the transmitter can be powered on and off independently of the receiver. The receiver then has to cope with partial messages.

Another complication is where binary data is transmitted: delimiters cannot then be used because their uniqueness cannot be guaranteed.

One solution to these problems is to design a communications protocol which synchronises the transmitter to the receiver. The receiver sends a message to the TX requesting transmission, and the TX responds with a message in a well-defined format. If the RX knows exactly how many characters to expect it can wait for that precise number.
Peter Hinch
Index to my micropython libraries.

User avatar
curt
Posts: 25
Joined: Thu Jul 29, 2021 3:52 am
Location: Big Lake, Alaska

Re: uart.read() is giving accumulated data

Post by curt » Sat Aug 21, 2021 5:58 pm

Could the streaming/buffer anomalies be bypassed by reading 1 character at a time until you get a timeout?

Curt

Post Reply