UART.readline() doesn't match the documentation

All ESP8266 boards running MicroPython.
Official boards are the Adafruit Huzzah and Feather boards.
Target audience: MicroPython users with an ESP8266 board.
Post Reply
JumpZero
Posts: 54
Joined: Mon Oct 30, 2017 5:54 am
Location: Arcachon - France

UART.readline() doesn't match the documentation

Post by JumpZero » Wed Feb 20, 2019 1:04 pm

Hello,

I'm playing with the ESP8266 uart and doing some communication tests with an arduino.
The hardware is ok the communication works fine (of course I use a 2k2 + 1k voltage divider for Arduino 5V to ESP 3.3V)
Correct me if I'm wrong but I notice that in the documentation here:
https://docs.micropython.org/en/latest/ ... T.readline
The statement:
Return value: the line read or None on timeout.
should read
Return value: the line read or None on timeout if no character available, or the characters available (maximum 16).
It's not a big deal, but you better know it.

Here after I try to explain the tests which bring me to that conclusion.
Here is the Arduino software:

Code: Select all

int index;

void setup() {
    Serial.begin(115200) ;
    index = 0 ;
}

void loop() {
  Serial.println("Hello! This message from arduino, let's test that!");
  Serial.print("index = ");
  Serial.println(index);
  index++;
  delay(5000);

}
So this code doesn't do much, it prints 2 lines ending with \r\n every 5 seconds
fisrt line is 50 characters, second line is just an index value incrementing every turn.


And here is the Micropython webrepl console:

Code: Select all

>>> import uos, machine
>>> uos.dupterm(None, 1)
>>> uart = machine.UART(0, baudrate=115200, bits=8, parity=None, stop=1, rxbuf=100, timeout=0, timeout_char=1)
>>> while True:
...     if uart.any():
...         print(uart.readline())
...
...
...
b"Hello! This message from arduino, let's test that!\r\n"
b'index = 83\r\n'
b'Hello! This message from arduino, le'
b'Hello! This mess'
b'age from arduino'
b", let's test tha"
b't!\r\n'
b'index = 88\r\n'
b'Hello! This mess'
b'age from arduino'
b", let's test tha"
b't!\r\n'
b'index = 89\r\n'
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
KeyboardInterrupt:
>>> uart = machine.UART(0, baudrate=115200, bits=8, parity=None, stop=1, rxbuf=100, timeout=5, timeout_char=1)
>>> while True:
...     if uart.any():
...         print(uart.readline())
...
...
...
b"Hello! This message from arduino, let's test that!\r\n"
b'index = 93\r\n'
b'Hello! This message from arduino, le'
b"Hello! This message from arduino, let's test that!\r\n"
b'index = 95\r\n'
b"Hello! This message from arduino, let's test that!\r\n"
b'index = 96\r\n'
b"Hello! This message from arduino, let's test that!\r\n"
b'index = 97\r\n'
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
KeyboardInterrupt:
>>>
In the first part after disabling the repl on uart0 with uos.dupterm(None, 1), I set the uart with timeout=0.
Then read the incoming lines. The first line is complete because already in the buffer (see index is 83 next will be 88) but all following lines are truncated.
uart.readline() returns as soon as called (because timeout is 0 I guess) with a line of 16 characters not ending with \n.

Then in second part (after a KeyboardInterrupt by Ctrl-C) I set the uart with timeout=5 and restart the same incoming lines reading.
And this time uart.readline() behaves as expected: it returns a line ending by \n.

So my conclusion is that when uart.readline() returns on timeout if some characters are available in the buffer it will return them (but 16 maximum).

I also did this test @9600 bauds with same result.

Also as I found mentioned here by Roberthh
viewtopic.php?f=16&t=5359&hilit=uos.dup ... 5&start=10
that uart.readline() is non blocking and again it's ok, but you better know it, because when like me if you come from Python pyserial (or C under Linux) the same readline function is blocking.

So may I sugest a documentation update? Of course if I'm not wrong with my conclusion.

And thanks to all Micropython developers, Micropython is awesome :-)
--
Jmp0

User avatar
jcf
Posts: 60
Joined: Wed Mar 03, 2021 11:14 am

Re: UART.readline() doesn't match the documentation

Post by jcf » Sun Apr 03, 2022 10:15 am

I have had some strange experiences with readline too.
I have 2 Picos, one is receiving messages from the other.
The messages are sent in 1 second intervals

The program on the receiving site sometimes hangs, the Pico is only accessible after several reset operations.
This is the code (reduced to find the error):

Code: Select all

from machine import UART
uart = UART(0, baudrate=9600, timeout = 10, timeout_char = 10)

while True:
    
    s = uart.readline() 
    if s:
        #print(s)
        uart.write(s)
The timeout values seem to be critical, and it is not exactly clear to me what they mean, and if they are in seconds or milliseconds, as the doc says.
What frustrates me most is that the error is not really reproducable, sometimes all goes well and sometimes the strange behavior occurs.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: UART.readline() doesn't match the documentation

Post by pythoncoder » Sun Apr 03, 2022 11:17 am

jcf wrote:
Sun Apr 03, 2022 10:15 am
I have had some strange experiences with readline too.
I have 2 Picos, one is receiving messages from the other.
The messages are sent in 1 second intervals

The program on the receiving site sometimes hangs, the Pico is only accessible after several reset operations.
...
There is a known issue affecting concurrent transmission and reception on the RP2 UARTS. I have submitted a PR which fixes this and another related problem. It is waiting for approval.
Peter Hinch
Index to my micropython libraries.

User avatar
jcf
Posts: 60
Joined: Wed Mar 03, 2021 11:14 am

Re: UART.readline() doesn't match the documentation

Post by jcf » Mon Apr 04, 2022 7:46 am

Thanks, Peter.
Maybe in the meantime I should work with uart.read() and put everything into a buffer.
This way I have the full control I suppose.

It was really frustrating that not only I didn't get my data, but the Pico was in a strange Nirvana state from which it was difficult to come back (not only in Thonny, but also by using the REPL throug a simple terminal).

User avatar
jcf
Posts: 60
Joined: Wed Mar 03, 2021 11:14 am

Re: UART.readline() doesn't match the documentation

Post by jcf » Mon Apr 04, 2022 8:03 am

I've tried this now:

Code: Select all

from machine import UART, Pin
uart = UART(0, baudrate=9600, timeout = 10, timeout_char = 10)

buffer = b""
buffer_ready = False

while True:   
    if uart.any():
        c = uart.read(1)
        if c == b'\n':
            buffer_ready = True
        else:    
            buffer += c
    if buffer_ready:
        print(buffer)
        buffer = b""
        buffer_ready = False
It seems to work

User avatar
jcf
Posts: 60
Joined: Wed Mar 03, 2021 11:14 am

Re: UART.readline() doesn't match the documentation

Post by jcf » Mon Apr 04, 2022 9:02 am

Another variant, this time class based:

Code: Select all

from machine import UART, Pin
uart = UART(0, baudrate=9600, timeout = 10, timeout_char = 10)

class ReadSerialLines():
    def __init__(self, uart):
        self.uart = uart
        self.buffer = b""
        self.buffer_ready = False
        self.c = b''
        
    def read(self):
        if self.uart.any():
        
            self.c = self.uart.read(1)
            if self.c == b'\n':
                self.buffer_ready = True
            else:    
                self.buffer += self.c
            
        if self.buffer_ready:
            s = self.buffer.decode('utf-8')
            
            self.buffer = b""
            self.buffer_ready = False
            return s
        else:
            return ""
serReader = ReadSerialLines(uart)

while True:   
    s = serReader.read()
    
    if s:
        print(s)
        uart.write(s) 
    

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: UART.readline() doesn't match the documentation

Post by pythoncoder » Mon Apr 04, 2022 9:38 am

I am puzzled by your description of the Pico requiring multiple resets. How are you resetting the device - with a hardware reset on the RUN line or software with ctrl-c?
Peter Hinch
Index to my micropython libraries.

User avatar
jcf
Posts: 60
Joined: Wed Mar 03, 2021 11:14 am

Re: UART.readline() doesn't match the documentation

Post by jcf » Tue Apr 05, 2022 8:31 am

I am puzzled too!
There is no appearant logic in the failures.

It goes like this:
I test the program under Thonny.
Several times everything goes well, I'm happy.
Then, just when I am sure I programmed the right solution, I start it again and, plop, I have the Pico not responding any more.

I have the impression it occurs often when I save a file on the Pico (though it is not reproducible).
Thonny hangs when trying to save.
Then I klick the Cancel button, klick the Close dialog button, klick wildly around and when nothing helps, I pull the USB connector out and plug it in again. Often, even then, there is no connection. It is as if Micropython itself would hang. As I said, there is no reproducible pattern, unfortunately.
The problem occured when I used uart.readline(), but I'm still not sure if it doesn't happen with the new code.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: UART.readline() doesn't match the documentation

Post by pythoncoder » Tue Apr 05, 2022 9:32 am

I am reading a lot of problems relating to IDE's. To enable the right team to be alerted it's important to determine whether a fault is in hardware, firmware or IDE.

In extensive work with RP2 I've never encountered a situation where, after a power cycle, I couldn't connect using rshell or mpremote. I suggest you try one of these tools. If the problem persists I'd first suspect the hardware. If you can produce a simple repro then it's a firmware issue.

The one case where connection to any MP platform can be difficult is where main.py immediately runs user code, but I'm sure you haven't fallen into that trap ;)
Peter Hinch
Index to my micropython libraries.

User avatar
jcf
Posts: 60
Joined: Wed Mar 03, 2021 11:14 am

Re: UART.readline() doesn't match the documentation

Post by jcf » Wed Apr 06, 2022 5:44 pm

The one case where connection to any MP platform can be difficult is where main.py immediately runs user code, but I'm sure you haven't fallen into that trap
Yes, I know this one :-)
To avoid it, I used a pushbutton to get out of the main while loop, if the Raspi would not react.
I did some more tests, and it seems as if my new buffer based code would work
[touching wood :-) ]
Normally I am very satisfied with Thonny.
I also tried communication through Gtkterm, without Thonny, when I got the problem with readline, and it happened there too. So it didn't seem that Thonny was to blame.

Post Reply