UART lost bytes / buffer overflow issue

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Post Reply
User avatar
bandaangosta
Posts: 6
Joined: Thu May 18, 2017 1:39 pm
Location: Santiago, Chile
Contact:

UART lost bytes / buffer overflow issue

Post by bandaangosta » Sun Apr 07, 2019 1:11 am

I have been struggling with an issue for a couple days. I am communicating my WEMOS D1 Mini Lite board to my PC through the onboard USB port (UART0). After understanding* the whole deal with the REPL and UART0, thanks to the good experts here (viewtopic.php?f=15&t=6182), I got stuck at a buffer issue that is actually quite simple, but I could not find any reference in the documentation so I don't know if I am doing something wrong or not.

On the PC side:

Code: Select all

# Master device (host) main program
import serial
import time

slave = serial.Serial('/dev/ttyUSB0', 115200, timeout=3)
try:
    time1 = time.time()

    while True:
        time2 = time.time()
        if time2 - time1 > 1:
            time1 = time2
            packet = '[{:.3f}] ABCDEFGHIJKLMNOPQRSTUVWXYZ\n'.format(time1)
            print('Sending packet: {}'.format(packet), end='')
            slave.write(packet.encode('utf-8'))

        # Uncomment following line to reduce host CPU usage
        time.sleep(.1)

except KeyboardInterrupt:
    print('Ctrl+C was pressed')

finally:
    if slave.is_open:
        slave.close()
        print(' Serial connection was closed')
On execution:

Code: Select all

Sending packet: [1554597073.694] ABCDEFGHIJKLMNOPQRSTUVWXYZ
Sending packet: [1554597074.697] ABCDEFGHIJKLMNOPQRSTUVWXYZ
Sending packet: [1554597075.699] ABCDEFGHIJKLMNOPQRSTUVWXYZ
Sending packet: [1554597076.702] ABCDEFGHIJKLMNOPQRSTUVWXYZ
^CCtrl+C was pressed
 Serial connection was closed
On the ESP8266 side, I disable REPL on UART0 on boot.py with

Code: Select all

uos.dupterm(None, 1) 
and use UART1 (Tx only) for debugging with the help of a cheap USB to TTL serial adapter, connected to the PC as well. ESP code is:

Code: Select all

# Slave device (ESP8266 board) main program
import machine
import time

led = machine.Pin(2, machine.Pin.OUT) # set to 2 for builtin LED in Wemos D1 mini lite
debugger = machine.UART(1, 115200)

def debugPrint(msg):
    debugger.write(str(msg) + '\r\n')

def main():
    try:
        uart = machine.UART(0)
        uart.init(115200, timeout=100)

        ticker = time.ticks_ms()

        debugPrint('Entering main loop')
        while True:
            if uart.any():
                msg = uart.readline()
                debugPrint(msg)

    except Exception as exc:
        debugPrint('Exception was raised')
        debugPrint(type(exc))
        debugPrint(str(exc))
    finally:
        # Blink repeteadly
        for i in range(8):
            led.value(not led.value()) # Toggle LED
            time.sleep(0.1)

# Run main loop
main()
After hard resetting the board, debugging messages show the following once host (PC) script is run:

Code: Select all

Entering main loop
b'[1554597073.694 ABCDEFGHIJKLMNPQRSTUVWXYZ\n'
b'[1554597074.697 ABCDEFGHIJKLMNPQRSTUVWXYZ\n'
b'[1554597075.699 ABCDEFGHIJKLMNPQRSTUVWXYZ\n'
b'[1554597076.702 ABCDEFGHIJKLMNPQRSTUVWXYZ\n'
Notice the missing closing bracket and the letter O. So, it appears as if every 16th byte is lost. A buffer overflow maybe?

My solution was to send one byte at a time from the host, adding a short delay between bytes. On the board, package is reconstructed:

PC code:

Code: Select all

 
   time1 = time.time()
    while True:
        time2 = time.time()
        if time2 - time1 > 1:
            time1 = time2
            packet = '[{:.3f}] ABCDEFGHIJKLMNOPQRSTUVWXYZ\n'.format(time1)
            print('Sending packet: {}'.format(packet), end='')

            for byte in list(packet):
                slave.write(byte.encode())
                time.sleep(.001)
PC console:

Code: Select all

Sending packet: [1554598416.067] ABCDEFGHIJKLMNOPQRSTUVWXYZ
Sending packet: [1554598417.123] ABCDEFGHIJKLMNOPQRSTUVWXYZ
Sending packet: [1554598418.179] ABCDEFGHIJKLMNOPQRSTUVWXYZ
^CCtrl+C was pressed
 Serial connection was closed
Board code:

Code: Select all

        buffer = []
        while True:
            if uart.any():
                c = uart.read(1)
                if c is not None:
                    if c.decode('utf-8') == '\n':
                        msg = ''.join([x.decode('utf-8') for x in buffer])
                        debugPrint(msg)
                        buffer = []
                    else:
                        buffer.append(c)
On board (debugging):

Code: Select all

Entering main loop
[1554598416.067] ABCDEFGHIJKLMNOPQRSTUVWXYZ
[1554598417.123] ABCDEFGHIJKLMNOPQRSTUVWXYZ
[1554598418.179] ABCDEFGHIJKLMNOPQRSTUVWXYZ
Is this expected behavior or am I hitting a timing or buffering issue due to some error in my code(s)?

Thanks!

*I still don't fully understand REPL redirection/duplication. What is the 2nd parameter of uos.dupterm() for again? I think the documentation could benefit from more examples.

User avatar
pythoncoder
Posts: 4571
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: UART lost bytes / buffer overflow issue

Post by pythoncoder » Sun Apr 07, 2019 8:32 am

It is possible that the ESP8266 can't cope with such a high baudrate. I would try testing at (say) 9600 to see if this affects things. I don't know if the ESP8266 port of machine supports the rxbuf UART constructor arg. If it does, try setting the value to a number > your message length.

These tests would give some indication of the cause.

Re dupterm args my interpretation of the docs is that the index arg enables concurrent duplication on more than one uart. Assume u0 and u1 are UART instances, then the following should achieve it on platforms supporting more than one slot:

Code: Select all

uos.dupterm(u0)  # on slot 0
uos.dupterm(u1, 1)  # slot 1
Note I haven't actually tried this.
Peter Hinch

User avatar
bandaangosta
Posts: 6
Joined: Thu May 18, 2017 1:39 pm
Location: Santiago, Chile
Contact:

Re: UART lost bytes / buffer overflow issue

Post by bandaangosta » Sun Apr 07, 2019 8:54 pm

pythoncoder wrote:
Sun Apr 07, 2019 8:32 am
It is possible that the ESP8266 can't cope with such a high baudrate. I would try testing at (say) 9600 to see if this affects things. I don't know if the ESP8266 port of machine supports the rxbuf UART constructor arg. If it does, try setting the value to a number > your message length.

These tests would give some indication of the cause.

Re dupterm args my interpretation of the docs is that the index arg enables concurrent duplication on more than one uart. Assume u0 and u1 are UART instances, then the following should achieve it on platforms supporting more than one slot:

Code: Select all

uos.dupterm(u0)  # on slot 0
uos.dupterm(u1, 1)  # slot 1
Note I haven't actually tried this.
Thanks, Peter. You just led me in the right direction about the rxbuf setting.
First I tried with 9600 baudrate, but same results.

As I am disabling the REPL on UART0, I have no access for the REPL on USB. But after enabling and setting webrepl, I could try a test code by Dave Hylands on viewtopic.php?f=2&t=1741&p=33589&hilit= ... fer#p33589

This gave me the full UART0 default settings given my uart.init(115200, timeout=100):

Code: Select all

events = [(UART(0, baudrate=115200, bits=8, parity=None, stop=1, rxbuf=15, timeout=100, timeout_char=1), 1)]
I then figured out I could have obtained the same information by just doing:

Code: Select all

>>> uart=machine.UART(0)
>>> uart
UART(0, baudrate=115200, bits=8, parity=None, stop=1, rxbuf=15, timeout=0, timeout_char=1)   
At least I got to learn how the select.poll() method works.

So, after setting the UART with uart.init(115200, timeout=100, rxbuf=CUSTOM_RX_BUF), everything works fine:

CUSTOM_RX_BUF = 15 (default). Fails.

Code: Select all

Entering main loop
b'[1554664963.371 ABCDEFGHIJKLMNPQRSTUVWXYZ\n'
b'[1554664964.373 ABCDEFGHIJKLMNPQRSTUVWXYZ\n'
b'[1554664965.375 ABCDEFGHIJKLMNPQRSTUVWXYZ\n'
CUSTOM_RX_BUF = 100 . Works as expected.

Code: Select all

Entering main loop
b'[1554665017.527] ABCDEFGHIJKLMNOPQRSTUVWXYZ\n'
b'[1554665018.529] ABCDEFGHIJKLMNOPQRSTUVWXYZ\n'
b'[1554665019.532] ABCDEFGHIJKLMNOPQRSTUVWXYZ\n'
So, this effectively solves my problem.

There is still something I don't get. CUSTOM_RX_BUF=15 (default value) explains the lost characters in my tests in the exact positions they appeared. However, CUSTOM_RX_BUF = 16 is enough to solve the issue (I used 100 above just to emphazise the bigger value). From your suggestion of "try setting the [rxbuf] value to a number > your message length", I was expecting that CUSTOM_RX_BUF = 16 would just move the "missing" character one space to the right, given my test string.

User avatar
pythoncoder
Posts: 4571
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: UART lost bytes / buffer overflow issue

Post by pythoncoder » Mon Apr 08, 2019 7:02 pm

bandaangosta wrote:
Sun Apr 07, 2019 8:54 pm
...
There is still something I don't get. CUSTOM_RX_BUF=15 (default value) explains the lost characters in my tests in the exact positions they appeared. However, CUSTOM_RX_BUF = 16 is enough to solve the issue (I used 100 above just to emphazise the bigger value). From your suggestion of "try setting the [rxbuf] value to a number > your message length", I was expecting that CUSTOM_RX_BUF = 16 would just move the "missing" character one space to the right, given my test string.
I would have expected that too. Puzzling.
Peter Hinch

Logician
Posts: 3
Joined: Mon May 11, 2020 10:48 am

Re: UART lost bytes / buffer overflow issue

Post by Logician » Wed May 13, 2020 9:30 pm

Thanks for this post!

I spend hours (and hours) trying to figure out why I was only ever getting 15 bytes in my rx buffer. Turns out that's the default limit when not specified in the UART instantiation.

kumy
Posts: 1
Joined: Fri Oct 16, 2020 8:21 am

Re: UART lost bytes / buffer overflow issue

Post by kumy » Fri Oct 16, 2020 8:30 am

Hi,
I hit that too. Setting RXBUF to 16 solved my issue.
Thanks!!!

Post Reply