working with UART and Bytearrays

The official pyboard running MicroPython.
This is the reference design and main target board for MicroPython.
You can buy one at the store.
Target audience: Users with a pyboard.
Sokrates
Posts: 15
Joined: Mon Dec 14, 2015 11:24 pm

working with UART and Bytearrays

Post by Sokrates » Tue Dec 22, 2015 7:50 pm

Hi all,

I'm working with bytes; I have to read 25bytes frames from Uart.

At the moment I'm using the following approach and it's working:

1)byte array init:
myFrame = bytearray()

2)read 1 byte from uart:
tmpByte = uart.read(1)

3) add the byte to the bytearry
myFrame.extend(tmpByte)

4) when i get the last byte of the frame, process the content and then destroy the byte array:
myFrame = bytearray()

I would like to adopt another approach in order to avoid destroying the bytearray, so I tried:

1) 25 byte array init:
myFrame = bytearray(25)

2)read 1 byte from uart:
tmpByte = uart.read(1)

3) add the byte to the bytearry in the "frameIndex" location
myFrame[frameIndex]=tmpByte

the problem is that i get the following error on the last operation: can't convert byte to int.

Shouldn't be tmpByte and myFrame[x] already od type byte?

Where is the error?

User avatar
pythoncoder
Posts: 2899
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: working with UART and Bytearrays

Post by pythoncoder » Wed Dec 23, 2015 5:52 am

I suggest you look at uart.readinto(buf[, nbytes])

To answer your question, uart.read(1) returns a bytes object with a single element (a bytes object is like bytearray, but is immutable). So you'd need to write uart.read(1)[0] to get a byte. But readinto() allows you to read a specified number of bytes into a pre-allocated buffer at a stroke and is the way I'd do it.

Code: Select all

myFrame = bytearray(25)
uart.readinto(myFrame) # job done
Peter Hinch

Sokrates
Posts: 15
Joined: Mon Dec 14, 2015 11:24 pm

Re: working with UART and Bytearrays

Post by Sokrates » Wed Dec 23, 2015 11:29 am

pythoncoder wrote:I suggest you look at uart.readinto(buf[, nbytes])

To answer your question, uart.read(1) returns a bytes object with a single element (a bytes object is like bytearray, but is immutable). So you'd need to write uart.read(1)[0] to get a byte. But readinto() allows you to read a specified number of bytes into a pre-allocated buffer at a stroke and is the way I'd do it.

Code: Select all

myFrame = bytearray(25)
uart.readinto(myFrame) # job done
Thanks a lot. I'll definitely follow your advice.

User avatar
EasyRider
Posts: 89
Joined: Wed Dec 30, 2015 8:17 am
Location: Land Down Under

Re: working with UART and Bytearrays

Post by EasyRider » Fri Jan 15, 2016 11:16 am

I am having an issue with reading bytearray with uart.
This is on pyboard 1.1 with todays firmware.

Once I sort this out I need to read variable length array as a whole not broken up.
But for initial simplicity I cant even read fixed length array.

Example of fixed 8 byte array:

Code: Select all

0x01, 0x03, 0x00, 0x00, 0x00, 0x01, 0x84, 0x0A
I have initially tried uart.read(8), expecting to receive 8 byte array but was receiving fairly random length arrays with correct byte order.

Code: Select all

b'01:03:00'
b'00:00:01:84'
b'0a'
b'01:03:00'
b'00:00:01:84'
b'0a'
b'01:03:00'
b'00:00:01:84'
b'0a'
b'01:03:00'
b'00:00:01'
b'84:0a'
b'01:03:00'
b'00:00:01'
b'84:0a'
b'01:03'
b'00:00:00:01'
b'84:0a'
b'01:03'
b'00:00:00:01'
b'84:0a'
b'01:03'
b'00:00:00:01'
b'84:0a'
b'01'
b'03:00:00:00'
b'01:84:0a'
b'01'
b'03:00:00:00'
b'01:84:0a'
b'01'
b'03:00:00:00'
b'01:84:0a'
b'01'
b'03:00:00:00'
b'01:84:0a'
b'01:03:00:00'
b'00:01:84:0a'
b'01:03:00:00'
b'00:01:84:0a'
b'01:03:00:00'
b'00:01:84:0a'
b'01:03:00'
b'00:00:01:84'
b'0a'
Then I tried Peter's advice to read into a buffer array, but am receiving bytes in incorrect order and in the random positions in the array buffer.

This is a simple code.

Code: Select all

from pyb import UART
import binascii

uart = UART(2, 38400)
reading = bytearray(8)

while 1:  

    if uart.any():

         uart.readinto(reading)
         print(binascii.hexlify(bytearray(reading), b":"))
         
        
This is a sample of printed inconsistent output, through REPL to terminal.

Code: Select all

b'00:00:01:84:00:0a:00:00'
b'84:0a:01:84:00:0a:00:00'
b'01:03:01:84:00:0a:00:00'
b'00:00:00:01:00:0a:00:00'
b'84:0a:00:01:00:0a:00:00'
b'01:03:00:01:00:0a:00:00'
b'00:00:00:01:00:0a:00:00'
b'84:0a:00:01:00:0a:00:00'
b'01:03:00:01:00:0a:00:00'
b'00:00:00:01:00:0a:00:00'
b'84:0a:00:01:00:0a:00:00'
b'01:03:00:01:00:0a:00:00'
b'00:00:00:01:00:0a:00:00'
b'84:0a:00:01:00:0a:00:00'
b'01:0a:00:01:00:0a:00:00'
b'03:00:00:00:00:0a:00:00'
b'01:84:0a:00:00:0a:00:00'
b'01:84:0a:00:00:0a:00:00'
b'03:00:00:00:00:0a:00:00'
b'01:84:0a:00:00:0a:00:00'
b'01:84:0a:00:00:0a:00:00'
b'03:00:00:00:00:0a:00:00'
b'01:84:0a:00:00:0a:00:00'
b'01:03:00:00:00:0a:00:00'
b'00:01:84:0a:00:0a:00:00'
b'01:03:00:00:00:0a:00:00'
b'00:01:84:0a:00:0a:00:00'
b'01:03:00:00:00:0a:00:00'
b'00:01:84:0a:00:0a:00:00'
b'01:03:00:00:00:0a:00:00'
b'00:01:84:0a:00:0a:00:00'
b'01:03:00:0a:00:0a:00:00'
b'00:00:01:84:00:0a:00:00'
b'0a:00:01:84:00:0a:00:00'
b'01:03:00:84:00:0a:00:00'
b'00:00:01:84:00:0a:00:00'
b'0a:00:01:84:00:0a:00:00'
b'01:03:00:84:00:0a:00:00'
b'00:00:01:84:00:0a:00:00'
b'0a:00:01:84:00:0a:00:00'
b'01:03:00:84:00:0a:00:00'
b'00:00:01:84:00:0a:00:00'
b'0a:00:01:84:00:0a:00:00'
b'01:03:00:84:00:0a:00:00'
b'00:00:01:84:00:0a:00:00'
b'84:0a:01:84:00:0a:00:00'
b'01:03:01:84:00:0a:00:00'
b'00:00:00:01:00:0a:00:00'
b'84:0a:00:01:00:0a:00:00'
b'01:03:00:01:00:0a:00:00'
b'00:00:00:01:00:0a:00:00'
b'84:0a:00:01:00:0a:00:00'
Would appreciate advice.

Regards
John

User avatar
EasyRider
Posts: 89
Joined: Wed Dec 30, 2015 8:17 am
Location: Land Down Under

Re: working with UART and Bytearrays

Post by EasyRider » Fri Jan 15, 2016 11:23 am

Further clarification to above post:

I have tried all sorts of delays between array transmissions to make sure that pyboard receive buffer doesn't get overrun , above results are with about 1 second delay between transmissions.

User avatar
dhylands
Posts: 2841
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: working with UART and Bytearrays

Post by dhylands » Fri Jan 15, 2016 4:49 pm

There are 3 things that can cause uart to return.

1 - A timeout occurs. This is controlled by the timeout= parameter to UART.init and defaults to 1000 milliseconds (1 second). The controls the time that the uart waits for the first character to arrive
2 - An inter-character timeout occurs. This is controlled by the timeout_char= parameter. Once the uart has started to receive characters this is the amount of time betweem characters. It looks like there is no way to turn this timeout off. You can pass in large values, like 65535 which will wait for upto 65 seconds between characters.
3 - The desired amount of data is read.

In any of the above scenarios, it will return the actual number of characters read.

When you use readinto it will always start at offset 0 of the buffer you pass in, so if you call uart.readinto(reading) then the newly read data will occupy positions 0 thru the number of the bytes read (minus 1).

If you want to fill the buffer and also deal with partial reads, then you'l need to use a combination of readinto and memoryview. Something like the following:

Code: Select all

import pyb
from binascii import hexlify

u = pyb.UART(1, 115200, timeout_char=500)

buf = bytearray(8)
mv = memoryview(buf)
idx = 0

while idx < len(buf):
    if u.any():
        bytes_read = u.readinto(mv[idx:])
        print('Got {} bytes of data'.format(bytes_read), hexlify(buf[idx:idx+bytes_read], b':'))
        idx += bytes_read
print('Final buffer =', hexlify(buf, ':'))
I used timeout_char of 500 just so that I could type in multiple characters at human speed and get the number of characters I needed. The results I got were:

Code: Select all

Got 1 bytes of data b'61'
Got 3 bytes of data b'62:63:64'
Got 2 bytes of data b'65:66'
Got 2 bytes of data b'67:68'
Final buffer = b'61:62:63:64:65:66:67:68'
With a normal bytearray when you slice it (accessing a sub-range is called a slice) by doing buf[idx:] that winds up creating a new bytearray object with the contents from idx to the end. So if you modify this new bytearray, it doesn't modify the original.

With a memoryview, when you slice it, it creates a sub-memoryview whose indicies start at zero, but whose data is the original data. This allows you to modify the original data. For example:

Code: Select all

>>> buf = bytearray('0123456789')
>>> buf[3:6] = b'abc'
>>> buf
bytearray(b'012abc6789')
>>> mv = memoryview(buf)
>>> bytes(mv)
b'012abc6789'
>>> mv[3:6] = b'abc'
>>> buf
bytearray(b'012abc6789')

User avatar
EasyRider
Posts: 89
Joined: Wed Dec 30, 2015 8:17 am
Location: Land Down Under

Re: working with UART and Bytearrays

Post by EasyRider » Fri Jan 15, 2016 9:38 pm

Thanks Dave,

It was a simple inter character delay causing issues, should have looked at UART setting options, once again learning pains.
At this stage 5ms wait for next character fixes the problem at 38400 baud, may have to play with this to fine tune it a bit more.

Regards
John

User avatar
EasyRider
Posts: 89
Joined: Wed Dec 30, 2015 8:17 am
Location: Land Down Under

Re: working with UART and Bytearrays

Post by EasyRider » Sat Jan 16, 2016 6:24 am

Still having issue reading separate arrays/packets.

Example code flow.

Code: Select all

uart = UART(2, 38400, timeout = 100, timeout_char = 10,read_buf_len=15) 

PYB1 writes send_bytearrayA
read response  read_bytearrayA, (about 5ms delayed). 

delay(anywhere 200-1000)

PYB1 writes send_bytearrayB
read response  read_bytearrayB, (about 5ms delayed).

delay(anywhere 200-1000)

Repeat loop.
Issue is that INSIDE THE LOOP BOTH read_bytearrays are read as a combined array (read_bytearrayA + read_bytearrayB), and placed in a single array, read_bytearrayA or read_bytearrayB?

Then when it repeats the loop it creates the break, but inside the loop it again reads 2 arrays as as arrayA + arrayB.

I can clearly observe the timing on the scope of both write and read signals that are correct.

I have tried to manipulate timeout, timeout_char ,read_buf_len in UART settings, but can't get the uart.read to create a break between reading separate arrays/packets.

How to uart.read sequential bytearrays into separate arrays/buffers?

Sorry to be a pain, but would really appreciate help.

Regards
John

User avatar
dhylands
Posts: 2841
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: working with UART and Bytearrays

Post by dhylands » Sat Jan 16, 2016 6:33 am

To have the the data go into separate bytearrays then you need to limit the size of the reads.

In my experience, trying to do multi-byte reads will inevitably screw up somewhere down the line because the timing won't be exactly what you expect.

I always write my uart receivers to either read one byte at a time and build up whatever data needs to be builtup, or possibly to read a larger buffer but still process it one character at a time.

User avatar
pythoncoder
Posts: 2899
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: working with UART and Bytearrays

Post by pythoncoder » Sat Jan 16, 2016 8:11 am

dhylands wrote:... For example:

Code: Select all

>>> buf = bytearray('0123456789')
>>> buf[3:6] = b'abc'
>>> buf
bytearray(b'012abc6789')
>>> mv = memoryview(buf)
>>> bytes(mv)
b'012abc6789'
>>> mv[3:6] = b'abc'
>>> buf
bytearray(b'012abc6789')
Perhaps not the most convincing demo of the memoryview as assigning to it and the buffer slice resulted in the same outcome ;) This perhaps illustrates it better:

Code: Select all

>>> a = bytearray(b'0123456789')
>>> m = memoryview(a)
>>> z = m[3:6] # z is not a copy
>>> z[:] = b'jkl'
>>> a
bytearray(b'012jkl6789')
>>> 
The point of course being that z is a window into the original bytearray. If z had been declared as a[3:6] it would have been a copy, and assigning to it would have left a unaffected:

Code: Select all

>>> p = a[7:9]
>>> p
bytearray(b'78')
>>> p[:] = b'xy'
>>> p
bytearray(b'xy') # p has changed
>>> a
bytearray(b'012jkl6789') # a has not
>>> 
Peter Hinch

Post Reply