problem with bytearray conversion

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
EasyRider
Posts: 94
Joined: Wed Dec 30, 2015 8:17 am

problem with bytearray conversion

Post by EasyRider » Mon Jan 04, 2016 10:54 am

Struck a strange output from bytearray conversion.
See bytes 13 and 14.
Appears to be wrong if byte 13 or 14 values exceed 15 (0xF).

I am actually using this in uart.write(bytearray(message)) where the output on uart port is OK as expected , irrispective of byte 13 or 14 value..
But a print(bytearray(message)) produces the strange output for bytes 13 and 14.

Same strange behavior is observed in immediate REPL mode.
Have not looked at other bytes.
Development is on STM32F407 Discovery, fairly recent firmware.
>>> message
[1, 16, 0, 0, 0, 3, 6, 0, 6, 0, 2, 0, 3, 143, 65]
>>> bytearray(message)
bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\x06\x00\x02\x00\x03\x8fA')
>>> message[13]=5
>>> bytearray(message)
bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\x06\x00\x02\x00\x03\x05A')
>>> message[14]=6
>>> bytearray(message)
bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\x06\x00\x02\x00\x03\x05\x06')
>>> message[13]=96
>>> bytearray(message)
bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\x06\x00\x02\x00\x03`\x06')
>>> message
[1, 16, 0, 0, 0, 3, 6, 0, 6, 0, 2, 0, 3, 96, 6]
>>> bytearray(message)
bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\x06\x00\x02\x00\x03`\x06')
>>>

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: problem with bytearray conversion

Post by pfalcon » Mon Jan 04, 2016 11:16 am

Can you describe what strangeness you see exactly? (Like: I see "<few, not many characters>", instead I expect to see "<few characters>").
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

Peter.Kenyon
Posts: 70
Joined: Wed Oct 14, 2015 5:07 pm

Re: problem with bytearray conversion

Post by Peter.Kenyon » Mon Jan 04, 2016 11:20 am

What you describe is what I get using python3 on a PC.
Isnt this just the way python stringifies byte arrays ie prints the char or if not printable then the \x value

EasyRider
Posts: 94
Joined: Wed Dec 30, 2015 8:17 am

Re: problem with bytearray conversion

Post by EasyRider » Mon Jan 04, 2016 11:35 am

>>> message
[1, 16, 0, 0, 0, 3, 6, 0, 6, 0, 2, 0, 3, 143, 65]
>>> bytearray(message)
bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\x06\x00\x02\x00\x03\x8fA')
I would expect byte 13 (dec 143) to be converted to (hex x8F).
Byte 14 (dec 65) to be (hex x41).
Not x8fA' ???

Here is another example where the conversion is OK, as I would expect it.
Conversion of byte 13 and 14 dec values to hex is OK.
dec144 == hex92, dec131 == hex83.
ytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\x1f\x00\x02\x00\x03\x92\x83')
[1, 16, 0, 0, 0, 3, 6, 0, 31, 0, 2, 0, 3, 146, 131]
But there is inconsistency in some packet conversions.
Am I missing something?

EasyRider
Posts: 94
Joined: Wed Dec 30, 2015 8:17 am

Re: problem with bytearray conversion

Post by EasyRider » Mon Jan 04, 2016 11:46 am

Here is a printout of some other conversions.

I would expect bytes 13 and and 14 to be converted from dec value to appropriate hex value.
bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\x02\x00\x02\x00\x03~\x81')
[1, 16, 0, 0, 0, 3, 6, 0, 2, 0, 2, 0, 3, 126, 129]


bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\x03\x00\x02\x00\x03CA')
[1, 16, 0, 0, 0, 3, 6, 0, 3, 0, 2, 0, 3, 67, 65]


bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\x04\x00\x02\x00\x03\xf6\x81')
[1, 16, 0, 0, 0, 3, 6, 0, 4, 0, 2, 0, 3, 246, 129]


bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\x05\x00\x02\x00\x03\xcbA')
[1, 16, 0, 0, 0, 3, 6, 0, 5, 0, 2, 0, 3, 203, 65]


bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\x06\x00\x02\x00\x03\x8fA')
[1, 16, 0, 0, 0, 3, 6, 0, 6, 0, 2, 0, 3, 143, 65]


bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\x07\x00\x02\x00\x03\xb2\x81')
[1, 16, 0, 0, 0, 3, 6, 0, 7, 0, 2, 0, 3, 178, 129]


bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\x08\x00\x02\x00\x03\xe6\x80')
[1, 16, 0, 0, 0, 3, 6, 0, 8, 0, 2, 0, 3, 230, 128]


bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\t\x00\x02\x00\x03\xdb@')
[1, 16, 0, 0, 0, 3, 6, 0, 9, 0, 2, 0, 3, 219, 64]


bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\n\x00\x02\x00\x03\x9f@')
[1, 16, 0, 0, 0, 3, 6, 0, 10, 0, 2, 0, 3, 159, 64]


bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\x0b\x00\x02\x00\x03\xa2\x80')
[1, 16, 0, 0, 0, 3, 6, 0, 11, 0, 2, 0, 3, 162, 128]


bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\x0c\x00\x02\x00\x03\x17@')
[1, 16, 0, 0, 0, 3, 6, 0, 12, 0, 2, 0, 3, 23, 64]


bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\r\x00\x02\x00\x03*\x80')
[1, 16, 0, 0, 0, 3, 6, 0, 13, 0, 2, 0, 3, 42, 128]


bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\x0e\x00\x02\x00\x03n\x80')
[1, 16, 0, 0, 0, 3, 6, 0, 14, 0, 2, 0, 3, 110, 128]


bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\x0f\x00\x02\x00\x03S@')
[1, 16, 0, 0, 0, 3, 6, 0, 15, 0, 2, 0, 3, 83, 64]


bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\x10\x00\x02\x00\x03\xc6\x82')
[1, 16, 0, 0, 0, 3, 6, 0, 16, 0, 2, 0, 3, 198, 130]


bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\x11\x00\x02\x00\x03\xfbB')
[1, 16, 0, 0, 0, 3, 6, 0, 17, 0, 2, 0, 3, 251, 66]


bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\x12\x00\x02\x00\x03\xbfB')
[1, 16, 0, 0, 0, 3, 6, 0, 18, 0, 2, 0, 3, 191, 66]

Peter.Kenyon
Posts: 70
Joined: Wed Oct 14, 2015 5:07 pm

Re: problem with bytearray conversion

Post by Peter.Kenyon » Mon Jan 04, 2016 1:07 pm

Well I agree it would be more consistent but note that
A is 65 decimal
B is 66 Decimal
@ is 64 decimal

Damien
Site Admin
Posts: 627
Joined: Mon Dec 09, 2013 5:02 pm

Re: problem with bytearray conversion

Post by Damien » Mon Jan 04, 2016 1:09 pm

There is no problem, it's just the way that bytes/bytearray objects are printed: Python tries to print printable characters (eg A-Z) and if it can't then it prints in hex format using \x.

If you want a nice print out of the bytes (or bytearray) then you can use binascii.hexlify:

Code: Select all

>>> binascii.hexlify(bytearray(b'\x01\x10\x00\x00\x00\x03\x06\x00\x02\x00\x02\x00\x03~\x81'))
b'011000000003060002000200037e81'

EasyRider
Posts: 94
Joined: Wed Dec 30, 2015 8:17 am

Re: problem with bytearray conversion

Post by EasyRider » Mon Jan 04, 2016 1:40 pm

Thanks Guys,

I am fairly new to Python and was stumped by the fact that bytearray worked OK in converting an Integer array to bytearray when writing to uart but not actually printing for debugging purposes.

Anyway, have found another option to convert and print correctly Integer array to hex string values.

print('\n'.join([hex(i) for i in message])), message being the Integer array.

Are there any other easier ways in MicroPython?

Regards
John

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: problem with bytearray conversion

Post by pfalcon » Mon Jan 04, 2016 4:57 pm

MicroPython follows Python in almost everything and the way to get hexadecimal string from byte strings is (mind the backtick character with hex code 0x60, decimal 96 which apparently confused you):

Code: Select all

>>> import binascii
>>> print(binascii.hexlify(bytearray(b"123`")))
b'31323360'
I recently also added an extension to CPython which allows to specify a separator char:

Code: Select all

>>> import binascii
>>> print(binascii.hexlify(bytearray(b"123`"), b":"))
b'31:32:33:60'
(That's fairly recent addition, ~ week, so probably not in your firmware, and once again, it's extension to standard Python, so use it cautiously). Otherwise, as you show yourself, Python is powerful expressive language, and it's a one-liner to do almost any conversion you like. E.g., extended form of binascii.hexlify() above can be easily replaced with (which also works in standard Python):

Code: Select all

print(":".join(["%02x" % a for a in bytearray(b"123`")]))
The benefit of binascii.hexlify() is memory efficiency (it uses as little memory as required to do conversion, and not a single byte more).
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

User avatar
dhylands
Posts: 3295
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: problem with bytearray conversion

Post by dhylands » Mon Jan 04, 2016 5:07 pm

I also put together a hex/ASCII dumper (ported from some C code I had).
https://github.com/dhylands/bioloid3/bl ... ump_mem.py

It also uses log.py from the same directory so that output can be redirected to say a uart.
https://github.com/dhylands/bioloid3/blob/master/log.py

Sample output:

Code: Select all

>>> from dump_mem import dump_mem
>>> b = bytearray(b'0123\x00\x01This is a test\nto see what happens\n')
>>> dump_mem(b, 'Buf')
Buf: 0000: 30 31 32 33 00 01 54 68 69 73 20 69 73 20 61 20 0123..This is a 
Buf: 0010: 74 65 73 74 0a 74 6f 20 73 65 65 20 77 68 61 74 test.to see what
Buf: 0020: 20 68 61 70 70 65 6e 73 0a                       happens.
I'll have to take a look at updating it to use hexlify

Post Reply