bits to bytes

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
martincho
Posts: 96
Joined: Mon May 16, 2022 9:59 pm

Re: bits to bytes

Post by martincho » Thu Jun 30, 2022 7:13 am

jimmo wrote:
Thu Jun 30, 2022 4:53 am
"you should be able to make your eyes only read the green text and still know exactly what the code does and why"
Could not agree more.

KJM
Posts: 158
Joined: Sun Nov 18, 2018 10:53 pm
Location: Sydney AU

Re: bits to bytes

Post by KJM » Wed Jul 06, 2022 8:56 am

After reading the learned comments I thought I had a handle on bytes & the different ways python displays them. But I still managed to fall at the first hurdle.

If I do

Code: Select all

bis='1111011111111010'; d=int(bis, 2); r=hex(d); print(r, repr(r))
then send r to the lora modem all good

Code: Select all

0xf7fa '0xf7fa'
but if I try

Code: Select all

r=0xf7fa; print(r, repr(r))
then try to send r to the lora modem I get

Code: Select all

63482 63482
TypeError: object with buffer protocol required
Why is the r I entered directly different to the one I constructed from the bit string?

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: bits to bytes

Post by Roberthh » Wed Jul 06, 2022 10:05 am

You'll see the difference by trying type(r) for both variants.
For r=0xf7fy, the type of r is <class 'int'>, for r=hex(d), the type of r is <class 'str'>. And str is an object with a buffer protocol, int is not.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: bits to bytes

Post by jimmo » Wed Jul 06, 2022 12:26 pm

KJM wrote:
Wed Jul 06, 2022 8:56 am
Why is the r I entered directly different to the one I constructed from the bit string?
As Robert said, the first is a string (the result of calling `hex()`), the second is an integer (an integer literal).

I suspect _neither_ of these are what you want though if your goal is to send a 16-bit (i.e. two byte) payload over the radio.

i.e. if you literally want to send a constructed sequence of bits, such that you end up transmitting 0b11110111 (247 decimal, 0xf7 hex) followed by 0b11111010 (250 decimal, 0xfa hex).

Code: Select all

data = bytes((0b11110111, 0b11111010,))
radio.send(data)

martincho
Posts: 96
Joined: Mon May 16, 2022 9:59 pm

Re: bits to bytes

Post by martincho » Wed Jul 06, 2022 3:50 pm

KJM wrote:
Wed Jul 06, 2022 8:56 am
After reading the learned comments I thought I had a handle on bytes & the different ways python displays them. But I still managed to fall at the first hurdle.
When talking to the outside world, the only data types you want to use are bytes and bytearrays.

I know it sounds simple. Once you make that a mental rule everything becomes much simpler. In other words, you can do whatever you want in between, yet, once you come into contact with a .read() or .write() method you will get or have to provide bytes or bytearray types.

I second Robert's suggestion to check data types with type(). This will clarify things over time. As I said, I don't enjoy/agree with the way 8 bit types have been engineered into Python 3. However, this isn't going to change, and so the only option is to understand how to live with what we have.

As a useful exercise, I would suggest converting bunch of examples of data types to and from bytes (or bytearrays). Here are a few ideas with some of them showing potential conversion solutions:

Code: Select all

print("\nNumbers range limited to 0 to 255")
list1 = [1, 2, 3, 4]
print(f"source: {list1}")

# To bytes
b_list1 = bytes(list1)
print(f"bytes: {b_list1}")

# Back to original form
o_list1 = list(b_list1)
print(f"back to original form: {o_list1}")


list2 = [1000, 5, 6, 7]   # List with numbers not range limited to 0 to 255 (this can be managed in many ways)

tuple1 = (1, 2, 3, 4)     # Numbers range limited to 0 to 255
typle2 = (1000, 5, 6, 7)  # Numbers not range limited to 0 to 255

list_16_bit = (4, 210)    # A 16 bit number broken into two 8-bit numbers in the 0 to 256 range

string1 = 'abcdefg'       # String we want to transmit
string2 = '1, 2, 3, 4'    # String that contains numbers we want to transmit as bytes, not ascii characters

int1 = 128                # Integer in the 0 to 255 range we want to transmit
int2 = 12300              # Integer in that fits into a 16 bit range we want to transmit
int3 = 0x80               # Entered as as hex value, still an integer in the 0 to 255 range

bin1 = 0b1000_0000              # Binary numbers in 0 to 255 range; just an integer (separator is optional)
bin2 = 0b1000_0000_0000_0000    # 16 bits
bin3 = int('10000000', 2)       # Another way to enter something in binary (separators work here just as well)

# Lists or tuples from binary representations in a string with a separator
binaries1 = "0000, 00001, 0010, 0100, 0101, 0110, 0111"
binaries2 = "0000.00001.0010.0100.0101.0110.0111"
bytes_from_binaries1 = bytes([int(n, 2) for n in binaries1.split(",")])
bytes_from_binaries2 = bytes((int(n, 2) for n in binaries2.split(".")))  # Using a tuple comprehension


print("\nA hypothetical packet stored in a dictionary")
d1 = {"start_of_packet": 0x02,
      "length": 5,
      "payload": (1, 2, 3, 4, 5),
      "crc16": 11022
      }

data_to_send = bytes([d1["start_of_packet"], d1["length"]] + list(d1["payload"]) + [int(d1["crc16"]/256), d1["crc16"]%256])
print(d1)
print(data_to_send)
Note: There are better ways to do some of this. I just threw this together real quick in hopes it might help.

KJM
Posts: 158
Joined: Sun Nov 18, 2018 10:53 pm
Location: Sydney AU

Re: bits to bytes

Post by KJM » Thu Jul 07, 2022 12:44 am

I hear you martincho but the lora radio won't let me use bytes, it has to be a buffer protocol. So the only way I can compress my 16 contact closures into a buffer protocol entity is to convert them to type string with hex and send that 2 byte string.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: bits to bytes

Post by jimmo » Thu Jul 07, 2022 12:48 am

I think the most important thing to learn about when doing networking code from Python is struct.pack.

See https://docs.python.org/3.5/library/str ... ule-struct and https://docs.micropython.org/en/latest/ ... truct.html

Don't worry about the underlying bytes, just pack them and unpack them according to the payload layout you define.

e.g. if you want to send an unsigned 16-bit value followed by a single byte over the network in big endian:

Code: Select all

msg = struct.pack('>HB', 0xabcd, 100)
radio.send(msg)
If you receive the same payload and want to convert it back

Code: Select all

msg = radio.receive()
a, b = struct.unpack('>HB', msg)

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: bits to bytes

Post by jimmo » Thu Jul 07, 2022 12:50 am

KJM wrote:
Thu Jul 07, 2022 12:44 am
I hear you martincho but the lora radio won't let me use bytes, it has to be a buffer protocol. So the only way I can compress my 16 contact closures into a buffer protocol entity is to convert them to type string with hex and send that 2 byte string.
Sorry I don't understand what you mean by this. Can you link to the lora library you're using?

The bytes and bytearray types implement the buffer protocol. As long as you can construct a bytes object containing your data you should be fine.

There's no way that "hex()" should be part of this process. It generates a string with "0x" on the front.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: bits to bytes

Post by jimmo » Thu Jul 07, 2022 12:53 am

In more detail. If you have sixteen contact closures to send, then you should make a single 16-bit integer which is the bitwise OR of the various contacts.

Code: Select all

v = 0
for i in range(16):
  v |= 1 if contacts[i].closed() else 0
  v <<= 1
then convert that integer into a two-byte payload in network (big-endian) order.

Code: Select all

msg = struct.pack('>H', v)
radio.send(msg)
There should be no strings, bin(), hex(), etc.

KJM
Posts: 158
Joined: Sun Nov 18, 2018 10:53 pm
Location: Sydney AU

Re: bits to bytes

Post by KJM » Thu Jul 07, 2022 2:00 am

We've come full circle now, struct.pack was where I started. With struct 16 contacts eg 1111 0111 1111 1010 converts to b'\xf7\xf0' <class 'bytes'> and takes 921 ms to send. But if I use the chr representation 1111 0111 1111 1010 converts to  <class 'str'> and takes 828 ms to send. I'm using a pycom lopy4. I don't know much about the lora lib in it, their socket interface to the lora chip keeps me insulated from it.

Post Reply