ubluetooth does not decode advertising payloads

Discussion about programs, libraries and tools that work with MicroPython. Mostly these are provided by a third party.
Target audience: All users and developers of MicroPython.
Post Reply
redyellow
Posts: 12
Joined: Tue Jun 30, 2020 10:19 am

ubluetooth does not decode advertising payloads

Post by redyellow » Fri Jul 03, 2020 6:12 am

I am using the BLE advertising helpers from GitHub in my central device scanning for peripherals.
However, services for advertising packets that have UUID16 are decoded.
For devices that have UUID128 services list are returned empty.

In the code below first the example payload is generated and decoded correctly.
Then I try to decode payloads that I captured during scanning.
The output is only a empty list.


Is this a bug or am doing something wrong?

Code: Select all

bytearray(b'\x02\x01\x06\x0c\tmicropython\x03\x03\x1a\x18\x11\x07\x9e\xca\xdc$\x0e\xe5\xa9\xe0\x93\xf3\xa3\xb5\x01\x00@n\x03\x19\x00\x00')
micropython
[UUID16(0x181a), UUID128('6e400001-b5a3-f393-e0a9-e50e24dcca9e')]
b'02010613ff4c000c0e083c0e80d244810d6c8dac99bf7e'

[]
b'02010613ff4c000c0e08440ec5aab68919bb7974850b98'

[]

Code: Select all

# Helpers for generating BLE advertising payloads.

from micropython import const
import struct
import bluetooth
import binascii

# Advertising payloads are repeated packets of the following form:
#   1 byte data length (N + 1)
#   1 byte type (see constants below)
#   N bytes type-specific data

_ADV_TYPE_FLAGS = const(0x01)
_ADV_TYPE_NAME = const(0x09)
_ADV_TYPE_UUID16_COMPLETE = const(0x3)
_ADV_TYPE_UUID32_COMPLETE = const(0x5)
_ADV_TYPE_UUID128_COMPLETE = const(0x7)
_ADV_TYPE_UUID16_MORE = const(0x2)
_ADV_TYPE_UUID32_MORE = const(0x4)
_ADV_TYPE_UUID128_MORE = const(0x6)
_ADV_TYPE_APPEARANCE = const(0x19)


# Generate a payload to be passed to gap_advertise(adv_data=...).
def advertising_payload(limited_disc=False, br_edr=False, name=None, services=None, appearance=0):
    payload = bytearray()

    def _append(adv_type, value):
        nonlocal payload
        payload += struct.pack("BB", len(value) + 1, adv_type) + value

    _append(
        _ADV_TYPE_FLAGS,
        struct.pack("B", (0x01 if limited_disc else 0x02) + (0x18 if br_edr else 0x04)),
    )

    if name:
        _append(_ADV_TYPE_NAME, name)

    if services:
        for uuid in services:
            b = bytes(uuid)
            if len(b) == 2:
                _append(_ADV_TYPE_UUID16_COMPLETE, b)
            elif len(b) == 4:
                _append(_ADV_TYPE_UUID32_COMPLETE, b)
            elif len(b) == 16:
                _append(_ADV_TYPE_UUID128_COMPLETE, b)

    # See org.bluetooth.characteristic.gap.appearance.xml
    _append(_ADV_TYPE_APPEARANCE, struct.pack("<h", appearance))

    return payload


def decode_field(payload, adv_type):
    i = 0
    result = []
    while i + 1 < len(payload):
        if payload[i + 1] == adv_type:
            result.append(payload[i + 2 : i + payload[i] + 1])
        i += 1 + payload[i]
    return result


def decode_name(payload):
    n = decode_field(payload, _ADV_TYPE_NAME)
    return str(n[0], "utf-8") if n else ""


def decode_services(payload):
    services = []
    for u in decode_field(payload, _ADV_TYPE_UUID16_COMPLETE):
        services.append(bluetooth.UUID(struct.unpack("<h", u)[0]))
    for u in decode_field(payload, _ADV_TYPE_UUID32_COMPLETE):
        services.append(bluetooth.UUID(struct.unpack("<d", u)[0]))
    for u in decode_field(payload, _ADV_TYPE_UUID128_COMPLETE):
        services.append(bluetooth.UUID(u))
    return services


def demo():
   #example Payload with UUID16 service
       payload = advertising_payload(
        name="micropython",
        services=[bluetooth.UUID(0x181A), bluetooth.UUID("6E400001-B5A3-F393-E0A9-E50E24DCCA9E")],
    )
    print(payload)
    print(decode_name(payload))
    print(decode_services(payload))
	
   #1 capture payload
    payload = bytearray(b'\x02\x01\x06\x13\xffL\x00\x0c\x0e\x08<\x0e\x80\xd2D\x81\rl\x8d\xac\x99\xbf~')
    print(binascii.hexlify(payload))
    print(decode_name(payload))
    print(decode_services(payload))
	
    #2 capture payload	
    payload = bytearray(b'\x02\x01\x06\x13\xffL\x00\x0c\x0e\x08D\x0e\xc5\xaa\xb6\x89\x19\xbbyt\x85\x0b\x98')
    print(binascii.hexlify(payload))
    print(decode_name(payload))
    print(decode_services(payload))


if __name__ == "__main__":
    demo()

EDIT: I am not sure if the UUID length is really causing this behaviour or if it is something else
Last edited by redyellow on Fri Jul 03, 2020 7:27 pm, edited 3 times in total.

redyellow
Posts: 12
Joined: Tue Jun 30, 2020 10:19 am

Re: ubluetooth does not decode advertising payloads

Post by redyellow » Fri Jul 03, 2020 3:34 pm

I looked into the bluetooth spec more closely and it looks like the recorded payloads did not advertise any service.
This is probably the reason why the services are empty.

However, that leaves me the question why my own bluetooth peripheral (Arduino Nano BLE) is not discovered by the central ( ESP32 running MicroPython) but discovered from nrf connect.

redyellow
Posts: 12
Joined: Tue Jun 30, 2020 10:19 am

Re: ubluetooth does not decode advertising payloads

Post by redyellow » Fri Jul 03, 2020 7:23 pm

Just for if anyone interested, the payload is structured as following (as stated in the ble_advertising example on GitHub:

Code: Select all

#   1 byte data length (N + 1)
#   1 byte type (see constants below)
#   N bytes type-specific data
So for the example payload
02,01,06,0c,09,6d,69,63,72,6f,70,79,74,68,6f,6e,03,03,1a,18,11,07,9e,ca,dc,24,0e,e5,a9,e0,93,f3,a3,b5,01,00,40,6e,03,19,00,00

contains the following structures:

02,01,06: Length 2 bytes, AD_TYPE, 6 (Bit 1 and 2) means LE General Discoverable Mode and BR/EDR is NOT supported.
0c,09,6d,69,63,72,6f,70,79,74,68,6f,6e: Length 12 bytes, AD_TYPE_NAME, "MicroPython"
03,03,1a,18: Length 3 bytes, AD_TYPE_16_COMPLETE, 181A UUID16 Service
11,07,9e,ca,dc,24,0e,e5,a9,e0,93,f3,a3,b5,01,00,40,6e: Length 17 bytes, AD_TYPE_128_COMPLETE, 6E400001-B5A3-F393-E0A9-E50E24DCCA9E UUID128 Service
03,19,00,00: Length 3 bytes, AD_TYPE_APPEARANCE: 0000

My captured (#1) payload 02,01,06,13,ff,4c,00,0c,0e,08,3c,0e,80,d2,44,81,0d,6c,8d,ac,99,bf,7e
02,01,06: Length 2 bytes, AD_TYPE, 6 (Bit 1 and 2) means LE General Discoverable Mode and BR/EDR is NOT supported.
13,ff,4c,00,0c,0e,08,3c,0e,80,d2,44,81,0d,6c,8d,ac,99,bf,7e: Length 19 bytes, AD_MANUFACTURER_SPECIFIC, some manufacturer specific data

So in consequence there is no service being advertised which means the service list is of course empty.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: ubluetooth does not decode advertising payloads

Post by jimmo » Sat Jul 04, 2020 6:27 am

Yep, that's exactly right. The advertising payload and the actual set of gatt services is completely independent.

The services listed in the advertising payload are just a hint to the scanner about what the device's capabilities are (to avoid needing to connect to a device to discover them fully).

Random trivial, on iOS, in order to scan while in the background, an app must specify a service uuid to detect in the advertising payload.

redyellow
Posts: 12
Joined: Tue Jun 30, 2020 10:19 am

Re: ubluetooth does not decode advertising payloads

Post by redyellow » Sat Jul 04, 2020 7:14 pm

I was able to track down further why the device I am looking for is not showing up as scan result with the standard ble_advertising helper on GitHub.

My initial suspicion was actually right. The device advertises with ADV_TYPE 6 which means incomplete UUID128 list of services.
The payload is as following:

Code: Select all

02,01,06,11,06,a6,7c,eb,91,f7,86,92,8f,1b,4b,b0,bf,1f,1b,b8,32
As the ble_advertising helper does not process ADV_TYPE 6 no services are returned.
I have tried to change it now but still no services are returned.

This how i changed the decode_services method by adding for processing UUID128_MORE (ADV_TYPE 6).

Code: Select all

# Helpers for generating BLE advertising payloads.

from micropython import const
import struct
import bluetooth
import binascii

# Advertising payloads are repeated packets of the following form:
#   1 byte data length (N + 1)
#   1 byte type (see constants below)
#   N bytes type-specific data

_ADV_TYPE_FLAGS = const(0x01)
_ADV_TYPE_NAME = const(0x09)
_ADV_TYPE_UUID16_COMPLETE = const(0x3)
_ADV_TYPE_UUID32_COMPLETE = const(0x5)
_ADV_TYPE_UUID128_COMPLETE = const(0x7)
_ADV_TYPE_UUID16_MORE = const(0x2)
_ADV_TYPE_UUID32_MORE = const(0x4)
_ADV_TYPE_UUID128_MORE = const(0x6)
_ADV_TYPE_APPEARANCE = const(0x19)


# Generate a payload to be passed to gap_advertise(adv_data=...).
def advertising_payload(limited_disc=False, br_edr=False, name=None, services=None, appearance=0):
    payload = bytearray()

    def _append(adv_type, value):
        nonlocal payload
        payload += struct.pack("BB", len(value) + 1, adv_type) + value

    _append(
        _ADV_TYPE_FLAGS,
        struct.pack("B", (0x01 if limited_disc else 0x02) + (0x18 if br_edr else 0x04)),
    )#



    if name:
        _append(_ADV_TYPE_NAME, name)

    if services:
        for uuid in services:
            b = bytes(uuid)
            if len(b) == 2:
                _append(_ADV_TYPE_UUID16_COMPLETE, b)
            elif len(b) == 4:
                _append(_ADV_TYPE_UUID32_COMPLETE, b)
            elif len(b) == 16:
                _append(_ADV_TYPE_UUID128_COMPLETE, b)

    # See org.bluetooth.characteristic.gap.appearance.xml
    _append(_ADV_TYPE_APPEARANCE, struct.pack("<h", appearance))

    return payload


def decode_field(payload, adv_type):
    i = 0
    result = []
    while i + 1 < len(payload):
        if payload[i + 1] == adv_type:
            result.append(payload[i + 2 : i + payload[i] + 1])
        i += 1 + payload[i]
    return result


def decode_name(payload):
    n = decode_field(payload, _ADV_TYPE_NAME)
    return str(n[0], "utf-8") if n else ""


def decode_services(payload):
    services = []
    for u in decode_field(payload, _ADV_TYPE_UUID16_COMPLETE):
        services.append(bluetooth.UUID(struct.unpack("<h", u)[0]))
    for u in decode_field(payload, _ADV_TYPE_UUID32_COMPLETE):
        services.append(bluetooth.UUID(struct.unpack("<d", u)[0]))
    for u in decode_field(payload, _ADV_TYPE_UUID128_COMPLETE):
        services.append(bluetooth.UUID(u))
    for u in decode_field(payload, _ADV_TYPE_UUID128_MORE):
        print("UUID128_MORE")
        services.append(bluetooth.UUID(u))    
    return services


def demo():
    payload = advertising_payload(
        name="micropython",
        services=[bluetooth.UUID(0x181A), bluetooth.UUID("6E400001-B5A3-F393-E0A9-E50E24DCCA9E")],
    )
    print(payload)
    print(decode_name(payload))
    print(decode_services(payload))

    payload = b'\x02\x01\x06\x11\x06\xa6|\xeb\x91\xf7\x86\x92\x8f\x1bK\xb0\xbf\x1f\x1b\xb82'
    print(binascii.hexlify(payload))
    print(decode_name(payload))
    print(decode_services(payload))

if __name__ == "__main__":
    demo()

Interestingly if I run the helper standalone with my payload is properly decoded.

Output from ble_advertising helper

Code: Select all

b'0201061106a67ceb91f786928f1b4bb0bf1f1bb832'

UUID128_MORE
[UUID128('32b81b1f-bfb0-4b1b-8f92-86f791eb7ca6')]
If I run it as a part of the central this is not the case...
The handler of the central looks like this

Code: Select all

def _irq(self, event, data):
        #print("IRQ: "+ str(event))

        if event == _IRQ_SCAN_RESULT:
            addr_type, addr, adv_type, rssi, adv_data = data
            print("----------")
            print("Address: " + str(binascii.hexlify(addr)))           
            print("ADV RAW: " +str(adv_data))
            print("ADV NAME: " + str(decode_name(adv_data)))   
            print("ADV Services: " + str(decode_services(adv_data)))
          
            if adv_type in (_ADV_IND, _ADV_DIRECT_IND,) and _MY_SERVICE in decode_services(
                adv_data
            ):
                # Found a potential device, remember it and stop scanning.
                self._addr_type = addr_type
                self._addr = bytes(
                    addr
                )  # Note: addr buffer is owned by caller so need to copy it.
                self._name = decode_name(adv_data) or "?"
                self._ble.gap_scan(None)

        elif event == _IRQ_SCAN_DONE:
            if self._scan_callback:
                if self._addr:
                    # Found a device during the scan (and the scan was explicitly stopped).
                    self._scan_callback(self._addr_type, self._addr, self._name)
                    self._scan_callback = None
                else:
                    # Scan timed out.
                    self._scan_callback(None, None, None)

        elif event == _IRQ_PERIPHERAL_CONNECT:
            # Connect successful.
            conn_handle, addr_type, addr, = data
            if addr_type == self._addr_type and addr == self._addr:
                self._conn_handle = conn_handle
                self._ble.gattc_discover_services(self._conn_handle)

        elif event == _IRQ_PERIPHERAL_DISCONNECT:
            # Disconnect (either initiated by us or the remote end).
            conn_handle, _, _, = data
            if conn_handle == self._conn_handle:
                # If it was initiated by us, it'll already be reset.
                self._reset()

        elif event == _IRQ_GATTC_SERVICE_RESULT:
            # Connected device returned a service.
            conn_handle, start_handle, end_handle, uuid = data
            if conn_handle == self._conn_handle and uuid == _MY_SERVICE:
                self._start_handle, self._end_handle = start_handle, end_handle

        elif event == _IRQ_GATTC_SERVICE_DONE:
            # Service query complete.
            if self._start_handle and self._end_handle:
                self._ble.gattc_discover_characteristics(
                    self._conn_handle, self._start_handle, self._end_handle
                )
            else:
                print("Failed to find environmental sensing service.")

        elif event == _IRQ_GATTC_CHARACTERISTIC_RESULT:
            # Connected device returned a characteristic.
            conn_handle, def_handle, value_handle, properties, uuid = data
            if conn_handle == self._conn_handle:
                self._value_handle.add(value_handle)

        elif event == _IRQ_GATTC_CHARACTERISTIC_DONE:
            # Characteristic query complete.
            if self._value_handle:
                # We've finished connecting and discovering device, fire the connect callback.
                if self._conn_callback:
                    self._conn_callback()
            else:
                print("Failed to find temperature characteristic.")

        elif event == _IRQ_GATTC_READ_RESULT:
            # A read completed successfully.
            conn_handle, value_handle, char_data = data
            if conn_handle == self._conn_handle and value_handle == self._value_handle:
                self._update_value(char_data)
                if self._read_callback:
                    self._read_callback(self._value)
                    self._read_callback = None

        elif event == _IRQ_GATTC_READ_DONE:
            # Read completed (no-op).
            print(len(data))
            conn_handle, value_handle, status, value1 , value2 = data
            print("Connection handle: " + str(conn_handle))
            print("Value handle: " + str(value_handle))
            print("Status: " + str(status))
            print("Value 1: " + str(value1))
            print("Value 2: " + str(value2))

        elif event == _IRQ_GATTC_NOTIFY:
            # extract connection handle, value_handle and data and process in callback
            conn_handle, value_handle, notify_data = data
            if conn_handle == self._conn_handle and value_handle == self._value_handle:
                self._update_value(notify_data)
                if self._notify_callback:
                    self._notify_callback(self._value)
What am I doing wrong that no services are returned?
Console Output when running the central:

Code: Select all

Address: b'ee62112eba41'
ADV RAW: b'\x02\x01\x06\x11\x06\xa6|\xeb\x91\xf7\x86\x92\x8f\x1bK\xb0\xbf\x1f\x1b\xb82'
ADV NAME: 
ADV Services: []

Additional question: What is the best way to debug? Logging library is not included in MicroPython, is it? If I use print statements in the ble_advertising helper oddly these are not shown on the console/serial output.

redyellow
Posts: 12
Joined: Tue Jun 30, 2020 10:19 am

Re: ubluetooth does not decode advertising payloads

Post by redyellow » Sun Jul 05, 2020 3:32 pm

I have done some further tests now by including the methods decode_services, decode_field, decode_name and the constants of the ADV_TYPES into my central device class directly I.e. not imported from ble_advertising.

This is working now i.e. services for ADV_TYPE UUID128_MORE are no properly decoded.

Honestly I am puzzled now. This must be some python/micropython thing I am not aware of. Can someone enlighten me why this is happening?

Post Reply