Could ujson.loads be improved faster?

[bhcuong2008]

Hi,
I test ujson.loads, then found it takes about 2-3ms in my cases. Here are my tests. It seems to be load time depending on input length rather than nested dictionary. Could it be improved under 1ms?

Code: Select all

def test_json(d):
   t0 = time.ticks_us()
   ret = json.loads(d)
   t1 = time.ticks_us()
   print('dt = %sms' % ((t1-t0)/1000,))
   return ret

>>> 
>>> d
'{"a": 2990, "b": {"c": 2, "c2": [[10,100], 1]}, "d": ["OFF", {"foo": "ON", "foo2": "OFF"}]}'
>>> test_json(d)
dt = 2.593ms
{'a': 2990, 'd': ['OFF', {'foo2': 'OFF', 'foo': 'ON'}], 'b': {'c': 2, 'c2': [[10, 100], 1]}}

>>> 
>>> d2
'{"battery":93,"battery_low":false,"contact":false,"linkquality":127,"tamper":false,"voltage":2900}'
>>> test_json(d2)
dt = 2.769ms
{'linkquality': 127, 'contact': False, 'battery': 93, 'battery_low': False, 'tamper': False, 'voltage': 2900}
>>>

Thanks

[pythoncoder]

The JSON protocol is designed for human readability rather than performance. There are other methods of data serialisation designed for performance. See the ustruct module. The binary format means that the ease of use of JSON is lost.

[bhcuong2008]

Hi pythoncoder,

Yes, I know ustruct. My need originates from receiving data from external devices / human through protocols as MQTT, websockets,...

When receiving their data, I parse it to store data as dictionary. This parsing process requires to use ujson to convert json string to dict for further manipulation.

I dont know how to do it in other ways or how to make it faster.

[stijn]

The parsing is written in C and while it might be optimised in size instead of performance, speeding it up would be a substantial effort. In other words: from the MicroPython developers there's probably no incentive to make this faster so if you want that, and you cannot change the data, there's not many options: do it yourself or get a faster device.

However since you mention MQTT and websockets I'm not sure what you are going to gain by speeding up JSON: any of those protocols have an inherent latencty to them which can be much more than this 2mSec you talk about.

[bhcuong2008]

@stijn, I see it. I must accept that truth. I could not change any code of ujson.

I got MQTT data from main app Arduino. I try to optimize code, do @native. And here is result I got.
For Zigbee contact sensor, its parsing time is ~5.5ms.
For Zigbee switch, its parsing time is ~7ms.

If I could reduce dict transformation time, its time will be reduced more. So my app could process more external devices.

By the way, do u know how to integrate MessagePack into MPY. I think MessagePack will be faster.
https://github.com/msgpack/msgpack-c

Thank you.

[stijn]

So my app could process more external devices.

Ah, right, then it might make sense to reduce processing time for serialization.

To get the C implementation in MicroPython you have to write a custom module. Too much to explain here, this should get you started: http://docs.micropython.org/en/latest/d ... dules.html

Or you could try the msgpack Python implementation which works with only minimal changes: https://github.com/micropython/micropython/issues/4241, but it's not going be faster than JSON probably.

[bhcuong2008]

@stijn, Thank you so much.

[pythoncoder]

Another possibility is protocol buffers, for which there is a "micro" implementation. This is discussed here.

[bhcuong2008]

Thank you for your guide. I will consult your docs.