MemoryView, uctypes.struct, array or other built-in Data Structure... Which is quickest?

[MicroP_W691]

Hi all,

I am working with CAN-Bus Data coming in at over 100hz (in bursts) and need to store all the messages as quickly as possible so that the next message can be read from the CAN Chip Buffer. But which way is the quickest way to store this data in micropython?

For those who haven't worked with CAN before, it comes as a series of 8-byte messages with specific IDs, so I will need to quickly store 8 bytes at a time.

These are my thoughts so far (based on my limited experience with micropython):
1) Built-in Data Types:

• List: Obviously appending to a list every time is slow as the memory has to be allocated but if the list was created beforehand with null
  values then filled like a buffer might not be as slow?
• Bytearray: The obvious choice to write to, but is it quick enough in micropython?
• Default_dict: Creating a default dictionary with byte arrays for each value, and writing each 8 Byte message into the byte array to be
  processed later (Not the best Idea I know...)

2) MemoryView Objects:

• I trialled memory view objects by 'pointing' them at a bytearray and it seemed to work well
• I couldn't use the slice notation which means I would have to copyt everything out of the buffer foirst before working on it

  Code: Select all

  >>> example_bytearray = bytearray(64)
  >>> mem_view_obj = memory_view(example_bytearray)
  >>> mem_view_obj = memoryview(example_bytearray)
  >>> print(mem_view_obj[8:16])
  <memoryview>
  >>> print(example_bytearray[8:16])
  bytearray(b'\x00\x00\x00\x00\x00\x00\x00\x00')
  >>>

3) Arrays:

• I haven't worked with arrays much in Python so are they quick to write to?
• What are their limitations in micropython?

4) uctypes.struct

• I am aware that this module was designed to interface with functions or libraries written in C, but if it offers the ability to create c-type structures does that mean it offers a similar speed?
• How difficult would it be to implement?

I am of course aware that the quickest way would just be to do it all in C, but most of the functionality written for parsing and sorting the other CAN-Bus messages has been written in micropython, so if possible I would like to keep it all together, rather than having the micropython listen for certain CAN-Bus messages, and a C Module listen for other messages...

I appreciate this is a bit of an abstract question but was hoping those with a bit more experience using micropython had come into a similar problem before and may already have worked out the quickest way of doing it!

[pythoncoder]

A pre-allocated array or bytearray is fast as elements occupy contiguous memory locations. They are as good as it gets in MicroPython.

The purpose of a memoryview is to provide allocation-free slicing. To send elements [4:8] of a bytearray to a function, the following would cause allocation:

Code: Select all

a = bytearray(range(20))
def func(arr):
    for x in arr:
        print(x)
func(a[4:8]) # allocates a new 4 element bytearray

On the other hand

Code: Select all

a = bytearray(20)
b = memoryview(a)
def func(arr):
    for x in arr:
        print(x)
func(b[4:8])  # does not allocate but passes a "window" into a

In both cases func() can iterate through the passed object or access individual elements.

[jimmo]

I couldn't use the slice notation which means I would have to copyt everything out of the buffer foirst before working on it

If I understand here the issue is just that print() doesn't print the contents, rather it prints <memoryview>. (This doesn't mean that slicing isn't working).

Can you explain what you mean by "so I will need to quickly store 8 bytes at a time.". If your goal is to concatenate a bunch of 8-byte sequences together, then pre-allocating a large bytearray and using the example from the docs would work:

Code: Select all

N = 100
buf = bytearray(8 * N)
m = memoryview(buf)
lst = [0, 0, 0, None]

for i in range(N):
  lst[3] = m[i * 8:i*8+8]
  can.recv(0, lst)

Something else to be aware of is that slicing assignment does something sensible too.

Code: Select all

>>> b = bytearray(range(20))
>>> b
bytearray(b'\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13')
>>> b[3:6] = b'abc'
>>> b
bytearray(b'\x00\x01\x02abc\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13')
>>> 

[MicroP_W691]

Thank you for both your replies @pythoncoder and @jimmo, it looks like a bytearray is going to be the best option. I

I see my mistake with the memory view object and using the 'print' function (thanks @jimmo), when using the method suggested by @pythoncoder it worked as expected. However, the memory view is only for viewing the data and can't be used to write to it, is that correct? When I try to write through the memory view object, I get this error:

Code: Select all

m = memoryview(buf)
m[0:8] = [1,2,3,4,5,6,7,8]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'memoryview' object doesn't support item assignment

I only planned on using the memory view object as it seems to not need to allocate any more memory after being created, so maybe I will try to write to the bytearray and read out through the memory view object. The data comes in so fast (40 CAN channels, most transmitting at 100hz) that I am struggling to process them in time, resulting in a buffer overflow on the CAN chip (and therefore lost messages), which is why I wanted to dump them quickly (8 bytes at a time) and then process them only when I need them.

I may have to look at storing and processing the channels in C though, then passing that data to the micropython application, as I don't think micropython will be quick enough for this task.