Can I specify the number of bytes to read into an over-sized buffer via SPI?

[jim]

I'm performing an SPI read within an ISR. The report lengths can vary quite a bit, depending on which signals the sensor is reporting and how many it might be bundling together in one report. The reports come in at roughly 200Hz. Each SPI transmission starts with a 4-byte header that includes the total number of bytes in the report payload that follows. I'd like to read this header and only read the necessary number of bytes into a buffer, as I cannot allocate a tailored buffer within the ISR. I can set the read buffer to the largest possible report size, but that burns a lot of time reading nothing. I tried passing a memoryview to the read call, but the system did not seem to like that.

[pythoncoder]

I think you need two buffers, one (for the header) of 4 bytes and the other of the maximum size a message can be. Do an SPI read of the header, then immediately do a read of the number of bytes specified in the header.

[jim]

This is the kicker. I can't figure out how to read only the number of bytes specified in the header. The spi.readinto call will keep reading to fill the buffer (which results in filling out the unnecessary part at the end with zeros). By using the largest possible buffer, I'm always taking the time to fill it, even if the actual data transmission is much shorter.

[pythoncoder]

If your header buf is 4 bytes long, .readinto will read exactly that number. Having interpreted the contents as a number N to read, you then have two options. You could then use .read(N) to read that number, but this has the drawback of allocating a buffer each time. The other approach is to use a memoryview into your buffer. This means you should be able to issue .readinto(mv[:N]) where mv is the memoryview. A memoryview allows you to create a slice into a buffer without copying.

[jim]

The .readinto(mv[0:N]) generates a heap allocation exception. Here's how I set it up:

In __init__():

Code: Select all

self.header_buffer = bytearray(self._HEADER_BUFFER_SIZE)	# (4 bytes)
self.cargo_buffer = bytearray(self._CARGO_BUFFER_SIZE)
self.cargo_memoryview = memoryview(self.cargo_buffer)
self.number_of_bytes = 0

and in the isr():

Code: Select all

self.spi.readinto(self.header_buffer)
self.number_of_bytes = (self.header_buffer[1] << 8) + self.header_buffer[0] - 4  # subtracting 4 bytes for the header that's already been read
self.spi.readinto(self.cargo_memoryview[:self.number_of_bytes])

[pythoncoder]

I have replicated this on a Pyboard. I'm surprised that an allocation is occurring. The docs would lead you to believe that it should not occur. I'll raise an issue on GitHub and we'll see what the maintainers think.

As a workround you could use micropython.schedule but maybe that would be too slow.

[pythoncoder]

I have raised this issue.

[jim]

Thanks much for creating the issue.

Indeed, we're streaming data too frequently for micropython.schedule(). As a workaround, I was able to pre-allocate buffers for all the potential buffer sizes (it turns out there are only a handful, but that might go up in the future), and use the buffer that matches the size in the header.

[pythoncoder]

It turns out that slicing the memoryview instantiates another memoryview. Hence, even though the data is not copied, an allocation occurs. There is no really satisfactory solution.

I was therefore going to suggest something akin to your solution, but pre-creating a number of memoryviews into a single buffer to save RAM:

Code: Select all

buf = bytearray(256)
mv = memoryview(buf)
mv16 = mv[:16]
mv32 = mv[:32]

[jim]

That's a good idea. Thanks.

For this particular case, it would be sufficient if an optional integer could be passed to spi.readinto() to specify the number of bytes to read.