Can I specify the number of bytes to read into an over-sized buffer via SPI?
Can I specify the number of bytes to read into an over-sized buffer via SPI?
I'm performing an SPI read within an ISR. The report lengths can vary quite a bit, depending on which signals the sensor is reporting and how many it might be bundling together in one report. The reports come in at roughly 200Hz. Each SPI transmission starts with a 4-byte header that includes the total number of bytes in the report payload that follows. I'd like to read this header and only read the necessary number of bytes into a buffer, as I cannot allocate a tailored buffer within the ISR. I can set the read buffer to the largest possible report size, but that burns a lot of time reading nothing. I tried passing a memoryview to the read call, but the system did not seem to like that.
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: Can I specify the number of bytes to read into an over-sized buffer via SPI?
I think you need two buffers, one (for the header) of 4 bytes and the other of the maximum size a message can be. Do an SPI read of the header, then immediately do a read of the number of bytes specified in the header.
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
Re: Can I specify the number of bytes to read into an over-sized buffer via SPI?
This is the kicker. I can't figure out how to read only the number of bytes specified in the header. The spi.readinto call will keep reading to fill the buffer (which results in filling out the unnecessary part at the end with zeros). By using the largest possible buffer, I'm always taking the time to fill it, even if the actual data transmission is much shorter.
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: Can I specify the number of bytes to read into an over-sized buffer via SPI?
If your header buf is 4 bytes long, .readinto will read exactly that number. Having interpreted the contents as a number N to read, you then have two options. You could then use .read(N) to read that number, but this has the drawback of allocating a buffer each time. The other approach is to use a memoryview into your buffer. This means you should be able to issue .readinto(mv[:N]) where mv is the memoryview. A memoryview allows you to create a slice into a buffer without copying.
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
Re: Can I specify the number of bytes to read into an over-sized buffer via SPI?
The .readinto(mv[0:N]) generates a heap allocation exception. Here's how I set it up:
In __init__():
and in the isr():
In __init__():
Code: Select all
self.header_buffer = bytearray(self._HEADER_BUFFER_SIZE) # (4 bytes)
self.cargo_buffer = bytearray(self._CARGO_BUFFER_SIZE)
self.cargo_memoryview = memoryview(self.cargo_buffer)
self.number_of_bytes = 0
Code: Select all
self.spi.readinto(self.header_buffer)
self.number_of_bytes = (self.header_buffer[1] << 8) + self.header_buffer[0] - 4 # subtracting 4 bytes for the header that's already been read
self.spi.readinto(self.cargo_memoryview[:self.number_of_bytes])
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: Can I specify the number of bytes to read into an over-sized buffer via SPI?
I have replicated this on a Pyboard. I'm surprised that an allocation is occurring. The docs would lead you to believe that it should not occur. I'll raise an issue on GitHub and we'll see what the maintainers think.
As a workround you could use micropython.schedule but maybe that would be too slow.
As a workround you could use micropython.schedule but maybe that would be too slow.
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: Can I specify the number of bytes to read into an over-sized buffer via SPI?
I have raised this issue.
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
Re: Can I specify the number of bytes to read into an over-sized buffer via SPI?
Thanks much for creating the issue.
Indeed, we're streaming data too frequently for micropython.schedule(). As a workaround, I was able to pre-allocate buffers for all the potential buffer sizes (it turns out there are only a handful, but that might go up in the future), and use the buffer that matches the size in the header.
Indeed, we're streaming data too frequently for micropython.schedule(). As a workaround, I was able to pre-allocate buffers for all the potential buffer sizes (it turns out there are only a handful, but that might go up in the future), and use the buffer that matches the size in the header.
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: Can I specify the number of bytes to read into an over-sized buffer via SPI?
It turns out that slicing the memoryview instantiates another memoryview. Hence, even though the data is not copied, an allocation occurs. There is no really satisfactory solution.
I was therefore going to suggest something akin to your solution, but pre-creating a number of memoryviews into a single buffer to save RAM:
I was therefore going to suggest something akin to your solution, but pre-creating a number of memoryviews into a single buffer to save RAM:
Code: Select all
buf = bytearray(256)
mv = memoryview(buf)
mv16 = mv[:16]
mv32 = mv[:32]
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
Re: Can I specify the number of bytes to read into an over-sized buffer via SPI?
That's a good idea. Thanks.
For this particular case, it would be sufficient if an optional integer could be passed to spi.readinto() to specify the number of bytes to read.
For this particular case, it would be sufficient if an optional integer could be passed to spi.readinto() to specify the number of bytes to read.