Transmit data from file via I2C

[blmorris]

I'm trying to figure out the best way to read data from a file to transmit via I2C.
I am running Micro Python on a board with an audio DSP chip (SigmaDSP series from Analog Devices.) DSP applications are developed for the SigmaDSP chip using the Sigma Studio IDE provided by ADI. The IDE compiles the application into a binary file that can be written to an EEPROM or else exported as text file (human-readable Hex format) that can be used by a micro controller.
Assuming that the DSP program file is already on the uPy filesystem, the simple way to solve the problem is to create a buffer object (bytes or byte array) and parse the program file into that buffer object and feed that to the I2C object to transmit. Given that the data already exists on the Flash filesystem, I am wondering if it is possible to create a buffer object that parses the data from the program file and feeds it directly to the I2C object without copying the entire DSP program into RAM as an intermediate step - the DSP program can be up to 8KB or so in size, so I would like to not waste RAM unnecessarily.
I'm a relative novice with Python; if it seems that my problem has an obvious answer then that is the answer I'm looking for
Thanks,
Bryan

[blmorris]

I realized that I can reformulate this as a more generic Python / uPy question.
The various serial interfaces provided by the pyb module (I2C, SPI, UART) all provide methods to transmit data from a buffer object, usually a bytes or bytearray object. These methods can be used to transmit blocks of data of indefinite length, really only limited by the size of the buffer object that can be held in RAM. These large data transfers can be useful for initializing peripherals that may be running substantial programs themselves; in my case, I need to initialize an audio DSP chip with up to 8.5KB of program + parameter data, which could potentially be done in a single data transfer.
The simple way to do this is to store the DSP program on the flash filesystem; at startup the data is read from the file into a buffer object which stores the entire block in RAM to send via I2C. 8.5KB isn't really that much when we have 192KB of RAM to work with, and it only needs to be done once, at startup, after which the RAM is presumably recovered by the garbage collector if it is needed. Still, I'm curious if there is a better way, perhaps by creating a new kind of buffer object (a new class written in Python?) which can stream the data from the file to a serial transmit method (as a single transfer) without requiring all of the data to be resident in RAM at once.
Any input is appreciated, as I truly have no idea if this is a really simple problem, a really hard one, or even just the wrong problem to try to solve. I'm coming to this as a hardware designer currently dependent on a colleague to develop code for my projects in C. Micro Python gives me the opportunity to develop some new ideas on our hardware, even though I'm still pretty much a novice in Python as well.
Thanks!
Bryan

[pfalcon]

I can tell you the Pythonic way to do it:

Code: Select all

import shutil
file = open("file", "rb")
i2c_device = i2c.open(address, other_params...)
shutil.copyfileobj(file, i2c_device, buffer_length)

The fact that it's not possible now to do it in Pythonic way is the reason why we have https://github.com/micropython/micropython/issues/588 open, where current "pyb" module interface is criticized. Note that there's good reason why currently "pyb" does it in non-pythonic way - it's efficiency. And surely, you can still trivially do the transfer you need, it's will be just whole ~4 lines, but Python reduces that triviality to 1 function call.

[blmorris]

Thanks for the reply. If I understand you correctly, the reason that I can't currently do it in the Pythonic way is that the current I2C and other serial device classes don't present a file-like interface.
That works well enough for me. As much as I would like to develop good Python programming habits, for now it is easy enough to work around as you pointed out. I'll keep an eye on this issue as my project develops.
One question- when you said that it is still possible to trivially do the transfer that I need, did you mean loading the data into a buffer in RAM, or did you mean that it can be streamed in a more RAM-eficient but not-very-pythonic way. I'm happy to do it the simple way for now, but if there is a more efficient way to do it as a single transfer then I'm still not seeing it.

[pfalcon]

Thanks for the reply. If I understand you correctly, the reason that I can't currently do it in the Pythonic way is that the current I2C and other serial device classes don't present a file-like interface.

Correct.

That works well enough for me. As much as I would like to develop good Python programming habits, for now it is easy enough to work around as you pointed out. I'll keep an eye on this issue as my project develops.
One question- when you said that it is still possible to trivially do the transfer that I need, did you mean loading the data into a buffer in RAM, or did you mean that it can be streamed in a more RAM-eficient but not-very-pythonic way. I'm happy to do it the simple way for now, but if there is a more efficient way to do it as a single transfer then I'm still not seeing it.

Well, you surely would need to load data into buffer in RAM. Depending on how disk controller is implemented, it might be formally possible to do DMA directly from disk controller into device. But mixing both filesystem support, and disk-device DMA support would be something non-orthodox (if any OS or RTOS supports that, let me know, that could set an objective to strive for).

And bothering with DMA for your case would probably be much more trouble than real need. So, buffer in RAM, but how big, depends actually on bus semantic (or how well it's implemented in "pyb" module). I didn't really have much practical experience with I2C, but with SPI, single transfer is when you keep SlaveSelect signal asserted during the entire transfer. But in http://micropython.org/doc/module/pyb/SPI , I don't see any way to control SS explicitly, which means you won't be able to read file by chunks into small buffer, than spool it to SPI, and repeat until end of file - you'll have to read entire, file, then send content at once to SPI. It might be the same for I2C (again, depending on its bus semantics, IIRC, there's kind of STOP condition, and again, I don't see any way to control that explicitly at http://micropython.org/doc/module/pyb/I2C).

Bottom line, so far there's just rough interface, and ideas how it might be improved, but what we need is that people who know hardware and Python well enough to actually start using it, to come up with actually better way to do it.

[blmorris]

with SPI, single transfer is when you keep SlaveSelect signal asserted during the entire transfer. But in http://micropython.org/doc/module/pyb/SPI , I don't see any way to control SS explicitly, which means you won't be able to read file by chunks into small buffer, than spool it to SPI, and repeat until end of file - you'll have to read entire, file, then send content at once to SPI. It might be the same for I2C (again, depending on its bus semantics, IIRC, there's kind of STOP condition, and again, I don't see any way to control that explicitly at http://micropython.org/doc/module/pyb/I2C).

I2C does have a similar issue to what you describe with SlaveSelect in SPI. In SPI the master asserts the SlaveSelect pin to start a transfer; in I2C the master transmits a slave's bus address (including one bit to indicate whether the transfer will be a read or write) followed by a starting memory address (if relevant) followed by the data transfer, which can be a single byte or an arbitrarily long sequence of bytes so long as the receiving device continues to acknowledge each byte and there is no stop condition.

The reason I want to use a single data transfer is that the program file for my DSP has the starting addresses for the program and parameter data built in to the data structure; breaking this into several smaller data transfers would require calculating a memory offset to start each transfer; not impossible, but why complicate things unnecessarily.

Thanks for your help, this has been very enlightening.

[rambo]

Caveat emptor: I have not tried this or even dug into documentation/code to check but in theory:

If the I2C interface takes anything that looks like a buffer you could make your own buffer class that works as a generator/iterable and buffers X bytes from the SD while returning a byte for each iteration until done (a particularly naive implementation could just read the bytes one at a time from the SD but I imagine this to be exceedingly inefficient).

[blmorris]

@rambo - Creating a new buffer class is the sort of approach that I was thinking of, although the idea of being able to do a file copy operation as per @pfalcon is would be very elegant if it could be made to work.
Would you be able to point to any code examples for creating a new buffer class like you describe? Even though my project doesn't strictly need it, it would be a good learning opportunity for me to dig deeper into Python to implement this, but I don't have a good idea where to start (other than looking up examples for generators and iterables.)
Thanks,
Bryan

[pfalcon]

If the I2C interface takes anything that looks like a buffer you could make your own buffer class that works as a generator/iterable and buffers X bytes from the SD while returning a byte for each iteration

Python indeed has "buffer protocol", but its data model is exactly what you would expect of buffer - continuous memory block with starting address and length. Anything else would be to slow for a "buffer". CPython buffer protocol is described here: https://docs.python.org/3/c-api/buffer.html , and MicroPython's is the same in spirit, but unbloated and simplified.

[blmorris]

Python indeed has "buffer protocol", but its data model is exactly what you would expect of buffer - continuous memory block with starting address and length. Anything else would be to slow for a "buffer".

So trying to create a new class presenting a buffer-like interface is a dead end. Good to know, I can just go ahead with the straightforward implementation without worrying that I've missed an obviously better one.
Thanks again!
Bryan