passing array of data to host computer

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
ryanGT
Posts: 24
Joined: Fri Jan 28, 2022 12:15 am

passing array of data to host computer

Post by ryanGT » Fri Jan 28, 2022 4:44 pm

I am using micropython with an arduino (spi and i2c) to perform real-time feedback control of a DC motor (or any simple dynamic system). I used to do this kind of thing with Raspberry Pi + Arduino, but trying to run Python in real-time on Raspbian is iffy (it mostly works, but glitches out for 6-10ms here and there). I am doing this control at 500 Hz (2ms time steps). I have everything working using micropython+ arduino, but now I need to graph the data. The data is stored in a ulab.numpy.ndarray and I need to pass it back to the host computer for plotting using numpy and matplotlib.

My first thought was to dump the data to a csv file and simply import and plot it on the host computer that way. But writing to csv seems slow and it seems like I have to reboot the pyboard to get the csv file to show up on the host computer.

Is there a better way to pass an array of data to the host computer so that it can be plotted using Python?

Thanks,
Ryan

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: passing array of data to host computer

Post by dhylands » Fri Jan 28, 2022 5:12 pm

You definitely don't want to be writing it to a file, especially if you're using USB Mass Storage. Having MicroPython write to the filesystem while it's mounted via MSC is basically a receipe for corruption.

I'd recommend that you send the data back to the host through the serial link. I'd probably start with sending ASCII data just to get things working and then switch to doing it in binary to reduce the amount of data which needs to be sent (i.e. you can send more data).

I wrote json-ipc https://github.com/dhylands/json-ipc which I probably wouldn't recommend that you use for sending your data, but the underlying mechanisms that it uses are what you're looking for.

ryanGT
Posts: 24
Joined: Fri Jan 28, 2022 12:15 am

Re: passing array of data to host computer

Post by ryanGT » Fri Jan 28, 2022 5:19 pm

I have used pyserial on a host computer to send and receive data from an Arduino. So, I am fine with this suggestion. How do I make the pyboard send data over serial that doesn't just go to the screen app on my mac? And how do I receive the data on the host computer (preferably in python)?

ryanGT
Posts: 24
Joined: Fri Jan 28, 2022 12:15 am

Re: passing array of data to host computer

Post by ryanGT » Fri Jan 28, 2022 5:23 pm

Just to clarify, are you suggesting that I use UART pins on the pyboard rather than the micro USB connection? If so, I think I would then need some kind of USB-to-uart connection for my macbook. But maybe you meant something else.

Thanks again,
Ryan

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: passing array of data to host computer

Post by dhylands » Fri Jan 28, 2022 5:50 pm

I was thinking of using the same serial-over-USB as the REPL is on. This requires that you disconnect the serial terminal that you use for accessing the REPL, but when your code is running, the REPL isn't. With rshell, I put a repl command right in it so you can easily toggle back and forth between REPL mode and what I'll call data mode.

ryanGT
Posts: 24
Joined: Fri Jan 28, 2022 12:15 am

Re: passing array of data to host computer

Post by ryanGT » Fri Jan 28, 2022 6:03 pm

That sounds perfect. Any suggestions on how to get started with toggling between REPL and data mode (or was I supposed to learn that from the other example you linked)? What functions do I use to send the data from micropython and what do I need to do to "listen" for it on the host computer? And can the host listen using python or jupyter?

ryanGT
Posts: 24
Joined: Fri Jan 28, 2022 12:15 am

Re: passing array of data to host computer

Post by ryanGT » Fri Jan 28, 2022 6:39 pm

I installed your rshell, which is much nicer than screen on my mac. Screen does not allow me to scroll back through previous output, so when I printed the data from micropython I could only see the last 20 rows or so. I can now copy and paste the data from rshell into a test editor on my mac. But I would very much like to automate getting the data to my mac.

I am currently running my test in main.py through a soft reset in REPL. Perhaps that is where my confusion starts. I currently think of running my code only from REPL.

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: passing array of data to host computer

Post by dhylands » Sat Jan 29, 2022 12:20 am

There isn't really anything to do on the MicroPython side to toggle between REPL and data mode. When you're executing your code on the MicroPython board then you're no longer in REPL mode.

Here's the functions that the MicroPython side uses: https://github.com/dhylands/json-ipc/bl ... sb_port.py and here's the functions that the host side uses: https://github.com/dhylands/json-ipc/bl ... al_port.py

Note that you can only have one program on the host side with the serial port open at a time.

ryanGT
Posts: 24
Joined: Fri Jan 28, 2022 12:15 am

Re: passing array of data to host computer

Post by ryanGT » Sat Jan 29, 2022 5:32 pm

Thanks for your help and all the work you put into rshell. Thanks for being patient with me. I have used python for a long time, but have only been working with micropython for a few days now.

I started reading through rshell.main.py. It is a lot of code, but well organized. I am trying to understand the workflow to perform the process you are describing. Part of my issue is that there are important differences with what I am currently doing, and I am not sure how to reconcile that. Here is my current process:

- main.py on my device runs my real-time test
- my device is a pyboard v1.1 if it matters
- I develop main.py and execute it using a soft reboot (control-D) until it is working
- after the execution of main.py on the device, the data I need it in a ulab.numpy.ndarray waiting for me to do something with it
- as a short term solution, I go through the ndarray (which is 2D) and print the rows for copy and paste from REPL into a text file on my host computer

To do what I think you are suggesting, I would need to do the following (I think):

- trigger the micropython device to run my micropython file that includes sending data over serial
- have a python script on the host computer that receives the data over python (after triggering the test on the device)
- plot the data on the host computer
- go into a shell on the host computer for REPL interaction

Passing a filename to rshell using -f seems to cause a file to be run on the host computer. That could be used for my second bullet point.

My current roadblock is that I don't know how to trigger a file to be run on the micropython device other than using control-D from REPL.


Thanks,
Ryan

ryanGT
Posts: 24
Joined: Fri Jan 28, 2022 12:15 am

Re: passing array of data to host computer

Post by ryanGT » Sat Jan 29, 2022 5:47 pm

I came across another thread where it was mentioned that rshell is built on pyboard. That lead me to try this, which works but seems like a long way around (kind of):

In [1]: from rshell import pyboard

In [2]: pyboard.execfile('main.py', device='/dev/tty.usbmodem2088369D58532')
loop_time = 998369

That allows me to repeatedly trigger a file to be run on the device from python on my host computer. But it doesn't feel like I am using rshell correctly......

Post Reply