How to handle data transfer from pyboard to PC

[Chemist]

Hello All,

I am going to build a web interface to interact with my sensor and a problem I stumbled upon is how to transfer data from the pyboard to PC. My sensor generates about 1300 values that are stored in a list so I am wondering if there is a way how to pass the whole list from the pyboard to PC? One method is to pass each value of the list using USB_VCP() and serial. Another method is to save the generated data in a file and then get the data from that file. But in both methods there is a need to use two python scripts at the same time. Ideally, I would like to click "Run" button on my web interface and data will be generated by the pyboard and plotted by PC on the web page. Any ideas how to trigger two events by one action ("Run" button)?

Thank you for your time.

[pythoncoder]

I suggest you look at Dave Hylands' pyboard.py: https://github.com/dhylands/upy-shell/b ... pyboard.py. You run it on the PC, the Pyboard executes the file and the output of the MicroPython print statements is returned to the PC program for you to process.

I'd be wary of the shared file approach: it brings up concurrency issues.

[dhylands]

Actually, pyboard.py originates from micropython/tools/pyboard.py

I just made a copy of it in rshell to make using rshell a bit easier.

The copy with rshell is actually a bit behind the version in the tools directory.

[Chemist]

Dear pythoncoder and dhylands,

Thank you very much for your replies. I am still having problems with transferring data from pyboard to PC. Here is a simplified version of my pb_get_data.py:

Code: Select all

def get_data():
    data= [i for i in range(10)]
    return data
get_data()

This is my pc_read_data.py which is run on PC(Gnu/Linux):

Code: Select all

import pyboard
pyboard.execfile('pb_get_data.py', device='/dev/ttyACM0')

The script runs without errors but how can I access get_data() inside pc_read_data.py for further processing? What I am missing? I tried to write data to a file but the file is not visible unless I reset the board. I guess, this is what pythoncoder meant by concurrency issues. I can use pyb.hard_reset() in the code but I think it is a bad idea to reset pyboard every time the data is generated. I would highly appreciate any ideas on how to solve this problem.

Thank you very much.

[pythoncoder]

Try this:

Code: Select all

def get_data():
    data= [i for i in range(10)]
    return data
print(get_data())

Code: Select all

>>> a =pyb.execfile('pb_get_data.py')
>>> a
b'[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\r\n'
>>> 

With the pickle (or json) module you should be able to return arbitrary Python objects as strings.

[Chemist]

Dear pythoncoder,

Thank you very much for your suggestion but unfortunately I get a different outcome.

pb_get_data.py:

Code: Select all

def get_data():
    data= [i for i in range(10)]
    return data
print(get_data())

pc_read_data.py:

Code: Select all

import pyboard
a = pybaord.execfile('pb_get_data.py')
print(a)

When I run in the terminal python3 pc_read_data.py I get:

Code: Select all

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
None

Thank you for your time.

[dhylands]

Using the pyboard.py from micropython/tools I was able to get the following to work:

Code: Select all

import pyboard

def execfile(filename, device='/dev/ttyACM0'):
    pyb = pyboard.Pyboard(device)
    pyb.enter_raw_repl()
    output = pyb.execfile(filename)
    pyb.exit_raw_repl()
    pyb.close()
    return output

a = eval(execfile('pb_get_data.py'))
print(a)

The problem with pyboard.execfile is that it prints the output rather then returning it. The execfile function shown above is essentially an exact copy, but it returns the output rather than printing it.

Using eval smooths over the differences between python2 and python3. And I get:

Code: Select all

bash> python2 pc_read_data.py 
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
bash> python3 pc_read_data.py 
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Without the eval, I get:

Code: Select all

bash> python2 pc_read_data.py 
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

bash> python3 pc_read_data.py 
b'[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\r\n'

[Chemist]

Dear dhylands,

Thank you so much for your help! Now it works (at least with python 3) !

Best,
Vitali

[pythoncoder]

@Chemist - apologies - I goofed. Sometimes I suspect I'm getting too old for this stuff
@dhylands Alas it doesn't seem to be very reliable. Repeated calls to any of the execfile variants fail from time to time. Note the following session at the Python 3.4.3 REPL with your version of execfile:

Code: Select all

>>> a=eval(execfile('pb_get_data.py'))
>>> a=eval(execfile('pb_get_data.py'))
>>> a=eval(execfile('pb_get_data.py'))
>>> a=eval(execfile('pb_get_data.py'))
>>> a=eval(execfile('pb_get_data.py'))
>>> a=eval(execfile('pb_get_data.py'))
>>> a=eval(execfile('pb_get_data.py'))
>>> a=eval(execfile('pb_get_data.py'))
b' exit\r\n>'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in execfile
  File "/home/adminpete/pyboard.py", line 171, in enter_raw_repl
    raise PyboardError('could not enter raw repl')
pyboard.PyboardError: could not enter raw repl
>>> a=eval(execfile('pb_get_data.py'))
>>> 

Also this session with pyboard.execfile

Code: Select all

>>> >>> pyboard.execfile('pb_get_data.py', device='/dev/pyboard')
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
pyboard.execfile('pb_get_data.py', device='/dev/pyboard')
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> a =pyboard.execfile('pb_get_data.py', device='/dev/pyboard')
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/serial/serialposix.py", line 275, in open
    self.fd = os.open(self.portstr, os.O_RDWR|os.O_NOCTTY|os.O_NONBLOCK)
OSError: [Errno 16] Device or resource busy: '/dev/pyboard'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/adminpete/pyboard.py", line 243, in execfile
    pyb = Pyboard(device, baudrate, user, password)
  File "/home/adminpete/pyboard.py", line 126, in __init__
    self.serial = serial.Serial(device, baudrate=baudrate, interCharTimeout=1)
  File "/usr/lib/python3/dist-packages/serial/serialutil.py", line 261, in __init__
    self.open()
  File "/usr/lib/python3/dist-packages/serial/serialposix.py", line 278, in open
    raise SerialException("could not open port %s: %s" % (self._port, msg))
serial.serialutil.SerialException: could not open port /dev/pyboard: [Errno 16] Device or resource busy: '/dev/pyboard'
>>> pyboard.execfile('pb_get_data.py', device='/dev/pyboard')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/adminpete/pyboard.py", line 244, in execfile
    pyb.enter_raw_repl()
  File "/home/adminpete/pyboard.py", line 168, in enter_raw_repl
    data = self.read_until(1, b'soft reboot\r\nraw REPL; CTRL-B to exit\r\n')
  File "/home/adminpete/pyboard.py", line 139, in read_until
    elif self.serial.inWaiting() > 0:
  File "/usr/lib/python3/dist-packages/serial/serialposix.py", line 435, in inWaiting
    s = fcntl.ioctl(self.fd, TIOCINQ, TIOCM_zero_str)
OSError: [Errno 5] Input/output error
>>> pyboard.execfile('pb_get_data.py', device='/dev/pyboard')
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/serial/serialposix.py", line 275, in open
    self.fd = os.open(self.portstr, os.O_RDWR|os.O_NOCTTY|os.O_NONBLOCK)
OSError: [Errno 16] Device or resource busy: '/dev/pyboard'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/adminpete/pyboard.py", line 243, in execfile
    pyb = Pyboard(device, baudrate, user, password)
  File "/home/adminpete/pyboard.py", line 126, in __init__
    self.serial = serial.Serial(device, baudrate=baudrate, interCharTimeout=1)
  File "/usr/lib/python3/dist-packages/serial/serialutil.py", line 261, in __init__
    self.open()
  File "/usr/lib/python3/dist-packages/serial/serialposix.py", line 278, in open
    raise SerialException("could not open port %s: %s" % (self._port, msg))
serial.serialutil.SerialException: could not open port /dev/pyboard: [Errno 16] Device or resource busy: '/dev/pyboard'
>>> pyboard.execfile('pb_get_data.py', device='/dev/pyboard')
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/serial/serialposix.py", line 275, in open
    self.fd = os.open(self.portstr, os.O_RDWR|os.O_NOCTTY|os.O_NONBLOCK)
OSError: [Errno 16] Device or resource busy: '/dev/pyboard'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/adminpete/pyboard.py", line 243, in execfile
    pyb = Pyboard(device, baudrate, user, password)
  File "/home/adminpete/pyboard.py", line 126, in __init__
    self.serial = serial.Serial(device, baudrate=baudrate, interCharTimeout=1)
  File "/usr/lib/python3/dist-packages/serial/serialutil.py", line 261, in __init__
    self.open()
  File "/usr/lib/python3/dist-packages/serial/serialposix.py", line 278, in open
    raise SerialException("could not open port %s: %s" % (self._port, msg))
serial.serialutil.SerialException: could not open port /dev/pyboard: [Errno 16] Device or resource busy: '/dev/pyboard'
>>> pyboard.execfile('pb_get_data.py', device='/dev/pyboard')
b' exit\r\n>'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/adminpete/pyboard.py", line 244, in execfile
    pyb.enter_raw_repl()
  File "/home/adminpete/pyboard.py", line 171, in enter_raw_repl
    raise PyboardError('could not enter raw repl')
pyboard.PyboardError: could not enter raw repl
>>> pyboard.execfile('pb_get_data.py', device='/dev/pyboard')
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> a=pyboard.execfile('pb_get_data.py', device='/dev/pyboard')
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

The errors seem always to right themselves eventually. I don't know if this relates to the work you are doing on serial port DMA.

Tested using a Pyboard with firmware built from source on 11th November, connected to a Linux box via USB.

[dhylands]

Hmm. I ran this:

Code: Select all

import pyboard

def execfile(filename, device='/dev/ttyACM0'):
    pyb = pyboard.Pyboard(device)
    pyb.enter_raw_repl()
    output = pyb.execfile(filename)
    pyb.exit_raw_repl()
    pyb.close()
    return output

for i in range(1000):
    print(i)
    a = execfile('pb_get_data.py')

and it ran all 1000 iterations fine. I was also using Python 3.4.3. I tried with and without my DMA code and it worked the same, but I didn't have an sdcard in my pyboard so the DMA code wouldn't have been executed anyways.

The fact the the open is failing sounds suspicious. You didn't happen to have that open in another window? linux seems to allow that. I was going to investigate having pyboard.py use the same locking mechanism that most of the serial programs use.