Sending large array of numbers to host computer

The official pyboard running MicroPython.
This is the reference design and main target board for MicroPython.
You can buy one at the store.
Target audience: Users with a pyboard.
Post Reply
Batman
Posts: 13
Joined: Sun Aug 26, 2018 6:04 pm

Sending large array of numbers to host computer

Post by Batman » Sun Sep 02, 2018 8:45 pm

I use the Pyboard's ADC's to fill up an array with 7500 unsigned short integers. Now I want to send the data to the host computer.

At the side of the Pyboard I'm using

port = pyb.USB_VCP()
port.send(data)

or

port = pyb.USB_VCP()
port.write(data)

I am having (at least two problems):

1) the data gets truncated. I am not sure what the difference between write() and send() is. However, both of the commands allow me to read out some of the data on the host computer. However, not all: the sent data seems to get truncated. This is, only part of the data arrives at the host computer. For example, only 3000 out of 7500 bytes. Does anybody have an example of how to send larger arrays of data to the host computer?

2) All print statements in my program seem to be sent to the serial port as well. This is great for debugging. However, for my use case, all the print statements end up making it more difficult to find my data in whether arrives at the host computer. It seems I need to implement my own 'mini protocol' that allows me to identify where my data (the 75000 numbers) start and end. Has anybody run into this issue before and found a robust solution?

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: Sending large array of numbers to host computer

Post by jickster » Sun Sep 02, 2018 11:38 pm

(1) try increasing the time out; currently it’s set to five seconds


Sent from my iPhone using Tapatalk Pro

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Sending large array of numbers to host computer

Post by pythoncoder » Mon Sep 03, 2018 5:55 am

@Batman It's not clear from your post how you're doing this. I created the following script and transferred it to the Pyboard as rats10.py:

Code: Select all

import array, ujson
a = array.array('h', range(7500))
print(ujson.dumps(a))
At the shell I issued:

Code: Select all

pyboard.py --device /dev/pyboard -c "import rats10"
I got back the full array without issue.

Re distinguishing between data and other print statements, I assume you're running Python on the PC. I would attempt a json load in a try-except block. This should reject all but valid json data.
Peter Hinch
Index to my micropython libraries.

starter111
Posts: 40
Joined: Wed Mar 08, 2017 7:24 am

Re: Sending large array of numbers to host computer

Post by starter111 » Thu Sep 06, 2018 5:15 pm

Here's what I did on pyboard side. Is slow but works, I was able to send larger data back to computer.

boot.py

Code: Select all

import machine
import pyb
sw = pyb.Switch()
pyb.LED(2).on()
pyb.delay(1500)
pyb.LED(2).off()

if not sw():    
    pyb.main('pyb_server.py')
pyb_server.py

Code: Select all

import pyb

class USB_Port:
    def __init__(self):
        self.serial = pyb.USB_VCP()
        self.serial.setinterrupt(-1)
        
    def any(self):
        if self.serial.any():
            return True
        else:
            return False

    def read_until(self, ending, timeout=100):
        data = self.serial.read(1)
        timeout_count = 0
        while True:
            if data.endswith(ending):
                data=data.replace(ending, b'')
                break
            elif self.serial.any():
                new_data = self.serial.read(1)
                data = data + new_data
                timeout_count = 0
            else:
                timeout_count += 1
                if timeout is not None and timeout_count >= timeout:
                    break
                pyb.delay(1)
        return data
                
    def send(self, data):
        self.serial.write(data)
        self.serial.write(b'END')

def run_cmd(cmd):
    try:
        exec('d={}'.format(cmd))
        return locals()['d']
    except Exception as e:
        return e    
    
def main(vcp):    
    while True:
        if vcp.any():
            cmd=vcp.read_until(b'END').decode('utf-8')
            data=run_cmd(cmd) 
            if data is not None:
                if not isinstance(data, bytes):
                    data = bytes(data, encoding='utf8')
                vcp.send(data)
        pyb.delay(1)
        
main(USB_Port())

Last edited by starter111 on Thu Sep 06, 2018 5:18 pm, edited 1 time in total.

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: Sending large array of numbers to host computer

Post by jickster » Thu Sep 06, 2018 5:16 pm

It’s easier to read your posts if you surround code with the code tags


Sent from my iPhone using Tapatalk Pro

zaord
Posts: 96
Joined: Fri Jan 31, 2020 3:56 pm

Re: Sending large array of numbers to host computer

Post by zaord » Fri Dec 31, 2021 8:09 am

pythoncoder wrote:
Mon Sep 03, 2018 5:55 am
@Batman It's not clear from your post how you're doing this. I created the following script and transferred it to the Pyboard as rats10.py:

Code: Select all

import array, ujson
a = array.array('h', range(7500))
print(ujson.dumps(a))
At the shell I issued:

Code: Select all

pyboard.py --device /dev/pyboard -c "import rats10"
I got back the full array without issue.

Re distinguishing between data and other print statements, I assume you're running Python on the PC. I would attempt a json load in a try-except block. This should reject all but valid json data.
@pythoncoder, I can't get make the json.loads() working with array on the PC side. Do you have any solution for using json working with arrays ?

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Sending large array of numbers to host computer

Post by pythoncoder » Fri Dec 31, 2021 12:18 pm

I made a mistake here: JSON doesn't support arrays and it's a quirk that ujson appeared to work here. The following worked under the Unix build of MicroPython. However it's a bit of a hack - I'm not well-up on this stuff and hopefully someone will come up with a better way. Under CPython I believe the best way is using subprocess: feel free to Google ;) Here is the hack.

On Pyboard, rats10.py

Code: Select all

import array
a = array.array('h', range(7500))
print(str(a))
On the PC:

Code: Select all

import os
s = """
from array import array
a = """
with open("result.py", "w") as f:
    f.write(s)
os.system('pyboard.py --device /dev/pyboard -c "import rats10" >> result.py')
from result import a
print(a[99])  # a is the array, retrieve an element
I'm sure there's a much more elegant way to do this on MicroPython...
Peter Hinch
Index to my micropython libraries.

teltonique21
Posts: 11
Joined: Fri Mar 04, 2022 9:47 am

Re: Sending large array of numbers to host computer

Post by teltonique21 » Tue May 24, 2022 3:35 pm

pythoncoder wrote:
Fri Dec 31, 2021 12:18 pm

Code: Select all

import os
os.system('pyboard.py --device /dev/pyboard -c "import rats10" >> result.py')
from result import a
print(a[99])  # a is the array, retrieve an element
I'm sure there's a much more elegant way to do this on MicroPython...
Is there a way to get the standard output of the launched command? I see you use os.system and redirect to a file which you subsequently read for output. Is there a direct way to get the output just like in Python with subprocess.Popen ?

Post Reply