Sending large array of numbers to host computer
Sending large array of numbers to host computer
I use the Pyboard's ADC's to fill up an array with 7500 unsigned short integers. Now I want to send the data to the host computer.
At the side of the Pyboard I'm using
port = pyb.USB_VCP()
port.send(data)
or
port = pyb.USB_VCP()
port.write(data)
I am having (at least two problems):
1) the data gets truncated. I am not sure what the difference between write() and send() is. However, both of the commands allow me to read out some of the data on the host computer. However, not all: the sent data seems to get truncated. This is, only part of the data arrives at the host computer. For example, only 3000 out of 7500 bytes. Does anybody have an example of how to send larger arrays of data to the host computer?
2) All print statements in my program seem to be sent to the serial port as well. This is great for debugging. However, for my use case, all the print statements end up making it more difficult to find my data in whether arrives at the host computer. It seems I need to implement my own 'mini protocol' that allows me to identify where my data (the 75000 numbers) start and end. Has anybody run into this issue before and found a robust solution?
At the side of the Pyboard I'm using
port = pyb.USB_VCP()
port.send(data)
or
port = pyb.USB_VCP()
port.write(data)
I am having (at least two problems):
1) the data gets truncated. I am not sure what the difference between write() and send() is. However, both of the commands allow me to read out some of the data on the host computer. However, not all: the sent data seems to get truncated. This is, only part of the data arrives at the host computer. For example, only 3000 out of 7500 bytes. Does anybody have an example of how to send larger arrays of data to the host computer?
2) All print statements in my program seem to be sent to the serial port as well. This is great for debugging. However, for my use case, all the print statements end up making it more difficult to find my data in whether arrives at the host computer. It seems I need to implement my own 'mini protocol' that allows me to identify where my data (the 75000 numbers) start and end. Has anybody run into this issue before and found a robust solution?
Re: Sending large array of numbers to host computer
(1) try increasing the time out; currently it’s set to five seconds
Sent from my iPhone using Tapatalk Pro
Sent from my iPhone using Tapatalk Pro
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: Sending large array of numbers to host computer
@Batman It's not clear from your post how you're doing this. I created the following script and transferred it to the Pyboard as rats10.py:
At the shell I issued:
I got back the full array without issue.
Re distinguishing between data and other print statements, I assume you're running Python on the PC. I would attempt a json load in a try-except block. This should reject all but valid json data.
Code: Select all
import array, ujson
a = array.array('h', range(7500))
print(ujson.dumps(a))
Code: Select all
pyboard.py --device /dev/pyboard -c "import rats10"
Re distinguishing between data and other print statements, I assume you're running Python on the PC. I would attempt a json load in a try-except block. This should reject all but valid json data.
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
-
- Posts: 40
- Joined: Wed Mar 08, 2017 7:24 am
Re: Sending large array of numbers to host computer
Here's what I did on pyboard side. Is slow but works, I was able to send larger data back to computer.
boot.py
pyb_server.py
boot.py
Code: Select all
import machine
import pyb
sw = pyb.Switch()
pyb.LED(2).on()
pyb.delay(1500)
pyb.LED(2).off()
if not sw():
pyb.main('pyb_server.py')
Code: Select all
import pyb
class USB_Port:
def __init__(self):
self.serial = pyb.USB_VCP()
self.serial.setinterrupt(-1)
def any(self):
if self.serial.any():
return True
else:
return False
def read_until(self, ending, timeout=100):
data = self.serial.read(1)
timeout_count = 0
while True:
if data.endswith(ending):
data=data.replace(ending, b'')
break
elif self.serial.any():
new_data = self.serial.read(1)
data = data + new_data
timeout_count = 0
else:
timeout_count += 1
if timeout is not None and timeout_count >= timeout:
break
pyb.delay(1)
return data
def send(self, data):
self.serial.write(data)
self.serial.write(b'END')
def run_cmd(cmd):
try:
exec('d={}'.format(cmd))
return locals()['d']
except Exception as e:
return e
def main(vcp):
while True:
if vcp.any():
cmd=vcp.read_until(b'END').decode('utf-8')
data=run_cmd(cmd)
if data is not None:
if not isinstance(data, bytes):
data = bytes(data, encoding='utf8')
vcp.send(data)
pyb.delay(1)
main(USB_Port())
Last edited by starter111 on Thu Sep 06, 2018 5:18 pm, edited 1 time in total.
Re: Sending large array of numbers to host computer
It’s easier to read your posts if you surround code with the code tags
Sent from my iPhone using Tapatalk Pro
Sent from my iPhone using Tapatalk Pro
Re: Sending large array of numbers to host computer
@pythoncoder, I can't get make the json.loads() working with array on the PC side. Do you have any solution for using json working with arrays ?pythoncoder wrote: ↑Mon Sep 03, 2018 5:55 am@Batman It's not clear from your post how you're doing this. I created the following script and transferred it to the Pyboard as rats10.py:At the shell I issued:Code: Select all
import array, ujson a = array.array('h', range(7500)) print(ujson.dumps(a))
I got back the full array without issue.Code: Select all
pyboard.py --device /dev/pyboard -c "import rats10"
Re distinguishing between data and other print statements, I assume you're running Python on the PC. I would attempt a json load in a try-except block. This should reject all but valid json data.
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: Sending large array of numbers to host computer
I made a mistake here: JSON doesn't support arrays and it's a quirk that ujson appeared to work here. The following worked under the Unix build of MicroPython. However it's a bit of a hack - I'm not well-up on this stuff and hopefully someone will come up with a better way. Under CPython I believe the best way is using subprocess: feel free to Google Here is the hack.
On Pyboard, rats10.py
On the PC:
I'm sure there's a much more elegant way to do this on MicroPython...
On Pyboard, rats10.py
Code: Select all
import array
a = array.array('h', range(7500))
print(str(a))
Code: Select all
import os
s = """
from array import array
a = """
with open("result.py", "w") as f:
f.write(s)
os.system('pyboard.py --device /dev/pyboard -c "import rats10" >> result.py')
from result import a
print(a[99]) # a is the array, retrieve an element
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
-
- Posts: 11
- Joined: Fri Mar 04, 2022 9:47 am
Re: Sending large array of numbers to host computer
Is there a way to get the standard output of the launched command? I see you use os.system and redirect to a file which you subsequently read for output. Is there a direct way to get the output just like in Python with subprocess.Popen ?pythoncoder wrote: ↑Fri Dec 31, 2021 12:18 pmI'm sure there's a much more elegant way to do this on MicroPython...Code: Select all
import os os.system('pyboard.py --device /dev/pyboard -c "import rats10" >> result.py') from result import a print(a[99]) # a is the array, retrieve an element