Issues with data-logging on Pyboard

The official pyboard running MicroPython.
This is the reference design and main target board for MicroPython.
You can buy one at the store.
Target audience: Users with a pyboard.
Post Reply
stucke02
Posts: 2
Joined: Sat Jun 25, 2022 6:44 am

Issues with data-logging on Pyboard

Post by stucke02 » Sat Jun 25, 2022 7:50 am

Hello!
My name is Sam, and I am a graduate student in mechanical engineering, trying to collect data from an analog pressure sensor for a personal research project. Earlier tonight, I was able to get a simple script working to record the sensor (and 3V3 reference) voltage, along with a timestamp. However, it only works for "small" numbers of data points. I have included an excerpt of my script and data.

The time frame I wish to collect over is likely on the order of ~100-200 seconds, and I found that if I make the elapsed time too large, a file will be created, but it will be empty-- a size of 0 KB. The script I am using is one that I repurposed after reading this article [/viewtopic.php?t=86]. I noticed that a similar thing occurred when I used the loop, with the data logging for smaller values of the loop iteration number, but failing for larger values. I have not pinpointed the exact number which it no longer records, but it cannot do 10 seconds.

I assume that I am running into some kind of memory limit, but as there is no error (that I can see). Does anyone know what limit I am running into? Some kind of time-out?

Once I can record data, I plan to write the data onto an SD card. If anyone has recommendations about using a queue/buffer, those would also be appreciated, as I believe this script may be somewhat slow. Me thinks the writing and collecting in the loop causes unwanted delay. But thats just my intuition. And if anyone has a script to do what I am trying to do, I would gladly take (a look at) it.

And lastly, thank you for your time! If anyone needs any clarifying info, I am happy to provide it.

Warm regards,
Sam

Code: Select all

[/# main.py -- put your code here!
import pyb
from pyb import Pin,ADC
pyb.LED(4).on
f = open('samm.dat', 'w')                 # open the file for writing

start=pyb.millis()
while pyb.elapsed_millis(start) < 100:
	time = pyb.millis()                       # get the current time
	adc=int(ADC(Pin('X21')).read())
	volt=int(ADC(Pin('X22')).read())

	# write time and x,y,z values to the file
	f.write('{} {} {}\n'.format(time, adc, volt))
	pyb.delay(5)                         
f.close()                                   # close the file
pyb.LED(4).off()]

# a representative output
2067 2217 4093
2073 2223 4095
2078 2224 4093
2083 2221 4095
2088 2220 4095
2093 2219 4093
2098 2221 4092
2103 2217 4093
2108 2221 4095
2113 2221 4093
2118 2221 4095
2123 2222 4093
2128 2221 4095
2133 2220 4095
2138 2221 4095

User avatar
scruss
Posts: 360
Joined: Sat Aug 12, 2017 2:27 pm
Location: Toronto, Canada
Contact:

Re: Issues with data-logging on Pyboard

Post by scruss » Sat Jun 25, 2022 6:34 pm

Hi Sam!

Interesting problem, thanks for sharing. What I would've given to have had hardware like a pyboard when I was doing my mechanical engineering graduate degree roughly 30 years ago!

On a pyboard v1.1, running MicroPython v1.19.1, I can modify your script so it reads:

Code: Select all

while pyb.elapsed_millis(start) < 100000:
(that is, 100 s) and it creates a "samm.dat" file that looks like this:

Code: Select all

1112133 653 735
1112144 603 811
1112149 733 895
1112154 833 959
  ... appx 17371 lines skipped ...
1212122 793 905
1212127 772 889
1212132 816 945
I've got no sensor attached, so nothing interesting is happening. My data is writing to the SD card.

So we're doing something different. The fact that you're getting zero byte files makes me think something is interrupting your program. Are you writing data to onboard flash, or to the SD card? There's probably not enough space on the built-in flash to do much logging.

How are you accessing the 'samm.dat' file? Although the pyboard's storage might appear as a USB drive, it's pretty fragile USB storage. You can't read or write files on the pyboard via USB while the pyboard is busy. I mean, your computer will let you try, but what you'll end up is not what you want.

A couple of hints:
  • If you have a long loop collecting data, it's probably better to put in a call to f.flush() now and again, perhaps once every 1000 lines or so. This will ensure that most of your data gets committed to storage.
  • Maybe turn off USB storage mode on your pyboard. Before you do this, it will fundamentally change how you access the device. You'll have to program via a MicroPython device-aware editor (like Thonny), and transferring files will no longer be drag and drop. I have my pyboard set up this way and file storage is reliable, if less convenient. To do this, change the line in boot.py that says: pyb.usb_mode('VCP+MSC') # act as a serial and a storage device to pyb.usb_mode('VCP')

stucke02
Posts: 2
Joined: Sat Jun 25, 2022 6:44 am

Re: Issues with data-logging on Pyboard

Post by stucke02 » Sat Jun 25, 2022 10:05 pm

Hi Scruss!

Absolutely. That's awesome! And definitely-- I feel pretty lucky to have such an abundance of technology and information! It's honestly mind blowing.

Are you writing data to onboard flash, or to the SD card? There's probably not enough space on the built-in flash to do much logging.

Indeed I am. I thought that since my files, at least starting out, were on the order of 25-900 bytes, that I would not exceed the flash's limit.
However, when I switched over to recording on the SD card, and it worked!!! I've included a link to a pic of some processed data.[img][/https://imgur.com/a/AOm5764] I now have no problems recording 60 seconds of data.

How are you accessing the 'samm.dat' file? Although the pyboard's storage might appear as a USB drive, it's pretty fragile USB storage. You can't read or write files on the pyboard via USB while the pyboard is busy. I mean, your computer will let you try, but what you'll end up is not what you want.

Previously, I was just copying the file from the flash to the desktop, and opening it there. I am now accessing the SD card directly, plugging it in after data collection. That's good to know-- I was not aware of it's fragility. And when you say it is fragile, are you referring to the inability to read/write from USB while the Pyboard is busy, or something else?

I appreciate the tips. When I call f.flush(), I assume that would pause data logging, or would it still record? From my cursory internet search I see that it will clear the internal buffer-- but I do not fully understand what that means. I am trying to capture a very rapid event surrounded by a relatively slow changing signal, so perhaps I will write some code to only flush under certain criteria. What do you think?

Thanks again for your help here Scruss!

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Issues with data-logging on Pyboard

Post by jimmo » Mon Jun 27, 2022 2:32 am

stucke02 wrote:
Sat Jun 25, 2022 10:05 pm
And when you say it is fragile, are you referring to the inability to read/write from USB while the Pyboard is busy, or something else?
The simple answer is that USB Mass Storage was never designed to have what is essentially a USB stick that modifies its blocks at runtime. The host drivers on your PC have no idea what's going on.

The other thing is that when problems happen (especially unfinished writes) it can end up with a situation where you have used blocks that are unreferenced. FAT has no way to clean this up automatically, but the result is that the disk appears to be full even though the "used bytes" is nowhere near the total bytes. I actually added a basic scandisk/fsck function to the FAT driver used by micropython (see https://github.com/micropython/oofatfs/ ... ff.c#L6037) but we don't use it in any of the released firmware.

I strongly recommend reformatting the device as LittleFS (see https://docs.micropython.org/en/latest/ ... l#littlefs) and using the mpremote tool (or pyboard.py or rshell) to access files to/from the device.

User avatar
scruss
Posts: 360
Joined: Sat Aug 12, 2017 2:27 pm
Location: Toronto, Canada
Contact:

Re: Issues with data-logging on Pyboard

Post by scruss » Thu Jun 30, 2022 8:22 am

stucke02 wrote:
Sat Jun 25, 2022 10:05 pm
I appreciate the tips. When I call f.flush(), I assume that would pause data logging, or would it still record?
Yes, it would pause very briefly.

Since you're using a pyboard, you've got access to ADC.read_timed(). This allows you to read an ADC at a precise frequency. It does log to RAM, though, so this might only work if you're able to detect the start of your fast-changing event.

User avatar
mathieu
Posts: 88
Joined: Fri Nov 10, 2017 9:57 pm

Re: Issues with data-logging on Pyboard

Post by mathieu » Sat Jul 02, 2022 8:19 pm

For the record, another option for logging sensor output during "short" experiments (up to a few hours) is to connect the board to a computer and have it just print out data as it is read instead of writing to a file. The computer can log this serial stream to a file using cat or something like it. In my experience that works rather well.

Post Reply