Logging ADC data to SD-card

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Post Reply
vikebo
Posts: 14
Joined: Sun Mar 15, 2015 8:15 pm
Location: Norway

Logging ADC data to SD-card

Post by vikebo » Sun Mar 15, 2015 9:08 pm

Hello,

I'm trying to store data read from the ADC to a file on the SD-card. X1 is connected to an external function generator configured to a 10 Hz triangular wave.

As this plot shows there are some discontinuities in the data:
adc.png
Data from ADC
adc.png (43.73 KiB) Viewed 5010 times
I also tried storing the variable of the for-loop to check if the problem persists without the ADC (zoomed in view):
variable.png
Data from loop variable
variable.png (22.31 KiB) Viewed 5010 times
Both scatter plots use time from pyb.micros() for the x-axis.

Any info on what the pyboard is doing when it's not storing values? Would be good to know if there is a way to get better real-time preformance with this simple method. I guess it is better to use read_timed and store values to a buffer, and then only store to file when the buffer is filled, but probably need to understand the limitations to get that working properly as well...

Code: Select all

import pyb

use_adc = True  # Read values from ADC
# use_adc = False  # Use loop variable instead of ADC values
length = 1000  # Number of values to store to file

adc = pyb.ADC('X1')

# Open file on SD-card in write mode
f = open('/sd/log_1.txt', 'w')

t_start = pyb.micros()  # Save start time

for i in range(length):
    t = pyb.micros() - t_start  # Calculate current time
    if (use_adc):
        value = adc.read()
        value_string = '%i, %i;' %(t, value)
    else:
        value_string = '%i, %i;' %(t, i)

    # Write string with time and data to file for scatter plot
    f.write(value_string)

f.close()
Eivind

User avatar
dhylands
Posts: 3177
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Logging ADC data to SD-card

Post by dhylands » Mon Mar 16, 2015 4:23 am

There is a timer tick interrupt which fires once per millisecond.

If you're connected via USB there is a periodic interrupt which is required to process USB stuff.

When writing to sdcard I believe that there is a periodic interrupt to ensure that buffers get flushed to the sdcard.

hansel
Posts: 13
Joined: Wed Feb 11, 2015 7:34 pm
Location: Rotterdam, NL
Contact:

Re: Logging ADC data to SD-card

Post by hansel » Mon Mar 16, 2015 7:34 am

It would be interesting to read (and timestamp) the ADC in a timer callback...

manitou
Posts: 67
Joined: Wed Feb 25, 2015 12:15 am

Re: Logging ADC data to SD-card

Post by manitou » Mon Mar 16, 2015 12:51 pm

I think the jitter from 1ms timer interrupt is only 10 to 30 microseconds. not sure about the USB jitter, but it doesn't look like program is doing serial i/o. then there is the garbage-collection jitter ????

but running the code and looking at the data, i wonder if the periodic delays are from the SD fat interface actually writing a 512-byte sector (taking about 1ms or 2ms in some cases) as dhylands suggests. if memory allows, storing the values in an array, and then doing the SD write's might remove the big jitter.
gap.png
gap.png (20.16 KiB) Viewed 4928 times
In the plot above, most of the inter-sample times are about 207 us. There are no negative gap times, so time is always increasing.
Last edited by manitou on Tue Mar 24, 2015 11:32 am, edited 2 times in total.

vikebo
Posts: 14
Joined: Sun Mar 15, 2015 8:15 pm
Location: Norway

Re: Logging ADC data to SD-card

Post by vikebo » Mon Mar 16, 2015 8:27 pm

Thank you for your replies.

As can be seen, there are two types of discontinuities. The short ones of ~1 ms are at ~8 ms intervals, and long ones of 10 - 12 ms with "random" intervals (sometimes several in a short time). I think the worst with these is that the timestamp counter for pyb.micros seems to freeze or run slower during the incidents, if it continued to run normally there should not be a vertical shift in the scatter plot since the x-axis is the timestamp. Also tried using elapsed_micros as well as elapsed_millis instead of subtracting starting time, both gave the same problems as before.

Regarding the serial port and USB, I used picocom during the tests, but I don't think I printed anything when the attached log files were stored. I tested again today without picocom connected, and the results were the same. Normally I use picocom to print debug info and do soft resets using CTRL-D.

I had some problems in the begining since I kept the SD-card on the pyboard mounted while logging, and the log files didn't show up on the computer. To avoid removing and inserting the SD-card all the time I edit the python file for the board in a local directory on the computer and run a short script called load.py to mount the SD-card while it is still in the pyboard, copy the latest file to the card and unmount it. Then I use the soft reset, and a similar python script to mount again, copy out the log file and plot the result using matplotlib.

load.py:

Code: Select all

import os

cwd = os.getcwd()
os.system('mount /media/usb_1')
copy_command = 'cp ' + cwd + '/main.py /media/usb_1/'
os.system(copy_command)
os.system('umount /media/usb_1')

vikebo
Posts: 14
Joined: Sun Mar 15, 2015 8:15 pm
Location: Norway

Re: Logging ADC data to SD-card

Post by vikebo » Mon Mar 16, 2015 8:51 pm

Also tried using timer callback to read the ADC today with better results. Inspired by one of Dave Hylands' examples, I made a class for timer 4.

I was able to achieve a sample rate slightly above 10 kHz, higher freq-settings for the timer gave just below 100 us between samples. As can be seen below things look much better, the frequency of the triangle wave has been increased from 10 Hz to 100 Hz. Having picocom connected during this test doesn't seem to affect the results. Of course storing to SD-card is done after all the sampling is done, so longer loggings may still give problems when data must be stored while new sampling is going on.

Haven't checked the priorities of timer interrupts for timer 3, which the documentation says is reserved for internal use, and timer 4 used in this test.
t4_callback_1.png
100 Hz input signal
t4_callback_1.png (71.75 KiB) Viewed 4959 times
t4_callback_2.png
Zoomed view of the same data
t4_callback_2.png (46.07 KiB) Viewed 4959 times

Code: Select all

import pyb
import micropython
micropython.alloc_emergency_exception_buf(200)

# Turn on LED to indicate logging
pyb.LED(2).on()

class T_4(object):
    def __init__(self):
        # Initialise timer 4 and ADC
        self.status = 'Sample'
        self.tim = pyb.Timer(4)
        self.tim.init(freq=10000)
        self.set_start()
        self.adc = pyb.ADC('X1')
        self.buf_index = 0
        self.buf_0 = [[0, 0] for i in range(1000)]
        self.tim.callback(self.t_4_callback)
        
    def set_start(self):
        # Remember time of start
        self.t_start = pyb.micros()

    def store_data(self):
        # Open log file on SD-card
        f = open('/sd/log_1.txt', 'w')

        # Store buffer data to log file
        for i in range(len(self.buf_0)):
            t = self.buf_0[i][0]
            d = self.buf_0[i][1]
            value_string = '%i, %i;' %(t, d)
            f.write(value_string)
            
        # Close file and blink LED
        f.close()
        pyb.LED(2).off()
        pyb.LED(3).on()
        pyb.delay(500)
        pyb.LED(3).off()
            
    def t_4_callback(self, tim):
        # Read ADC value and current time
        value = self.adc.read()
        t = pyb.micros() - self.t_start

        # Add value and time to buffer
        self.buf_0[self.buf_index][0] = t
        self.buf_0[self.buf_index][1] = value

        # Increment buffer index until buffer is filled,
        # then disable interrupt and change status
        self.buf_index += 1
        if (self.buf_index >= len(self.buf_0)):
            self.tim.callback(None)
            self.status = 'Store'
            
t = T_4()
pyb.delay(1000)

# Check if logging is finished, store data if so
# Was unable to call store data or call method storing data from callback
go_on = True
while (go_on):
    if(t.status == 'Store'):
        t.store_data()
        t.status = 'Done'
        go_on = False
    else:
        pyb.delay(100)

vikebo
Posts: 14
Joined: Sun Mar 15, 2015 8:15 pm
Location: Norway

Re: Logging ADC data to SD-card

Post by vikebo » Mon Mar 23, 2015 10:01 pm

Manitou, didn't see your plot before today, it illustrates it very well. Probably a good idea to store data in a buffer first. I want to find the different options for sampling and storing data at a few kilosamples for some minutes, so data will have to be written while still measuring. Also looking into using bluetooth to send data out, but storing to the sd-card would be simpler and a stand-alone data logger have more uses.

Did another test using timer 2 as the timestamp instead of the micros function, but keeping the values from micros to compare them (red in the plots). Seems like the timer for micros does stop during the jitter, while timer 2 still runs. ADC measurements (blue) are missing during jitter, but the timestamp looks ok. At least for this kind of use this looks like a bug related to micros, but I don't know of all the uses for the function.

Code: Select all

import pyb

pyb.LED(2).on()
use_adc = True  # Read values from ADC
length = 10000  # Number of values to store to file

adc = pyb.ADC('X1')

# Open file on SD-card in write mode
f = open('/sd/log_1.txt', 'w')

t_start_mu = pyb.micros()  # Save start time
t_2 = pyb.Timer(2, prescaler = 83, period = 0x3fffffff)
t_2.counter(0)
t_start = t_2.counter()

for i in range(length):
    t_mu = pyb.elapsed_micros(t_start) - t_start_mu
    t = t_2.counter() - t_start  # Calculate current time
    if (use_adc):
        value = adc.read()
        value_string = '%i, %i, %i;' %(t, value, t_mu)
    else:
        value_string = '%i, %i, %i;' %(t, i, t_mu)

    # Write string with time and data to file for scatter plot
    f.write(value_string)

f.close()
pyb.LED(2).off()
figure_1.png
figure_1.png (53.25 KiB) Viewed 4881 times
figure_2.png
figure_2.png (37.15 KiB) Viewed 4881 times

manitou
Posts: 67
Joined: Wed Feb 25, 2015 12:15 am

Re: Logging ADC data to SD-card

Post by manitou » Tue Mar 24, 2015 2:24 am

you mentioned read_timed, indeed with adc.read_timed(buf, freq) and two buffers, you might be able to read the ADC into one buffer ("in the background") while writing the other buffer to the SD. you'd have choose buffer size and/or frequency to cover the SD-write time.
Edit: never mind, read_timed is not asynchronous (though it seems that an asynchronous version could be developed in the firmware -- or NOT, DMA version wouldn't convert the values correctly into the buffer, ref adc.c)

http://docs.micropython.org/en/latest/l ... b.ADC.html

User avatar
pythoncoder
Posts: 3420
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Logging ADC data to SD-card

Post by pythoncoder » Tue Mar 24, 2015 7:50 am

The classic way to deal with this problem is a ring buffer or alternatively a two phase buffer. You can then read data continuously while perfroming periodic time consuming block writes.

A ring buffer has separate read and write pointers (or indexes). You read ADC values on a timer interrupt. The interrupt handler writes the data through the write pointer, increments the pointer, and checks if it's gone past the end of the buffer. If it has, it sets it back to the start. The main program loop checks the amount of unwritten data in the buffer: if it's above a threshold it writes out a block of size equal to the threshold value. The check is done by comparing the read and write indexes to see how far ahead the write index has got. You subtract the read index from the write index. If the result is negative, the write index has wrapped round and you need to add the buffer length to the result (modulo arithmetic).

Handling wrap-round for the read pointer is simple if you choose a threshold of half of the buffer size: it will only ever have two values after completion of a read (0 or len(buf)//2) and won't wrap until the last element has been read. In other words you only need to perform the check at the end and the read activity always handles contiguous elements. This works for thresholds of other integer fractions of the size.

A two phase buffer uses two separate buffers with read and write alternating between the two. There's little to choose between them and I generally use ring buffers. They are actually much simpler to implement than my explanation suggests, once you've got your head round the idea.

In either case the size of the buffer is determined by the amount of time it takes to write out the amount of data you've chosen. This should be a worst case including provision for garbage collection - there's no penalty in too large a buffer aside from RAM use. In terms of implementation it's likely that an array will provide better performance than a Python list.
Peter Hinch

manitou
Posts: 67
Joined: Wed Feb 25, 2015 12:15 am

Re: Logging ADC data to SD-card

Post by manitou » Tue Apr 14, 2015 12:33 am

vikebo wrote: Did another test using timer 2 as the timestamp instead of the micros function, but keeping the values from micros to compare them (red in the plots). Seems like the timer for micros does stop during the jitter, while timer 2 still runs.
Wow, you are right about micros() losing interrupts. The SD driver disables IRQ's and I'm guessing that is the cause of both the pauses in the data collection and lost micros. Here are some data of the inter-sample gap times for timer2 (counting microseconds) and micros.

Code: Select all

sample timer2   micros()
...
29       241     240
30       240     241
31       241     241
32       241     241
33       241     241
34       242     242
35    102721    1721
36       245     244
37       243     243
...
The "normal" inter-sample times are nearly identical, and the spikes (longer gaps) are synchronized, but often timer2 reveals that many milliseconds (102 ms in the example above) have elapsed in the SD driver :!:

Edit: writing to /flash files can also "stop" micros/millis due to flash-erase operation stalling the CPU (unless your code is in RAM)

Post Reply