data write speeds and broken SD cards

[Turbinenreiter]

24 Bytes per cycle, 1k cycles per second - I would fill this up in less than two seconds.

Last thing I worked on: http://fovs.de/category/general/
Rockets are awesome.
Solid fuels motors have very high g-loads (tough water rockets have higher loads - but only very short, about 0.2 seconds) and pretty nasty vibrations. That said, in our experiment that wasn't to big of a problem.

[pythoncoder]

Awesome! I had no idea your rockets were on that scale.

Regards, Pete

[Turbinenreiter]

Oh, no they aren't.
What I'm using the pyboard for are water rockets. On the big one we had something like an Atmeg8 - but as mechanical engineer didn't work with that, I just build the structure.

[Turbinenreiter]

Ok, so on the issue of the breaking SD cards and breaking SD card connectors (#2 today )

Is it possible to remap the SD card to other pins, so I can make a little daugtherboard to stack on top of the pyboard, and have the sd card connector there?

https://github.com/micropython/micropyt ... l/sdcard.c

Code: Select all

void sdcard_init(void) {
    GPIO_InitTypeDef GPIO_Init_Structure;

    // invalidate the sd_handle
    sd_handle.Instance = NULL;

    // configure SD GPIO
    // we do this here an not in HAL_SD_MspInit because it apparently
    // makes it more robust to have the pins always pulled high
    GPIO_Init_Structure.Mode = GPIO_MODE_AF_PP;
    GPIO_Init_Structure.Pull = GPIO_PULLUP;
    GPIO_Init_Structure.Speed = GPIO_SPEED_HIGH;
    GPIO_Init_Structure.Alternate = GPIO_AF12_SDIO;
    GPIO_Init_Structure.Pin = GPIO_PIN_8 | GPIO_PIN_9 | GPIO_PIN_10 | GPIO_PIN_11 | GPIO_PIN_12;
    HAL_GPIO_Init(GPIOC, &GPIO_Init_Structure);
    GPIO_Init_Structure.Pin = GPIO_PIN_2;
    HAL_GPIO_Init(GPIOD, &GPIO_Init_Structure);

    // configure the SD card detect pin
    // we do this here so we can detect if the SD card is inserted before powering it on
    GPIO_Init_Structure.Mode = GPIO_MODE_INPUT;
    GPIO_Init_Structure.Pull = MICROPY_HW_SDCARD_DETECT_PULL;
    GPIO_Init_Structure.Speed = GPIO_SPEED_HIGH;
    GPIO_Init_Structure.Pin = MICROPY_HW_SDCARD_DETECT_PIN.pin_mask;
    HAL_GPIO_Init(MICROPY_HW_SDCARD_DETECT_PIN.gpio, &GPIO_Init_Structure);
}

Am I rigth in thinking that this is the part I'd need to change to remap?

edit:
Answer: No, as this uses SDIO, SD card on the header pins would have to use SPI - SD card over SPI hast yet to be implemented.

[Damien]

Answer: No, as this uses SDIO, SD card on the header pins would have to use SPI - SD card over SPI hast yet to be implemented.

Right. The hardware has builtin SDIO support and the driver is designed around that. The pins are hardwired.

It's not a priority for me, but if someone wants to submit a driver for SD over SPI, then we can incorporate that into master. With such a driver, we could "mount" SD cards over SPI in the root directory, something like:

Code: Select all

>>> pyb.mount_sd(pyb.SPI(1))
>>> os.listdir('/')
['flash', 'sd', 'spi1']

[Turbinenreiter]

Thanks to the sheer awesomeness of our beloved devs, sd via spi is now implemented.

It's also been 16 days since this was implemented, but I only discovered it today, despite following the project on Github, which makes me think we have to find a way to better communicate such stuff WITHOUT putting that strain on the devs. Maybe I will try to make a 'what's new' post on the forum once a week or something like that. I think if you sit down once a week and go through the project, closed pulls and issues, you can get that info quiet easily. In fact, I will start right now.

[Martin]

@Turbinenreiter: I pretty much want to do the same thing, high speed logging of accelerometer data onto the SD card. Would you share your code? Would help me quite a bit - I'm new to pyboard!

BTW I want to measure the effectiveness of my bike suspension fork. I intend to place the accelerometer at the top of the fork and see how resulting acceleration changes depending on air pressure and fork type. Without having tried, I assume the acceleration is beyond the +-8 or 16 g typical for most sensors, so I ordered a few ADXL375 (+-200g) instead. Here http://www.veloplus.ch/pdf/veloplus/wei ... sattel.pdf is an interesting measurement of acceleration - on your back, depending on seatpost suspension. I just purchased an ADXL345 breakout board for starters, it is pin (and maybe software) compatible with the ADXL375. For connecting the part, I'm not sure whether I just fix the ADXL375 upside down on a small PCB and solder thin wires directly at the part or remove the ADXL345 from the breakout board and solder the ADXL375 instead - I have equipment for both, I'll see...
If I would do this in C on the STM32, I'd have the following structure:
- Initialize the sensor for 3200 Hz output rate, use FiFo, generate Interrupt when FiFo is full
- interface via SPI (> 2 MHz)
- In the interrupt routine, do 32 times an SPI read (6 bytes each for XYZ 16 bit data), maybe DMA (but not necessarily), unfortunately you can't do a single read for reading out the FiFo and unfortunately2 SPI reads at 2 MHz require an additional delay between reads as the FiFo is so slow.
- Write data in a buffer, when 512 bytes are full, write to SD card (SPI) via DMA.

Not sure if this can be done on the pyboard. Of course a few missed samples would't be the end of the world in this application, but most elegant would be if a seamless recording would work. I am open to suggestions here...

[Turbinenreiter]

@Martin

I managed to get about 900Hz at one time. I since moved to using a Teensy for other reasons.
The main trick to log fast is quite simple - store in binary format and know the buffer size of the sd card.

I don't have much code to share - it's basically a loop with a single

Code: Select all

log.write(b'')

in it. Actually, the SDDatalogger in the example section is the first version of my logger.

I'm also going to use the ADXL375 in the next version.

The only challenge in doing this the way you describe is that you can't allocate memory on the heap in an interrupt routine, but I think it should be possible to do all that without allocating.

When I started with logging, I only got about 50Hz. I managed to speed it up to about 900Hz quickly, but what I was missing the most was profiling tools, like profile or line_profiler.

Code: Select all

# datalogger.py
# Logs the data from the acceleromter to a file on the SD-card

import struct
import pyb
from bmp180 import BMP180
from mpu9150 import MPU9150
from rcv import Reciever
import os
import gc
import sys

def main():

    # create objects
    accel = pyb.Accel()
    blue = pyb.LED(4)
    switch = pyb.Switch()
    cam = pyb.Servo(2)
    pyb.delay(10)
    baro = BMP180('X')
    pyb.delay(10)
    imu = MPU9150('X', 1)
    rcv = Reciever()

    # settings
    baro.oversample_sett = 3
    #servo.calibration(860, 2160, 1500)

    # test i2c
    if len(baro._bmp_i2c.scan()) != 4:
        pyb.LED(2).on()
        print('sensors failed')
    else:
        print('sensors ready')

    # create dir
    if 'log' not in os.listdir('/sd/'):
        os.mkdir('/sd/log')
        print('created log dir')
    else:
        print('log dir already there')

    config = open('/sd/log/config.txt', 'w')
    config.write('AC1, AC2, AC3, AC4, AC5, AC6, B1, B2, MB, MC, MD, oversample_sett, accel_range, gyro_range\n')
    con = baro.compvaldump() + [imu.accel_range(), imu.gyro_range()]
    config.write('{},{},{},{},{},{},{},{},{},{},{},{},{},{}'.format(*con))
    config.close()
    print('wrote config file')

    # loop
    while True:

        # wait for interrupt
        pyb.wfi()

        # start if switch is pressed
        if (rcv.ch[1][0] < -45) or switch():                            # switch(): rcv.ch[1][0] < -45
            pyb.delay(300)                      # delay avoids detection of multiple presses
            blue.on()                           # blue LED indicates file open
            pyb.LED(3).off()
            filename = '/sd/log/log'+str(len(os.listdir('log'))+1)+'.bin'
            print('opening '+filename)
            log = open(filename, 'wb')
            i = 0
            trigger = b'\x00'
            cam.angle(90)
            pyb.delay(100)
            cam.angle(0)
            gc.collect()
            t0 = pyb.millis()/1000
            # until switch is pressed again
            while (rcv.ch[1][0] < 45) and (not switch()):                      #not switch(): rcv.ch[1][0] < 45

                t_start_ms = pyb.millis()
                next(baro.gauge())
                log.write(b''.join([struct.pack('I', t_start_ms),
                                    bytes(accel.filtered_xyz()),
                                    imu.get_accel_raw(),
                                    imu.get_gyro_raw(),
                                    baro.UT_raw,
                                    baro.MSB_raw,
                                    baro.LSB_raw,
                                    baro.XLSB_raw,
                                    trigger]))
                i = i+1
                if i%100 == 0:
                    trigger = para(baro.altitude)
                    dt = t_start_ms/1000-t0
                    sys.stdout.write('writing '+
                                     filename+
                                     ' %d lines, %d s, %d Hz \r'
                                     % (i, dt, i/dt))
                    pyb.LED(1).toggle()

            # end after switch is pressed again
            log.close()                         # close file
            blue.off()                          # blue LED indicates file closed
            print('\nclosed '+filename)
            cam.angle(90)
            pyb.delay(100)
            cam.angle(0)
            pyb.delay(300)                      # delay avoids detection of multiple presses

main()

[Martin]

Actually it turned out to be much simpler than I thought. Instead of using interrupts and DMA, I just enabled the ADXL345 FiFo and configured the INT1 output for "watermark" mode with 20 entries (fifo is 32 entries). I set the ADXL345 for 3200 Hz sampling and monitor the INT1 line. If it is 1, I read 20 samples from the FiFo and write them to the SD card. This is pretty simple and works out quite well:

Code: Select all

f=open('/sd/test.bin','wb')
for k in range(160*5):
    # wait for FiFo to be filled
    while int1pin.value()==False:
        pass
    # read 20 values and write to SD card
    for i in range(20):
        nCS.low();spi.send_recv(CMD_RD,b7);nCS.high()
        f.write(b7)

f.close()

I just record 5 seconds here, I'll have to add stopping with a switch and incrementing the file name.

What was a great help (and what I recommend to anyone) is hooking up a logic analyzer to the SPI pins. I used the Saleae logic, quite affordable and with SPI (and others) protocol analyzers. Here's a screenshot:

Unbenannt.JPG (154.08 KiB) Viewed 9325 times

The ADXL345 is connected via SPI at 5 MHz clock rate (I2C is too slow and I hate the protocol). At 3200 Hz sampling rate, the INT1 goes high every 20/3200 = 6.25 ms. The readout/file write takes 1.3 ms, so there's plenty of time. If the write buffer has filled up to 512, a sector is written to the SD card, which takes about 1.2 ms (427 kB/s).

I wrote a small import function in Matlab (I use it daily and I'm a total Python newbie, so this was quicker), here's a sample figure:

The attachment Unbenannt.JPG is no longer available

(somehow this image inlining doesn't seem to work in the preview)
Next I'll have to replace the ADXL345 on the board with the ADXL375 (which I got as samples from Analog), QFN rework...

[Turbinenreiter]

Nice!
I'm gonna have good use for this soon.