best way to store datestamps in upython

[philwilkinson40]

This is a general question from a novice. What is the optimal approach (in Micropython) to store a series of timestamps?

My use case is a bike counter. A radar sensor triggers a Pin on the rising edge of a ESP32 dev board, waking it from DeepSleep. I am recording the time of the external Pin interrupt only, no other data. The accuracy desired is seconds only. I am using an external RTC only to ensure reasonable accuracy over long periods and during power outages.

I am currently just recording the timestamp as the datetime tuple returned using urtc and ds1307 drivers. I am saving this onto a micro SD card.
Would it be better to convert the datetime tuple to an integer used in utime.time() for storage?

[jickster]

This is a general question from a novice. What is the optimal approach (in Micropython) to store a series of timestamps?

For what purpose?
File-storage efficiency? Runtime access efficiency?

When you access "what is the optimal approach to do XYZ . . ." you have to ask for what purpose you're optimizing otherwise the question is incomplete.

[pythoncoder]

A practical consideration is how you want to access the file and whether you want it to be human-readable. If the file is only going to be read by code, writing the data as integers to a binary file is simple, efficient in time and file size, and enables random access. For a human-readable file I'd use tuples serialised with ujson. But records could then only be accessed sequentially (because the serialised tuples may vary in length).

[philwilkinson40]

thanks @jickster and @pythoncoder for responding.
The use case at present is simply recording a timestamp on an uSD card. At some point this uSD will be retrieved and data extraction and manipulation will outside the system. So I suppose I meant that 'optimising' means

simple, efficient in time and file size, and enables random access

I am also looking at transmitting the data periodically across a LowPowerWideAreaNetwork (LoRa) where payloads are extremely constrained also. It sounds like the INT is the best option, converting it later to a human readable format if needed.

for interest/laugh my initial attempt at code is below.

• I am using a WEMOS D32Pro with an internal uSD card reader

• Pin 34 has a HFS-DC06H radar unit connected which reads high on activation

• external DS1307 attached

• I am using LoBo port, though I think my question is relevant for all ports

Code: Select all

def sd_data(data):
    import uos
    uos.sdconfig(uos.SDMODE_SPI, clk=18, mosi=23, miso=19, cs=4)
    uos.mountsd() #not sure if i need to do this everytime after power boot
    f = open('/sd/data.txt', 'a')
    f.write(data)
    f.close()
    uos.umountsd()

def goto_sleep():
    rtc = machine.RTC()
    rtc.wake_on_ext0(machine.Pin(34), 1)
    machine.deepsleep(0)

def get_urtc_time():
    import urtc
    i2c = machine.I2C(0, sda=21, scl=22)
    rtc = urtc.DS1307(i2c)
    now=rtc.datetime()
    return (urtc.tuple2seconds(now))

import machine
import utime
import sys
import urtc

if machine.wake_reason()[0]==0: #0=='Power on reset'
    utime.sleep(20) # top allow ctrl+C into REPL
    sys.exit() #for testing only

if machine.wake_reason()[1]==1: #1=='EXT_0 wake-up'
    output = get_urtc_time()
    sd_data(output)
    goto_sleep()

[pythoncoder]

Looks fine to me

As a general point when doing file operations it's best to use a context manager. This guarantees that the file will be closed even if an exception occurs:

Code: Select all

with open('/sd/data.txt', 'a') as f:
    f.write(data)
    # possible other file operations
uos.umountsd() # The file is closed now

So, for example, if you interrupted the program with <ctrl>c while the file was open, it would be closed.

Incidentally you are evidently storing the integers as their ASCII representation, which is fine but isn't ideal for random access. My comments above were intended to describe storing them in binary form in a random access file. This is more space efficient, and since every integer can be guaranteed to consume exactly 4 bytes makes random access straightforward.

You might want to consider binary for radio transmission if minimising the payload size is crucial.