best way to store datestamps in upython

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Post Reply
User avatar
philwilkinson40
Posts: 59
Joined: Tue Nov 14, 2017 3:11 am
Location: Perth, Australia

best way to store datestamps in upython

Post by philwilkinson40 » Wed Oct 17, 2018 11:14 am

This is a general question from a novice. What is the optimal approach (in Micropython) to store a series of timestamps?

My use case is a bike counter. A radar sensor triggers a Pin on the rising edge of a ESP32 dev board, waking it from DeepSleep. I am recording the time of the external Pin interrupt only, no other data. The accuracy desired is seconds only. I am using an external RTC only to ensure reasonable accuracy over long periods and during power outages.

I am currently just recording the timestamp as the datetime tuple returned using urtc and ds1307 drivers. I am saving this onto a micro SD card.
Would it be better to convert the datetime tuple to an integer used in utime.time() for storage?

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: best way to store datestamps in upython

Post by jickster » Wed Oct 17, 2018 2:47 pm

philwilkinson40 wrote:
Wed Oct 17, 2018 11:14 am
This is a general question from a novice. What is the optimal approach (in Micropython) to store a series of timestamps?
For what purpose?
File-storage efficiency? Runtime access efficiency?

When you access "what is the optimal approach to do XYZ . . ." you have to ask for what purpose you're optimizing otherwise the question is incomplete.

User avatar
pythoncoder
Posts: 3538
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: best way to store datestamps in upython

Post by pythoncoder » Wed Oct 17, 2018 4:08 pm

A practical consideration is how you want to access the file and whether you want it to be human-readable. If the file is only going to be read by code, writing the data as integers to a binary file is simple, efficient in time and file size, and enables random access. For a human-readable file I'd use tuples serialised with ujson. But records could then only be accessed sequentially (because the serialised tuples may vary in length).
Peter Hinch

User avatar
philwilkinson40
Posts: 59
Joined: Tue Nov 14, 2017 3:11 am
Location: Perth, Australia

Re: best way to store datestamps in upython

Post by philwilkinson40 » Wed Oct 17, 2018 11:45 pm

thanks @jickster and @pythoncoder for responding.
The use case at present is simply recording a timestamp on an uSD card. At some point this uSD will be retrieved and data extraction and manipulation will outside the system. So I suppose I meant that 'optimising' means
simple, efficient in time and file size, and enables random access
I am also looking at transmitting the data periodically across a LowPowerWideAreaNetwork (LoRa) where payloads are extremely constrained also. It sounds like the INT is the best option, converting it later to a human readable format if needed.

for interest/laugh my initial attempt at code is below.
  • I am using a WEMOS D32Pro with an internal uSD card reader
  • Pin 34 has a HFS-DC06H radar unit connected which reads high on activation
  • external DS1307 attached
  • I am using LoBo port, though I think my question is relevant for all ports

Code: Select all

def sd_data(data):
    import uos
    uos.sdconfig(uos.SDMODE_SPI, clk=18, mosi=23, miso=19, cs=4)
    uos.mountsd() #not sure if i need to do this everytime after power boot
    f = open('/sd/data.txt', 'a')
    f.write(data)
    f.close()
    uos.umountsd()

def goto_sleep():
    rtc = machine.RTC()
    rtc.wake_on_ext0(machine.Pin(34), 1)
    machine.deepsleep(0)

def get_urtc_time():
    import urtc
    i2c = machine.I2C(0, sda=21, scl=22)
    rtc = urtc.DS1307(i2c)
    now=rtc.datetime()
    return (urtc.tuple2seconds(now))

import machine
import utime
import sys
import urtc

if machine.wake_reason()[0]==0: #0=='Power on reset'
    utime.sleep(20) # top allow ctrl+C into REPL
    sys.exit() #for testing only

if machine.wake_reason()[1]==1: #1=='EXT_0 wake-up'
    output = get_urtc_time()
    sd_data(output)
    goto_sleep()

User avatar
pythoncoder
Posts: 3538
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: best way to store datestamps in upython

Post by pythoncoder » Thu Oct 18, 2018 5:23 am

Looks fine to me :D

As a general point when doing file operations it's best to use a context manager. This guarantees that the file will be closed even if an exception occurs:

Code: Select all

with open('/sd/data.txt', 'a') as f:
    f.write(data)
    # possible other file operations
uos.umountsd() # The file is closed now
So, for example, if you interrupted the program with <ctrl>c while the file was open, it would be closed.

Incidentally you are evidently storing the integers as their ASCII representation, which is fine but isn't ideal for random access. My comments above were intended to describe storing them in binary form in a random access file. This is more space efficient, and since every integer can be guaranteed to consume exactly 4 bytes makes random access straightforward.

You might want to consider binary for radio transmission if minimising the payload size is crucial.
Peter Hinch

Post Reply