Page 1 of 1
best way to store datestamps in upython
Posted: Wed Oct 17, 2018 11:14 am
by philwilkinson40
This is a general question from a novice. What is the optimal approach (in Micropython) to store a series of timestamps?
My use case is a bike counter. A radar sensor triggers a Pin on the rising edge of a ESP32 dev board, waking it from DeepSleep. I am recording the time of the external Pin interrupt only, no other data. The accuracy desired is seconds only. I am using an external RTC only to ensure reasonable accuracy over long periods and during power outages.
I am currently just recording the timestamp as the datetime tuple returned using
urtc and
ds1307 drivers. I am saving this onto a micro SD card.
Would it be better to convert the datetime tuple to
an integer used in utime.time() for storage?
Re: best way to store datestamps in upython
Posted: Wed Oct 17, 2018 2:47 pm
by jickster
philwilkinson40 wrote: ↑Wed Oct 17, 2018 11:14 am
This is a general question from a novice. What is the optimal approach (in Micropython) to store a series of timestamps?
For what purpose?
File-storage efficiency? Runtime access efficiency?
When you access "what is the optimal approach to do XYZ . . ." you have to ask for what purpose you're optimizing otherwise the question is incomplete.
Re: best way to store datestamps in upython
Posted: Wed Oct 17, 2018 4:08 pm
by pythoncoder
A practical consideration is how you want to access the file and whether you want it to be human-readable. If the file is only going to be read by code, writing the data as integers to a binary file is simple, efficient in time and file size, and enables random access. For a human-readable file I'd use tuples serialised with ujson. But records could then only be accessed sequentially (because the serialised tuples may vary in length).
Re: best way to store datestamps in upython
Posted: Wed Oct 17, 2018 11:45 pm
by philwilkinson40
thanks @jickster and @pythoncoder for responding.
The use case at present is simply recording a timestamp on an uSD card. At some point this uSD will be retrieved and data extraction and manipulation will outside the system. So I suppose I meant that 'optimising' means
simple, efficient in time and file size, and enables random access
I am also looking at transmitting the data periodically across a LowPowerWideAreaNetwork (LoRa) where payloads are extremely constrained also. It sounds like the INT is the best option, converting it later to a human readable format if needed.
for interest/laugh my initial attempt at code is below.
- I am using a WEMOS D32Pro with an internal uSD card reader
- Pin 34 has a HFS-DC06H radar unit connected which reads high on activation
- I am using LoBo port, though I think my question is relevant for all ports
Code: Select all
def sd_data(data):
import uos
uos.sdconfig(uos.SDMODE_SPI, clk=18, mosi=23, miso=19, cs=4)
uos.mountsd() #not sure if i need to do this everytime after power boot
f = open('/sd/data.txt', 'a')
f.write(data)
f.close()
uos.umountsd()
def goto_sleep():
rtc = machine.RTC()
rtc.wake_on_ext0(machine.Pin(34), 1)
machine.deepsleep(0)
def get_urtc_time():
import urtc
i2c = machine.I2C(0, sda=21, scl=22)
rtc = urtc.DS1307(i2c)
now=rtc.datetime()
return (urtc.tuple2seconds(now))
import machine
import utime
import sys
import urtc
if machine.wake_reason()[0]==0: #0=='Power on reset'
utime.sleep(20) # top allow ctrl+C into REPL
sys.exit() #for testing only
if machine.wake_reason()[1]==1: #1=='EXT_0 wake-up'
output = get_urtc_time()
sd_data(output)
goto_sleep()
Re: best way to store datestamps in upython
Posted: Thu Oct 18, 2018 5:23 am
by pythoncoder
Looks fine to me
As a general point when doing file operations it's best to use a context manager. This guarantees that the file will be closed even if an exception occurs:
Code: Select all
with open('/sd/data.txt', 'a') as f:
f.write(data)
# possible other file operations
uos.umountsd() # The file is closed now
So, for example, if you interrupted the program with <ctrl>c while the file was open, it would be closed.
Incidentally you are evidently storing the integers as their ASCII representation, which is fine but isn't ideal for random access. My comments above were intended to describe storing them in binary form in a random access file. This is more space efficient, and since every integer can be guaranteed to consume exactly 4 bytes makes random access straightforward.
You might want to consider binary for radio transmission if minimising the payload size is crucial.