Long delays in handling "large" datasets

All ESP32 boards running MicroPython.
Target audience: MicroPython users with an ESP32 board.
Post Reply
Daniel97
Posts: 3
Joined: Thu Mar 26, 2020 2:54 pm

Long delays in handling "large" datasets

Post by Daniel97 » Thu Mar 26, 2020 7:12 pm

Hello everyone,
I'm using a ESP32 to transmit serveral kB worth of sensor data via WiFi.
I store these in a textfile where one line represents a sample:

Code: Select all

...
22.5,42.0,1002.2,8.3
22.5,42.3,1002.2,8.3
22.5,42.5,1002.2,128.3
...
I need to push a fresh datavector on the file, while popping the oldest. My solution for this is working, but really slow. Adding one new element takes about 10s.
I basically use a new file to copy the lines to, and destroy the "donor" file at the end:

Code: Select all

def HandleMeasBuffer(newData):
  print("start transfer")
  foundFiles = os.listdir()
  
  if "log_A.txt" in foundFiles:
    log_A = True
    file1 = open("log_A.txt", 'r')
    file2 = open("log_B.txt", 'w+')
  else:
    log_A = False
    file1 = open("log_B.txt", 'r')
    file2 = open("log_A.txt", 'w+')

  for num, f1 in enumerate(file1):
    if num > 0: #forget first, oldest line
        file2.write(f1)

  file2.write("\n"+newData)
  file1.close()
  file2.close()
  if log_A:
    os.remove("log_A.txt")
  else:
    os.remove("log_B.txt")
  print("end transfer")
    
Does anyone have any suggestions how to improve the performance of the script? Or is there a better way of doing this task?
Sorry for the dumb question, I'm really not used to think much about system resources when it comes to regular Python :oops:
Thanks in advance for your help

OutoftheBOTS_
Posts: 847
Joined: Mon Nov 20, 2017 10:18 am

Re: Long delays in handling "large" datasets

Post by OutoftheBOTS_ » Thu Mar 26, 2020 9:45 pm

10s seems quite long ans surprised me.

I would look at moving blocks of data not 1 line at a time.

Daniel97
Posts: 3
Joined: Thu Mar 26, 2020 2:54 pm

Re: Long delays in handling "large" datasets

Post by Daniel97 » Thu Mar 26, 2020 10:18 pm

Thanks for your reply! I will try that out tomorrow.

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Long delays in handling "large" datasets

Post by dhylands » Fri Mar 27, 2020 4:23 am

I would only append to the file. When it gets to some size (or number of entries) then close the file and open a new open. To get rid of old stuff, delete the oldest file. Trying to treat a file like some type of in-memory structure is a mistake.

Daniel97
Posts: 3
Joined: Thu Mar 26, 2020 2:54 pm

Re: Long delays in handling "large" datasets

Post by Daniel97 » Fri Mar 27, 2020 5:06 pm

I played around a bit and ended up using dhylands advice.
Runtime is about 15 times faster. Thanks guys

Post Reply