Save data to flash

[Jurkylius]

Hello,
now I using json config file to save some settings but after when I write too many data to file I have problem with memory.
Can I read only specific row in file and not whole file?
I rewrite some library from arduino c and want open dict of specific key. See next:

Code: Select all

def begin(self, key, readOnly = True):
        self.readOnly = readOnly
        if(self.data == None):
            with open(self.fileName, 'r') as data_file:
                self.data = ujson.loads(data_file.read())
                data_file.close()
        if(self.data != None):
            self.key = key
            self.actualData = self.data[key]
            if(self.actualData != None):
                return True
        return False

[dhylands]

The JSON file format doesn't really lend itself to reading just one particular entry.

You really want a file format (probably binary) where each entry takes up the same amount of space. Then if you want entry 15, you can calculate the offset within the file where entry 15 starts and read a single entry into memory.

Even if you used a streaming JSON reader, updating an entry would also be problematic (you'd wind up having to read/write) the entire file.

Using a binary format where each entry takes up the same amount of space, allows for updates which don't require reading/writing the entire file.

[pythoncoder]

I think your answer may be the btree module which enables you to create a dictionary on a filesystem and efficiently access it by key.

If you must encode a dict there are more efficient ways than JSON: see my comments on Protocol Buffers here. This also discusses `ustruct` which is a way to write pure binary fixed record size data as recommended by @dhylands. JSON is designed to be human readable and is not efficient. CPython's Pickle is much better (non human-readable), but MicroPython's is not.

A fixed record size binary file implies replacing the dict with a set of key-value pairs and iterating over them until you match your key - unless you have a means of computing the offset from the key, in which case you only need to store the values and it becomes highly efficient. This works well with highly structured data such as bitmapped fonts.

I suspect that your data may not be amenable to fixed size records.

If, even with better encoding, the dict is still too big then you'll have to consider other data structures which can be read incrementally such as key-value pairs. Or perhaps the dict could be split into several smaller ones. You trade substantially longer access time for smaller RAM usage.

[Jurkylius]

Thanks for answers.
I try using btree database and have problem with save data.
Than I want ask for write mode. If I use open with "w+b" mode than I save new data and can update existing or .. ?
I create simple data library.

Code: Select all

import btree

class DataThree:
    
    database = None
    file = None
    actualKey = None
    
    def __init__(self):
        pass
    
    def openDatabase(self, readMode):
        if not readMode:
            self.file = open("config","w+b")
        else:
            try:
                self.file = open("config","r+b")
            except OSError:
                self.file = open("config","w+b")
        self.database = btree.open(self.file)
        return True if self.database != None else False
            
    def begin(self,key,readMode = True):
        self.actualKey=key
        return self.openDatabase(readMode)
    
    def put(self,key,data):
        if(self.database != None):
            self.database[(self.actualKey+key).encode("utf-8")] = data.encode("utf-8") if type(data) is str else str(data).encode("utf-8")
        else:
            print('Databaza sa nenasla!')
            
    def getString(self,key):
        try:
            if(self.database != None):
                if(self.database[(self.actualKey+key).encode("utf-8")] in self.database):
                    return self.database[(self.actualKey+key)].decode("utf-8")
        except KeyError:
            return None
        
    
    def getInt(self,key,defaultValue=None):
        data = self.getString(key)
        if(data != None):
            return int(self.getString(key))
        else:
            return defaultValue
    
    def getBool(self,key,defaultValue=None):
        data = self.getString(key)
        if(data != None):
            return data.lower() in ("true","yes","t","1")
        else:
            return defaultValue
        
    def end(self):
        self.database.flush()
        self.database.close()
        self.file.close()

After I use save data:

Code: Select all

if(pref.begin(definition.TYPE_DAY_NIGHT.lower(), False)):
     pref.put("hour_on", self.hourOpk)
     pref.put("min_on", self.minOpk)
     pref.put("hour_off", self.hourCyk)
     pref.put("min_off", self.minCyk)
     pref.end()

[jimmo]

Yes, the "+" in "w+b" is "update" mode which avoids the file being truncated.