[SOLVED] Using btree for key-value-database (OSError: 0)

[Roberthh]

You cannot trade flash against RAM. Whatever you did with _boot.py, boot.py and main.y does not help you substantially with RAM shortage. Better leave it as it is. The only thing that helps with RAM is to put as much as possible scripts into frozen bytecode and avoind duplicating of data in you code.
Due to that small heap size, pure Python scripts on ESP2866 are limited to about 250 lines of code, as a rule of thumb. Using precompiled code doubles that size. If you need to run larger scripts, you have to use another device. The ESP32 port gives you about 90 k of heap = about 1000 lines of code in pure Python. There is a port of Micropython on ESP32 which supports 4 MB of RAM. But these devices are rare and this special port deviates quite a bit from the ports supported by Damien and Paul. The newer devices of pycio.io will have 4 MB of RAM, but there is not firmware support yet (and no BTREE module).

[crizeo]

Ok, thanks for the answer! I will have a look at the ESP32 and precompile my code for the ESP8266 meanwhile.

I read in some other post that it might help splitting my code into multiple modules. Do you think that this could help? I got ~50 constants (const()), four classes and a main function (setting up the objects and running a main loop).

[pythoncoder]

...Due to that small heap size, pure Python scripts on ESP2866 are limited to about 250 lines of code, as a rule of thumb. Using precompiled code doubles that size...

Interesting observation. I'm beyond that limit with frozen bytecode. My application comprises several modules. The largest is about 450LOC, with two others on the order of 200LOC each, plus smaller modules. It uses about 20K of RAM (16K free) - this is periodically reported at runtime - and seems stable.

I'm still unclear as to whether frozen bytecode on the ESP8266 is moved to RAM prior to execution. I think some experimentation is called for.

[Roberthh]

@pythoncoder: the 250 LOC number is for pure Python code imported & compiled from the FAT file system ofthe ESP8266, not for frozen bytecode. That resides in flash and is executed virtually from flash. While the flash chip is physically connected via SPI, it is mapped into the address space of the CPU and is directly adressable, at least the lowest 1 MByte. The ESP caches the active section of flash in a dedicated RAM area, reserved for that purpose. Therefore the binary code as well as frozen bytecode it looks like executed from flash, and the code size limit for frozen bytecode is the 1MByte address range for direct adressable flash memory. The data used by the code has to reside in RAM.

For example, my little on-board editor has about 1000 LOC, but embedded as frozen bytecode and imported it takes about 1.5 k RAM for itself, most of that for a screen buffer.

@crizeo: Deferred import of modules NOT in frozen bytecode helps sometimes, since the compile phase during impoert needs some RAM, but you'll never get beyond the size of pure precompiled code files. Frozen bytecode is the only measure that breaks the RAM memory restrictions for code.

[pythoncoder]

Thanks, Robert - very informative.

[cefn]

The data used by the code has to reside in RAM.

However can bytes and strings declared in source files be kept and read from flash?

[Roberthh]

String and bytes constants declared in frozen bytecode will not be copied to RAM by importing. However if you assign these to anothr object, they will be copied. You can avoid that, when you need slices of these, by using memoryview, e.g.:

Code: Select all

# within the frozen bytecode
data = "very large string .................... and much more"

# when using
slice = memoryview(data[start:stop])

slice will not be a copy of the data, but a kind of pointer object, which can be used for assignment and in function call.

[deshipu]

I believe you meant:

Code: Select all

slice = memoryview(data)[start:stop]

The order of operations is important here.

[Roberthh]

Yup!

[beantree]

This fix has not solved my problem.

Code: Select all

>>> records()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 11, in records
  File "db.py", line 43, in db_open
OSError: 0

Code: Select all

class db():
    def __init__(self): 
        self._dbfile='/sd/mydb'
            
    def db_open(self):

        try:
            self.f = open(self._dbfile, "r+b")
        except OSError:
            self.f = open(self._dbfile, "w+b")

        # Now open a database itself
        self.db = btree.open(self.f, pagesize=1024)
        

The error refers to the last line.

My code is sending 100 records from the database via GPRS and then deleting those records. I have only about 1600 records in the database, each has 69 bytes.
The error does not occur straight away, but persists when it does until I do a hard reset. Ctrl-d does not clear the issue.
Any ideas of what I might try?