[SOLVED] Using btree for key-value-database (OSError: 0)

All ESP8266 boards running MicroPython.
Official boards are the Adafruit Huzzah and Feather boards.
Target audience: MicroPython users with an ESP8266 board.
crizeo
Posts: 42
Joined: Sun Aug 06, 2017 12:55 pm
Location: Germany

Re: Using btree for key-value-database (OSError: 0)

Post by crizeo » Tue Aug 08, 2017 3:00 pm

Sounds bad...

Do you know any alternative that I could use?

User avatar
Roberthh
Posts: 2222
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: Using btree for key-value-database (OSError: 0)

Post by Roberthh » Tue Aug 08, 2017 3:24 pm

I raised an issue in the github repository. Paul or Damien should see it. Maybe they know an answer.

User avatar
Roberthh
Posts: 2222
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: Using btree for key-value-database (OSError: 0)

Post by Roberthh » Tue Aug 08, 2017 4:34 pm

Got it working: you have to set pagesize in btree.open to a small value, like 512 or 1024, like:

Code: Select all

db = btree.open(dbfile, pagesize=1024)

crizeo
Posts: 42
Joined: Sun Aug 06, 2017 12:55 pm
Location: Germany

Re: Using btree for key-value-database (OSError: 0)

Post by crizeo » Tue Aug 08, 2017 5:42 pm

:o

You are awesome! Thank you so much for your help!!!

User avatar
Roberthh
Posts: 2222
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: Using btree for key-value-database (OSError: 0)

Post by Roberthh » Tue Aug 08, 2017 6:43 pm

90 minutes ago I started to create the 500000 entry database on the ESP8266. Being at 125000 now. On the PC it takes like 20 seconds.

P.S.: Finally, it was indeed a RTFM thing. But I understood it only after sneaking through the btree code.

Update: After almost 13 hours now at 400,000 entries, and still building up. It's getting very slow. There's no real value letting it go up to 500,000, but I'll let it go.
Last edited by Roberthh on Wed Aug 09, 2017 5:55 am, edited 1 time in total.

crizeo
Posts: 42
Joined: Sun Aug 06, 2017 12:55 pm
Location: Germany

Re: Using btree for key-value-database (OSError: 0)

Post by crizeo » Tue Aug 08, 2017 7:21 pm

Fortunately I can create it on my PC and only have to do few changes, so this will work for me. You saved me :D

User avatar
pythoncoder
Posts: 4641
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: [SOLVED] Using btree for key-value-database (OSError: 0)

Post by pythoncoder » Wed Aug 09, 2017 5:30 am

@Roberthh Well done! That really needs documenting.
Peter Hinch

User avatar
Roberthh
Posts: 2222
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: [SOLVED] Using btree for key-value-database (OSError: 0)

Post by Roberthh » Wed Aug 09, 2017 5:40 am

@pythoncoder Documentation is one good option. I think it's better to change the default value, maybe depending on the port. That's an pretty easy change, and may be better, considering that many people read the docs only as last resort. I do not want to exempt myself from that (although my last resort is digging in the sources).

User avatar
pythoncoder
Posts: 4641
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: [SOLVED] Using btree for key-value-database (OSError: 0)

Post by pythoncoder » Wed Aug 09, 2017 5:58 am

Agreed.
Peter Hinch

crizeo
Posts: 42
Joined: Sun Aug 06, 2017 12:55 pm
Location: Germany

Re: [SOLVED] Using btree for key-value-database (OSError: 0)

Post by crizeo » Thu Aug 10, 2017 7:15 pm

Another question regarding RAM: Is it possible to increase heap size if I do not need any flash memory?

I modified the firmware so that I got the sdcard module frozen (and deleted some modules that I don't need). The inisetup.py also does not create the boot.py. I modified _boot.py so that my sdcard will be mounted and boot.py/main.py will be ran from there. Btw: Would it be useful to increase the "SDK params storage" value in flashbdev (currently set to 20480)? Simply changing static heap in main.c from 36 to 48 * 4096 did not do the trick of course... Flash memory size is 860*4096 B (which I don't required).

I would like to run even very large scripts (plain python code located on the sd card), but don't need the on-chip flash memory for my files in return. Currently, my 14 kB file fails (11 kB was working). Is there anything (apart from precompiling/freezing) I can do about this?
Last edited by crizeo on Thu Aug 10, 2017 8:38 pm, edited 1 time in total.

Post Reply