2nd ESP8266 projects stretch goal - Embedded Database (btree module)

All ESP8266 boards running MicroPython.
Official boards are the Adafruit Huzzah and Feather boards.
Target audience: MicroPython users with an ESP8266 board.
pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

2nd ESP8266 projects stretch goal - Embedded Database (btree module)

Post by pfalcon » Thu Jun 16, 2016 8:32 pm

While MQTT client waits for more feedback and testing, work on another stretch goal - a simple embedded database suitable even for the smallest devices - has started. As with MQTT, this is more of a progress report than announcement of ready to use feature, but if you're a developer interested in this area, you can find more details at https://github.com/micropython/micropython/issues/2188 . As usual, initial prototyping and development happens with unix port, and bringing this down to esp8266 will certainly require additional porting and optimization, so may take some time.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: 2nd ESP8266 projects stretch goal - Embedded Database (btree module)

Post by pfalcon » Sat Jul 02, 2016 2:47 pm

The "btree" module is now enabled for "unix" port. Note that it has "provisional" status and some API details WILL change - like way(s) to construct btree object and constant names. But the main data access interface should be quite stable - it follow Python's "dictionary protocol" as close as possible, with extensions done in a natural way to supported ordered key range enumeration. You can see usage example in a unit test for the module, https://github.com/micropython/micropyt ... /btree1.py .

It will take more effort (and time) to port module to baremetal ports (like ESP8266), and that's why I say that details of API will change (as a result of this porting). But all in all, if you're interested in usage of MicroPython as a full-stack application framework (and native database support is important part of that), you can start to play with it using the unix port.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: 2nd ESP8266 projects stretch goal - Embedded Database (btree module)

Post by pfalcon » Mon Aug 01, 2016 5:37 pm

Btree module has now been refactored to support esp8266 and was enabled in the master. It passes basic unit test, but more testing is required for it. If you're interested in database support, feel free to give it a try. A usage example is here: https://github.com/micropython/micropyt ... /btree1.py
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

mad474
Posts: 60
Joined: Sun Dec 29, 2013 7:48 pm

Re: 2nd ESP8266 projects stretch goal - Embedded Database (btree module)

Post by mad474 » Mon Aug 08, 2016 8:45 pm

Congratulations Paul! I've been playing with all available db-methods of todays build since an hour and didn't manage get the ESP8266 in to trouble (besides expected and well-established memory issues: Testing on a 1MB ESP-01).

Heaps of shame on me (who was one of the loudest crying for a DB) for not having time in these months to invest more than a few hours of poking around. For me and others(?) with a bad conscience some kind of sponsoring method may be a big relief ;) - no serious.

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: 2nd ESP8266 projects stretch goal - Embedded Database (btree module)

Post by pfalcon » Tue Aug 09, 2016 8:00 pm

mad474 wrote:Congratulations Paul! I've been playing with all available db-methods of todays build since an hour and didn't manage get the ESP8266 in to trouble (besides expected and well-established memory issues: Testing on a 1MB ESP-01).

Heaps of shame on me (who was one of the loudest crying for a DB) for not having time in these months to invest more than a few hours of poking around. For me and others(?) with a bad conscience some kind of sponsoring method may be a big relief ;) - no serious.
Thanks for trying it out, it's actually quite helpful to know it has worked for somebody before the release. Otherwise, I'm pretty got used to the situation when some feature gets some attention/a kind of adoption month(s) after it was released (when developers are already deep into something else, so can't even give enough/immediate attention to shower of questions starting ;-) ). Anyway, as you know, I was quite happy to work on an embedded DB, and wanted to deliver it early exactly to let people to start playing with it sooner, as it has big potential not just for ESP8266, but for MicroPython as a whole.

Definitely don't feel bad and feel free to look into it at your own pace (it's a summer after all). You can always clear conscience by posting examples and answering other users' questions ;-). And well, currently it's all OK with MicroPython development - the Kickstarter campaign gave enough momentum for now. The key is sustainability, so if later we need to think how to keep up MicroPython further, your ideas may be helpful. Thanks for your support!
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

User avatar
jgriessen
Posts: 191
Joined: Mon Sep 29, 2014 4:20 pm
Contact:

Re: 2nd ESP8266 projects stretch goal - Embedded Database (btree module)

Post by jgriessen » Sat Aug 13, 2016 2:53 pm

But all in all, if you're interested in usage of MicroPython as a full-stack application framework (and native database support is important part of that), you can start to play with it using the unix port.
And now the ESP8266! This IS getting interesting. I can imagine a vending machine with its own website that connects to a customer's phone or BLE dongle for a low cost money transaction. Maybe something similar to https://www.dotdashpay.com/ but less expensive and probably for bitcoin and ethereum also.

That might want the next ESP32 with BLE, since that is one money dongle comm method.
John Griessen blog.kitmatic.com

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: 2nd ESP8266 projects stretch goal - Embedded Database (btree module)

Post by pfalcon » Tue Apr 04, 2017 4:05 am

It occurred to me that all this time btree module stayed undocumented. Sorry for this omission, docs are now added: http://docs.micropython.org/en/latest/e ... btree.html . Proofreading is welcome.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

SpotlightKid
Posts: 463
Joined: Wed Apr 08, 2015 5:19 am

Re: 2nd ESP8266 projects stretch goal - Embedded Database (btree module)

Post by SpotlightKid » Tue Apr 04, 2017 12:08 pm

One remark about the docs: I don't see any mention of what the implicit sort order of keys is. If it is alphabetical, which I assume, which encoding is used and is it by code point or lexicographical? Can the sort method be changed?

Apart from that the docs have the (usual ;-)) inconsistent use of single and double backticks, which is all over the docs. I admit, it's easy to get this wrong, especially when one often writes Markdown at other times.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: 2nd ESP8266 projects stretch goal - Embedded Database (btree module)

Post by pythoncoder » Tue Apr 04, 2017 4:31 pm

Does Python support multiple ways of sorting bytes instances?
Peter Hinch
Index to my micropython libraries.

SpotlightKid
Posts: 463
Joined: Wed Apr 08, 2015 5:19 am

Re: 2nd ESP8266 projects stretch goal - Embedded Database (btree module)

Post by SpotlightKid » Tue Apr 04, 2017 4:56 pm

I am not sure. Anyway, I was guessing that the sorting would be implemented in C.

Post Reply