Page 1 of 1

transferring btree files from Python to Micropython

Posted: Sun Jan 10, 2021 11:45 am
by tcornall
Hi All and especially to the developers and maintainers of the btree module.
For Micropython apps, btree is a wonderful boon, allowing a file-backed dict-like database that can be huge compared to the usually trivial memory space of a Micropython board (I'm using an openMV board)

However, I'd love to be able to process big stuff, like Openstreetmaps xml map databases, into btree files using Python on a desktop, then transfer the files to the Micropython board for actual run-time use.

(Low powered, large-ish screen, e-ink based GPS device, anyone?)

My problem is that although I have got code to work on Linux, using berkeleydb.btopen() and on openMV Micropython using btree.open(), I cannot seem to make the files interchangeable.

I kinda expect them to be, because Micropython btree is reportedly based on berkelydb, but 'based on' does not mean compatible necessarily...

Bottom-line is: Does anyone know if this should work and I am just messing up the implementation/file-copy or is it known to be a futile goal?

If it isn't ever going to work, anyone got any suggestion on how I can do this? It does need to be something like btree because the map is never going to fit into the openMV memory all at once... besides, btree on Micropython is kinda lovely. Simple, efficient and available.

Here's some code, because we all love code:

Code: Select all

   #For the openMV Micropython
import btree
#now test btree writing a simple file
dbf=open('fred.btr', 'wrb')  #wrb so i can write it
db=btree.open(dbf)
nid = 1
nid_bytes=nid.to_bytes(6,'little') # 6 bytes, little endian. To match the Linux implementation
ln='one'
db[nid_bytes]=bytes(ln, 'utf-8')  # have to encode everything in bytes
print(db[nid_bytes])
db.close()
dbf.close()
# now see if you can reopen and read it
dbf=open('fred.btr', 'rb')  #rb read-only
db=btree.open(dbf)  #rb so it is read only
print(db[nid_bytes])
for key, value in db.items():
    print('key=',key,'value=',value)
dbf.close()
db.close()
That works and produces:
b'one'
b'one'
key= b'\x01\x00\x00\x00\x00\x00' value= b'one'

Then on a Linux, the code is fairly similar:

Code: Select all

  # for the Linux
import berkeleydb
db=berkeleydb.btopen('testLinuxBerkeleydb.btr', 'n')    #always make a new one
nid = 1
nid_bytes=nid.to_bytes(6,'little')
ln='one'
db[nid_bytes]=bytes(ln, 'utf-8')  # does setting the size and endianness of the key make a difference? Yes it does.
print(db[nid_bytes])

db.close()
db=berkeleydb.btopen('testLinuxBerkeleydb.btr', 'r')  # read-only
print(db[nid_bytes])
for key, value in db.items():
    print('key=',key,'value=',value)
db.close()
That also works (bar cut and paste errors), and produces the same output as the Micropython version:
b'one'
b'one'
key= b'\x01\x00\x00\x00\x00\x00' value= b'one'

Trouble is that when I do a hexdump on the files, they are quite different. Also, when I copy the Linux produced file to the openMV board and try to read it (with code designed to open it read-only and NOT to overwrite it) it fails and gives an OSError 79 (on which I can't find any doco...)

Re: transferring btree files from Python to Micropython

Posted: Mon Jan 11, 2021 12:58 am
by jimmo
tcornall wrote:
Sun Jan 10, 2021 11:45 am
I kinda expect them to be, because Micropython btree is reportedly based on berkelydb, but 'based on' does not mean compatible necessarily...
This could be difficult to track down, there could be many reasons.

One possibility worth investigating is to use the Unix (or Windows) port of MicroPython to do the generation of the btree file (as it uses the exact same btree implementation as the embedded ports).

It might require a two-step process -- CPython to something that MicroPython supports (e.g. JSON), then Unix-MicroPython from JSON to btree.

Re: transferring btree files from Python to Micropython

Posted: Mon Jan 11, 2021 2:47 am
by tcornall
Ah, @jimmo, I did look briefly into a Micropython that could run on the Windows or Linux, but thought it was too fiddly. Good suggestion though, I'll have another look.

I had another thought I liked too. What if the Python code output text that was actually executable Micropython code? i.e. use the Python on the PC to read the xml, then write out a complete micropython program as text strings, with all the variables fully expanded into literal Micropython commands to create the btree. Then run that program on the Micropython board to actually do it. Should work, as long as limits on Micropython code files isn't too weeny.
.... e.g.

Code: Select all

#Python code to make a Micropython program with literal values from xml
print("import btree") #output the Micropython code (actually write into a file)
#.... here do the thing you gotta do to read the xml and get variables.......
print("f=open('filename')")  
print("d=btree.open(f)")
nid=some_variable_from_xml   
nid_bytes=nid.to_bytes(6,'little')   #convert to bytes for btree
ln=some_string_from_xml
ln_bytes=bytes(ln, 'utf-8')
print("d[{}]={}".format(nid_bytes,ln_bytes)) #MP code to start btree database filling-in
#...... much much more filling in the database.......
print("d.close()")
print("f.close()")
I don't think I've come across a coding pattern like this before. It's not a compiler, though it is similar. There should be name for it though.

Re: transferring btree files from Python to Micropython

Posted: Mon Jan 11, 2021 6:31 am
by tcornall
Yeah, that idea of getting the Python to output Micropython code and then running that did work to make the btree file on the openMV board, although I see memory issues happening when I run the Micropython if I try to include all 30,000 entries from the map in the btree, also if I don't call flush() quite frequently. I can add about 800 items to the btree file per program which is not great... but it does work, and each program is pretty fast to run.

I could write the btree entries out to a text file from Python, then read them into the Micropython I suppose, which would keep the program line-count down, if that is the issue. i.e. define my own simple intermediate file format with the corresponding btree database entry on one line.
I wonder if I'd still have the same memory issues then.
Early experiments found I could use the Micropython to read the large xml file one line at a time and if it was a 'node' on that line, store it into a btree. That worked ok for 30,000 nodes. Hmmm, so why do I have issues with my programatic approach? Mystery.

What I really need is a board that'll run true Python with really large memory an' a SD card and' low power mode an' SPI an' a RTC to wake it up an' .....
Pi Zero maybe, but it is too hungry for current when running and doesn't have a nice 'sleep' pow-power mode. Just off and on.

Re: transferring btree files from Python to Micropython

Posted: Mon Jan 11, 2021 8:37 am
by pythoncoder
tcornall wrote:
Mon Jan 11, 2021 2:47 am
Ah, @jimmo, I did look briefly into a Micropython that could run on the Windows or Linux...
The Unix build is easy to run under Linux. Compile it and it just works.

Using CPython to creating MicroPython source code dynamically has been done before, e.g. here. In this instance the aim is to produce MicroPython source files which can be frozen as bytecode, which is highly RAM efficient for various reasons, not least because the code is precompiled to bytecode as part of the freezing process. Sending sourcecode to a MicroPython host is costly in RAM because the onboard compiler has to run to convert it to bytecode.

Re: transferring btree files from Python to Micropython

Posted: Mon Jan 11, 2021 11:44 am
by tcornall
Thanks pythoncoder.
In the end, the way I solved this was to use CPython on Windows PC to rapidly do the xml parsing, write out to a simply parsed intermediate text file, copy that to the embedded Micropython board's SD card, then read the textfile into a Micropython program and build the btree file onto the SD card from it. Somewhat like jimmo suggested.

The approach of writing out executable MicroPython was plagued by memory faults, probably caused what pythoncoder was alluding to.