Compile Btree on STM32

Discussion and questions about boards that can run MicroPython but don't have a dedicated forum.
Target audience: Everyone interested in running MicroPython on other hardware.
Post Reply
pyro25
Posts: 5
Joined: Mon Jul 20, 2020 2:57 pm

Compile Btree on STM32

Post by pyro25 » Mon Jul 20, 2020 3:03 pm

Hi !

I'm looking to compile btree in the stm32 port, which is disabled by default. When activating it, it goes as below
I'm running Ubuntu 16.04.4 LTS

my current state of brainstorming goes like
* modbtree is a submodule of micropython - possible to modify, just not ideal
* newlib is deb that's part of ubuntu - does not sound good to modify
* it would be cool to be able to patch outside of these two...

Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
CC ../../extmod/modbtree.c
In file included from ../../extmod/modbtree.c:37:0:
../../lib/berkeley-db-1.xx/PORT/include/db.h:59:17: error: conflicting types for 'int32_t'
typedef int int32_t;
^
In file included from /usr/include/newlib/sys/types.h:63:0,
from /usr/include/newlib/stdio.h:48,
from ../../extmod/modbtree.c:27:
/usr/include/newlib/sys/_stdint.h:31:19: note: previous declaration of 'int32_t' was here
typedef __int32_t int32_t ;
^
../../py/mkrules.mk:63: recipe for target 'build-BLACK_F407VE/extmod/modbtree.o' failed
make: *** [build-BLACK_F407VE/extmod/modbtree.o] Error 1

User avatar
jimmo
Posts: 1769
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Compile Btree on STM32

Post by jimmo » Tue Jul 21, 2020 4:56 am

pyro25 wrote:
Mon Jul 20, 2020 3:03 pm
When activating it, it goes as below
Can you include what steps you used to activate it. I'm not sure why you're getting the error you're seeing, but it appears that modbtree isn't really supported on the STM32 port. However, the instructions here look like they should work: viewtopic.php?f=2&t=4500&p=36006#p36006

User avatar
pythoncoder
Posts: 4374
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Compile Btree on STM32

Post by pythoncoder » Tue Jul 21, 2020 7:35 am

I raised this issue. In his response @Damien explained how it can be built as a native mpy module. That said, I haven't actually got this to work:

Code: Select all

[adminpete@capybara]: /mnt/qnap2/data/Projects/MicroPython/micropython/examples/natmod/btree
$ make ARCH=armv7m
LINK build/btree_c.o
Traceback (most recent call last):
  File "../../../tools/mpy_ld.py", line 1081, in <module>
    main()
  File "../../../tools/mpy_ld.py", line 1077, in main
    do_link(args)
  File "../../../tools/mpy_ld.py", line 1047, in do_link
    load_object_file(env, file)
  File "../../../tools/mpy_ld.py", line 680, in load_object_file
    if s.data_size == 0:
AttributeError: 'Section' object has no attribute 'data_size'
../../../py/dynruntime.mk:139: recipe for target 'build/btree_armv7m.native.mpy' failed
make: *** [build/btree_armv7m.native.mpy] Error 1
Peter Hinch

pyro25
Posts: 5
Joined: Mon Jul 20, 2020 2:57 pm

Re: Compile Btree on STM32

Post by pyro25 » Wed Jul 22, 2020 11:22 pm

Thanks for the replies !

1) I did keep trying with the post jimmo mentionned. It breaks at the following. I don't exactly know what to think about this. Like, maybe I can hard-code an error code ?

CC ../../extmod/modbtree.c
CC ../../lib/berkeley-db-1.xx/btree/bt_close.c
CC ../../lib/berkeley-db-1.xx/btree/bt_conv.c
CC ../../lib/berkeley-db-1.xx/btree/bt_debug.c
CC ../../lib/berkeley-db-1.xx/btree/bt_delete.c
CC ../../lib/berkeley-db-1.xx/btree/bt_get.c
CC ../../lib/berkeley-db-1.xx/btree/bt_open.c
CC ../../lib/berkeley-db-1.xx/btree/bt_overflow.c
CC ../../lib/berkeley-db-1.xx/btree/bt_page.c
CC ../../lib/berkeley-db-1.xx/btree/bt_put.c
CC ../../lib/berkeley-db-1.xx/btree/bt_search.c
CC ../../lib/berkeley-db-1.xx/btree/bt_seq.c
CC ../../lib/berkeley-db-1.xx/btree/bt_split.c
CC ../../lib/berkeley-db-1.xx/btree/bt_utils.c
CC ../../lib/berkeley-db-1.xx/mpool/mpool.c
CC mbedtls/mbedtls_port.c
CC build-BLACK_F407VE/pins_BLACK_F407VE.c
LINK build-BLACK_F407VE/firmware.elf
build-BLACK_F407VE/py/stream.o: In function `mp_stream_posix_write':
stream.c:(.text.mp_stream_posix_write+0xe): undefined reference to `__errno'
build-BLACK_F407VE/py/stream.o: In function `mp_stream_posix_read':
stream.c:(.text.mp_stream_posix_read+0xe): undefined reference to `__errno'
build-BLACK_F407VE/py/stream.o: In function `mp_stream_posix_lseek':
stream.c:(.text.mp_stream_posix_lseek+0xe): undefined reference to `__errno'
build-BLACK_F407VE/py/stream.o: In function `mp_stream_posix_fsync':
stream.c:(.text.mp_stream_posix_fsync+0xa): undefined reference to `__errno'
build-BLACK_F407VE/posix_helpers.o: In function `malloc':
posix_helpers.c:(.text.malloc+0xc): undefined reference to `__errno'
build-BLACK_F407VE/posix_helpers.o:posix_helpers.c:(.text.realloc+0xc): more undefined references to `__errno' follow
Makefile:616: recipe for target 'build-BLACK_F407VE/firmware.elf' failed
make: *** [build-BLACK_F407VE/firmware.elf] Error 1

2) pythoncoder: cool too ! It breaks for me with
But I'll keep digging this

GEN build/btree_armv7m.config.h
CC btree_c.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_close.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_conv.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_delete.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_get.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_open.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_overflow.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_page.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_put.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_search.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_seq.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_split.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_utils.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/mpool/mpool.c
LINK build/btree_c.o
Traceback (most recent call last):
File "../../../tools/mpy_ld.py", line 1081, in <module>
main()
File "../../../tools/mpy_ld.py", line 1077, in main
do_link(args)
File "../../../tools/mpy_ld.py", line 1047, in do_link
load_object_file(env, file)
File "../../../tools/mpy_ld.py", line 674, in load_object_file
symtab = list(elf.get_section_by_name(".symtab").iter_symbols())
AttributeError: 'NoneType' object has no attribute 'iter_symbols'
../../../py/dynruntime.mk:139: recipe for target 'build/btree_armv7m.native.mpy' failed
make: *** [build/btree_armv7m.native.mpy] Error 1

pyro25
Posts: 5
Joined: Mon Jul 20, 2020 2:57 pm

Re: Compile Btree on STM32

Post by pyro25 » Wed Jul 22, 2020 11:35 pm

Thanks for the replies !

1) jimmo, I pretty much followed the steps in the post you mentionned. mpconfigport.mk is the key file to activate it
I made some progress, but now I get

CC ../../lib/berkeley-db-1.xx/btree/bt_open.c
CC ../../lib/berkeley-db-1.xx/btree/bt_overflow.c
CC ../../lib/berkeley-db-1.xx/btree/bt_page.c
CC ../../lib/berkeley-db-1.xx/btree/bt_put.c
CC ../../lib/berkeley-db-1.xx/btree/bt_search.c
CC ../../lib/berkeley-db-1.xx/btree/bt_seq.c
CC ../../lib/berkeley-db-1.xx/btree/bt_split.c
CC ../../lib/berkeley-db-1.xx/btree/bt_utils.c
CC ../../lib/berkeley-db-1.xx/mpool/mpool.c
CC mbedtls/mbedtls_port.c
CC build-BLACK_F407VE/pins_BLACK_F407VE.c
LINK build-BLACK_F407VE/firmware.elf
build-BLACK_F407VE/py/stream.o: In function `mp_stream_posix_write':
stream.c:(.text.mp_stream_posix_write+0xe): undefined reference to `__errno'
build-BLACK_F407VE/py/stream.o: In function `mp_stream_posix_read':
stream.c:(.text.mp_stream_posix_read+0xe): undefined reference to `__errno'
build-BLACK_F407VE/py/stream.o: In function `mp_stream_posix_lseek':
stream.c:(.text.mp_stream_posix_lseek+0xe): undefined reference to `__errno'
build-BLACK_F407VE/py/stream.o: In function `mp_stream_posix_fsync':
stream.c:(.text.mp_stream_posix_fsync+0xa): undefined reference to `__errno'
build-BLACK_F407VE/posix_helpers.o: In function `malloc':
posix_helpers.c:(.text.malloc+0xc): undefined reference to `__errno'
build-BLACK_F407VE/posix_helpers.o:posix_helpers.c:(.text.realloc+0xc): more undefined references to `__errno' follow
Makefile:616: recipe for target 'build-BLACK_F407VE/firmware.elf' failed
make: *** [build-BLACK_F407VE/firmware.elf] Error 1

I'm currently considering hard-coding an error code ? I really don't know that part

2) pythoncoder, thank you, cool stuff ! I'll keep on digging this. Currently, I'm getting this

CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_search.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_seq.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_split.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_utils.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/mpool/mpool.c
LINK build/btree_c.o
Traceback (most recent call last):
File "../../../tools/mpy_ld.py", line 1081, in <module>
main()
File "../../../tools/mpy_ld.py", line 1077, in main
do_link(args)
File "../../../tools/mpy_ld.py", line 1047, in do_link
load_object_file(env, file)
File "../../../tools/mpy_ld.py", line 674, in load_object_file
symtab = list(elf.get_section_by_name(".symtab").iter_symbols())
AttributeError: 'NoneType' object has no attribute 'iter_symbols'
../../../py/dynruntime.mk:139: recipe for target 'build/btree_armv7m.native.mpy' failed
make: *** [build/btree_armv7m.native.mpy] Error 1

User avatar
jimmo
Posts: 1769
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Compile Btree on STM32

Post by jimmo » Wed Jul 22, 2020 11:52 pm

The mp_stream_posix_ errors should be fixed by setting MICROPY_STREAMS_POSIX_API

What's going on here is that the Berkeley DB library is expecting to be running on a unix-like environment, so it expects some basic functions to be available. On ESP8266, ESP32, Windows and Unix ports (where btree is enabled by default) these functions are provided by the OS.

On STM32 there is no OS, so you need extra wrappers and stuff to provide them (that's the posix_helpers.c copied from ESP8266). MICROPY_STREAMS_POSIX_API tells the build that you have these functions available, which enables the posix version of the MicroPython streams interface which modbtree uses.

pyro25
Posts: 5
Joined: Mon Jul 20, 2020 2:57 pm

Re: Compile Btree on STM32

Post by pyro25 » Fri Jul 24, 2020 6:30 pm

Thanks a lot for the explanation, very interesting

I guess I'm not far from breaking this, I get the following with, at the end of mphalport.c

extern int mp_stream_errno;
int *__errno() {
return &mp_stream_errno;
}

CC ../../lib/berkeley-db-1.xx/btree/bt_open.c
CC ../../lib/berkeley-db-1.xx/btree/bt_overflow.c
CC ../../lib/berkeley-db-1.xx/btree/bt_page.c
CC ../../lib/berkeley-db-1.xx/btree/bt_put.c
CC ../../lib/berkeley-db-1.xx/btree/bt_search.c
CC ../../lib/berkeley-db-1.xx/btree/bt_seq.c
CC ../../lib/berkeley-db-1.xx/btree/bt_split.c
CC ../../lib/berkeley-db-1.xx/btree/bt_utils.c
CC ../../lib/berkeley-db-1.xx/mpool/mpool.c
CC mbedtls/mbedtls_port.c
CC build-BLACK_F407VE/pins_BLACK_F407VE.c
LINK build-BLACK_F407VE/firmware.elf
build-BLACK_F407VE/mphalport.o: In function `__errno':
mphalport.c:(.text.__errno+0x4): undefined reference to `mp_stream_errno'
Makefile:616: recipe for target 'build-BLACK_F407VE/firmware.elf' failed
make: *** [build-BLACK_F407VE/firmware.elf] Error 1

pyro25
Posts: 5
Joined: Mon Jul 20, 2020 2:57 pm

Re: Compile Btree on STM32

Post by pyro25 » Fri Jul 24, 2020 11:23 pm


miltmobley
Posts: 25
Joined: Mon Mar 07, 2016 11:44 pm

Re: Compile Btree on STM32

Post by miltmobley » Sun Jul 26, 2020 2:50 am

Here's two cents worth:

If you are trying to compile a module for an STM32 processor, this is a cross-compile and you should not be accessing any Linux system header files in the build. Of course Linux would allow you to do something like set up symbolic links to make the linux source appear to be in the micropython tree, but that is not likely to work. It may be better to copy the btree source into the micropython tree and try to port it.
But someone else has already implemented btree for cpython, see: https://pypi.org/project/micropython-cpython-btree/

User avatar
jimmo
Posts: 1769
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Compile Btree on STM32

Post by jimmo » Mon Jul 27, 2020 3:15 am

miltmobley wrote:
Sun Jul 26, 2020 2:50 am
If you are trying to compile a module for an STM32 processor, this is a cross-compile and you should not be accessing any Linux system header files in the build. Of course Linux would allow you to do something like set up symbolic links to make the linux source appear to be in the micropython tree, but that is not likely to work. It may be better to copy the btree source into the micropython tree and try to port it.
This isn't accessing Linux system headers specifically, it's accessing whatever you provide for libc etc. The issue is more that the btree library that is already a submodule in the micropython tree expects certain things to be available. It's not just the headers but also the implementation.

Post Reply