MicroPython Forum (Archive)

Hi !

I'm looking to compile btree in the stm32 port, which is disabled by default. When activating it, it goes as below
I'm running Ubuntu 16.04.4 LTS

my current state of brainstorming goes like
* modbtree is a submodule of micropython - possible to modify, just not ideal
* newlib is deb that's part of ubuntu - does not sound good to modify
* it would be cool to be able to patch outside of these two...

Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
CC ../../extmod/modbtree.c
In file included from ../../extmod/modbtree.c:37:0:
../../lib/berkeley-db-1.xx/PORT/include/db.h:59:17: error: conflicting types for 'int32_t'
typedef int int32_t;
^
In file included from /usr/include/newlib/sys/types.h:63:0,
from /usr/include/newlib/stdio.h:48,
from ../../extmod/modbtree.c:27:
/usr/include/newlib/sys/_stdint.h:31:19: note: previous declaration of 'int32_t' was here
typedef __int32_t int32_t ;
^
../../py/mkrules.mk:63: recipe for target 'build-BLACK_F407VE/extmod/modbtree.o' failed
make: *** [build-BLACK_F407VE/extmod/modbtree.o] Error 1

pyro25 wrote: ↑

Mon Jul 20, 2020 3:03 pm

When activating it, it goes as below

Can you include what steps you used to activate it. I'm not sure why you're getting the error you're seeing, but it appears that modbtree isn't really supported on the STM32 port. However, the instructions here look like they should work: viewtopic.php?f=2&t=4500&p=36006#p36006

I raised this issue. In his response @Damien explained how it can be built as a native mpy module. That said, I haven't actually got this to work:

Thanks for the replies !

1) I did keep trying with the post jimmo mentionned. It breaks at the following. I don't exactly know what to think about this. Like, maybe I can hard-code an error code ?

CC ../../extmod/modbtree.c
CC ../../lib/berkeley-db-1.xx/btree/bt_close.c
CC ../../lib/berkeley-db-1.xx/btree/bt_conv.c
CC ../../lib/berkeley-db-1.xx/btree/bt_debug.c
CC ../../lib/berkeley-db-1.xx/btree/bt_delete.c
CC ../../lib/berkeley-db-1.xx/btree/bt_get.c
CC ../../lib/berkeley-db-1.xx/btree/bt_open.c
CC ../../lib/berkeley-db-1.xx/btree/bt_overflow.c
CC ../../lib/berkeley-db-1.xx/btree/bt_page.c
CC ../../lib/berkeley-db-1.xx/btree/bt_put.c
CC ../../lib/berkeley-db-1.xx/btree/bt_search.c
CC ../../lib/berkeley-db-1.xx/btree/bt_seq.c
CC ../../lib/berkeley-db-1.xx/btree/bt_split.c
CC ../../lib/berkeley-db-1.xx/btree/bt_utils.c
CC ../../lib/berkeley-db-1.xx/mpool/mpool.c
CC mbedtls/mbedtls_port.c
CC build-BLACK_F407VE/pins_BLACK_F407VE.c
LINK build-BLACK_F407VE/firmware.elf
build-BLACK_F407VE/py/stream.o: In function `mp_stream_posix_write':
stream.c:(.text.mp_stream_posix_write+0xe): undefined reference to `__errno'
build-BLACK_F407VE/py/stream.o: In function `mp_stream_posix_read':
stream.c:(.text.mp_stream_posix_read+0xe): undefined reference to `__errno'
build-BLACK_F407VE/py/stream.o: In function `mp_stream_posix_lseek':
stream.c:(.text.mp_stream_posix_lseek+0xe): undefined reference to `__errno'
build-BLACK_F407VE/py/stream.o: In function `mp_stream_posix_fsync':
stream.c:(.text.mp_stream_posix_fsync+0xa): undefined reference to `__errno'
build-BLACK_F407VE/posix_helpers.o: In function `malloc':
posix_helpers.c:(.text.malloc+0xc): undefined reference to `__errno'
build-BLACK_F407VE/posix_helpers.o:posix_helpers.c:(.text.realloc+0xc): more undefined references to `__errno' follow
Makefile:616: recipe for target 'build-BLACK_F407VE/firmware.elf' failed
make: *** [build-BLACK_F407VE/firmware.elf] Error 1

2) pythoncoder: cool too ! It breaks for me with
But I'll keep digging this

GEN build/btree_armv7m.config.h
CC btree_c.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_close.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_conv.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_delete.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_get.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_open.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_overflow.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_page.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_put.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_search.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_seq.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_split.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_utils.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/mpool/mpool.c
LINK build/btree_c.o
Traceback (most recent call last):
File "../../../tools/mpy_ld.py", line 1081, in <module>
main()
File "../../../tools/mpy_ld.py", line 1077, in main
do_link(args)
File "../../../tools/mpy_ld.py", line 1047, in do_link
load_object_file(env, file)
File "../../../tools/mpy_ld.py", line 674, in load_object_file
symtab = list(elf.get_section_by_name(".symtab").iter_symbols())
AttributeError: 'NoneType' object has no attribute 'iter_symbols'
../../../py/dynruntime.mk:139: recipe for target 'build/btree_armv7m.native.mpy' failed
make: *** [build/btree_armv7m.native.mpy] Error 1

Thanks for the replies !

1) jimmo, I pretty much followed the steps in the post you mentionned. mpconfigport.mk is the key file to activate it
I made some progress, but now I get

CC ../../lib/berkeley-db-1.xx/btree/bt_open.c
CC ../../lib/berkeley-db-1.xx/btree/bt_overflow.c
CC ../../lib/berkeley-db-1.xx/btree/bt_page.c
CC ../../lib/berkeley-db-1.xx/btree/bt_put.c
CC ../../lib/berkeley-db-1.xx/btree/bt_search.c
CC ../../lib/berkeley-db-1.xx/btree/bt_seq.c
CC ../../lib/berkeley-db-1.xx/btree/bt_split.c
CC ../../lib/berkeley-db-1.xx/btree/bt_utils.c
CC ../../lib/berkeley-db-1.xx/mpool/mpool.c
CC mbedtls/mbedtls_port.c
CC build-BLACK_F407VE/pins_BLACK_F407VE.c
LINK build-BLACK_F407VE/firmware.elf
build-BLACK_F407VE/py/stream.o: In function `mp_stream_posix_write':
stream.c:(.text.mp_stream_posix_write+0xe): undefined reference to `__errno'
build-BLACK_F407VE/py/stream.o: In function `mp_stream_posix_read':
stream.c:(.text.mp_stream_posix_read+0xe): undefined reference to `__errno'
build-BLACK_F407VE/py/stream.o: In function `mp_stream_posix_lseek':
stream.c:(.text.mp_stream_posix_lseek+0xe): undefined reference to `__errno'
build-BLACK_F407VE/py/stream.o: In function `mp_stream_posix_fsync':
stream.c:(.text.mp_stream_posix_fsync+0xa): undefined reference to `__errno'
build-BLACK_F407VE/posix_helpers.o: In function `malloc':
posix_helpers.c:(.text.malloc+0xc): undefined reference to `__errno'
build-BLACK_F407VE/posix_helpers.o:posix_helpers.c:(.text.realloc+0xc): more undefined references to `__errno' follow
Makefile:616: recipe for target 'build-BLACK_F407VE/firmware.elf' failed
make: *** [build-BLACK_F407VE/firmware.elf] Error 1

I'm currently considering hard-coding an error code ? I really don't know that part

2) pythoncoder, thank you, cool stuff ! I'll keep on digging this. Currently, I'm getting this

CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_search.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_seq.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_split.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/btree/bt_utils.c
CC /home/laurent/Projects/stm32/micropython/lib/berkeley-db-1.xx/mpool/mpool.c
LINK build/btree_c.o
Traceback (most recent call last):
File "../../../tools/mpy_ld.py", line 1081, in <module>
main()
File "../../../tools/mpy_ld.py", line 1077, in main
do_link(args)
File "../../../tools/mpy_ld.py", line 1047, in do_link
load_object_file(env, file)
File "../../../tools/mpy_ld.py", line 674, in load_object_file
symtab = list(elf.get_section_by_name(".symtab").iter_symbols())
AttributeError: 'NoneType' object has no attribute 'iter_symbols'
../../../py/dynruntime.mk:139: recipe for target 'build/btree_armv7m.native.mpy' failed
make: *** [build/btree_armv7m.native.mpy] Error 1

The mp_stream_posix_ errors should be fixed by setting MICROPY_STREAMS_POSIX_API

What's going on here is that the Berkeley DB library is expecting to be running on a unix-like environment, so it expects some basic functions to be available. On ESP8266, ESP32, Windows and Unix ports (where btree is enabled by default) these functions are provided by the OS.

On STM32 there is no OS, so you need extra wrappers and stuff to provide them (that's the posix_helpers.c copied from ESP8266). MICROPY_STREAMS_POSIX_API tells the build that you have these functions available, which enables the posix version of the MicroPython streams interface which modbtree uses.

Thanks a lot for the explanation, very interesting

I guess I'm not far from breaking this, I get the following with, at the end of mphalport.c

extern int mp_stream_errno;
int *__errno() {
return &mp_stream_errno;
}

CC ../../lib/berkeley-db-1.xx/btree/bt_open.c
CC ../../lib/berkeley-db-1.xx/btree/bt_overflow.c
CC ../../lib/berkeley-db-1.xx/btree/bt_page.c
CC ../../lib/berkeley-db-1.xx/btree/bt_put.c
CC ../../lib/berkeley-db-1.xx/btree/bt_search.c
CC ../../lib/berkeley-db-1.xx/btree/bt_seq.c
CC ../../lib/berkeley-db-1.xx/btree/bt_split.c
CC ../../lib/berkeley-db-1.xx/btree/bt_utils.c
CC ../../lib/berkeley-db-1.xx/mpool/mpool.c
CC mbedtls/mbedtls_port.c
CC build-BLACK_F407VE/pins_BLACK_F407VE.c
LINK build-BLACK_F407VE/firmware.elf
build-BLACK_F407VE/mphalport.o: In function `__errno':
mphalport.c:(.text.__errno+0x4): undefined reference to `mp_stream_errno'
Makefile:616: recipe for target 'build-BLACK_F407VE/firmware.elf' failed
make: *** [build-BLACK_F407VE/firmware.elf] Error 1

All right, got it. Fixed using

https://github.com/micropython/micropython/pull/5959

Here's two cents worth:

If you are trying to compile a module for an STM32 processor, this is a cross-compile and you should not be accessing any Linux system header files in the build. Of course Linux would allow you to do something like set up symbolic links to make the linux source appear to be in the micropython tree, but that is not likely to work. It may be better to copy the btree source into the micropython tree and try to port it.
But someone else has already implemented btree for cpython, see: https://pypi.org/project/micropython-cpython-btree/

miltmobley wrote: ↑

Sun Jul 26, 2020 2:50 am

If you are trying to compile a module for an STM32 processor, this is a cross-compile and you should not be accessing any Linux system header files in the build. Of course Linux would allow you to do something like set up symbolic links to make the linux source appear to be in the micropython tree, but that is not likely to work. It may be better to copy the btree source into the micropython tree and try to port it.

This isn't accessing Linux system headers specifically, it's accessing whatever you provide for libc etc. The issue is more that the btree library that is already a submodule in the micropython tree expects certain things to be available. It's not just the headers but also the implementation.