Adding External C Modules with external functions to mp as a library

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
cduran
Posts: 67
Joined: Thu Mar 17, 2016 4:52 pm

Adding External C Modules with external functions to mp as a library

Post by cduran » Tue Mar 10, 2020 11:29 pm

I have a unique problem that I am trying to solve. I'm currently building mp 1.12 as a library which I then import into my existing project. My project is arm based and has no OS so I'm using the minimal build as a start.

So far I'm able to build an mp lib and run script strings just fine. Where I'm having problems is adding a C module to the library.

I am following the instructions here, with one exception: http://docs.micropython.org/en/latest/d ... dules.html

The exception is that the only function in the module requires a lot of libraries and code from the project I am added mp to. So in my case this code:

Code: Select all

// This is the function which will be called from Python as example.add_ints(a, b).
STATIC mp_obj_t example_add_ints(mp_obj_t a_obj, mp_obj_t b_obj) {
    // Extract the ints from the micropython input objects
    int a = mp_obj_get_int(a_obj);
    int b = mp_obj_get_int(b_obj);

    // Calculate the addition and convert to MicroPython object.
    return mp_obj_new_int(a + b);
}
is replaced with a header file containing

Code: Select all

STATIC mp_obj_t example_add_ints(mp_obj_t a_obj, mp_obj_t b_obj);
The body of this function is part of my code. The library builds with no errors, but when I build my project I get an undefined reference error for the function in my module. Any ideas, suggestions?

I've included my following
mpconfigport.h
micropython.mk
helpers.h - module code
helpers.c

Here's my Makefile for the library:

Code: Select all

#include <stdint.h>

// options to control how MicroPython is built

// You can disable the built-in MicroPython compiler by setting the following
// config option to 0.  If you do this then you won't get a REPL prompt, but you
// will still be able to execute pre-compiled scripts, compiled with mpy-cross.
#define MICROPY_ENABLE_COMPILER     (1)

#define MICROPY_QSTR_BYTES_IN_HASH  (1)
#define MICROPY_QSTR_EXTRA_POOL     mp_qstr_frozen_const_pool
#define MICROPY_ALLOC_PATH_MAX      (256)
#define MICROPY_ALLOC_PARSE_CHUNK_INIT (16)
#define MICROPY_EMIT_X64            (0)
#define MICROPY_EMIT_THUMB          (0)
#define MICROPY_EMIT_INLINE_THUMB   (0)
#define MICROPY_COMP_MODULE_CONST   (0)
#define MICROPY_COMP_CONST          (0)
#define MICROPY_COMP_DOUBLE_TUPLE_ASSIGN (0)
#define MICROPY_COMP_TRIPLE_TUPLE_ASSIGN (0)
#define MICROPY_MEM_STATS           (0)
#define MICROPY_DEBUG_PRINTERS      (0)
#define MICROPY_ENABLE_GC           (1)
#define MICROPY_GC_ALLOC_THRESHOLD  (0)
#define MICROPY_REPL_EVENT_DRIVEN   (0)
#define MICROPY_HELPER_REPL         (1)
#define MICROPY_HELPER_LEXER_UNIX   (0)
#define MICROPY_ENABLE_SOURCE_LINE  (0)
#define MICROPY_ENABLE_DOC_STRING   (0)
#define MICROPY_ERROR_REPORTING     (MICROPY_ERROR_REPORTING_TERSE)
#define MICROPY_BUILTIN_METHOD_CHECK_SELF_ARG (0)
#define MICROPY_PY_ASYNC_AWAIT      (0)
#define MICROPY_PY_BUILTINS_BYTEARRAY (0)
#define MICROPY_PY_BUILTINS_DICT_FROMKEYS (0)
#define MICROPY_PY_BUILTINS_MEMORYVIEW (0)
#define MICROPY_PY_BUILTINS_ENUMERATE (0)
#define MICROPY_PY_BUILTINS_FILTER  (0)
#define MICROPY_PY_BUILTINS_FROZENSET (0)
#define MICROPY_PY_BUILTINS_REVERSED (0)
#define MICROPY_PY_BUILTINS_SET     (0)
#define MICROPY_PY_BUILTINS_SLICE   (0)
#define MICROPY_PY_BUILTINS_PROPERTY (0)
#define MICROPY_PY_BUILTINS_MIN_MAX (0)
#define MICROPY_PY_BUILTINS_STR_COUNT (0)
#define MICROPY_PY_BUILTINS_STR_OP_MODULO (0)
#define MICROPY_PY___FILE__         (0)
#define MICROPY_PY_GC               (0)
#define MICROPY_PY_ARRAY            (0)
#define MICROPY_PY_ATTRTUPLE        (0)
#define MICROPY_PY_COLLECTIONS      (0)
#define MICROPY_PY_MATH             (0)
#define MICROPY_PY_CMATH            (0)
#define MICROPY_PY_IO               (0)
#define MICROPY_PY_STRUCT           (0)
#define MICROPY_PY_SYS              (0)
#define MICROPY_MODULE_FROZEN_MPY   (1)
#define MICROPY_CPYTHON_COMPAT      (0)
#define MICROPY_LONGINT_IMPL        (MICROPY_LONGINT_IMPL_NONE)
#define MICROPY_FLOAT_IMPL          (MICROPY_FLOAT_IMPL_NONE)
#define MODULE_HELPERS_ENABLED		(1)

// type definitions for the specific machine

#define MICROPY_MAKE_POINTER_CALLABLE(p) ((void*)((mp_uint_t)(p) | 1))

// This port is intended to be 32-bit, but unfortunately, int32_t for
// different targets may be defined in different ways - either as int
// or as long. This requires different printf formatting specifiers
// to print such value. So, we avoid int32_t and use int directly.
#define UINT_FMT "%u"
#define INT_FMT "%d"
typedef int mp_int_t; // must be pointer size
typedef unsigned mp_uint_t; // must be pointer size

typedef long mp_off_t;

#define MP_PLAT_PRINT_STRN(str, len) mp_hal_stdout_tx_strn_cooked(str, len)

// extra built in names to add to the global namespace
#define MICROPY_PORT_BUILTINS \
    //{ MP_ROM_QSTR(MP_QSTR_open), MP_ROM_PTR(&mp_builtin_open_obj) },

// We need to provide a declaration/definition of alloca()
#include <alloca.h>

#define MICROPY_HW_BOARD_NAME "minimal"
#define MICROPY_HW_MCU_NAME "unknown-cpu"

#ifdef __linux__
#define MICROPY_MIN_USE_STDOUT (1)
#endif

#ifdef __thumb__
#define MICROPY_MIN_USE_CORTEX_CPU (1)
#define MICROPY_MIN_USE_STM32_MCU (1)
#endif

#define MP_STATE_PORT MP_STATE_VM

#define MICROPY_PORT_ROOT_POINTERS \
    const char *readline_hist[8];
My micropython.mk file for the module:

Code: Select all

HELPERS_MOD_DIR := $(USERMOD_DIR)

# Add all C files to SRC_USERMOD.
SRC_USERMOD += $(HELPERS_MOD_DIR)/helpers.c

# We can add our module folder to include paths if needed
# This is not actually needed in this example.
#CFLAGS_USERMOD += -I$(HELPERS_MOD_DIR)
My module source:

Code: Select all

// helpers.h
// --------------------------------------------------------
// This is our helper function
STATIC mp_obj_t helpers_c_helper(mp_obj_t op_code, mp_obj_t mp_arg1, mp_obj_t mp_arg2);

// helpers.c
// --------------------------------------------------------
#include "py/obj.h"
#include "py/runtime.h"
#include "py/builtin.h"
#include "helpers.h"

// Module usage:
//
// import helpers
// helpers.c_helper(a, b, c) 


// Define a Python reference to the function above
STATIC MP_DEFINE_CONST_FUN_OBJ_3(helpers_c_helper_obj, helpers_c_helper);

// Define all properties of the example module.
// Table entries are key/value pairs of the attribute name (a string)
// and the MicroPython object reference.
// All identifiers and strings are written as MP_QSTR_xxx and will be
// optimized to word-sized integers by the build system (interned strings).
STATIC const mp_rom_map_elem_t helpers_module_globals_table[] = {
    { MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_helpers) },
    { MP_ROM_QSTR(MP_QSTR_c_helper), MP_ROM_PTR(&helpers_c_helper_obj) },
};
STATIC MP_DEFINE_CONST_DICT(helpers_module_globals, helpers_module_globals_table);

// Define module object.
const mp_obj_module_t helpers_user_cmodule = {
    .base = { &mp_type_module },
    .globals = (mp_obj_dict_t*)&helpers_module_globals,
};

// Register the module to make it available in Python
MP_REGISTER_MODULE(MP_QSTR_helpers, helpers_user_cmodule, MODULE_HELPERS_ENABLED);

stijn
Posts: 462
Joined: Thu Apr 24, 2014 9:13 am

Re: Adding External C Modules with external functions to mp as a library

Post by stijn » Wed Mar 11, 2020 8:40 am

The code you show doesn't contain a definition of helpers_c_helper, only a declaration, hence the undefined reference?

Also I don't know all effects of declaring static functions in header files, but least to say is that it's unusual and probably not intended?

cduran
Posts: 67
Joined: Thu Mar 17, 2016 4:52 pm

Re: Adding External C Modules with external functions to mp as a library

Post by cduran » Wed Mar 11, 2020 1:37 pm

stijn wrote:
Wed Mar 11, 2020 8:40 am
The code you show doesn't contain a definition of helpers_c_helper, only a declaration, hence the undefined reference?

Also I don't know all effects of declaring static functions in header files, but least to say is that it's unusual and probably not intended?
I updated my code as follows and still getting an undefined reference. helpers_c_helper is fully defined.

Code: Select all

// helpers.h
// This is our helper function
//extern mp_obj_t helpers_c_helper(mp_obj_t op_code, mp_obj_t mp_arg1, mp_obj_t mp_arg2);

void set_helper_callback(mp_obj_t (*function)(mp_obj_t, mp_obj_t, mp_obj_t));
//-------------------------------------------------------------------------------------------------------------------

// helpers.c
#include "py/obj.h"
#include "py/runtime.h"
#include "py/builtin.h"
#include "helpers.h"

// Module usage:
//
// import helpers
// helpers.c_helper(a, b, c)

// Null function
mp_obj_t do_nothing(mp_obj_t a_obj, mp_obj_t b_obj, mp_obj_t c_obj)
{
	return mp_obj_new_int(0);
}

mp_obj_t (*fun_ptr)(mp_obj_t, mp_obj_t, mp_obj_t) = &do_nothing;

void set_helper_callback(mp_obj_t (*function)(mp_obj_t, mp_obj_t, mp_obj_t))
{
	fun_ptr = function;
}

STATIC mp_obj_t helpers_c_helper(mp_obj_t a_obj, mp_obj_t b_obj, mp_obj_t c_obj)
{
    return (*fun_ptr)(a_obj, b_obj, c_obj);
}

// Define a Python reference to the function above
STATIC MP_DEFINE_CONST_FUN_OBJ_3(helpers_c_helper_obj, helpers_c_helper);

// Define all properties of the example module.
// Table entries are key/value pairs of the attribute name (a string)
// and the MicroPython object reference.
// All identifiers and strings are written as MP_QSTR_xxx and will be
// optimized to word-sized integers by the build system (interned strings).
STATIC const mp_rom_map_elem_t helpers_module_globals_table[] = {
    { MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_helpers) },
    { MP_ROM_QSTR(MP_QSTR_c_helper), MP_ROM_PTR(&helpers_c_helper_obj) },
};
STATIC MP_DEFINE_CONST_DICT(helpers_module_globals, helpers_module_globals_table);

// Define module object.
const mp_obj_module_t helpers_user_cmodule = {
    .base = { &mp_type_module },
    .globals = (mp_obj_dict_t*)&helpers_module_globals,
};

// Register the module to make it available in Python
MP_REGISTER_MODULE(MP_QSTR_helpers, helpers_user_cmodule, MODULE_HELPERS_ENABLED);

stijn
Posts: 462
Joined: Thu Apr 24, 2014 9:13 am

Re: Adding External C Modules with external functions to mp as a library

Post by stijn » Wed Mar 11, 2020 3:09 pm

Aha, looks like you found a bug. I don't know why, but the culprit is defining MODULE_HELPERS_ENABLED in mpconfigport: if you remove that line and instead use CFLAGS_EXTRA=-DMODULE_HELPERS_ENABLED=1 on the commandline it works. Tried with the unix port and it doesn't have that problem though.


User avatar
dhylands
Posts: 3385
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Adding External C Modules with external functions to mp as a library

Post by dhylands » Wed Mar 11, 2020 4:19 pm

You should also look at the linker command line (i.e. build with make V=1).

The GNU linker has some peculiarities about the order that object files need to appear in order to resolve symbols.

So lets say you have a foo.o file which needs to call some_function and some_function is defined in bar.o

Then foo.o needs to occur on the linker command line before bar.o (or the library containing bar.o)

cduran
Posts: 67
Joined: Thu Mar 17, 2016 4:52 pm

Re: Adding External C Modules with external functions to mp as a library

Post by cduran » Wed Mar 11, 2020 5:07 pm

stijn wrote:
Wed Mar 11, 2020 3:09 pm
Aha, looks like you found a bug. I don't know why, but the culprit is defining MODULE_HELPERS_ENABLED in mpconfigport: if you remove that line and instead use CFLAGS_EXTRA=-DMODULE_HELPERS_ENABLED=1 on the commandline it works. Tried with the unix port and it doesn't have that problem though.
I actually found that the libs makefile wasnt building my modules c file. I fixed the issue, however, now the c module executes when called, but when it returns from the module function the firmware hangs. Any thoughts?

cduran
Posts: 67
Joined: Thu Mar 17, 2016 4:52 pm

Re: Adding External C Modules with external functions to mp as a library

Post by cduran » Wed Mar 11, 2020 5:13 pm

dhylands wrote:
Wed Mar 11, 2020 4:19 pm
You should also look at the linker command line (i.e. build with make V=1).

The GNU linker has some peculiarities about the order that object files need to appear in order to resolve symbols.

So lets say you have a foo.o file which needs to call some_function and some_function is defined in bar.o

Then foo.o needs to occur on the linker command line before bar.o (or the library containing bar.o)
I basically discovered that the actual c file for the c module wasn't being compiled into the library, I edited the makefile to fix the issue.

Of course that now uncovered another issue, the c module function executes fine but hangs the firmware when it returns. Any ideas what could be wrong?

User avatar
dhylands
Posts: 3385
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Adding External C Modules with external functions to mp as a library

Post by dhylands » Wed Mar 11, 2020 7:46 pm

It's hard to say, especially without seeing the code. The C code is probably corrupting something (likely the stack).

I'd probably start by making the C code do nothing and just return. And then start adding functionality back in to figure out where it's messing things up.

cduran
Posts: 67
Joined: Thu Mar 17, 2016 4:52 pm

Re: Adding External C Modules with external functions to mp as a library

Post by cduran » Wed Mar 11, 2020 7:52 pm

dhylands wrote:
Wed Mar 11, 2020 7:46 pm
It's hard to say, especially without seeing the code. The C code is probably corrupting something (likely the stack).

I'd probably start by making the C code do nothing and just return. And then start adding functionality back in to figure out where it's messing things up.
In particular this line is causing the hang, I've tried using a constant and still have a hang.

return mp_obj_new_int_from_uint(aCounter);
Last edited by cduran on Wed Mar 11, 2020 9:44 pm, edited 1 time in total.

Post Reply