Implementing a C function that takes a Python function as argument

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Turbinenreiter
Posts: 288
Joined: Sun May 04, 2014 8:54 am

Implementing a C function that takes a Python function as argument

Post by Turbinenreiter » Wed Dec 14, 2016 12:43 pm

I'm experimenting with writing a dbus module for the unix port of MicroPython, but the question is a general one, so just ignore the dbus variable names.

Consider the following code:

Code: Select all

STATIC mp_obj_t mod_dbus_test(mp_obj_t input) {
    
    return;
}
STATIC MP_DEFINE_CONST_FUN_OBJ_1(mod_dbus_test_obj, mod_dbus_test);
This function I can call from MicroPython like:

Code: Select all

dbus.test(foo)
where foo is a function.

What I'm trying to figure out now is how I can then run this passed function and get it's results, like this:

Code: Select all

STATIC mp_obj_t mod_dbus_test(mp_obj_t input) {
    # run input function
    return # return input functions results;
}
STATIC MP_DEFINE_CONST_FUN_OBJ_1(mod_dbus_test_obj, mod_dbus_test);
Also, is there a way to get information about the function passed in, like it's name and number of arguments?

The idea is to write a function that takes a Python function as argument and exposes it on the dbus. This can then be applied as a decorator to Python functions.

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Implementing a C function that takes a Python function as argument

Post by dhylands » Wed Dec 14, 2016 4:33 pm

This thread includes some sample code which shows how to pass a python function to a C function:
http://forum.micropython.org/viewtopic.php?f=2&t=1411

I just rebased to the latest MicroPython and you can find the code here:
https://github.com/dhylands/micropython ... eabf8f0f51

I verified that it compiles and there's no reason for it not to run, but I didn't actually verify that it runs after the latest tweak.

Turbinenreiter
Posts: 288
Joined: Sun May 04, 2014 8:54 am

Re: Implementing a C function that takes a Python function as argument

Post by Turbinenreiter » Thu Dec 15, 2016 1:50 pm

Thank you!

I can now register a method and run a dbus Service from MicroPython.

Turbinenreiter
Posts: 288
Joined: Sun May 04, 2014 8:54 am

Re: Implementing a C function that takes a Python function as argument

Post by Turbinenreiter » Sat Dec 17, 2016 10:12 pm

I'm now trying to dynamically create the sd_bus_vtable from input I get from the MicroPython script and it really got my head spinning.

Code: Select all

sd_bus_vtable *test_vtable = NULL;
int tab_len = 2;
static sd_bus_vtable start = SD_BUS_VTABLE_START(0);
static sd_bus_vtable end = SD_BUS_VTABLE_END;

STATIC mp_obj_t mod_dbus_register(mp_obj_t input) {
    MP_STATE_PORT(c_dbus_method_obj) = input;
    qstr qstr_fun_name = mp_obj_fun_get_name(input);

    sd_bus_vtable *old_vtable = test_vtable;
    test_vtable = (sd_bus_vtable*)m_malloc(sizeof(struct sd_bus_vtable)*(++tab_len));

    sd_bus_vtable new_elem = SD_BUS_METHOD(qstr_str(qstr_fun_name),
                                           "x",
                                           "x",
                                           method_call,
                                           SD_BUS_VTABLE_UNPRIVILEGED);

    test_vtable[0] = start;
    int i;
    for(i=1; i<tab_len-1; i++) {
        test_vtable[i] = start;//old_vtable[i];
    }
    test_vtable[tab_len-2] = new_elem;
    test_vtable[tab_len-1] = end;

    m_free(old_vtable, sizeof(old_vtable));

    return MP_OBJ_NEW_QSTR(qstr_fun_name);
}
STATIC MP_DEFINE_CONST_FUN_OBJ_1(mod_dbus_register_obj, mod_dbus_register);
This very example actually segfaults. A different implementation ran, but the vtable it wrote was wrong.
Basically, all I want to do is add a struct to an array of structs. Will pick up tomorrow with a fresh brain, but maybe someone easily spots my mistake.

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Implementing a C function that takes a Python function as argument

Post by dhylands » Sat Dec 17, 2016 11:01 pm

All C structs which represent python objects need to have a mp_obj_base_t base as the first element. Any remaining elements can be whatever your code desires.

The Timer class allocates timers dynamically. Here's where the timer struct is declared:
https://github.com/micropython/micropyt ... #L129-L137

and here's where it's allocated and initialized:
https://github.com/micropython/micropyt ... #L644-L647

We memset the entire structure to make sure that we don't have uninitialized data which happens to look like a pointer and prevent the GC from freeing something.

If you allocate python objects, then they should be "findable" by the GC through a root pointer (or chain of objects which starts with a root pointer). If your function just allocates the object and then passes it off to python code, then you don't need to do anything else (since the python state will be keeping the pointer to the object). If your function allocates something that you want to keep around and it isn't passed off to the python code, then you need to keep track of it using through a root pointer someplace. The simplest way is to add a pointer/struct to the MICROPY_PORT_ROOT_POINTERS macro in the mpconfigport.h file, like this:
https://github.com/micropython/micropyt ... #L209-L236

Otherwise, when the GC does a garbage collection, it will free your object if it doesn't find anything pointing to it.

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Implementing a C function that takes a Python function as argument

Post by dhylands » Sat Dec 17, 2016 11:18 pm

You might want to consider using realloc to expand your table. This will either expand in place (if it can) or allocate new and copy the old portion over.

When you call m_free and pass in sizeof(old_vtable), since old_vtable is a pointer it will have a size of 4 or 8 bytes.

Where does it segfault? Since you're presumably running this under linux, you should be able to run the debugger to find out.

The very first time this is called, old_vtable will be NULL, and I don't see any checks which compensate for that.

I think your for loop should be using tab_len-2 (not -1). The very first time the for loop is called, it shouldn't be copying any elements. tab_len will be equal to 3. so you want it to look like: for (i = 1; i < 1; i++)

Turbinenreiter
Posts: 288
Joined: Sun May 04, 2014 8:54 am

Re: Implementing a C function that takes a Python function as argument

Post by Turbinenreiter » Sun Dec 18, 2016 9:54 pm

realloc solved my problem.

I don't actually expose the vtable to pythong, rather I have the register function used to register python functions to the dbus append it. I got it working now, but there might even be an easier way, as I by now found a function in sd-bus to add to the vtable.

Next step is figuring out how to handle different data types and number of arguments. Also function names of arbitrary length. As of now, this works:

Code: Select all

import dbus

def foo(inp):
    return(inp+1)

def bar(inp):
    return(inp-1)

dbus.register(foo, 'x', 'x')
dbus.register(bar, 'x', 'x')

dbus.run('space.plamauer.test', '/space/plamauer/test')
I will push this to my fork on GitHub later, for now code is below.

Code: Select all

#include "py/mpconfig.h"

#include "py/obj.h"
#include "py/runtime.h"

#include <systemd/sd-bus.h>

sd_bus_vtable *test_vtable = NULL;
mp_obj_t *fun_table = NULL;
int tab_len = 0;
static sd_bus_vtable start = SD_BUS_VTABLE_START(0);
static sd_bus_vtable end = SD_BUS_VTABLE_END;

static int method_call(sd_bus_message *m, void *userdata, sd_bus_error *ret_error) {
    int64_t inp, outp;
    int r, fun_no;
    fun_no = 0;

    /* Read the parameters */
    r = sd_bus_message_read(m, "x", &inp);
    const char *member = sd_bus_message_get_member(m);

    for(i=0; i<tab_len; i++) {
        if(strcmp(member, test_vtable[i+1].x.method.member == 0)) {
            fun_no = i;
            break;
        }
    }

    mp_obj_t inp_obj = mp_obj_new_int_from_ll(inp);
    if (r < 0) {
            fprintf(stderr, "Failed to parse parameters: %s\n", strerror(-r));
            return r;
    }

    outp = mp_obj_get_int(mp_call_function_1(fun_table[fun_no],
                                             inp_obj));

    /* Reply with the response */
    return sd_bus_reply_method_return(m, "x", outp);
}

STATIC mp_obj_t mod_dbus_register(mp_obj_t function, mp_obj_t inp, mp_obj_t outp) {
    qstr qstr_fun_name = mp_obj_fun_get_name(function);
    qstr qstr_inp = mp_obj_str_get_qstr(inp);
    qstr qstr_outp = mp_obj_str_get_qstr(outp);

    fun_table = m_realloc(fun_table,
                          sizeof(mp_obj_t)*tab_len,
                          sizeof(mp_obj_t)*tab_len+1);
    fun_table[tab_len] = function;

    // sd_bus_add_object_vtable
    test_vtable = m_realloc(test_vtable,
                            sizeof(struct sd_bus_vtable)*(tab_len+2),
                            sizeof(struct sd_bus_vtable)*(tab_len+3));
    tab_len++;

    sd_bus_vtable new_elem = SD_BUS_METHOD(qstr_str(qstr_fun_name),
                                           qstr_str(qstr_inp),
                                           qstr_str(qstr_outp),
                                           method_call,
                                           SD_BUS_VTABLE_UNPRIVILEGED);

    test_vtable[0] = start;
    test_vtable[tab_len] = new_elem;
    test_vtable[tab_len+1] = end;

    return MP_OBJ_NEW_QSTR(qstr_fun_name);//mp_const_none;
}
STATIC MP_DEFINE_CONST_FUN_OBJ_3(mod_dbus_register_obj, mod_dbus_register);

STATIC mp_obj_t mod_dbus_run(mp_obj_t interface_name, mp_obj_t object_path) {
    const char *name = mp_obj_str_get_str(interface_name);
    const char *path = mp_obj_str_get_str(object_path);

    sd_bus_slot *slot = NULL;
    sd_bus *bus = NULL;
    int r;

    /* Connect to the user bus */
    r = sd_bus_open_user(&bus);
    if (r < 0) {
            fprintf(stderr, "Failed to connect to system bus: %s\n", strerror(-r));
            goto finish;
    }

    /* Install the object */
    r = sd_bus_add_object_vtable(bus,
                                 &slot,
                                 path,
                                 name,
                                 test_vtable,
                                 NULL);
    if (r < 0) {
            fprintf(stderr, "Failed to issue method call: %s\n", strerror(-r));
            goto finish;
    }


    /* Request a name */
    r = sd_bus_request_name(bus, name, 0);
    if (r < 0) {
            fprintf(stderr, "Failed to acquire service name: %s\n", strerror(-r));
            goto finish;
    }

    for (;;) {
            /* Process requests */
            r = sd_bus_process(bus, NULL);
            if (r < 0) {
                    fprintf(stderr, "Failed to process bus: %s\n", strerror(-r));
                    goto finish;
            }
            if (r > 0) /* we processed a request, try to process another one, right-away */
                    continue;

            /* Wait for the next request to process */
            r = sd_bus_wait(bus, (uint64_t) -1);
            if (r < 0) {
                    fprintf(stderr, "Failed to wait on bus: %s\n", strerror(-r));
                    goto finish;
            }
    }

finish:
    sd_bus_slot_unref(slot);
    sd_bus_unref(bus);
    return MP_OBJ_NEW_SMALL_INT(r);
}
STATIC MP_DEFINE_CONST_FUN_OBJ_2(mod_dbus_run_obj, mod_dbus_run);

STATIC const mp_rom_map_elem_t mp_module_dbus_globals_table[] = {
    { MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_dbus) },
    { MP_ROM_QSTR(MP_QSTR_register), (mp_obj_t)&mod_dbus_register_obj },
    { MP_ROM_QSTR(MP_QSTR_run), (mp_obj_t)&mod_dbus_run_obj },
};

STATIC MP_DEFINE_CONST_DICT(mp_module_dbus_globals, mp_module_dbus_globals_table);

const mp_obj_module_t mp_module_dbus = {
    .base = { &mp_type_module },
    .globals = (mp_obj_dict_t*)&mp_module_dbus_globals,
};
//edit:
fixed some bugs in the code


Turbinenreiter
Posts: 288
Joined: Sun May 04, 2014 8:54 am

Re: Implementing a C function that takes a Python function as argument

Post by Turbinenreiter » Thu Jan 05, 2017 4:44 pm

I expanded this quite a bit, supporting different data types now as well as 0 to 1 input arguments.

Currently I'm struggling to return arrays.

I get a list from MicroPython like so:

Code: Select all

out_obj = mp_obj_new_list(out_len, mp_call_function_0(fun_table[fun_no]));
The items in that list I have to convert to the needed datatypes and then send using this function:

Code: Select all

sd_bus_reply_method_return(m, out_type, out_len, val1, val2, val3, ...);
On the sd-bus side, I'm afraid I will have to construct the message myself instead of using the reply_method_return function, as it doesn't allow passing an array in.
On the MicroPython side I have to figure out how to access the contents of the list in out_obj.

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Implementing a C function that takes a Python function as argument

Post by dhylands » Thu Jan 05, 2017 6:27 pm

Here's an example of allocating and returning an array (a list):
https://github.com/micropython/micropyt ... #L303-L312

A list is mutable, and a tuple is immutable, so you can choose your poison depending on how you want it treated.

This example:
https://github.com/micropython/micropyt ... #L487-L531
shows returning and processing a tuple.

Here are the public tuple/list functions:
https://github.com/micropython/micropyt ... #L633-L634 (constructors)
https://github.com/micropython/micropyt ... #L716-L729 (accessors)

Keep in mind that python lists and tuples are themselves python objects, and contain python objects as elements.

Your example:

Code: Select all

out_obj = mp_obj_new_list(out_len, mp_call_function_0(fun_table[fun_no]));
doesn't look right to me.

mp_obj_new_list expects a pointer to a C array of python objects. Python functions can't return one of those. mp_call_function_0 is going to return a single python object (which may be a list). You can either just use the return value directly (if it's already a list) or if it's return an object you want in the list, then you'll need to construct your array of objects (in which case the length passed to mp_obj_new_list should be 1.

Post Reply