gc.collect() triggers finalizer method too early?m_new_obj_with_finaliser()

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
User avatar
braiins
Posts: 19
Joined: Mon Feb 23, 2015 1:09 pm
Contact:

gc.collect() triggers finalizer method too early?m_new_obj_with_finaliser()

Post by braiins » Thu Feb 25, 2016 3:18 pm

Hi,

I have a question regarding the behavior of the garbage collector with regards to handling objects created by

Code: Select all

m_new_obj_with_finaliser() 
function.

Scenario:
- we have a custom built-in class that creates the instance of the object using the above function as part of the make_new() implementation
- the class also has a __del__ method for cleanup/shutdown of the hardware that it controls

Runtime test:
- have micropython use some of its heap - issuing couple of syntax errors/import errors etc.
- create an instance of the class in repl and keep the reference in repl's local variable (u)
- trigger gc.collect()

Code: Select all

>>> u=hal.UART(1, 9600, timeoutx=-1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: extra keyword arguments given
>>> u=hal.UART(1, 9600, timeoutx=-1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: extra keyword arguments given
>>> gc.mem_free()
13936
>>> u=hal.UART(1, 9600, timeout=-1)
>>> gc.collect()
Problem:
The gc.collect() now attempts to finalise the object and kill it. How is it possible? Do I misunderstand anything about the use of the finaliser?

We have started a wiki page: http://wiki.micropython.org/Developers/PortingNotes that covers topics that we had to go through while working on the port for FreeRTOS. The reason is that I haven't found any developer related documentation except for this forum. It is a work-in-progress, I will be adding elements throught the development process.

Thanks for any input on this topic.

Best regards,

Jan
braiins - Your partner in embedded world - http://www.braiins.cz

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: gc.collect() triggers finalizer method too early?m_new_obj_with_finaliser()

Post by dhylands » Thu Feb 25, 2016 6:48 pm

Did you have a print in your finaliser? How do you know your finaliser was called?

I don't see any evidence in your log. It would be good to have the make_new method also do a print which includes the address of the object (i.e. the pointer returned by mp_new_obj_with_finalizer and also have your __del__ method print the pointer value for self.

I tried on the pyboard, and for objects created in the REPL, I'm seeing the opposite problem. My finaliser isn't getting called at all. It gets called fine for objects created in a function that are then collected.

Can you share your source for the C code?

What does your C gc_collect function look like?

User avatar
braiins
Posts: 19
Joined: Mon Feb 23, 2015 1:09 pm
Contact:

Re: gc.collect() triggers finalizer method too early?m_new_obj_with_finaliser()

Post by braiins » Thu Feb 25, 2016 9:02 pm

Hi,

I will probably step back and do some testing on an STM32F4 discovery board. This is a custom port to an obsolete platform from TI/Stellaris (LM3s9D92). Please, check my comments below
dhylands wrote:Did you have a print in your finaliser? How do you know your finaliser was called?

I don't see any evidence in your log. It would be good to have the make_new method also do a print which includes the address of the object (i.e. the pointer returned by mp_new_obj_with_finalizer and also have your __del__ method print the pointer value for self.
,

I am connected with my gdb session to the target. So, I have set a breakpoint on the __del__ method implementation and from there on I was following a backtrace.
dhylands wrote: I tried on the pyboard, and for objects created in the REPL, I'm seeing the opposite problem. My finaliser isn't getting called at all. It gets called fine for objects created in a function that are then collected.

Can you share your source for the C code?

What does your C gc_collect function look like?
On the other hand, if I don't perform any action in the REPL - so no heap memory gets used initially and I just instantiate the class, and throw away the local variable (that retains the instance reference), it would trigger the finaliser correctly. If I keep the variable to reference the instance it won't trigger the finaliser as long as I don't make REPL use some more heap space - e.g. by some of the actions shown in the log in the original post

The project is not ready to be completely published yet, below is the current platform specific part of the GC that my port provides. As you can see it is fairly simple, I have had no issues with it until now. Maybe it is just me not understanding the API properly.

Code: Select all

#include <stdint.h>

#include "py/gc.h"

/* FreeRTOS includes */
#include <FreeRTOS.h>

#include "lib-rtos/assert.h"

#include "micropython-freertos-hal/mp/gc_helper.h"


/** End of the stack of the micropython task */
static uint32_t gc_stack_end;



/**
 * Initialize the garbage collector with the bottom of our stack
 *
 * @param sp - stack pointer that donotes the current top of the stack
 */
void gc_collect_init(mp_uint_t sp) {

    void *mp_heap;

    // GC init
    mp_heap = pvPortMalloc(CONFIG_MICROPYTHON_FREERTOS_HAL_HEAP_SIZE);

    assert_ptr(mp_heap);

    gc_init(mp_heap, mp_heap + CONFIG_MICROPYTHON_FREERTOS_HAL_HEAP_SIZE);
    gc_stack_end = sp;
}


/**
 * @todo: Add timing information
 */
void gc_collect(void) {
    // TODO: move this to architecture specific section (ARM-Cortex)
    // get the registers and the sp
    mp_uint_t regs[10];

    /* start the GC */
    gc_collect_start();

    mp_uint_t sp = gc_helper_get_regs_and_sp(regs);


    /* perform garbage collection on the stack including the registers */
    gc_collect_root((void**)sp, ((uint32_t)gc_stack_end - sp) / sizeof(uint32_t));

    /* end the GC */
    gc_collect_end();
}
This is the code for the make_new method:

Code: Select all

STATIC mp_obj_t mp_uart_make_new(mp_obj_t type_in, mp_uint_t n_args, mp_uint_t n_kw, const mp_obj_t *args) {
    mp_uart_obj_t *self;
    int uart_id = 0;

    // check arguments
    mp_arg_check_num(n_args, n_kw, 1, MP_OBJ_FUN_ARGS_MAX, true);


    uart_id = mp_obj_get_int(args[0]);
    if (uart_id < 0 || uart_id >= MAX_UART_ID) {
        nlr_raise(mp_obj_new_exception_msg_varg(&mp_type_ValueError, "UART(%d) does not exist", uart_id));
    }

    /* create new UART object - we have an explicit finaliser */
    self = m_new_obj_with_finaliser(mp_uart_obj_t);
    self->base.type = &mp_uart_type;
    self->uart_id = uart_id;

    if (n_args > 1 || n_kw > 0) {
        // start the peripheral
        mp_map_t kw_args;
        mp_map_init_fixed_table(&kw_args, n_kw, args + n_args);
        mp_uart_init_helper(self, n_args - 1, args + 1, &kw_args);
    }

    return self;
}
In every case, thank you for any input on this.

Best regards,

Jan
braiins - Your partner in embedded world - http://www.braiins.cz

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: gc.collect() triggers finalizer method too early?m_new_obj_with_finaliser()

Post by dhylands » Thu Feb 25, 2016 9:44 pm

braiins wrote:
dhylands wrote: I tried on the pyboard, and for objects created in the REPL, I'm seeing the opposite problem. My finaliser isn't getting called at all. It gets called fine for objects created in a function that are then collected.

Can you share your source for the C code?

What does your C gc_collect function look like?
On the other hand, if I don't perform any action in the REPL - so no heap memory gets used initially and I just instantiate the class, and throw away the local variable (that retains the instance reference), it would trigger the finaliser correctly. If I keep the variable to reference the instance it won't trigger the finaliser as long as I don't make REPL use some more heap space - e.g. by some of the actions shown in the log in the original post
The REPL assigns the variable _ to any results which would otherwise be thrown away (just something to be aware of). So if you do:

Code: Select all

>>> 57
57
>>> _
57
>>> x = 56
>>> _
57
so _ will be holding on to the 57 (or any other allocated object) until it gets assigned to something else.
The project is not ready to be completely published yet, below is the current platform specific part of the GC that my port provides. As you can see it is fairly simple, I have had no issues with it until now. Maybe it is just me not understanding the API properly.

Code: Select all

#include <stdint.h>

#include "py/gc.h"

/* FreeRTOS includes */
#include <FreeRTOS.h>

#include "lib-rtos/assert.h"

#include "micropython-freertos-hal/mp/gc_helper.h"


/** End of the stack of the micropython task */
static uint32_t gc_stack_end;



/**
 * Initialize the garbage collector with the bottom of our stack
 *
 * @param sp - stack pointer that donotes the current top of the stack
 */
void gc_collect_init(mp_uint_t sp) {

    void *mp_heap;

    // GC init
    mp_heap = pvPortMalloc(CONFIG_MICROPYTHON_FREERTOS_HAL_HEAP_SIZE);

    assert_ptr(mp_heap);

    gc_init(mp_heap, mp_heap + CONFIG_MICROPYTHON_FREERTOS_HAL_HEAP_SIZE);
    gc_stack_end = sp;
}


/**
 * @todo: Add timing information
 */
void gc_collect(void) {
    // TODO: move this to architecture specific section (ARM-Cortex)
    // get the registers and the sp
    mp_uint_t regs[10];

    /* start the GC */
    gc_collect_start();

    mp_uint_t sp = gc_helper_get_regs_and_sp(regs);


    /* perform garbage collection on the stack including the registers */
    gc_collect_root((void**)sp, ((uint32_t)gc_stack_end - sp) / sizeof(uint32_t));

    /* end the GC */
    gc_collect_end();
}
This checking from the current sp to the end of RAM, which is correct for the pyboard, where there is just one global stack. With FreeRTOS, aren't the stacks allocated per thread? Or maybe you're still running on the default stack?

I took a look at your make_new method and that looks fine to me.

I'm pretty sure that the WiPy port uses FreeRTOS, so you may want to take a look at that for hints (I haven't had the time to spend any time looking at this level of detail myself).

User avatar
braiins
Posts: 19
Joined: Mon Feb 23, 2015 1:09 pm
Contact:

Re: gc.collect() triggers finalizer method too early?m_new_obj_with_finaliser()

Post by braiins » Fri Feb 26, 2016 6:35 am

Hi Dave,
dhylands wrote: The REPL assigns the variable _ to any results which would otherwise be thrown away (just something to be aware of). So if you do:

Code: Select all

>>> 57
57
>>> _
57
>>> x = 56
>>> _
57
so _ will be holding on to the 57 (or any other allocated object) until it gets assigned to something else.
I didn't know about this, but such reference should not really count and should get collected, right? I guess, I will have to dig into the sources to see how exactly the GC works. I am sure Damien would be able to shed some light into this.
dhylands wrote: This checking from the current sp to the end of RAM, which is correct for the pyboard, where there is just one global stack. With FreeRTOS, aren't the stacks allocated per thread? Or maybe you're still running on the default stack?

I took a look at your make_new method and that looks fine to me.

I'm pretty sure that the WiPy port uses FreeRTOS, so you may want to take a look at that for hints (I haven't had the time to spend any time looking at this level of detail myself).
Actually, the inspiration comes from the WiPy port. One of the steps that gc_init() does is to record the current SP. The current stackpointer is the very bottom of the stack for the micropython task and thus is stored as gc_stack_end. So, the garbage collector (except for the heap) also sweeps only the entire stack of the micropython task (which is the current SP up to gc_stack_end), not the complete RAM. I have not really changed any of this.

As of the heap allocation, I didn't want to change any of our freertos linker configurations as I find it redundant and have simply taken the heap by allocating space from FreeRTOS's heap. I find this approach more clean and readable, no linker script magic.


Anyway, thanks for your input - especially the 'underscore' variable ;-)

Cheers,

Jan
braiins - Your partner in embedded world - http://www.braiins.cz

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: gc.collect() triggers finalizer method too early?m_new_obj_with_finaliser()

Post by dhylands » Fri Feb 26, 2016 8:26 am

braiins wrote:Hi Dave,
dhylands wrote: The REPL assigns the variable _ to any results which would otherwise be thrown away (just something to be aware of). So if you do:

Code: Select all

>>> 57
57
>>> _
57
>>> x = 56
>>> _
57
so _ will be holding on to the 57 (or any other allocated object) until it gets assigned to something else.
I didn't know about this, but such reference should not really count and should get collected, right? I guess, I will have to dig into the sources to see how exactly the GC works. I am sure Damien would be able to shed some light into this.
The gc is totally independent of how the program works, and just looks for pointers (through the stack and other root pointer regions). Since _ is a variable in the python space holding a reference to a variable, it will prevent that variable from getting collected. This is expected behaviour. If it were collected then that would be very bad (since _ would now contain a pointer to freed memory rather than a real object).

The REPL (including the _ variable) is documented here:
http://docs.micropython.org/en/latest/r ... /repl.html

User avatar
braiins
Posts: 19
Joined: Mon Feb 23, 2015 1:09 pm
Contact:

Re: gc.collect() triggers finalizer method too early?m_new_obj_with_finaliser()

Post by braiins » Fri Feb 26, 2016 8:34 am

Hi Dave,

I will post a reply to myself and have a good news. Everything was actually working as expected! I will document it here just for reference. The example I gave created an instance of UART and stored it into local variable u. The next step: I tried calling the constructor with wrong parameters, however, due to the way how make_new() method is implemented, the instance would actually get created. However, the init_helper() method would eventually fail after parsing the arguments. The resulting error obviously didn't result into an assignment to variable x

Therefore, the garbage collector was actually doing its work and was doing it right - it would collect the second instance. I will reorganize the driver a bit to prevent the instance from creation.

Code: Select all

MicroPython v1.5.1 on 2016-02-26; eks-lm3s9d92 with lm3s9d92
Type "help()" for more information.
>>> import hal
>>> import gc
>>> u=hal.UART(1,9600)
>>> x=hal.UART(1,9600,time)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'time' is not defined
>>> x=hal.UART(1,9600,time=5)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: extra keyword arguments given
>>> x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'x' is not defined
>>> u
UART(1)
>>> gc.collect()
Running mp_uart___del__
>>> u
UART(1)
>>> gc.collect()
>>> u.read(3)
b'asd'
>>> u.read(1)
b'f'
>>> u.read(1)
b''
Jan
braiins - Your partner in embedded world - http://www.braiins.cz

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: gc.collect() triggers finalizer method too early?m_new_obj_with_finaliser()

Post by dhylands » Fri Feb 26, 2016 8:38 am

It's so nice when "sanity" returns. By "sanity" I mean things are understood clearly. I hate it when programs behave in mysterious ("insane") ways for no apparent reason.

User avatar
braiins
Posts: 19
Joined: Mon Feb 23, 2015 1:09 pm
Contact:

Re: gc.collect() triggers finalizer method too early?m_new_obj_with_finaliser()

Post by braiins » Fri Feb 26, 2016 1:02 pm

dhylands wrote:It's so nice when "sanity" returns. By "sanity" I mean things are understood clearly. I hate it when programs behave in mysterious ("insane") ways for no apparent reason.
YES ;-)

I now have the make_new() method roughly in this form. The init helper is used to perform preliminary argument parsing. I believe the small performance penalty is acceptable:

Code: Select all

STATIC mp_obj_t upy_uart__make_new(mp_obj_t type_in, mp_uint_t n_args, mp_uint_t n_kw, const mp_obj_t *args) {
    upy_uart__obj_t *self;
    mp_map_t kw_args;

    /* Check argument count and map the keyword arguments */
    mp_arg_check_num(n_args, n_kw, 1, MP_OBJ_FUN_ARGS_MAX, true);
    mp_map_init_fixed_table(&kw_args, n_kw, args + n_args);

    /* Use init helper to preliminary parse the arguments before any further
     * action. This prevents undesired allocation of the object */
    upy_uart__init_helper(NULL, n_args - 1, args + 1, &kw_args, true);

    /* create new UART object - we have an explicit finaliser */
    self = m_new_obj_with_finaliser(upy_uart__obj_t);
    self->base.type = &upy_uart__type;
    self->uart.id = mp_obj_get_int(args[0]);
    self->initialized = false;


    upy_uart__init_helper(self, n_args - 1, args + 1, &kw_args, false);

    return self;
}
braiins - Your partner in embedded world - http://www.braiins.cz

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: gc.collect() triggers finalizer method too early?m_new_obj_with_finaliser()

Post by pfalcon » Sat Feb 27, 2016 8:35 am

braiins wrote: We have started a wiki page: http://wiki.micropython.org/Developers/PortingNotes that covers topics that we had to go through while working on the port for FreeRTOS. The reason is that I haven't found any developer related documentation except for this forum. It is a work-in-progress, I will be adding elements throught the development process.
There're both Developers' and Users' wiki, and I would say that their breakdown and location is quite logical: Github, which hosts source code, also hosts Developers' wiki: https://github.com/micropython/micropython/wiki . http://www.micropython.org, which is user-facing site, hosts Users' wiki.

One of the problem of wiki is incomplete and/or stale content. Let me give an example from your page:
mp_obj_t

All Micro Python objects can be casted to this type (effectively void *)
This isn't true. mp_obj_t is an opaque type, defined as not compatible with any other type, and requiring MP_OBJ_FROM_PTR(), MP_OBJ_TO_PTR() macros to be accessed. Historically, it was defined as void*, and some ports still use shortcuts to access it, but it's no longer correct as of now. Regarding "outdated" part, see a page I created during my initial code acquaintance: https://github.com/micropython/micropyt ... /Internals . It wasn't edited for 2 years now, so can be expected to contain stale material now, and warns about that (surprisingly, what's written there still holds, but only because there's just couple of facts listed on that page).

Development documentation is very important, but incorrect documentation is worse than no documentation (in which case interested parties will go straight to the source code, and find it pretty well commented/documented, suggestions for improvements in that area are welcome). So, we encourage content like above (http://wiki.micropython.org/Developers/PortingNotes) to be posted as a 3rd-party blog post, to separate it from official information of MicroPython project, and specify exact version of MicroPython it was written against (blog engine should timestamp the post itself to complete the temporal references).
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

Post Reply