help is appreciated - memory allocated by m_malloc() is automatically free'ed

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
jickster
Posts: 167
Joined: Thu Sep 07, 2017 8:57 pm

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by jickster » Fri Apr 13, 2018 8:49 pm

dhylands wrote:
Fri Apr 13, 2018 12:08 am
By Python reference, I meant some python code which is storing a reference to the object (could be anyplace that python stores things).

If the GC detects a pointers pointing to an object in the heap as part of its scan (it looks through the root pointers, the stack, and any python objects which have previously been referenced) then it adds that block to the collection of objects to "not free" and also scans the object for pointers to other objects that it may contain.
I initialize my module hierarchy using mp_store_name
// Add the device module to the main dictionary
mp_store_name(MP_QSTR_device, (mp_obj_t)&device_module);
When I do `dir(device)`, it shows what I expect given the c-code I've written:
['CAN', '__name__']

When I do `gc.collect()` and then do `dir(device)`, my c-code segfaults i.e. hits a trap.

Why?

Isn't this the very definition of a chain?
"device" points to "CAN" points to ...

I need the real technical reason why this is happening and how to avoid it.
Last edited by jickster on Fri Apr 13, 2018 9:47 pm, edited 1 time in total.

jickster
Posts: 167
Joined: Thu Sep 07, 2017 8:57 pm

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by jickster » Fri Apr 13, 2018 8:50 pm

stijn wrote:
Tue Feb 06, 2018 10:01 am
I think I mentioned this before, but here it is again: such documentation of internals is not needed and hence makes no sense. In my opinion.
It is not needed because the most of the MicroPython code is particularly well written and readable and has comments where needed. Apart from that, if you need to know how the internals work you probably plan on changing them or working with them. In order to be able to do that you should have a basic understanding of the code. If you cannot get that from the code itself you probably shouldn't be trying to fiddle with it in the first place and you're up for some more studying. Documentation of internals isn't going to help with that: it can't really do much more than rather self-explanatory code. Note this doesn't mean you can just look any part of the code and grasp what it does 5 minutes later. But it is a utopy to believe that documentation of the internals would somehow enable that. It just takes some effort. But you'll learn from it.
I initialize my module hierarchy using mp_store_name

Code: Select all

// Add the device module to the main dictionary
mp_store_name(MP_QSTR_device, (mp_obj_t)&device_module);
When I do `dir(device)`, it shows what I expect given the c-code I've written:
['CAN', '__name__']

When I do `gc.collect()` and then do `dir(device)`, my c-code segfaults i.e. hits a trap which means that "device" was garbage collected but "device" was linked to a Python reference.

Isn't this the very definition of a chain?
"device" points to "CAN" points to ...

Documentation of internals would help with understanding why gc.collect() obliterates my CHAINED references.

User avatar
dhylands
Posts: 2708
Joined: Mon Jan 06, 2014 6:08 pm
Location: Shuswap, BC, Canada
Contact:

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by dhylands » Fri Apr 13, 2018 9:53 pm

mp_store_name stores a reference into the locals dict.

How was device_module allocated? and why are you casting it to mp_obj_t?

modules are normally declared const and come from flash, and flash objects aren't garbage collected.

So I'm confused by your example.

jickster
Posts: 167
Joined: Thu Sep 07, 2017 8:57 pm

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by jickster » Sat Apr 14, 2018 11:45 pm

dhylands wrote:
Fri Apr 13, 2018 9:53 pm
mp_store_name stores a reference into the locals dict.

How was device_module allocated? and why are you casting it to mp_obj_t?

modules are normally declared const and come from flash, and flash objects aren't garbage collected.

So I'm confused by your example.
It was allocated globally just like the other definitions of "mp_obj_module"

Code: Select all

mp_obj_dict_t atiupy_device_dict;
mp_obj_module_t device_module = {
	.base = { &mp_type_module },
	.globals = (mp_obj_dict_t*)&atiupy_device_dict,
};
I'm casting it because

Code: Select all

void mp_store_name(qstr qst, mp_obj_t obj);
void mp_store_global(qstr qst, mp_obj_t obj);
I changed it to const and I still have the same issue

Code: Select all

const mp_obj_module_t device_module = {
	.base = { &mp_type_module },
	.globals = (mp_obj_dict_t*)&atiupy_device_dict,
};
...
	// Add the device module to the main dictionary
	mp_store_name(MP_QSTR_device, (mp_obj_t)&device_module);
The very first thing I execute is "gc.collect()" and then if I do "dir(device)", my C-debugger hits a trap that indicates invalid memory access.

Here's my gc_collect().

Code: Select all

void gc_collect(void)
{
	// get current time, in case we want to time the GC
#if 0
	uint32_t start = mp_hal_ticks_us();
#endif

	// start the GC
	gc_collect_start();

	// get the registers and the sp
	mp_uint_t regs[10];
	mp_uint_t sp = gc_helper_get_regs_and_sp(regs);

	// trace the stack, including the registers (since they live on the stack in this function)
//#if MICROPY_PY_THREAD
	gc_collect_root((void**)sp, ((uint32_t)MP_STATE_THREAD(stack_top) - sp) / sizeof(uint32_t));
//#else
//	gc_collect_root((void**)sp, ((uint32_t)&g_uPyStackEnd - sp) / sizeof(uint32_t));
//#endif

	// trace root pointers from any threads
#if MICROPY_PY_THREAD
	mp_thread_gc_others();
#endif

	// end the GC
	gc_collect_end();

#if 0
	// print GC info
	uint32_t ticks = mp_hal_ticks_us() - start;
	gc_info_t info;
	gc_info(&info);
	printf("GC@%lu %lums\n", start, ticks);
	printf(" " UINT_FMT " total\n", info.total);
	printf(" " UINT_FMT " : " UINT_FMT "\n", info.used, info.free);
	printf(" 1=" UINT_FMT " 2=" UINT_FMT " m=" UINT_FMT "\n", info.num_1block, info.num_2block, info.max_block);
#endif
}
My processor is an M4 so the assembly function gc_helper_get_regs_and_sp() is the same one as the one in the folder stmhal

It's not "device_module" that's the problem but the things that are in "device_module"'s dictionary.

Normally, the hierarchy looks like this:

Code: Select all

device.CAN.
    CAN1
    CAN2
    message
    
everything below "device" is created dynamically and placed into the dictionary of its parent using mp_obj_dict_store

Code: Select all

mp_obj_dict_store(MP_OBJ_FROM_PTR(mp_obj_periph_type_module->globals), MP_OBJ_NEW_QSTR(periph_ch_qstr), mp_obj_periph_ch);
What must be happening is that I'm doing this wrong i.e. the "chain" is not being created from "device" down to the "device.CAN.xyz" and everything downstream of "device" is being garbage collected.
Last edited by jickster on Sun Apr 15, 2018 1:08 am, edited 1 time in total.

User avatar
dhylands
Posts: 2708
Joined: Mon Jan 06, 2014 6:08 pm
Location: Shuswap, BC, Canada
Contact:

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by dhylands » Sun Apr 15, 2018 1:07 am

I suspect that the problem is related to when the call to mp_store_name is being called.

The normal way to register a module is to add it to the MICROPY_PORT_BUILTIN_MODULES found in mpconfigport.h

Normally, the module object itself is declared const so that it comes from flash.

mp_store_name is normally called as part of the bytecode machinery, and I've not seen it used outside that context.

So I suspect that the issue is that mp_store_name is trying to store the device object into whatever mp_locals_get() returns and that might not be valid in the context that you're trying to use.

jickster
Posts: 167
Joined: Thu Sep 07, 2017 8:57 pm

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by jickster » Sun Apr 15, 2018 1:09 am

dhylands wrote:
Sun Apr 15, 2018 1:07 am
I suspect that the problem is related to when the call to mp_store_name is being called.

The normal way to register a module is to add it to the MICROPY_PORT_BUILTIN_MODULES found in mpconfigport.h

Normally, the module object itself is declared const so that it comes from flash.

mp_store_name is normally called as part of the bytecode machinery, and I've not seen it used outside that context.

So I suspect that the issue is that mp_store_name is trying to store the device object into whatever mp_locals_get() returns and that might not be valid in the context that you're trying to use.
Ok so then mp_store_global()? I'll try that.

I'll try that but why does mp_store_name "work" until GC? Don't care too much but I am curious.

We want it to be dynamic so that configuration of our different products is done via code, not config files that are auto-generated based off an external program.

jickster
Posts: 167
Joined: Thu Sep 07, 2017 8:57 pm

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by jickster » Sun Apr 15, 2018 1:16 am

dhylands wrote:
Sun Apr 15, 2018 1:07 am
I suspect that the problem is related to when the call to mp_store_name is being called.

The normal way to register a module is to add it to the MICROPY_PORT_BUILTIN_MODULES found in mpconfigport.h

Normally, the module object itself is declared const so that it comes from flash.

mp_store_name is normally called as part of the bytecode machinery, and I've not seen it used outside that context.

So I suspect that the issue is that mp_store_name is trying to store the device object into whatever mp_locals_get() returns and that might not be valid in the context that you're trying to use.
So it didn't work (same issue) but I didn't expect things to be different.
In "mp_init()", MP_STATE_THREAD(dict_locals) and MP_STATE_THREAD(dict_globals) are set to the same thing.

Code: Select all

mp_locals_set(&MP_STATE_VM(dict_main));
mp_globals_set(&MP_STATE_VM(dict_main));
And both those are set to "vm.dict_main"

I think I figured it out but can't test right now: I used wrong API.

I think I need to use mp_module_register

stijn
Posts: 219
Joined: Thu Apr 24, 2014 9:13 am

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by stijn » Sun Apr 15, 2018 6:40 am

Yes, see objmodule.h, mp_obj_new_module / mp_module_register is the way to go.

jickster
Posts: 167
Joined: Thu Sep 07, 2017 8:57 pm

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by jickster » Mon Apr 16, 2018 3:12 pm

stijn wrote:
Sun Apr 15, 2018 6:40 am
Yes, see objmodule.h, mp_obj_new_module / mp_module_register is the way to go.
Ok but why don'tmp_store_name or mp_store_global work?

Both are set to dict_main in mp_init

Code: Select all

    // initialise the __main__ module
    mp_obj_dict_init(&MP_STATE_VM(dict_main), 1);
    mp_obj_dict_store(MP_OBJ_FROM_PTR(&MP_STATE_VM(dict_main)), MP_OBJ_NEW_QSTR(MP_QSTR___name__), MP_OBJ_NEW_QSTR(MP_QSTR___main__));

    // locals = globals for outer module (see Objects/frameobject.c/PyFrame_New())
    mp_locals_set(&MP_STATE_VM(dict_main));
    mp_globals_set(&MP_STATE_VM(dict_main));
And mp_obj_dict_t dict_main; is in the root pointers section

Code: Select all

 // dictionary for the __main__ module
    mp_obj_dict_t dict_main;

    // these two lists must be initialised per port, after the call to mp_init
    mp_obj_list_t mp_sys_path_obj;
    mp_obj_list_t mp_sys_argv_obj;

    // dictionary for overridden builtins
    #if MICROPY_CAN_OVERRIDE_BUILTINS
    mp_obj_dict_t *mp_module_builtins_override_dict;
    #endif

    // include any root pointers defined by a port
    MICROPY_PORT_ROOT_POINTERS

    // root pointers for extmod

    #if MICROPY_PY_OS_DUPTERM
    mp_obj_t term_obj;
    mp_obj_t dupterm_arr_obj;
    #endif

    #if MICROPY_PY_LWIP_SLIP
    mp_obj_t lwip_slip_stream;
    #endif

    #if MICROPY_VFS
    struct _mp_vfs_mount_t *vfs_cur;
    struct _mp_vfs_mount_t *vfs_mount_table;
    #endif

    //
    // END ROOT POINTER SECTION
    ////////////////////////////////////////////////////////////

jickster
Posts: 167
Joined: Thu Sep 07, 2017 8:57 pm

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by jickster » Mon Apr 16, 2018 3:30 pm

stijn wrote:
Sun Apr 15, 2018 6:40 am
Yes, see objmodule.h, mp_obj_new_module / mp_module_register is the way to go.
mp_module_register also doesn't work i.e.

// c-code
mp_module_register(MP_QSTR_device, (mp_obj_t)&device_module);

# Python
import device
dir(device) # works
import gc
gc.collect()
dir(device) # c-code gives same memory segfault as using mp_store_name, mp_store_global

Post Reply