Page 1 of 5

help is appreciated - memory allocated by m_malloc() is automatically free'ed

Posted: Fri Nov 17, 2017 6:53 am
by RockySong
Hi,
I discover this issue when I was creating an OLED display driver : I used m_malloc() to allocate a framebuffer of 2kB for an OLED display. Then, after a short while, I found the OLED display get messed at the zones where I never draw onto. Later, I set a watchpoint to an address within the framebuffer that I'm sure that will not be drawn, and the watchpoint often get hit from GC related functions and other places in my test python script.

The code looks like "fb = (COLOR_T*) m_malloc(128 * 128 / 8);"

Later, I use static allocating, the issue nolonger happens.
The code looks like “
static uint8_t vbuf[128 * 128 / 8];
fb = (COLOR_T*)vbuf;


Could any one guide me in which cases the memory allocated by m_malloc() will be automatically gc'ed?

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Posted: Fri Nov 17, 2017 10:58 am
by pythoncoder
The gc will de-allocate a memory block if there is no Python reference to it which is currently in scope. So I think you need a Python function which allocates the RAM and returns a reference to it, which you can store in a variable.

Have you considered using the official framebuf module?

help is appreciated - memory allocated by m_malloc() is automatically free'ed

Posted: Fri Nov 17, 2017 2:41 pm
by jickster
pythoncoder wrote:The gc will de-allocate a memory block if there is no Python reference to it which is currently in scope. So I think you need a Python function which allocates the RAM and returns a reference to it, which you can store in a variable.

Have you considered using the official framebuf module?
Which function do you call for returning the refernence?


Sent from my iPhone using Tapatalk

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Posted: Fri Nov 17, 2017 4:22 pm
by dhylands
If there is no python reference to the object allocated then there needs to be a pointer stored in the root pointers.

For example, if we look in the stm32 timer.c file, you'll see references to the MP_STATE_PORT macro:
https://github.com/micropython/micropyt ... mer.c#L722

The MP_STATE_PORT macro is defined to be MP_STATE_VM:
https://github.com/micropython/micropyt ... ort.h#L248
and just below that you'll see MICROPY_PORT_ROOT_POINTERS which includes the pyb_timer_obj_all member used in timer.c

And then over in py/mpstate.h MICROPY_PORT_ROOT_POINTERS is included in the mp_state_vm_t struct
https://github.com/micropython/micropyt ... ate.h#L166
and the MP_STATE_VM macro which pulls its members from mp_state_ctx which includes and mp_state_vm_t
https://github.com/micropython/micropyt ... ate.h#L241

This collection of root pointers is scanned by the garbage collector and any pointers included in it are not freed.

So, the summary is add a member to the MICROPY_PORT_ROOT_POINTERS ans wrap all of your references with the MP_STATE_PORT macro in order to create "root pointers".

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Posted: Mon Nov 20, 2017 2:35 am
by RockySong
Thank you all, I've solved this issue by using root pointers! I indeed see multiple usage of MP_STATE_PORT macro but I didn't understand why user should add them to core struct, now I got it ---- it is to 'fool' the GC :)

However, this issue comes back (memory is GC'ed) if I use ctrl-D to soft reset, and then run the same script again. Running gui test script after core reset or power on reset never reproduce the issue.
If I use ctrl-C to break the script then run it again, this issue never happens either.
If I free the memory allocated by m_malloc(), then run the script again (will call m_malloc() again), the issue never happens either.

I guess ctrl-D soft reset free'ed the memory. So in init() of the OLED driver, I unconditionally call m_free then m_malloc, this workaround can solve the issue, but I don't know if this is a graceful way and if there is potential risk underneeth.

@pythoncoder,
I port OLED driver with a more complicated graphical library that can manage window, draw more primitives and has built-in raster fonts. The library is "SWIM", which can be used in NXP platforms.

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Posted: Mon Nov 20, 2017 7:11 pm
by dhylands
Yes - soft reset causes the entire heap to be re-initialized.

I think that any code which adds a root pointer should also add an init function to zero the root pointer.

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Posted: Thu Dec 14, 2017 9:23 pm
by jickster
dhylands wrote:
Mon Nov 20, 2017 7:11 pm
Yes - soft reset causes the entire heap to be re-initialized.

I think that any code which adds a root pointer should also add an init function to zero the root pointer.
Does that mean that all calls to m_new that aren't root-pointers will get deleted next time the gc is called?

Code: Select all

// Init the vstr so it allocs exactly given number of bytes.  Set length to zero.
void vstr_init(vstr_t *vstr, size_t alloc) {
    if (alloc < 1) {
        alloc = 1;
    }
    vstr->alloc = alloc;
    vstr->len = 0;
    vstr->buf = m_new(char, vstr->alloc);
    vstr->fixed_buf = false;
}
Will vstr->buf be deleted?

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Posted: Mon Dec 18, 2017 7:25 pm
by dhylands
jickster wrote:
Thu Dec 14, 2017 9:23 pm
dhylands wrote:
Mon Nov 20, 2017 7:11 pm
Yes - soft reset causes the entire heap to be re-initialized.

I think that any code which adds a root pointer should also add an init function to zero the root pointer.
Does that mean that all calls to m_new that aren't root-pointers will get deleted next time the gc is called?

Code: Select all

// Init the vstr so it allocs exactly given number of bytes.  Set length to zero.
void vstr_init(vstr_t *vstr, size_t alloc) {
    if (alloc < 1) {
        alloc = 1;
    }
    vstr->alloc = alloc;
    vstr->len = 0;
    vstr->buf = m_new(char, vstr->alloc);
    vstr->fixed_buf = false;
}
Will vstr->buf be deleted?
vstr->buf will be deleted when there are no more pointers pointing to it. (i.e. when vstr->buf no longer points to your buffer). The garbage collector follows pointers found in allocated blocks, so if there is a chain of pointers starting in the root block to your buffer, then your buffer won't be freed, until there is no longer a chain (i.e. the object that was pointing to your buffer no longer points to your buffer). This could happen if vstr is deleted (explicitly or by the GC), or if vstr->buf is redirected to point to another buffer.

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Posted: Tue Dec 19, 2017 9:39 pm
by jickster
dhylands wrote:
Mon Dec 18, 2017 7:25 pm
jickster wrote:
Thu Dec 14, 2017 9:23 pm
dhylands wrote:
Mon Nov 20, 2017 7:11 pm
Yes - soft reset causes the entire heap to be re-initialized.

I think that any code which adds a root pointer should also add an init function to zero the root pointer.
Does that mean that all calls to m_new that aren't root-pointers will get deleted next time the gc is called?

Code: Select all

// Init the vstr so it allocs exactly given number of bytes.  Set length to zero.
void vstr_init(vstr_t *vstr, size_t alloc) {
    if (alloc < 1) {
        alloc = 1;
    }
    vstr->alloc = alloc;
    vstr->len = 0;
    vstr->buf = m_new(char, vstr->alloc);
    vstr->fixed_buf = false;
}
Will vstr->buf be deleted?
vstr->buf will be deleted when there are no more pointers pointing to it. (i.e. when vstr->buf no longer points to your buffer).
How would the gc know whether vstr->buf (or any other pointer) is pointing to the buffer . . . unless vstr->buf is a root pointer?

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Posted: Tue Dec 19, 2017 10:56 pm
by dhylands
The GC follows pointers. If a root pointer points into the heap then the GC knows that heap block is being used. It will then scan that heap block for other pointers into the heap and repeat. When it's finished any heap blocks which weren't visited aren't being pointed at and can be freed.