help is appreciated - memory allocated by m_malloc() is automatically free'ed

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
RockySong
Posts: 4
Joined: Mon Oct 02, 2017 1:55 am

help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by RockySong » Fri Nov 17, 2017 6:53 am

Hi,
I discover this issue when I was creating an OLED display driver : I used m_malloc() to allocate a framebuffer of 2kB for an OLED display. Then, after a short while, I found the OLED display get messed at the zones where I never draw onto. Later, I set a watchpoint to an address within the framebuffer that I'm sure that will not be drawn, and the watchpoint often get hit from GC related functions and other places in my test python script.

The code looks like "fb = (COLOR_T*) m_malloc(128 * 128 / 8);"

Later, I use static allocating, the issue nolonger happens.
The code looks like “
static uint8_t vbuf[128 * 128 / 8];
fb = (COLOR_T*)vbuf;


Could any one guide me in which cases the memory allocated by m_malloc() will be automatically gc'ed?

User avatar
pythoncoder
Posts: 2974
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by pythoncoder » Fri Nov 17, 2017 10:58 am

The gc will de-allocate a memory block if there is no Python reference to it which is currently in scope. So I think you need a Python function which allocates the RAM and returns a reference to it, which you can store in a variable.

Have you considered using the official framebuf module?
Peter Hinch

jickster
Posts: 506
Joined: Thu Sep 07, 2017 8:57 pm

help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by jickster » Fri Nov 17, 2017 2:41 pm

pythoncoder wrote:The gc will de-allocate a memory block if there is no Python reference to it which is currently in scope. So I think you need a Python function which allocates the RAM and returns a reference to it, which you can store in a variable.

Have you considered using the official framebuf module?
Which function do you call for returning the refernence?


Sent from my iPhone using Tapatalk

User avatar
dhylands
Posts: 2848
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by dhylands » Fri Nov 17, 2017 4:22 pm

If there is no python reference to the object allocated then there needs to be a pointer stored in the root pointers.

For example, if we look in the stm32 timer.c file, you'll see references to the MP_STATE_PORT macro:
https://github.com/micropython/micropyt ... mer.c#L722

The MP_STATE_PORT macro is defined to be MP_STATE_VM:
https://github.com/micropython/micropyt ... ort.h#L248
and just below that you'll see MICROPY_PORT_ROOT_POINTERS which includes the pyb_timer_obj_all member used in timer.c

And then over in py/mpstate.h MICROPY_PORT_ROOT_POINTERS is included in the mp_state_vm_t struct
https://github.com/micropython/micropyt ... ate.h#L166
and the MP_STATE_VM macro which pulls its members from mp_state_ctx which includes and mp_state_vm_t
https://github.com/micropython/micropyt ... ate.h#L241

This collection of root pointers is scanned by the garbage collector and any pointers included in it are not freed.

So, the summary is add a member to the MICROPY_PORT_ROOT_POINTERS ans wrap all of your references with the MP_STATE_PORT macro in order to create "root pointers".

RockySong
Posts: 4
Joined: Mon Oct 02, 2017 1:55 am

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by RockySong » Mon Nov 20, 2017 2:35 am

Thank you all, I've solved this issue by using root pointers! I indeed see multiple usage of MP_STATE_PORT macro but I didn't understand why user should add them to core struct, now I got it ---- it is to 'fool' the GC :)

However, this issue comes back (memory is GC'ed) if I use ctrl-D to soft reset, and then run the same script again. Running gui test script after core reset or power on reset never reproduce the issue.
If I use ctrl-C to break the script then run it again, this issue never happens either.
If I free the memory allocated by m_malloc(), then run the script again (will call m_malloc() again), the issue never happens either.

I guess ctrl-D soft reset free'ed the memory. So in init() of the OLED driver, I unconditionally call m_free then m_malloc, this workaround can solve the issue, but I don't know if this is a graceful way and if there is potential risk underneeth.

@pythoncoder,
I port OLED driver with a more complicated graphical library that can manage window, draw more primitives and has built-in raster fonts. The library is "SWIM", which can be used in NXP platforms.

User avatar
dhylands
Posts: 2848
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by dhylands » Mon Nov 20, 2017 7:11 pm

Yes - soft reset causes the entire heap to be re-initialized.

I think that any code which adds a root pointer should also add an init function to zero the root pointer.

jickster
Posts: 506
Joined: Thu Sep 07, 2017 8:57 pm

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by jickster » Thu Dec 14, 2017 9:23 pm

dhylands wrote:
Mon Nov 20, 2017 7:11 pm
Yes - soft reset causes the entire heap to be re-initialized.

I think that any code which adds a root pointer should also add an init function to zero the root pointer.
Does that mean that all calls to m_new that aren't root-pointers will get deleted next time the gc is called?

Code: Select all

// Init the vstr so it allocs exactly given number of bytes.  Set length to zero.
void vstr_init(vstr_t *vstr, size_t alloc) {
    if (alloc < 1) {
        alloc = 1;
    }
    vstr->alloc = alloc;
    vstr->len = 0;
    vstr->buf = m_new(char, vstr->alloc);
    vstr->fixed_buf = false;
}
Will vstr->buf be deleted?

User avatar
dhylands
Posts: 2848
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by dhylands » Mon Dec 18, 2017 7:25 pm

jickster wrote:
Thu Dec 14, 2017 9:23 pm
dhylands wrote:
Mon Nov 20, 2017 7:11 pm
Yes - soft reset causes the entire heap to be re-initialized.

I think that any code which adds a root pointer should also add an init function to zero the root pointer.
Does that mean that all calls to m_new that aren't root-pointers will get deleted next time the gc is called?

Code: Select all

// Init the vstr so it allocs exactly given number of bytes.  Set length to zero.
void vstr_init(vstr_t *vstr, size_t alloc) {
    if (alloc < 1) {
        alloc = 1;
    }
    vstr->alloc = alloc;
    vstr->len = 0;
    vstr->buf = m_new(char, vstr->alloc);
    vstr->fixed_buf = false;
}
Will vstr->buf be deleted?
vstr->buf will be deleted when there are no more pointers pointing to it. (i.e. when vstr->buf no longer points to your buffer). The garbage collector follows pointers found in allocated blocks, so if there is a chain of pointers starting in the root block to your buffer, then your buffer won't be freed, until there is no longer a chain (i.e. the object that was pointing to your buffer no longer points to your buffer). This could happen if vstr is deleted (explicitly or by the GC), or if vstr->buf is redirected to point to another buffer.

jickster
Posts: 506
Joined: Thu Sep 07, 2017 8:57 pm

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by jickster » Tue Dec 19, 2017 9:39 pm

dhylands wrote:
Mon Dec 18, 2017 7:25 pm
jickster wrote:
Thu Dec 14, 2017 9:23 pm
dhylands wrote:
Mon Nov 20, 2017 7:11 pm
Yes - soft reset causes the entire heap to be re-initialized.

I think that any code which adds a root pointer should also add an init function to zero the root pointer.
Does that mean that all calls to m_new that aren't root-pointers will get deleted next time the gc is called?

Code: Select all

// Init the vstr so it allocs exactly given number of bytes.  Set length to zero.
void vstr_init(vstr_t *vstr, size_t alloc) {
    if (alloc < 1) {
        alloc = 1;
    }
    vstr->alloc = alloc;
    vstr->len = 0;
    vstr->buf = m_new(char, vstr->alloc);
    vstr->fixed_buf = false;
}
Will vstr->buf be deleted?
vstr->buf will be deleted when there are no more pointers pointing to it. (i.e. when vstr->buf no longer points to your buffer).
How would the gc know whether vstr->buf (or any other pointer) is pointing to the buffer . . . unless vstr->buf is a root pointer?

User avatar
dhylands
Posts: 2848
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by dhylands » Tue Dec 19, 2017 10:56 pm

The GC follows pointers. If a root pointer points into the heap then the GC knows that heap block is being used. It will then scan that heap block for other pointers into the heap and repeat. When it's finished any heap blocks which weren't visited aren't being pointed at and can be freed.

Post Reply