help is appreciated - memory allocated by m_malloc() is automatically free'ed

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by jickster » Wed Dec 20, 2017 5:11 pm

dhylands wrote:
Tue Dec 19, 2017 10:56 pm
The GC follows pointers. If a root pointer points into the heap then the GC knows that heap block is being used. It will then scan that heap block for other pointers into the heap and repeat. When it's finished any heap blocks which weren't visited aren't being pointed at and can be freed.
So only if it's a root pointer it can do the "following", right?

If I have this code

Code: Select all

char * temp = m_new(char, 10);
gc_collect();
then will the buffer returned to char * temp will be gone after gc_collect() is executed?

User avatar
dhylands
Posts: 3234
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by dhylands » Wed Dec 20, 2017 9:12 pm

jickster wrote:
Wed Dec 20, 2017 5:11 pm
dhylands wrote:
Tue Dec 19, 2017 10:56 pm
The GC follows pointers. If a root pointer points into the heap then the GC knows that heap block is being used. It will then scan that heap block for other pointers into the heap and repeat. When it's finished any heap blocks which weren't visited aren't being pointed at and can be freed.
So only if it's a root pointer it can do the "following", right?

If I have this code

Code: Select all

char * temp = m_new(char, 10);
gc_collect();
then will the buffer returned to char * temp will be gone after gc_collect() is executed?
OK - I simplified it a bit. The actual code, also treats the contents of the stack and the contents of the current registers as potential sources of root pointers. This means that the buffer allocated in your example will not be freed (since temp will either be stored on the stack or in a register depending on the code generation).

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by jickster » Mon Feb 05, 2018 9:20 pm

dhylands wrote:
Wed Dec 20, 2017 9:12 pm
jickster wrote:
Wed Dec 20, 2017 5:11 pm
dhylands wrote:
Tue Dec 19, 2017 10:56 pm
The GC follows pointers. If a root pointer points into the heap then the GC knows that heap block is being used. It will then scan that heap block for other pointers into the heap and repeat. When it's finished any heap blocks which weren't visited aren't being pointed at and can be freed.
So only if it's a root pointer it can do the "following", right?

If I have this code

Code: Select all

char * temp = m_new(char, 10);
gc_collect();
then will the buffer returned to char * temp will be gone after gc_collect() is executed?
OK - I simplified it a bit. The actual code, also treats the contents of the stack and the contents of the current registers as potential sources of root pointers. This means that the buffer allocated in your example will not be freed (since temp will either be stored on the stack or in a register depending on the code generation).
Is there an effort to produce documentation for the code?

"Look at the code" isn't documentation.
"Read the forum" isn't documentation.

User avatar
pythoncoder
Posts: 3651
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by pythoncoder » Tue Feb 06, 2018 6:30 am

I think you're asking a lot for detailed documentation of MicroPython internals. There are only a handful of busy people with the knowledge to write it.

Out of interest does such documentation exist for CPython?
Peter Hinch

Online
stijn
Posts: 366
Joined: Thu Apr 24, 2014 9:13 am

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by stijn » Tue Feb 06, 2018 10:01 am

I think I mentioned this before, but here it is again: such documentation of internals is not needed and hence makes no sense. In my opinion.
It is not needed because the most of the MicroPython code is particularly well written and readable and has comments where needed. Apart from that, if you need to know how the internals work you probably plan on changing them or working with them. In order to be able to do that you should have a basic understanding of the code. If you cannot get that from the code itself you probably shouldn't be trying to fiddle with it in the first place and you're up for some more studying. Documentation of internals isn't going to help with that: it can't really do much more than rather self-explanatory code. Note this doesn't mean you can just look any part of the code and grasp what it does 5 minutes later. But it is a utopy to believe that documentation of the internals would somehow enable that. It just takes some effort. But you'll learn from it.

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by jickster » Wed Apr 11, 2018 10:41 pm

pythoncoder wrote:
Fri Nov 17, 2017 10:58 am
The gc will de-allocate a memory block if there is no Python reference to it which is currently in scope. So I think you need a Python function which allocates the RAM and returns a reference to it, which you can store in a variable.

Have you considered using the official framebuf module?
Give an example of a Python function which allocates the RAM and returns a reference to it

User avatar
pythoncoder
Posts: 3651
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by pythoncoder » Thu Apr 12, 2018 6:32 am

It's down to the Python variable scope rules.

Code: Select all

def allocate_buffer():
    return byterray(100)  # Allocates and returns a reference

def bar():
    buf = allocate_buffer()
    # as long as buf is in scope it won't be garbage collected

def foo():
    bar()
    # At this point buf is out of scope and may be gc'd
Peter Hinch

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by jickster » Thu Apr 12, 2018 7:39 pm

pythoncoder wrote:
Fri Nov 17, 2017 10:58 am
The gc will de-allocate a memory block if there is no Python reference to it which is currently in scope. So I think you need a Python function which allocates the RAM and returns a reference to it, which you can store in a variable.

Have you considered using the official framebuf module?
Notice the comment // need to store this to prevent GC from reclaiming buf
How does storing that prevent GC from reclaiming the memory?

modframebuf.c

Code: Select all

typedef struct _mp_obj_framebuf_t {
    mp_obj_base_t base;
    mp_obj_t buf_obj; // need to store this to prevent GC from reclaiming buf
    void *buf;
    uint16_t width, height, stride;
    uint8_t format;
} mp_obj_framebuf_t;
It's just referenced once in

Code: Select all

STATIC mp_obj_t framebuf_make_new(const mp_obj_type_t *type, size_t n_args, size_t n_kw, const mp_obj_t *args) {
    mp_arg_check_num(n_args, n_kw, 4, 5, false);

    mp_obj_framebuf_t *o = m_new_obj(mp_obj_framebuf_t);
    o->base.type = type;
    o->buf_obj = args[0];
    ...
With respect to Python code, we're storing the pointer to the bytearray(10 * 100 * 2)

Code: Select all

fbuf = FrameBuffer(bytearray(10 * 100 * 2), 10, 100, framebuf.RGB565)
So how is storing it preventing it from being GCd?

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by jickster » Thu Apr 12, 2018 7:51 pm

dhylands wrote:
Fri Nov 17, 2017 4:22 pm
If there is no python reference to the object allocated then there needs to be a pointer stored in the root pointers.
By Python reference, do you mean an entry in the list of items obtained from the dict() command?

User avatar
dhylands
Posts: 3234
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: help is appreciated - memory allocated by m_malloc() is automatically free'ed

Post by dhylands » Fri Apr 13, 2018 12:08 am

By Python reference, I meant some python code which is storing a reference to the object (could be anyplace that python stores things).

If the GC detects a pointers pointing to an object in the heap as part of its scan (it looks through the root pointers, the stack, and any python objects which have previously been referenced) then it adds that block to the collection of objects to "not free" and also scans the object for pointers to other objects that it may contain.

Post Reply