[SOLVED]del (mp_delete_name, mp_delete_global) - why not call m_del

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
Post Reply
jickster
Posts: 628
Joined: Thu Sep 07, 2017 8:57 pm

[SOLVED]del (mp_delete_name, mp_delete_global) - why not call m_del

Post by jickster » Tue May 01, 2018 7:12 pm

In .py, when you say

Code: Select all

m1 = 5
del m1
The memory doesn't get reclaimed when del is called, it merely gets marked as invalid and the gc needs to run to actually free that block.

Code: Select all

STATIC mp_obj_t dict_get_helper(size_t n_args, const mp_obj_t *args, mp_map_lookup_kind_t lookup_kind) {
    mp_check_self(MP_OBJ_IS_DICT_TYPE(args[0]));
    mp_obj_dict_t *self = MP_OBJ_TO_PTR(args[0]);
    mp_map_elem_t *elem = mp_map_lookup(&self->map, args[1], lookup_kind);
    mp_obj_t value;
    if (elem == NULL || elem->value == MP_OBJ_NULL) {
        if (n_args == 2) {
            if (lookup_kind == MP_MAP_LOOKUP_REMOVE_IF_FOUND) {
                nlr_raise(mp_obj_new_exception_arg1(&mp_type_KeyError, args[1]));
            } else {
                value = mp_const_none;
            }
        } else {
            value = args[2];
        }
        if (lookup_kind == MP_MAP_LOOKUP_ADD_IF_NOT_FOUND) {
            elem->value = value;
        }
    } else {
        value = elem->value;
        if (lookup_kind == MP_MAP_LOOKUP_REMOVE_IF_FOUND) {
            elem->value = MP_OBJ_NULL; // so that GC can collect the deleted value
        }
    }
    return value;
}

Code: Select all

elem->value = MP_OBJ_NULL; // so that GC can collect the deleted value
del -> mp_delete_name() -> mp_obj_dict_delete() -> dict_get_helper(2, args, MP_MAP_LOOKUP_REMOVE_IF_FOUND);

I care about this because I have the following code

Code: Select all

while True:
    m1 = my_queue1.get()    
    my_queue2.put(m1)
    del m1
.get() - allocates memory
so after enough time, the gc will be triggered EVEN IF I call "del m1"

I would like for "del m1" to also call "m_free" to free underlying memory.
This would eliminate the need to call gc (and the resulting overhead)
Last edited by jickster on Thu May 03, 2018 4:12 pm, edited 2 times in total.

stijn
Posts: 321
Joined: Thu Apr 24, 2014 9:13 am

Re: del (mp_delete_name, mp_delete_global) - why not call m_del

Post by stijn » Tue May 01, 2018 7:32 pm

This would eliminate the need to call gc (and the resulting overhead)
Not sure if it's that simple. For this to work, wouldn't 'del m1' have to notify the gc in some way (possibly involving lookup, marking block as freed already, etc) else the next gc cycle would trigger m_free again because it wouldn't know that happened already? In other words, performance-wise it might not matter whether del() calls into the memory manager, or the next gc cycle does that.

Anyway, that is just how I think it is, no idea if that is really correct. But having a C++ background with automatic cleanup being baked into the language, I can understand why you're having problems. Then again, when starting Python I had to get used to it and I just settled for using context managers when really needed, or else insert gc.collect() calls when appropriate, and never looked back. Though your mileage may vary.

User avatar
dhylands
Posts: 3146
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: del (mp_delete_name, mp_delete_global) - why not call m_del

Post by dhylands » Tue May 01, 2018 7:40 pm

In general del can't call m_free because it doesn't know whether its the last reference to the block of memory.

For example:

Code: Select all

m1 = 5
m2 = [m1]
del m1
If del were to call m_free then m2 would contain a pointer to freed data, which is bad.

jickster
Posts: 628
Joined: Thu Sep 07, 2017 8:57 pm

Re: del (mp_delete_name, mp_delete_global) - why not call m_del

Post by jickster » Tue May 01, 2018 8:52 pm

dhylands wrote:
Tue May 01, 2018 7:40 pm
In general del can't call m_free because it doesn't know whether its the last reference to the block of memory.

For example:

Code: Select all

m1 = 5
m2 = [m1]
del m1
If del were to call m_free then m2 would contain a pointer to freed data, which is bad.
I agree that merely adding "gc_free('m1')" to the existing code is not enough to deal with your scenario but two points:

(1) so what? you play with fire, better be careful.
It could be an optional feature for performance sensitive applications like long-running loops that would benefit from not having GC "randomly" run.

(2) to make it work for all scenarios, you'd have to do a reverse dict lookup - use the pointer as the key - in local i.e. current function scope dictionary along with any "local" dictionaries in the context of the callers'.
But it's more complicated: "m2" is mapped to a list so I can't merely compare the pointers, the values, in the dict.
I'd have to implement something like what the GC does with root pointers when searching for whether "m1" is being referenced somewhere else.

So option (2) will never be implemented but (1) is still useful for performance-sensitive loops.

Thoughts?

User avatar
dhylands
Posts: 3146
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: del (mp_delete_name, mp_delete_global) - why not call m_del

Post by dhylands » Tue May 01, 2018 10:01 pm

to make it work for all scenarios you need to essentially do all of the work that gc_collect does, so you can just call gc.collect()

If you really want a function that does del followed by free, then I'd recommend that you just write your own C extension which does exactly that.

jickster
Posts: 628
Joined: Thu Sep 07, 2017 8:57 pm

Re: del (mp_delete_name, mp_delete_global) - why not call m_del

Post by jickster » Wed May 02, 2018 12:01 am

dhylands wrote:
Tue May 01, 2018 10:01 pm
to make it work for all scenarios you need to essentially do all of the work that gc_collect does, so you can just call gc.collect()

If you really want a function that does del followed by free, then I'd recommend that you just write your own C extension which does exactly that.
Thanks for the confirmation.

Do you think I should do it as an additional bytecode or as C-module?

I’m leaning toward C-module so I don’t have to screw with modifying/testing the compiler.

User avatar
dhylands
Posts: 3146
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: del (mp_delete_name, mp_delete_global) - why not call m_del

Post by dhylands » Wed May 02, 2018 12:34 am

I'd definitely do it as a C module. Messing with the compiler/parser seems like a lot more potential to introduce weird errors.

jickster
Posts: 628
Joined: Thu Sep 07, 2017 8:57 pm

Re: del (mp_delete_name, mp_delete_global) - why not call m_del

Post by jickster » Wed May 02, 2018 1:29 am

dhylands wrote:
Wed May 02, 2018 12:34 am
I'd definitely do it as a C module. Messing with the compiler/parser seems like a lot more potential to introduce weird errors.
This wouldn’t be an issue if uPy used reference counting.

Why doesn’t it?

Sure there’s ?more RAM usage? but runtime is vastly improved.

User avatar
dhylands
Posts: 3146
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: del (mp_delete_name, mp_delete_global) - why not call m_del

Post by dhylands » Wed May 02, 2018 5:13 am

Because upy runs in a limited RAM environment and doing it the way it's done saves ram.

Post Reply