[SOLVED]del (mp_delete_name, mp_delete_global) - why not call m_del

[jickster]

In .py, when you say

Code: Select all

m1 = 5
del m1

The memory doesn't get reclaimed when del is called, it merely gets marked as invalid and the gc needs to run to actually free that block.

Code: Select all

STATIC mp_obj_t dict_get_helper(size_t n_args, const mp_obj_t *args, mp_map_lookup_kind_t lookup_kind) {
    mp_check_self(MP_OBJ_IS_DICT_TYPE(args[0]));
    mp_obj_dict_t *self = MP_OBJ_TO_PTR(args[0]);
    mp_map_elem_t *elem = mp_map_lookup(&self->map, args[1], lookup_kind);
    mp_obj_t value;
    if (elem == NULL || elem->value == MP_OBJ_NULL) {
        if (n_args == 2) {
            if (lookup_kind == MP_MAP_LOOKUP_REMOVE_IF_FOUND) {
                nlr_raise(mp_obj_new_exception_arg1(&mp_type_KeyError, args[1]));
            } else {
                value = mp_const_none;
            }
        } else {
            value = args[2];
        }
        if (lookup_kind == MP_MAP_LOOKUP_ADD_IF_NOT_FOUND) {
            elem->value = value;
        }
    } else {
        value = elem->value;
        if (lookup_kind == MP_MAP_LOOKUP_REMOVE_IF_FOUND) {
            elem->value = MP_OBJ_NULL; // so that GC can collect the deleted value
        }
    }
    return value;
}

Code: Select all

elem->value = MP_OBJ_NULL; // so that GC can collect the deleted value

del -> mp_delete_name() -> mp_obj_dict_delete() -> dict_get_helper(2, args, MP_MAP_LOOKUP_REMOVE_IF_FOUND);

I care about this because I have the following code

Code: Select all

while True:
    m1 = my_queue1.get()    
    my_queue2.put(m1)
    del m1

.get() - allocates memory
so after enough time, the gc will be triggered EVEN IF I call "del m1"

I would like for "del m1" to also call "m_free" to free underlying memory.
This would eliminate the need to call gc (and the resulting overhead)

[stijn]

This would eliminate the need to call gc (and the resulting overhead)

Not sure if it's that simple. For this to work, wouldn't 'del m1' have to notify the gc in some way (possibly involving lookup, marking block as freed already, etc) else the next gc cycle would trigger m_free again because it wouldn't know that happened already? In other words, performance-wise it might not matter whether del() calls into the memory manager, or the next gc cycle does that.

Anyway, that is just how I think it is, no idea if that is really correct. But having a C++ background with automatic cleanup being baked into the language, I can understand why you're having problems. Then again, when starting Python I had to get used to it and I just settled for using context managers when really needed, or else insert gc.collect() calls when appropriate, and never looked back. Though your mileage may vary.

[dhylands]

In general del can't call m_free because it doesn't know whether its the last reference to the block of memory.

For example:

Code: Select all

m1 = 5
m2 = [m1]
del m1

If del were to call m_free then m2 would contain a pointer to freed data, which is bad.

[jickster]

In general del can't call m_free because it doesn't know whether its the last reference to the block of memory.

For example:

Code: Select all

m1 = 5
m2 = [m1]
del m1

If del were to call m_free then m2 would contain a pointer to freed data, which is bad.

I agree that merely adding "gc_free('m1')" to the existing code is not enough to deal with your scenario but two points:

(1) so what? you play with fire, better be careful.
It could be an optional feature for performance sensitive applications like long-running loops that would benefit from not having GC "randomly" run.

(2) to make it work for all scenarios, you'd have to do a reverse dict lookup - use the pointer as the key - in local i.e. current function scope dictionary along with any "local" dictionaries in the context of the callers'.
But it's more complicated: "m2" is mapped to a list so I can't merely compare the pointers, the values, in the dict.
I'd have to implement something like what the GC does with root pointers when searching for whether "m1" is being referenced somewhere else.

So option (2) will never be implemented but (1) is still useful for performance-sensitive loops.

Thoughts?

[dhylands]

to make it work for all scenarios you need to essentially do all of the work that gc_collect does, so you can just call gc.collect()

If you really want a function that does del followed by free, then I'd recommend that you just write your own C extension which does exactly that.

[jickster]

to make it work for all scenarios you need to essentially do all of the work that gc_collect does, so you can just call gc.collect()

If you really want a function that does del followed by free, then I'd recommend that you just write your own C extension which does exactly that.

Thanks for the confirmation.

Do you think I should do it as an additional bytecode or as C-module?

I’m leaning toward C-module so I don’t have to screw with modifying/testing the compiler.

[dhylands]

I'd definitely do it as a C module. Messing with the compiler/parser seems like a lot more potential to introduce weird errors.

[jickster]

I'd definitely do it as a C module. Messing with the compiler/parser seems like a lot more potential to introduce weird errors.

This wouldn’t be an issue if uPy used reference counting.

Why doesn’t it?

Sure there’s ?more RAM usage? but runtime is vastly improved.

[dhylands]

Because upy runs in a limited RAM environment and doing it the way it's done saves ram.