help with leaking memory management

[pythoncoder]

[deleted after brain revived]I was talking nonsense here, I'm afraid

[Roberthh]

I think that's' what I have seen with a test program which constantly allocated an frees memory in a loop. In that loop, I printed the amount of free memory told by mem_free(). That went down until no more memory was available, and then went up again, and so on. The problem seems to arise, when the memory is already scattered with small allocated blocks, and then a larger one is requested. Then, gc.collect() seems not to be able to resolve this. Running gc.collect often seems to reduce the risk.

[pythoncoder]

@Roberthh Yes, I've come across this a few times.

[dhylands]

@dhylands [...]Doing a gc collect often, in some circumstances can dramatically reduce heap fragmentation [...]

A further thought. When the VM attempts to allocate memory and the allocation fails, should it not trigger a collection and then retry? This would avoid the "out of memory" exception being thrown in such cases.

Issuing gc.collect() regularly would still be a good idea, but it strikes me that the above change would avoid needless runtime failures.

Currently that's exactly how the allocator works. If the allocation fails, it tries a gc and then retries the allocation.

But if the heap was badly fragmented then it won't help, since after the gc you wind up with a heap which has lots of tiny holes rather that one big free space.

[pythoncoder]

... if the heap was badly fragmented then it won't help, since after the gc you wind up with a heap which has lots of tiny holes rather that one big free space.

I realised that after posting. Call it a "senior moment"

[bmarkus]

... if the heap was badly fragmented then it won't help, since after the gc you wind up with a heap which has lots of tiny holes rather that one big free space.

I realised that after posting. Call it a "senior moment"

In such case is there a solution to merge them or the current GC implementation can't manage the situation and no way out in a running program except restart?

[dhylands]

The current GC can't move the blocks without also updating all of the pointers which point to the blocks.

The real problem is that in the current architecture, there is no way to distinguish a pointer containing the address of a block, and a collection of bytes that just happens to have the same value.

If one of your data structures happened to contain an integer (or just an array of bytes) which happened to coincide with the address of a block then the current GC will conservatively keep the block and prevent it from being freed.

[mianos]

The current GC can't move the blocks without also updating all of the pointers which point to the blocks.

The real problem is that in the current architecture, there is no way to distinguish a pointer containing the address of a block, and a collection of bytes that just happens to have the same value.

If one of your data structures happened to contain an integer (or just an array of bytes) which happened to coincide with the address of a block then the current GC will conservatively keep the block and prevent it from being freed.

Obviously this is not the problem here. If doing a garbage collect means there is not a leak with millions of iterations there are no miss recognized pointers.
It also means there are not items being permanently allocated.
It also means the memory is not being fragmented by permanent user allocated variables as there are none. It is being fragmented by its own block pointers because they are never coalesced.

[dhylands]

No -not the general problem, but this is why blocks can't be moved after they've been allocated.

[mianos]

No -not the general problem, but this is why blocks can't be moved after they've been allocated.

Yep, 100% agree. Unless you have a extra level of indirection, which would mean a lot more work in C.