help with leaking memory management
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: help with leaking memory management
[deleted after brain revived]I was talking nonsense here, I'm afraid
Last edited by pythoncoder on Tue Jan 26, 2016 6:34 pm, edited 1 time in total.
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
Re: help with leaking memory management
I think that's' what I have seen with a test program which constantly allocated an frees memory in a loop. In that loop, I printed the amount of free memory told by mem_free(). That went down until no more memory was available, and then went up again, and so on. The problem seems to arise, when the memory is already scattered with small allocated blocks, and then a larger one is requested. Then, gc.collect() seems not to be able to resolve this. Running gc.collect often seems to reduce the risk.
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: help with leaking memory management
@Roberthh Yes, I've come across this a few times.
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
Re: help with leaking memory management
Currently that's exactly how the allocator works. If the allocation fails, it tries a gc and then retries the allocation.pythoncoder wrote:A further thought. When the VM attempts to allocate memory and the allocation fails, should it not trigger a collection and then retry? This would avoid the "out of memory" exception being thrown in such cases.dhylands wrote:@dhylands [...]Doing a gc collect often, in some circumstances can dramatically reduce heap fragmentation [...]
Issuing gc.collect() regularly would still be a good idea, but it strikes me that the above change would avoid needless runtime failures.
But if the heap was badly fragmented then it won't help, since after the gc you wind up with a heap which has lots of tiny holes rather that one big free space.
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: help with leaking memory management
I realised that after posting. Call it a "senior moment"dhylands wrote:... if the heap was badly fragmented then it won't help, since after the gc you wind up with a heap which has lots of tiny holes rather that one big free space.
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
Re: help with leaking memory management
In such case is there a solution to merge them or the current GC implementation can't manage the situation and no way out in a running program except restart?pythoncoder wrote:I realised that after posting. Call it a "senior moment"dhylands wrote:... if the heap was badly fragmented then it won't help, since after the gc you wind up with a heap which has lots of tiny holes rather that one big free space.
Tiny Core Linux (piCore) developer
HAM radio call: HA5DI (Béla)
HAM radio call: HA5DI (Béla)
Re: help with leaking memory management
The current GC can't move the blocks without also updating all of the pointers which point to the blocks.
The real problem is that in the current architecture, there is no way to distinguish a pointer containing the address of a block, and a collection of bytes that just happens to have the same value.
If one of your data structures happened to contain an integer (or just an array of bytes) which happened to coincide with the address of a block then the current GC will conservatively keep the block and prevent it from being freed.
The real problem is that in the current architecture, there is no way to distinguish a pointer containing the address of a block, and a collection of bytes that just happens to have the same value.
If one of your data structures happened to contain an integer (or just an array of bytes) which happened to coincide with the address of a block then the current GC will conservatively keep the block and prevent it from being freed.
Re: help with leaking memory management
Obviously this is not the problem here. If doing a garbage collect means there is not a leak with millions of iterations there are no miss recognized pointers.dhylands wrote:The current GC can't move the blocks without also updating all of the pointers which point to the blocks.
The real problem is that in the current architecture, there is no way to distinguish a pointer containing the address of a block, and a collection of bytes that just happens to have the same value.
If one of your data structures happened to contain an integer (or just an array of bytes) which happened to coincide with the address of a block then the current GC will conservatively keep the block and prevent it from being freed.
It also means there are not items being permanently allocated.
It also means the memory is not being fragmented by permanent user allocated variables as there are none. It is being fragmented by its own block pointers because they are never coalesced.
Re: help with leaking memory management
No -not the general problem, but this is why blocks can't be moved after they've been allocated.
Re: help with leaking memory management
Yep, 100% agree. Unless you have a extra level of indirection, which would mean a lot more work in C.dhylands wrote:No -not the general problem, but this is why blocks can't be moved after they've been allocated.