Memmory Allocation Error although plenty of RAM left...

[Johnny010]

Hello. I am new to Python (I am a Java/Android/SpringBoot dev by day).
I am fetching strings from a file (lines) using the `with open __ as file:` command.
I am using the garbage collector's `mem_free()` method to allocate 20% of the free memmory as a "buffer" for the data.
The code is meant to fetch lines untill the length of the data is >= to the "20% of free mem buffer" size.

I keep getting a MemmoryException although the GC seems to be working and also there is plenty of RAM left. Any ideas?

Thanks for any help! The GC is working...i can see where mem_free gets to around 644bytes and then it suddenly frees it back up to 64000 ish.

Code: Select all

 def getAllFromIndex(self, model, first):
        print(self._instance)
        print("GET FROM " + first)
        print(model + " file size = " + str(uos.stat("/database/" + model)[6]) + " bytes")
        print("Free Heap Size = " + str(gc.mem_free()))
        bufferRamSize = gc.mem_free() * 0.2
        print("Fetching to 0.2x Free HEAP = " + str(bufferRamSize))

        try:
            data = ""
            with open("/database/" + model, "r") as file:
                print("Mem free = " + str(gc.mem_free()))
                for line in file:
                    if len(data) >= bufferRamSize:
                        print("Break!")
                        break
                    print(str(len(data)) + " / "+str(gc.mem_free()))
                    data = data + line
                data = "[" + data + "]"
                print("GOT DATA!")
                gc.collect()
                print(data)
                file.close()

                return data
        except Exception as e:
            print("Whoopsies! " + str(e))

================== Log Serial Output: ================

Code: Select all

<GETDaoService object at 3ffec960>
GET FROM 10
user file size = 58774 bytes
Free Heap Size = 68560
Fetching to 0.2x Free HEAP = 13696.0
Mem free = 68256
0 / 68032
58 / 67824
116 / 67472
174 / 67072
232 / 66608
290 / 66080
348 / 65504

..........
........
.....
..

13340 / 43072
13398 / 29440
13456 / 42832
13514 / 42720
13572 / 28912
13630 / 42512
13688 / 42384
Break!
Whoopsies! memory allocation failed, allocating 13749 bytes

[Johnny010]

Ah ok nevermind...
Think I have fixes it. Guessing it was during:
data = "[" + data + "]"
not being able to find a RAM allocation with enough room that isn't fragmented.
I called gc.collect() again before modifying the data and it "seems" to work.

PS: My BBCode will not turn on...so [code] isnt working...

============MODIFIED CODE==========

def getAllFromIndex(self, model, first):
gc.collect()
print(self._instance)
print("GET FROM " + first)
print(model + " file size = " + str(uos.stat("/database/" + model)[6]) + " bytes")
print("Free Heap Size = " + str(gc.mem_free()))
bufferRamSize = gc.mem_free() * 0.2
print("Fetching to 0.2x Free HEAP = " + str(bufferRamSize))

# Get total File Size (Lines):
totalLines = 0

with open("/database/" + model, "r") as file:
for line in file:
totalLines = totalLines + 1

gc.collect()
try:
data = ""
with open("/database/" + model, "r") as file:
print("Mem free = " + str(gc.mem_free()))
for line in file:
if len(data) >= bufferRamSize:
print("Break!")
break
data = data + line
gc.collect()
print(str(len(data)) + "bytes / " + str(gc.mem_free()) + "bytes")
data = "[" + data + "]"
gc.collect()
file.close()

return data
except Exception as e:
print("Whoopsies! " + str(e))

[jickster]

Unfortunately the current GC does not compact so even if you set the GC to perform collection every single time an allocation/deallocation is performed, you can still run out of memory.


Sent from my iPhone using Tapatalk Pro

[Johnny010]

Unfortunately the current GC does not compact so even if you set the GC to perform collection every single time an allocation/deallocation is performed, you can still run out of memory.

Yeah, noticed this now with more testing. The results were "sporadic". Sometimes working and sometimes not.
I am guessing I should really have a private buffer initialised to store the data?

How do I go about say putting aside a block of memory (I'd do this at the start of main.py when I am initialsing singletons such as the class I am having issues with)?

[pagano.paganino]

try with gc.threshold

Code: Select all

import gc
gc.threshold((gc.mem_alloc() + gc.mem_free()) // 10)

try to calibrate the right size for you instead of

Code: Select all

(gc.mem_alloc() + gc.mem_free()) // 10)

I suggest you try with a very low value (4096) and increase it.

[pythoncoder]

@Johnny010 There are optimisations you can make. I suggest you read this doc.

[Johnny010]

@Johnny010 There are optimisations you can make. I suggest you read this doc.

Thank you.

I was trying to build a webserver framework for the ESP32 wit a DAO that does a "GET ALL / SELECT * " from a file containing models on separate lines. Now seeing that memory fragmentation is an issue, going to stick with a smaller paged like request...where it fetches in batches of 10 (user configurable). I can catch the exception and send back a simple "413 - Server could not handle this request/Too large"....but I guess as the memory gets more and more fragmented, then even smaller requests will fail. Note I was thinking of maybe having a buffer somehow.
Are variables only tagged for garbage collection once set to None?

Could I for example have a buffer bytearray in my singleton:

Code: Select all

self.buffer = bytearray(10000) 

Then when I readlines from the file...I add the string to the buffer?

With the buffer as a "private class field" (I don't know the python terminology....I am Java guy) in a singleton, am I right in thinking the garbage collector *should* leave it alone?

[kevinkk525]

Garbage collection will never touch a referenced object. Therefore your buffer will stay there.

But if you use readline() for your file and send that line and not reference it after that, it will be garbage collected and should not occupy and RAM or contribute to RAM fragmentation.

[jickster]

Unfortunately the current GC does not compact so even if you set the GC to perform collection every single time an allocation/deallocation is performed, you can still run out of memory.

Yeah, noticed this now with more testing. The results were "sporadic". Sometimes working and sometimes not.
I am guessing I should really have a private buffer initialised to store the data?

How do I go about say putting aside a block of memory (I'd do this at the start of main.py when I am initialsing singletons such as the class I am having issues with)?

What do you want to do with the block of memory?

Sounds like you want to create another heap from which you can allocate later?

Not possible now.

But you can achieve the same thing indirectly:
(1) allocate a chunk of heap as a byte array
(2) then free it when you want to allocate your actual object
(3) allocate actual object

(1) you have to keep a reference around to allocates byte array meaning you have to have to assign it to a normal python variable and make sure it doesn’t go out of scope.

(2) to free, call del on the byteareay from (1) then gc.collect() but make sure you haven’t assigned the bytearray to another variable because then there’ll be 2 or more references and you’ll have to del all those python variables.

(3) by now you’ve freed a contiguous chunk so allocating should work.


Sent from my iPhone using Tapatalk Pro

[jickster]

Implementing compacting gc would make this process unnecessary and many people’s lives easier.


Sent from my iPhone using Tapatalk Pro