stdin memory leak

[aviator_11]

Are there any caveats or tricks to using sys.stdin.read() and readline() in a way that preserves memory? I am seeing a memory leak ( I believe) when reading certain ASCII characters using sys.stdin.read(). I am running Micropython v1.18 on a Pico with RP2040.

Code: Select all

import sys
import gc

while True:
        sys.stdin.read(1)
        print (gc.mem_free())

Sending any combination of characters (b,e,f,n,r,s,t,v,x,y) results in a printout of free memory that does not change. Sending a character not on that list results in a decrease of free memory. If I use readline() instead, I see a decrease in available memory regardless of characters sent.

I have also tested this outside of an IDE (loading the pico with this code, disconnecting from Thonny, and using terminal) and see the same result, so I don't believe it is Thonny related.

[Roberthh]

What you see is not a memory leak but the normal behavior of dynamic memory. Memory for objects will be allocated, until the pool is exhausted and garbage collection runs. You can trigger that by calling gc.collect(). If you add that to your loop, you will notice that the amount of free memory stays stable within a certain range. Interesting that memory is not affected for some characters. But maybe these are interned strings.
There is sys.stdin.readinto(), which reads into a preallocated bufffer. For that, no new memory has to be allocated. Sample:

Code: Select all

import sys
import gc
b=bytearray(1)

while True:
        sys.stdin.readinto(b)
        print (gc.mem_free())

[aviator_11]

Thank you for the helpful explanation! This is exactly what I needed.

This code is actually just a snippet from a larger application, and I did try gc.collect(), but it seemed to cause instability in the second thread (using the second core of the RP4040 to ping a sensor and update a global). I haven't had much luck understanding if gc.collect() really does cause an issue when multithreading, but hopefully if I am not leaking memory, I won't have to deal with that anymore

[aviator_11]

Am I doing something similar when attempting to decode/concatenate the byte array into a string? I see a similar "memory loss" issue when attempting to convert to string or concatenate the bytes to a string for comparison later in the program.

Code: Select all

import sys
import gc

b = bytearray(1)

string = ""

while True:
        sys.stdin.readinto(b)
        string = b.decode()
        print (gc.mem_free())
 

[Roberthh]

There was a code change with respect to GC and Threads. Try a nightly build if the strange behaviour is still present.
And yes, the decode and assignment creates a new object with uses dynamic memory.

[aviator_11]

Thanks again - this makes sense. I'll try some nightly builds.

I am converting this code from circuitpython and did not notice memory allocation issues with essentially the same code. Dynamic memory allocation must be handled very differently there.

Is the safe path forward is to avoid most string conversion methods like decode() or periodically force gc.collect() in micropython if I am periodically performing string operations (str()/decode()/bytes()/format())? My plan is to rewrite the entire application using bytearrays instead.

[Roberthh]

CircuitPython uses the core components of MicroPython. Object and memory handling should be generally the same. The parameters may be differently configured.

[scruss]

Has CircuitPython gained the ability to use more than one core? Until recently, it couldn't do that

[aviator_11]

Unfortunately, no multicore support yet on circuitpython, hence the switch. My code in circuitpython was a single routine that scanned the port, grabbed sensor values, and reported back, obviously loop time greatly held back by how long the onewire sensor was taking to update.

For anyone curious: Here is the snippet from the original circuitpython code that runs with no indication of memory allocation issues. One (relevant?) difference is that I have the secondary USB_CDC port enabled through boot.py rather than attempting everything through the REPL port, so I have the full uart functionality available including a buffer reset. I'm assuming that I've had memory allocation issues in circuitpython too, as I'm decoding and encoding strings repeatedly to/from bytearrays, but circuitpython must be handling memory differently in a way that I never had a pico shut down on me.

Code: Select all

uart = usb_cdc.data

while True:
    #reset buffer
    message = 0
    #check number of bytes at port
    count = uart.in_waiting
    if count > 0:
        message = uart.read(count)
        uart.reset_input_buffer()
        message = message.decode()
        print(message)
     #----if there is a command, respond to it-----------
    if message is not 0:
        response = getresponse(str(message))
        print(response)
        if response is not None:
            uart.write(bytearray(str(response).encode()))
        else:
            uart.write(bytearray("Invalid".encode()))

    #----loop delay---------------------------------
    time.sleep(0.25) # 0.25 seconds
        

[Roberthh]

You must make a difference between the way Python (and Micropython) handles memory, and a hopefully solved problem with MicroPython and threads. All of Python, CircuitPyhton and MicroPython handle memory in a dynamic way. Memory for objects is allocated from a heap, reducing the amount of allocateable memory on the heap. If objects are not use any more, the memory used by them is a subject to be returned to the free space, which is done by garbage collection when needed. Using the short script from your first post shows the same pattern of decreasing free memory, when used with CircuitPyhon.