Clearing a bytearray like a list?

[V!nce]

Hello,
I have a program that needs that writes measures into a bugger that i then flushed into a file every half second. Because I'm using multiple uasyncio coroutines, I cannot simply create a fixed size array/bytearray into which I put my data throughout the half second (I tried). There are two main reasons to that: there is a slight frequency variability from cycle to cycle (so a different number of values to save) and I just want to write the bytearray so I don't want to parse it to remove the empty tail values, also managing the indexing isn't easy...

What I currently do is create an empty global bytearray, then extend it throughout the cycle and finally empty it once it has been written into the file. In this thread viewtopic.php?t=2480, I found an efficient way of clearing the bytearray by doing :

Code: Select all

my_bytearray[::] = b''

I thought this method was truncating the bytearray, keeping it at the same place in memory and overwriting old values with new ones. However, by printing the available memory over time, I saw that it was actually decreasing until the garbage collector is automatically called and frees up new memory. When using a list, memory is freed everytime I call the .clear() method which is the behavior I would like to implement. However, the method bytearray.clear() only exists in CPython and not in micropython. Do you have any idea on how to achieve this? By the way, I cannot use a list for this application because the byterray is so much lighter I can actually record multiple seconds without having to clear it and running out of memory...

My code actually works well but I don't find it elegant and the idea of filling up the RAM with discarded data feels very wrong.

[pythoncoder]

Why clear it at all? I would allocate a bytearray at the start of the script. The BA would be large enough to accommodate the worst-case amount of data. Also have a global int which stores the current index of the last populated byte and a memoryview into the BA.

As you fill the BA increment the index. When you come to write it out, take a non-allocating slice:

Code: Select all

ba = bytearray(5000)
idx = 0
mv = memoryview(ba)

def write_data():
    global idx
    with open(my_file, "a") as f:
        f.write(mv[:idx])  # Slice does not allocate
    idx = 0

[martincho]

Hello,
I have a program that needs that writes measures into a bugger that i then flushed into a file every half second. Because I'm using multiple uasyncio coroutines, I cannot simply create a fixed size array/bytearray into which I put my data throughout the half second (I tried). There are two main reasons to that: there is a slight frequency variability from cycle to cycle (so a different number of values to save) and I just want to write the bytearray so I don't want to parse it to remove the empty tail values, also managing the indexing isn't easy...

Well, as Peter suggests, one-time allocation and a pointer (index) into the array is the efficient way to do this. If your application needs to consume or process data from the front you'll need head and tail pointers. The next level up would be a ring buffer.

[V!nce]

Thank you both for your replies.

I already tried this implementation but at the time I made a mistake and discarded the idea. Today I tried to implement it again. It works BUT I still have a memory leak... As an example, here, each half second I lose around Ko of RAM. I know printing values will consume a bit of RAM everytime but 2Ko seems excessive. It is way more in the full version of the script which seems to fill up the RAM in 2 or 3 seconds...

Code: Select all

buf = bytearray(1000)
print("Buffer length", len(buf))
mbuf = memoryview(buf)
idx = 0

# I use asyncio coroutines to get values from sensors. They are all based on this simple while loop
async def eda():
    global mbuf
    global idx
    s = ADC(Pin(32))
    while True:
        mbuf[idx:idx+3] = pack("<Bh", 2, s.read_u16())
        idx+=3
        await uasyncio.sleep_ms(20)
        
# In parallel, I have a second thread running the the writing loop. I found it has less impact on performance than running another coroutine.
def thread_write(filename):
    global running
    global mbuf
    global idx
    with open(filename, "wb") as f: 
        while running:
            sleep_ms(500)
            f.write(mbuf[:idx])
            idx = 0
            print("Used memory: ", gc.mem_alloc())
    return

Here is the output when running to eda loops:

Code: Select all

Used memory:  18304
Used memory:  20256
Used memory:  22208
Used memory:  24160
Used memory:  26112
Used memory:  28064
Used memory:  30016
Used memory:  31968
Used memory:  34016
Used memory:  35968
Used memory:  37920
Used memory:  39872
Used memory:  41872
Used memory:  43824
Used memory:  45776
Used memory:  47728
Used memory:  49680

Any idea? I am quite puzzled by this...

[V!nce]

Ok, I found that the leak seems to come from the packing step as well as from the reading of some sensors like the MPU6050 accelerometer...

[pythoncoder]

Two points. It's worth issuing gc.collect() before taking any measurement of free or allocated RAM.

Secondly, your use of threading looks hazardous. I would do the writing as a coroutine. If you must use threads you need a lock to ensure that you don't write out data which is half way through being updated.

Thirdly, do you really mean to overwrite the file rather than append to it?

[V!nce]

Isn't calling the GC before showing the available memory cheating? It doesn't remedy the fact that the heap is filling up with no obvious reason to me and introduces a blocking step.

From my benchmarks, using a thread instead of another coroutine is faster and therefore yields better results. It doesn't seem to be a problem when using the buffer extending method nor the pre-allocated buffer. I can retry with a coroutine but it doesn't change much.

Surprisingly, using the fixed size buffer is not faster than extending and clearing it as I described before. But also, using the pre-allocated bytearray doesn't help at all with the problem of the heap filling up in 2 secs. I never get an error because the GC is always called automatically when the heap is full but I would like to avoid that behaviour if possible.

Finally, I check the presence of the file and delete it at the begining and at the end of the script because I'm still prototyping. Writing or appending doesn't make a difference.

[pythoncoder]

Writing non-allocating applications is quite difficult. You need to acquire a lot of knowledge about when MP allocates. The way I've done this is to run code fragments in an ISR on a platform that supports hard IRQ's.

But in general there is no need to worry unduly about allocation - just let the GC do its job. Calling gc.collect() periodically can have benefits. If the GC doesn't have much to do, it blocks for a much shorter time. Often application performance benefits if you collect at times of your own choosing. And regular collection reduces RAM fragmentation.

The point of issuing gc.collect() prior to checking RAM usage is to check for actual memory leaks where RAM is allocated yet isn't collected.

[jimmo]

One thing to watch out for with memoryview is that slicing a memoryview will create a new memoryview, which will be a small allocation.

In Peter's example code

Code: Select all

f.write(mv[:idx])  # Slice does not allocate

that comment is unfortunately not true, for two reasons:
* The actual slice object needs to be allocated.
* It creates a new memoryview object with the new start/length.

The problem in general is that something might take a reference to it, so although this seems like an obvious optimisation (i.e. it should stack-allocate rather than heap-allocate a temporary memoryview) but in general the analysis isn't simple.

We have considered a micropython-extension to memoryview to allow either modifying the memoryview bounds, or a way to do assignment without allocation (e.g. `mv.assign(start, end, data)`), but I think we'd prefer CPython implemented this first rather than doing more micropython extensions.

I'm not sure if that explains your specific problem with the heap growing though.

The only way to write allocation-free code is to avoid slicing (by always pre-creating the memoryviews for the desired offsets), or to only read/write individual bytes (indexing doesn't allocate, only slicing), or exclusively use readinto to write into the pre-allocated memoryviews.

[martincho]

Thanks for the clarification. I pretty much assumed that was the case. I took Peter's comment to mean exactly this.