reducing memory

[JimTal001]

I am currently at/near the limit of the pyboard memory when processing data and now I've been tasked with more than doubling the volume of data to process on the pyboard. Currently I read in a file which contains around 2016 floating point values and store the data in a floating point array of size 7x288. The new data requirement is to store an array of 16x288. How much memory can I save by saving the data as integers (unsigned is ok)?

Thanks
Jim

[dhylands]

I'm going to guess quite a bit.

floats require a memory allocation, which means that each float will take up a block of memory (or 16 bytes).

small integers (anything that fits in 31-bit signed number) will take no additional memory. As soon as you cross out of the small int category then it will require a memory allocation (so at least a block).

[Roberthh]

Hi Dave, not sure if you're fully right. Just gave it a try: if I allocate an array of 1000 floats, it will take a little bit more than 4000 bytes of memory. The same for a list of 1000 floats.

[dhylands]

Here's what I see on my pyboard with the following file:

Code: Select all

import gc

def alloc_small_ints():
    nums = []
    for i in range(1000):
        nums.append(i)
    return nums

def alloc_floats():
    nums = []
    for i in range(1000):
        nums.append(i * 1.0)
    return nums

def alloc_big_ints():
    nums = []
    for i in range(1000):
        nums.append(i << 32)
    return nums

def main():

    for i in range(10):
        gc.collect()
        before1 = gc.mem_free();
        nums = alloc_small_ints()
        gc.collect()
        after1 = gc.mem_free();

        nums = None
        gc.collect()

        before2 = gc.mem_free()
        nums = alloc_big_ints()
        gc.collect()
        after2 = gc.mem_free();

        nums = None
        gc.collect()

        before3 = gc.mem_free()
        nums = alloc_floats()
        gc.collect()
        after3 = gc.mem_free();

        nums = None
        gc.collect()

        print('small_ints =', before1 - after1)
        print('big_ints =', before2 - after2)
        print('floats =', before3 - after3)

main()

and this produces the following output:

Code: Select all

>>> import mem_test
small_ints = 4112
big_ints = 36128
floats = 20112
small_ints = 4112
big_ints = 36112
floats = 20112
small_ints = 4112
big_ints = 36112
floats = 20112
small_ints = 4112
big_ints = 36112
floats = 20112
small_ints = 4112
big_ints = 36112
floats = 20112
small_ints = 4112
big_ints = 36112
floats = 20112
small_ints = 4112
big_ints = 36112
floats = 20112
small_ints = 4112
big_ints = 36112
floats = 20112
small_ints = 4112
big_ints = 36112
floats = 20112
small_ints = 4112
big_ints = 36112
floats = 20112

The nums array takes up 4112 (since the small ints are stored directly into the array). The big_ints take (36112 - 4112) = 32000 bytes or 32 bytes per number. Looking at the mpz object I see that there is mpz_t which contains a pointer to the allocated digits. So any non-trivial mpz_t will require 2 blocks. The float allocations require 20112 - 4112 = 16000 bytes or 16 bytes per float (1 block).

[dhylands]

Looking at this, I think that there is the potential for optimization the mpz (at least for some cases) by allowing the first few data objects to come from within the same block as the header object (when they'll fit) and fall back to the exsting mechanism when they don't.

[Roberthh]

I tried this in an ESP8266 (my PyBoards are still in use):

Code: Select all

>>> gc.collect();gc.mem_free()
23568
>>> a=[0.0 for _ in range(1000)]
>>> gc.collect();gc.mem_free()
19360
>>> len(a)
1000
>>> a[0]
0.0
>>> a[999]
0.0
>>> 

or:

Code: Select all

>>> gc.collect();gc.mem_free()
23488
>>> a = array.array("f", [0.0 for _ in range(1000)])
>>> gc.collect();gc.mem_free()
19344
>>> type(a)
<class 'array'>
>>> len(a)
1000

[dhylands]

I think that on the esp8266 that floats are "boxed" i.e. stored similarly to small ints, but the floats are only 30-bit floats.

The pyboard uses OBJ_REPR_A, and the esp8266 uses OBJ_REPR_C. These are described here:
https://github.com/micropython/micropyt ... .h#L52-L90

[Roberthh]

Interesting: I just repeated my test on other board I have, the esp32 and the teensy 3.6, and Pyboard (I carried the one I have here, which is inside a 20kg steel box, to my desk). And they all show the similar behaviour for allocating a 1000 element float array with the array lib:
- esp82866 4208 bytes
- PyBoard 4208 bytes
- esp32 4224 bytes
- teensy 3.6: 4160 bytes
So the floats seem to be quite packed. And that does not change substantially if I modify the array content.
.

[pythoncoder]

My ARM Thumb Assembler applications on the Pyboard use the fact that arrays of floats created with the array module store values in contiguous word locations. So I'd expect an array of N floats on the Pyboard to use roughly 4*N bytes.

[dhylands]

Ahh - So I think it comes down to the difference between an array and a list. I was using a list, and not an array, but due to the [] and my C background, I commonly mistake a list for an array. Yeah an array is 'packed' so each entry would only take up the 32-bits of the float.