reducing memory
reducing memory
I am currently at/near the limit of the pyboard memory when processing data and now I've been tasked with more than doubling the volume of data to process on the pyboard. Currently I read in a file which contains around 2016 floating point values and store the data in a floating point array of size 7x288. The new data requirement is to store an array of 16x288. How much memory can I save by saving the data as integers (unsigned is ok)?
Thanks
Jim
Thanks
Jim
Re: reducing memory
I'm going to guess quite a bit.
floats require a memory allocation, which means that each float will take up a block of memory (or 16 bytes).
small integers (anything that fits in 31-bit signed number) will take no additional memory. As soon as you cross out of the small int category then it will require a memory allocation (so at least a block).
floats require a memory allocation, which means that each float will take up a block of memory (or 16 bytes).
small integers (anything that fits in 31-bit signed number) will take no additional memory. As soon as you cross out of the small int category then it will require a memory allocation (so at least a block).
Re: reducing memory
Hi Dave, not sure if you're fully right. Just gave it a try: if I allocate an array of 1000 floats, it will take a little bit more than 4000 bytes of memory. The same for a list of 1000 floats.
Re: reducing memory
Here's what I see on my pyboard with the following file: and this produces the following output: The nums array takes up 4112 (since the small ints are stored directly into the array). The big_ints take (36112 - 4112) = 32000 bytes or 32 bytes per number. Looking at the mpz object I see that there is mpz_t which contains a pointer to the allocated digits. So any non-trivial mpz_t will require 2 blocks. The float allocations require 20112 - 4112 = 16000 bytes or 16 bytes per float (1 block).
Code: Select all
import gc
def alloc_small_ints():
nums = []
for i in range(1000):
nums.append(i)
return nums
def alloc_floats():
nums = []
for i in range(1000):
nums.append(i * 1.0)
return nums
def alloc_big_ints():
nums = []
for i in range(1000):
nums.append(i << 32)
return nums
def main():
for i in range(10):
gc.collect()
before1 = gc.mem_free();
nums = alloc_small_ints()
gc.collect()
after1 = gc.mem_free();
nums = None
gc.collect()
before2 = gc.mem_free()
nums = alloc_big_ints()
gc.collect()
after2 = gc.mem_free();
nums = None
gc.collect()
before3 = gc.mem_free()
nums = alloc_floats()
gc.collect()
after3 = gc.mem_free();
nums = None
gc.collect()
print('small_ints =', before1 - after1)
print('big_ints =', before2 - after2)
print('floats =', before3 - after3)
main()
Code: Select all
>>> import mem_test
small_ints = 4112
big_ints = 36128
floats = 20112
small_ints = 4112
big_ints = 36112
floats = 20112
small_ints = 4112
big_ints = 36112
floats = 20112
small_ints = 4112
big_ints = 36112
floats = 20112
small_ints = 4112
big_ints = 36112
floats = 20112
small_ints = 4112
big_ints = 36112
floats = 20112
small_ints = 4112
big_ints = 36112
floats = 20112
small_ints = 4112
big_ints = 36112
floats = 20112
small_ints = 4112
big_ints = 36112
floats = 20112
small_ints = 4112
big_ints = 36112
floats = 20112
Re: reducing memory
Looking at this, I think that there is the potential for optimization the mpz (at least for some cases) by allowing the first few data objects to come from within the same block as the header object (when they'll fit) and fall back to the exsting mechanism when they don't.
Re: reducing memory
I tried this in an ESP8266 (my PyBoards are still in use):
or:
Code: Select all
>>> gc.collect();gc.mem_free()
23568
>>> a=[0.0 for _ in range(1000)]
>>> gc.collect();gc.mem_free()
19360
>>> len(a)
1000
>>> a[0]
0.0
>>> a[999]
0.0
>>>
Code: Select all
>>> gc.collect();gc.mem_free()
23488
>>> a = array.array("f", [0.0 for _ in range(1000)])
>>> gc.collect();gc.mem_free()
19344
>>> type(a)
<class 'array'>
>>> len(a)
1000
Re: reducing memory
I think that on the esp8266 that floats are "boxed" i.e. stored similarly to small ints, but the floats are only 30-bit floats.
The pyboard uses OBJ_REPR_A, and the esp8266 uses OBJ_REPR_C. These are described here:
https://github.com/micropython/micropyt ... .h#L52-L90
The pyboard uses OBJ_REPR_A, and the esp8266 uses OBJ_REPR_C. These are described here:
https://github.com/micropython/micropyt ... .h#L52-L90
Re: reducing memory
Interesting: I just repeated my test on other board I have, the esp32 and the teensy 3.6, and Pyboard (I carried the one I have here, which is inside a 20kg steel box, to my desk). And they all show the similar behaviour for allocating a 1000 element float array with the array lib:
- esp82866 4208 bytes
- PyBoard 4208 bytes
- esp32 4224 bytes
- teensy 3.6: 4160 bytes
So the floats seem to be quite packed. And that does not change substantially if I modify the array content.
.
- esp82866 4208 bytes
- PyBoard 4208 bytes
- esp32 4224 bytes
- teensy 3.6: 4160 bytes
So the floats seem to be quite packed. And that does not change substantially if I modify the array content.
.
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: reducing memory
My ARM Thumb Assembler applications on the Pyboard use the fact that arrays of floats created with the array module store values in contiguous word locations. So I'd expect an array of N floats on the Pyboard to use roughly 4*N bytes.
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
Re: reducing memory
Ahh - So I think it comes down to the difference between an array and a list. I was using a list, and not an array, but due to the [] and my C background, I commonly mistake a list for an array. Yeah an array is 'packed' so each entry would only take up the 32-bits of the float.