Page 1 of 1
Memory problem : iterate a file line by line
Posted: Sat Feb 15, 2020 12:26 pm
by Alexdd50
Hello,
I have a memory problem with my ESP32, I download a file that is around 1000 lines and want to iterate it line by line.
My code is :
Code: Select all
for line in st.splitlines():
action ...
And the error :
Code: Select all
MemoryError: memory allocation failed, allocating %u bytes
I tried gc.collect() before the loop, but it didn't solved the problem
Is there a way to iterate the file without using a lot of memory ?
Re: Memory problem : iterate a file line by line
Posted: Sat Feb 15, 2020 12:58 pm
by jimmo
Alexdd50 wrote: ↑Sat Feb 15, 2020 12:26 pm
My code is :
Code: Select all
for line in st.splitlines():
action ...
And the error :
Code: Select all
MemoryError: memory allocation failed, allocating %u bytes
What this code has to do is:
- Read the whole file into a string
- Make a new 1000-element list, and then create a short string for each line.
So the total memory used is 1000 * 4 + 2 * 1000 * N (where N is the average line length).
Instead what you want to do is use the built-in iterator on the file object:
Code: Select all
f = open('file.txt', 'r')
for line in f:
... do something with line
This will only use enough memory as it takes to store a single line of the file.
Re: Memory problem : iterate a file line by line
Posted: Sun Feb 16, 2020 8:59 am
by Alexdd50
That worked for this problem ! Thanks
But another appears related to the memory too.
My code suddenly stops in the middle of the process and I have this error with always the same number of bytes :
Code: Select all
MemoryError: memory allocation failed, allocating 1336 bytes
And after I can't do nothing in the console, it always display me this error.
The function that causes this problem is just an assignment of 2 variables that are not big ...
How can I solve this ?
Re: Memory problem : iterate a file line by line
Posted: Sun Feb 16, 2020 9:03 am
by pythoncoder
I doubt anyone can answer that without seeing your code. I strongly recommend you read
the docs to learn the techniques of coding to reduce RAM usage.