Memory problem : iterate a file line by line

All ESP32 boards running MicroPython.
Target audience: MicroPython users with an ESP32 board.
Post Reply
Alexdd50
Posts: 6
Joined: Thu Jan 30, 2020 1:23 pm

Memory problem : iterate a file line by line

Post by Alexdd50 » Sat Feb 15, 2020 12:26 pm

Hello,

I have a memory problem with my ESP32, I download a file that is around 1000 lines and want to iterate it line by line.
My code is :

Code: Select all

for line in st.splitlines():
	action ...
And the error :

Code: Select all

MemoryError: memory allocation failed, allocating %u bytes
I tried gc.collect() before the loop, but it didn't solved the problem
Is there a way to iterate the file without using a lot of memory ?

User avatar
jimmo
Posts: 1688
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Memory problem : iterate a file line by line

Post by jimmo » Sat Feb 15, 2020 12:58 pm

Alexdd50 wrote:
Sat Feb 15, 2020 12:26 pm
My code is :

Code: Select all

for line in st.splitlines():
	action ...
And the error :

Code: Select all

MemoryError: memory allocation failed, allocating %u bytes
What this code has to do is:
- Read the whole file into a string
- Make a new 1000-element list, and then create a short string for each line.

So the total memory used is 1000 * 4 + 2 * 1000 * N (where N is the average line length).

Instead what you want to do is use the built-in iterator on the file object:

Code: Select all

f = open('file.txt', 'r')
for line in f:
  ... do something with line
This will only use enough memory as it takes to store a single line of the file.

Alexdd50
Posts: 6
Joined: Thu Jan 30, 2020 1:23 pm

Re: Memory problem : iterate a file line by line

Post by Alexdd50 » Sun Feb 16, 2020 8:59 am

That worked for this problem ! Thanks

But another appears related to the memory too.
My code suddenly stops in the middle of the process and I have this error with always the same number of bytes :

Code: Select all

MemoryError: memory allocation failed, allocating 1336 bytes
And after I can't do nothing in the console, it always display me this error.
The function that causes this problem is just an assignment of 2 variables that are not big ...
How can I solve this ?

User avatar
pythoncoder
Posts: 4249
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Memory problem : iterate a file line by line

Post by pythoncoder » Sun Feb 16, 2020 9:03 am

I doubt anyone can answer that without seeing your code. I strongly recommend you read the docs to learn the techniques of coding to reduce RAM usage.
Peter Hinch

Post Reply