Memory problem : iterate a file line by line

[Alexdd50]

Hello,

I have a memory problem with my ESP32, I download a file that is around 1000 lines and want to iterate it line by line.
My code is :

Code: Select all

for line in st.splitlines():
	action ...

And the error :

Code: Select all

MemoryError: memory allocation failed, allocating %u bytes

I tried gc.collect() before the loop, but it didn't solved the problem
Is there a way to iterate the file without using a lot of memory ?

[jimmo]

My code is :

Code: Select all

for line in st.splitlines():
	action ...

And the error :

Code: Select all

MemoryError: memory allocation failed, allocating %u bytes

What this code has to do is:
- Read the whole file into a string
- Make a new 1000-element list, and then create a short string for each line.

So the total memory used is 1000 * 4 + 2 * 1000 * N (where N is the average line length).

Instead what you want to do is use the built-in iterator on the file object:

Code: Select all

f = open('file.txt', 'r')
for line in f:
  ... do something with line

This will only use enough memory as it takes to store a single line of the file.

[Alexdd50]

That worked for this problem ! Thanks

But another appears related to the memory too.
My code suddenly stops in the middle of the process and I have this error with always the same number of bytes :

Code: Select all

MemoryError: memory allocation failed, allocating 1336 bytes

And after I can't do nothing in the console, it always display me this error.
The function that causes this problem is just an assignment of 2 variables that are not big ...
How can I solve this ?

[pythoncoder]

I doubt anyone can answer that without seeing your code. I strongly recommend you read the docs to learn the techniques of coding to reduce RAM usage.