ESP8266 sockets - memory being all used up. FIXED

Discussion and questions about boards that can run MicroPython but don't have a dedicated forum.
Target audience: Everyone interested in running MicroPython on other hardware.
Post Reply
mianos
Posts: 84
Joined: Sat Aug 22, 2015 6:42 am

ESP8266 sockets - memory being all used up. FIXED

Post by mianos » Sun Sep 06, 2015 12:39 am

If I call this in a loop:
def doit(arg):
soc = esp.socket()
soc.__del__()
gc.collect()


And call gc.collect() in my outer loop, the memory is slowly used up. Is this expected?
Should I be trying to add code to the del so it manually marks some of the allocated elements in the socket free?
Are there any good ways to work out the object types in the managed memory?
Last edited by mianos on Sun Oct 04, 2015 11:43 pm, edited 1 time in total.

mianos
Posts: 84
Joined: Sat Aug 22, 2015 6:42 am

Re: ESP8266 sockets - memory being all used up.

Post by mianos » Mon Sep 07, 2015 12:09 pm

Me again. I am trying to track down why, when I use the external timer from my code https://github.com/micropython/micropython/pull/1453
It seems the garbage collector never actually has anything freeable.
For example, I have enabled the gc debugging. Once the string is printed I would assume it would be collectable.
I may be on the wrong track here. Once I come out of the uart loop and back to the system scheduled loop
maybe there is some other issue with the stack? It seems the garbage collector should just run through the object list that has been built on the pre-allocated heap.
The garbage collect does not seem to have any knowledge of the stack in the existing thread.

(ps. I bought a micropython board to contribute but I really want to get the ESP board going well).


gc_alloc(16 bytes -> 1 blocks)
gc_alloc(3fff4310)
gc_alloc(18 bytes -> 2 blocks)
gc_alloc(3fff4450)
b'HTTP/1.0 200 OK\r\n'
gc_alloc(16 bytes -> 1 blocks)
gc_alloc(3fff4350)
gc_alloc(146 bytes -> 10 blocks)
gc_alloc(3fff47c0)
b'Date: Mon, 07 Sep 2015 11:42:17 GMT\r\nServer: WSGIServer/0.1 Python/2.7.3\r\nContent-type: text/html\r\nContent-Length: 24\r\n\r\n<html>Hello world</html>'
disconnected
gc_alloc(40 bytes -> 3 blocks)
gc_alloc(3fff4640)
gc_alloc(28 bytes -> 2 blocks)
gc_alloc(3fff4600)
gc_alloc(32 bytes -> 2 blocks)
gc_alloc(3fff4670)
gc_alloc(16 bytes -> 1 blocks)
gc_alloc(3fff4370)
gc_sweep(3fff4310)
gc_sweep(3fff4350)
gc_sweep(3fff4370)
gc_sweep(3fff4420)
gc_sweep(3fff5600)
gc_sweep(3fff5610)
gc_sweep(3fff5620)
gc_sweep(3fff5630)
gc_sweep(3fff5640)
gc_sweep(3fff5650)
gc_sweep(3fff5660)
gc_sweep(3fff5670)
gc_sweep(3fff5680)
gc_sweep(3fff5690)
GC: total: 16128, used: 2224, free: 13904
No. of 1-blocks: 27, 2-blocks: 13, max blk sz: 11
connected

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: ESP8266 sockets - memory being all used up.

Post by pfalcon » Wed Sep 09, 2015 10:56 am

Once the string is printed I would assume it would be collectable.
It would be collectable, yes. It wouldn't be collected right away, because uPy uses garbage collection, not reference counting (which means deallocating old objects when there's not enough memory, or on exlicit gc.collect()). And it uses conservative GC, which means any particular object may be not freed (as eagerly as you expect, or at all).
Once I come out of the uart loop and back to the system scheduled loop
maybe there is some other issue with the stack?
Maybe.
The garbage collect does not seem to have any knowledge of the stack in the existing thread.
The effect of this would be opposite of what you describe - some yet-live objects would be collected (because GC misses refs to them on the stack).

Code: Select all

gc_alloc(16 bytes -> 1 blocks)
gc_alloc(3fff4310)
gc_sweep(3fff4310)
So, what you can see in this excerpt is object being allocated, and later swept (final phase of GC).
(ps. I bought a micropython board to contribute but I really want to get the ESP board going well).
Everyone would, it's just with all the effort already spent on the port, even more is required to get something really working out of it, and that will require outgrowing vendor framework, which is actually more disabling, than enabling, to the capabilities of the chip. That means many-many days, or if you have dayjob, many-many nights. Welcome to the club ;-).

In my list, next thing which should be done to esp8266 port is making it run testsuite (using pyboard.py tool, like other boards). That will allow to really assess what works in the port, and what not.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

User avatar
platforma
Posts: 258
Joined: Thu May 28, 2015 5:08 pm
Location: Japan

Re: ESP8266 sockets - memory being all used up.

Post by platforma » Thu Sep 10, 2015 9:02 pm

mianos wrote: (ps. I bought a micropython board to contribute but I really want to get the ESP board going well).
Yes, same story for me, I'd like to find some more time to pick up an issue or functionality that needs work on ESP port but it's a little difficult to see what's being worked on at the moment and what actually needs doing. I don't know if that deserves a separate board on the forum or perhaps it would be good to set up irc or similar.

mianos
Posts: 84
Joined: Sat Aug 22, 2015 6:42 am

Re: ESP8266 sockets - memory being all used up. FIXED

Post by mianos » Sun Oct 04, 2015 11:53 pm

With this:
https://github.com/micropython/micropython/issues/1479
and this:
https://github.com/micropython/micropython/issues/1486

My branch works perfectly now. I have just run 10 GETs a second for 3 days and it has been faultless.

You need to call the garbage collector if you are using the ESP internal scheduler. I am doing at the end of my os_timer callback.
I'll not be bothering with trying to get this pushed as the changes required to get the push approved will leave it not usable for me.

I'll try and keep my fork up to date with the master.
https://github.com/mianos/micropython

With my esp.os_timer and these changes this board is becoming really nice to use.

Post Reply