help with leaking memory management

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
mianos
Posts: 84
Joined: Sat Aug 22, 2015 6:42 am

help with leaking memory management

Post by mianos » Sat Jan 23, 2016 9:38 am

I am at a halt with my ESP8266 version.

I can't get a long running program that uses lists to keep running as it runs out of memory.

I have found if I have a struct that contains a list

Code: Select all

typedef struct _esp_ws_obj_t {
    mp_obj_base_t base;
     mp_obj_t header_key;
    mp_obj_t header_val;
    mp_obj_t header_kp;
    mp_obj_t headers;
 
I have a mini state based web server that receives multiple requests and, hopefully dispatches the request type, headers as a list and body to a dispatcher.
In my code I was planning to add the header keys and values to a ist.
(The keys and values themselves should be mostly the same so interning them should use next to no memory),
I found if I create a list, as follows (I have reduced the running code and it fails even without any items added to the list).

Code: Select all

...
                    pesp->header_val = mp_obj_new_str(header_val, strlen(header_val), true);
                    pesp->header_kp = mp_obj_new_list(0, NULL);
and at the end of each loop reset the value as follows:

Code: Select all

    ctx->header_val = mp_const_none;
    ctx->header_kp = mp_const_none;
As far as I can tell, this should orphan the mp_obj_new_list item and the collector should find it unreferenced and free it up.

If I do a few hundred requests and stop, a pub.info(1)
shows this:

Code: Select all

qstr:
  n_pool=1
  n_qstr=9
  n_str_data_bytes=123
  n_total_bytes=219
GC:
  19520 total
  9424 : 10096
  1=534 2=5 m=13
GC memory layout; from 3fff4d70:
f4d70: hthhhBhtBhhtttLhLhtthttttttthtttttttttttthLhLhLhLhthLhLhLhLhLhLh
f5170: LhttttttthttttthLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLh
f5570: LhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLh
f5970: LhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLh
f5d70: LhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLh
f6170: LhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLh
f6570: LhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLh
f6970: LhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLh
f6d70: LhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhLhBh
f7170: hBhhB...h.ht....ht....htt.......................................
       (9 lines all free)
f9970: ....
Then after a gc.collect()

Code: Select all

qstr:
  n_pool=1
  n_qstr=9
  n_str_data_bytes=123
  n_total_bytes=219
GC:
  19520 total
  5312 : 14208
  1=272 2=6 m=13
GC memory layout; from 3fff4d70:
f4d70: hthhhhhtB.htttLh.htthttttttthtttttttttttthLh.h.h.hth.h.h.h.h.h.h
f5170: .httttttthttttth.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h
f5570: .h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h
f5970: .h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h
f5d70: .h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h
f6170: .h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h
f6570: .h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h
f6970: .h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h
f6d70: .h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h.h
f7170: htth.hhB..ht....ht..hthtt.......................................
       (9 lines all free)
f9970: ....
I can only guess at what this means, as far as I can see the list is getting freed, but the 'h' records are filling up the memory.
The basic file is here:
https://github.com/mianos/micropython/b ... d_esp_ws.c

Can anyone help me with an idea here? I ran into this issue with the modesp sockets so I thought I'd try something similar in a separate file but I seem to have hit the same problem.

It would be a shame to give in at this point as my ESP8266 version is really fun now I pretty much everything on it.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: help with leaking memory management

Post by pythoncoder » Sat Jan 23, 2016 11:43 am

For whatever reason the heap is getting fragmented. Might it be worth issuing gc.collect() in your code to try to pre-empt this - perhaps once per loop? I'm guessing wildly here, though ;)
Peter Hinch
Index to my micropython libraries.

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: help with leaking memory management

Post by dhylands » Sat Jan 23, 2016 4:49 pm

The fact that the 'h' nodes are being left behind means that there is something in memory which contains a reference to them.

Since you're using a list structure, its possible that they are mostly linked together with a link pointing to the head item or something.

To debug this you probably need to examine the contents of the blocks which are being left behind and seeing that they contain. I'll guess you'll find a pointer in one block that points to the next block or something like that. Or there is another data structure which still have the pointers to these.

mianos
Posts: 84
Joined: Sat Aug 22, 2015 6:42 am

Re: help with leaking memory management

Post by mianos » Sat Jan 23, 2016 8:15 pm

dhylands wrote:The fact that the 'h' nodes are being left behind means that there is something in memory which contains a reference to them.
I am not doing anything but allocating the empty list and then assigning it to a member then nulling the member. There is no other code.
Is there a way to find what the 'h' is pointing to?

mianos
Posts: 84
Joined: Sat Aug 22, 2015 6:42 am

Re: help with leaking memory management

Post by mianos » Sat Jan 23, 2016 9:56 pm

pythoncoder wrote:For whatever reason the heap is getting fragmented. Might it be worth issuing gc.collect() in your code to try to pre-empt this - perhaps once per loop? I'm guessing wildly here, though ;)
Good guess. When I wrote my ESP os_timer module I got told off for manually doing a collection at the end of handling a timer event. Of course, if I didn't, it quickly ran out of ram so I put it back in.
You were correct. If I do a collect at the end of the web loop I can do millions and never run out of ram.
It's not a problem with my code at all, more a limitation of the gc. I've done 200,000 requests now and it's still only using the same few hundred bytes of ram.
Every time I have these problems I look closer and closer at the gc, one day I am going to understand it as much as the rest of the stuff in micropython I use every evening. :)

A question to maybe Damien, would it be possible to sweep the block and coalesce all these redundant items? Maybe that should be my next challenge.

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: help with leaking memory management

Post by dhylands » Sat Jan 23, 2016 10:09 pm

The problem with doing coalescing is that in order to move something you need to update all of the pointers which point at the block being moved.

It is possible to do by having something called a master table, and use pointer to pointers. But that requires some fairly extensive changes to the entire code base.

Now if the things that are causing the fragmentation were allocated by you, then you could always make a pass through your own data structures and make a copy of everything, doing a collect periodically. This would cause the objects that are scattered throughout the heap to now be allocated in the lower part of the heap.

mianos
Posts: 84
Joined: Sat Aug 22, 2015 6:42 am

Re: help with leaking memory management

Post by mianos » Sun Jan 24, 2016 12:15 am

The issue here is maybe a subset. Everything in the gc heap management table is freed. The list pointers used by the memory manager itself are what has filled the table. There is no user allocated data.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: help with leaking memory management

Post by pythoncoder » Mon Jan 25, 2016 12:01 pm

@mianos I don't claim any deep knowledge of garbage collection on the Pyboard but I have some empirical experience. In programs with a continuously running loop I've found it advantageous to do gc.collect() on every loop. This has two benefits. On some memory hungry programs it prevents memory errors, as you've found. And on time critical loops a forced gc.collect() runs quicker (on the order of 1mS) than an automatic one which will occur at an unpredictable point in the program when more garbage has accumulated. Only the second of these makes sense to me.

@dhylands Why does regularly issuing collect() prevent memory errors? I'd thought that when the VM fails to allocate space on the heap the memory allocation would automatically do a GC before trying again. I suspect there's something I haven't grasped about the lifetime of references to objects on the heap - but if a program is fixed by the sole addition of regular collect() calls I find it puzzling.
Peter Hinch
Index to my micropython libraries.

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: help with leaking memory management

Post by dhylands » Mon Jan 25, 2016 5:16 pm

pythoncoder wrote:@dhylands Why does regularly issuing collect() prevent memory errors? I'd thought that when the VM fails to allocate space on the heap the memory allocation would automatically do a GC before trying again. I suspect there's something I haven't grasped about the lifetime of references to objects on the heap - but if a program is fixed by the sole addition of regular collect() calls I find it puzzling.
Doing a gc collect often, in some circumstances can dramatically reduce heap fragmentation.

For example, suppose you do a loop 100 times, and each time through the loop you allocate one temp heap block and one more or less permanent heap block. At the end of the loop your heap will contain 100 temp blocks interleaved with 100 permanent blocks. If you did a collect, the 100 temp blocks get freed, leaving behind 100 free blocks of length 1.

If you did a collect each iteration through the loop, then your 100 permanent blocks would all be fairly contiguous and you would have a large free block area.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: help with leaking memory management

Post by pythoncoder » Tue Jan 26, 2016 7:47 am

Got it ;)
Peter Hinch
Index to my micropython libraries.

Post Reply