Memory allocation failed, when loading a html page

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
mardi2020
Posts: 15
Joined: Fri Mar 06, 2020 8:48 am

Memory allocation failed, when loading a html page

Post by mardi2020 » Sat Mar 07, 2020 3:01 pm

Hi,

i try to implement an oauth2 workflow but i'm currently stuck quite at the beginning.

I load a website via https to receive a form where i can put user and password in when i load the website i run out of memory.
Sending the same request via postman returns a reponse with ~20KBytes (19kb body, 1 1kb header).
I hoped with ~100kb RAM i would be able to load the page :-( but i am not getting past 3kb.

Do you have some suggestions/ideas for me?
I actually don't need the full page - i just need to extract from infos from hidden forms, but i have no idea to extract them on the fly. Therefore i wanted to download the full page and extract with some xml parser.

Code: Select all

<input type="hidden" name="SMAUTHREASON" value="">
<input type="hidden" name="target" value="">

So far i used several request.py libs, currently i am using: https://github.com/lucien2k/wipy-urllib ... equests.py
beause it reads the data in 1KB chunks in a loop. before i was using urequest.py from https://github.com/pfalcon/pycopy-lib/t ... est/urllib

Snippet from the loop after sending the request of the urequest.py

Code: Select all

        
        s.write(request) # this sends the request
        received_bytes = 0
        while 1:
            recv = s.read(1024)
            received_bytes += 1024
            if len(recv) == 0: break
            self.text += recv.decode()
            _LOGGER.debug('received bytes %s' % received_bytes)
            memorystate.memStatus()
i added after each read from the network a info where i show the available memory before and after a gc.collect()

Code: Select all

DEBUG:urequests2:received bytes 1024
DEBUG:memorystate:free mem 43904 bytes
DEBUG:memorystate:used mem 75184 bytes
DEBUG:memorystate:gc.collect -> free mem 61280 bytes
DEBUG:memorystate:used mem 57824 bytes
DEBUG:urequests2:received bytes 2048
DEBUG:memorystate:free mem 57776 bytes
DEBUG:memorystate:used mem 61312 bytes
DEBUG:memorystate:gc.collect -> free mem 59216 bytes
DEBUG:memorystate:used mem 59888 bytes
DEBUG:urequests2:received bytes 3072
DEBUG:memorystate:free mem 54688 bytes
DEBUG:memorystate:used mem 64400 bytes
DEBUG:memorystate:gc.collect -> free mem 58192 bytes
DEBUG:memorystate:used mem 60912 bytes
Traceback (most recent call last):
  File "main.py", line 23, in <module>
  File "oauth.py", line 225, in request_initial_token
  File "urequests2.py", line 156, in urlopen
  File "urequests2.py", line 74, in __init__
MemoryError: memory allocation failed, allocating 4097 bytes

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Memory allocation failed, when loading a html page

Post by jimmo » Mon Mar 16, 2020 12:44 am

mardi2020 wrote:
Sat Mar 07, 2020 3:01 pm
I hoped with ~100kb RAM i would be able to load the page :-( but i am not getting past 3kb.
Which board are you using? Is the request using SSL?

mardi2020
Posts: 15
Joined: Fri Mar 06, 2020 8:48 am

Re: Memory allocation failed, when loading a html page

Post by mardi2020 » Mon Mar 16, 2020 5:02 am

jimmo wrote:
Mon Mar 16, 2020 12:44 am
mardi2020 wrote:
Sat Mar 07, 2020 3:01 pm
I hoped with ~100kb RAM i would be able to load the page :-( but i am not getting past 3kb.
Which board are you using? Is the request using SSL?
I'm using a ESP32 heltec wifi kit 32
The request uses ssl.

My workaround
At the moment i' reading the stream byte by byte and extract only matching start/end Tags of the html file.
this might not be the cleanest solution, but it works.

Code: Select all

    def extractTargetTag(self, _startStr, _endStr):
        # we prepare an array to store
        results = []
        # prepare regex to reduces spaces to one space
        # and remove cr/linefeeds
        removeWhiteSpaces = ure.compile("(  )+")
        removeCR = ure.compile("[\r\n]")
        endOfStream = False
        _startStrBytes = bytes(_startStr, 'utf-8')
        _endStrBytes = bytes(_endStr, 'utf-8')
        # start with mininum length of the search String
        # if it is smaller than the start - end
        pageStreamBytes = self.raw.read(len(_startStr))
        if len(pageStreamBytes)<len(_startStr):
            endOfStream = True
        # we must convert the searchstring to bytes als not for all charcters uft-8 encoding is working
        # like in Curacao (special c)
        while not endOfStream:
            if (pageStreamBytes == _startStrBytes):
                # we found a matching string
                # print('Start found %s ' % pageStreamBytes.decode('utf-8'))
                # we need to find the endtag
                endOfTag = False
                read = self.raw.read(len(_endStr))
                if len(read) == 0: 
                    endOfStream = True
                pageStreamBytes += read
                while ((not endOfStream) and (not endOfTag)):
                    # comparing the string with the find method is easier
                    # than comparing the bytes
                    if (pageStreamBytes.decode('utf-8')).find(_endStr)>0:
                        endOfTag = True
                        result = removeWhiteSpaces.sub('', pageStreamBytes.decode('utf-8'))
                        result = removeCR.sub('', result)
                        results.append(result)
                        # _LOGGER.debug('Result: %s' % result)
                        # clear mem from string operations
                        # calling the GC seems to cause the ESP32 to stall therefore not called here 
                        # memorystate.memStatus()
                    else:
                        # read and Append
                        read = self.raw.read(1)
                        if len(read) == 0:
                            endOfStream = True
                        else:
                            pageStreamBytes += read
                            # print('End not Found %s' % pageStreamBytes.decode('utf-8'))
            else:
                # we did not find a matching string
                # and reduce by one character before we add the next
                # print('not found %s' % pageStream)
                pageStreamBytes = pageStreamBytes[1:len(_startStrBytes)]
            read = self.raw.read(1)
            if (len(read) == 0):
                endOfStream = True
            pageStreamBytes = pageStreamBytes + read
        self.close()
        return results

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Memory allocation failed, when loading a html page

Post by jimmo » Mon Mar 16, 2020 5:31 am

I think you're probably seeing memory fragmentation. The issue is that you can have lots of memory free, but spread out in small chunks over the heap.

At the point that your program fails, can you run:

Code: Select all

import micropython
micropython.mem_info(1)
and show us the results.

Even better, if you can share your entire code that might be useful to help debug. I looked at the request library you linked to but can't seem to match it up with the stack trace in the error message.

mardi2020
Posts: 15
Joined: Fri Mar 06, 2020 8:48 am

Re: Memory allocation failed, when loading a html page

Post by mardi2020 » Mon Mar 16, 2020 3:53 pm

Hi,

thanks for your effort and i appreciate your help.

I can't reproduce it with the same code, because i changed it a lot the last days, and i can't share a running example - because i can't share with you some of the access secrets.

I did a lot of changes since my post (changed library) and with the method mentioned above - reading and parsing the stream - i have a working solution (which extends the library - but is not called in the example below). There fore it is not necessary for you to put more effort in my problem.

But if you are interested - this it how meminfo looks like - i cant interpret the result.

Code: Select all

Traceback (most recent call last):
  File "main.py", line 23, in <module>
  File "oauth.py", line 221, in request_initial_token
  File "uurequests.py", line 101, in text
  File "uurequests.py", line 34, in content
MemoryError: memory allocation failed, allocating 4864 bytes
MicroPython v1.12 on 2019-12-20; ESP32 module with ESP32
Type "help()" for more information.
>>> import micropython
>>> micropython.mem_info(1)
stack: 752 out of 15360
GC: total: 118976, used: 64160, free: 54816
 No. of 1-blocks: 701, 2-blocks: 187, max blk sz: 288, max free sz: 264
GC memory layout; from 3ffc2f30:
00000: MDShhSMhhDThSh=hhh==============================================
00400: ================================================================
00800: ================================================================
00c00: ================================================================
01000: =========================hh==BMDSh====h=hSh=h==========h=DBBBhBh
01400: ======hh=hSDD..h=======h=====h==DBBB..BBBBBBBBBh===BBMD....SB=h=
01800: h========SShh===========MDSSShh=.h===hBhBBTMD..h==============Dh
01c00: BBhSB.hh========MD....h=h=.hB.ShBBBh====.h======h=Sh=h=D.h===h==
02000: ==h====...h===hB.B=hhhBh==DhSh===BSB=...hST..Bh==h===h==========
02400: ====h==hBShShh=======h==hh===h==========h==================h===B
02800: MD...Sh==Sh=S..........SSS.h====================.....h===h======
02c00: =====S.........h.MD..h.S.......hDSh===h=h=hh.hSh=SShhh==.SSh=h=h
03000: ===S.hhh===S.h==h=.hSh=S.h=Sh==.....B..S.......Sh=======Sh=h====
03400: ===..........h==hh===h=hhh==hhhh=hh=====hh=hh=hh=hh=hh=hh=hh==hh
03800: ============================h=========h=======BBB=.h......hSh===
03c00: ===Sh===========.S......h=======.hS.....h=.Sh=======h==h=======S
04000: h==h====h====..h=h===h=S.Sh=======.....Sh=h====........h.......h
04400: ========================================...h=.........SSh=======
04800: ===========............hS....h=...S...........h=======hSh===ShSh
04c00: hh=======h=h=hh.SSSh==hSh..h=Shh=Sh.............................
05000: .........Sh==h===========h======h=h=========h=hh=hh=hh=hh=======
05400: hh===h=Shh==hh==hh==========h==h=====hh=====hhDhh....hh=======..
05800: ........................h...h============h==============........
05c00: hh=Sh=======........S....hS..h...h=======Sh========h========h===
06000: ==h=======..........Sh==h==================.....................
06400: ..Sh===h==============..........hh........h.....hh...........hh=
06800: ...hhh...h.....h.....................S....h........SSh=.........
06c00: .....h........SSh=............h=..SBBBB=SSh=h========h=B.B=....S
07000: Sh=MD..D.SSSh======Sh=====h.hDS.Sh=Sh=SSh=....MDh.hh=Sh=h=......
07400: ...S.h=.DDDDDBBSSh=h=======h=======Sh==........LL.....S.h=h=====
07800: ==SShShh=======Sh=h=======SSh=h=ShSh=h=h=Sh=....h==h=h=======h=h
07c00: =ShSSSSh=h=======hh=Sh============================h=..h=h=======
08000: =....Sh==h===============SSSh=B=hBBDLB=BBBLD.h===h====h=h===L.h=
08400: ==h====.h======Sh====Sh=ShSh=h=======.hShh==..Sh.SSh==h==Lh=====
08800: Lh===h==DBB.B=.B=.h=B.Dh==h=.h=h==.....SSh=B=.h===.D.h=B=..D.h==
08c00: =====h======..h=h=======Sh=h===B.....h=BhhBh..DBhBT.Sh=h===h===h
09000: BSh=...h=====.DB.BThDSSh=..Sh.SS.Sh=======SSh=h==Bh.h===T.B..h=S
09400: h=hhh==h===hSh.......h=ShShSh==...h=h=SSSh=...Sh..DSh=h=...h==hh
09800: =........Sh=...Sh...Sh=.....S.SSh=h=....ShSShh===hh=hh=hhhh==ShS
09c00: h==Shh=======SShh====h=====h===hDBB=Th===.Sh=DB..Th===.DBB=Th=h=
0a000: ==SDBB=ThD.h===============Sh=Shh===BBBTh===...Sh=ShShhh=h==h==h
0a400: ==h==.hhh=h=======Sh=h=======Sh=h=======Sh=h==h==.B=BBh==..h=B=.
0a800: SB=h.B=B=h.h====SDSDBBBSh=h.h==h===BSh=h======hSSh=h============
0ac00: ======Sh=..B=.SB=BhSh==B=.SB=BBBBB.B=B.Sh=BBBBBBhBDBhh===h===BBB
0b000: BB...B==...B==SShShh===h==..h==Bh========BB............h====BBSh
0b400: h========hh============h====h==h===========h====h===========h===
0b800: ===========h==h========h======hh==hh==h=h====hh=================
0bc00: ================================================================
0c000: =======================================h==================h=hh==
0c400: hh==============.........................................h======
0c800: =h==============================================================
0cc00: ==================h==================...........................
0d000: ..................h===B.B=Bh===========B....h==...Sh...h=hh.h===
0d400: ===========h===..Sh..SSh==h........Sh==.Sh....Sh==.......ShSh==.
0d800: S...Sh=.......Sh..Sh............................................
0dc00: ....Sh..h================================================.......
0e000: .....h==========================================================
0e400: =====h=h=hh=hh=hh=hh===hh==hh===hh====================hh=hh=h...
       (3 lines all free)
0f400: ............................................Sh=========.........
0f800: ....................Sh..S...............................Sh====..
0fc00: ............h=======............................................
10000: .............................................h==================
10400: ================================================================
10800: ======..........................................................
       (3 lines all free)
11800: ..............h=================================================
11c00: ===========================================================.....
12000: .........Sh===Sh.....................Sh==Shh==.....Sh=...Sh=....
12400: ....Sh=..Sh.......Sh=...Shh==....Sh...Sh........h========.Sh...h
12800: =======.........................................................
12c00: ..............................Sh=......Sh=....h.......h===......
13000: ................................................................
13400: ........................................h=======................
13800: ................................................................
13c00: ...............................hhh==============================
14000: ================================================================
14400: ================================================================
14800: ================================================================
14c00: ================================================================
15000: =...........h===.................................h..............
15400: ................................................................
15800: .....................................................Sh=======..
15c00: .......h=.......................................................
16000: ..............................................................h=
16400: .......................................h=======...............Sh
16800: ....................................Sh..........................
16c00: .............Sh.................................................
17000: Sh.....................h===================....................S
17400: h....................................Sh.........................
17800: ....................................Sh..........................
17c00: ...........................hh..................hh..........h=...
18000: .hh.........h=....h=...hh.......................................
18400: .................h..............................................
18800: ................................................................
18c00: ................h...............................................
19000: ................................................................
19400: ............................hh..................................
19800: .h=.............................................................
19c00: ................................................................
1a000: ............................................h=====..............
1a400: ............................h======h=====h====h=================
1a800: ================h===hh========hh=============================hh=
1ac00: ====hh==========h=ShShShShh========hh====h=h==============h=h==h
1b000: ....h=======h=h=======h=ShShh===========h=h=====================
1b400: =========h=h=====hh=====hh=====hh======hh====hh====hh====hh=====
1b800: ==hh====hh====hh==hh====hh==hh=hh=hhhhhh===hh===hh====hh======hh
1bc00: ====hShShh======h=ShShShShShh==hhhhhh==hShhhhhh=hh==hh=hh=hh==hh
1c000: =hh=hh==hh==h...................................................
       (3 lines all free)
1d000: ............


User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Memory allocation failed, when loading a html page

Post by jimmo » Tue Mar 17, 2020 1:43 am

mardi2020 wrote:
Mon Mar 16, 2020 3:53 pm
But if you are interested - this it how meminfo looks like - i cant interpret the result.
It's showing a map of which part of RAM is used. The dots are free and anything else is in-use. The === show larger allocations.

So the thing to note here is that there are lots of small allocations all over the place and not a lot of large free regions. The thing to test now is to run
gc.collect()
then
micropython.mem_info(1)

and see if it clears up all of those scattered small allocations.

I'd like to make a more concise and useful guide for how to mitigate these sorts of issues, but in the meantime here's a very informal talk I gave about this (and it includes a bit of an explanation of the mem_info output). https://www.youtube.com/watch?v=H_xq8IYjh2w

This thread viewtopic.php?f=2&t=7345& also covers a bunch of these topics (and a lot of other stuff too).

mardi2020
Posts: 15
Joined: Fri Mar 06, 2020 8:48 am

Re: Memory allocation failed, when loading a html page

Post by mardi2020 » Tue Mar 17, 2020 6:01 am

Hi,

this would be the result before and after gc.collect().
For me it doe not look really better after gc.collect()

Code: Select all

MemoryError: memory allocation failed, allocating 4864 bytes
MicroPython v1.12 on 2019-12-20; ESP32 module with ESP32
Type "help()" for more information.
>>> import micropython
>>> micropython.mem_info(1)
stack: 752 out of 15360
GC: total: 118976, used: 65376, free: 53600
 No. of 1-blocks: 716, 2-blocks: 195, max blk sz: 288, max free sz: 279
GC memory layout; from 3ffc2f30:
00000: MDShhSMhhDThSh=hhh==============================================
00400: ================================================================
00800: ================================================================
00c00: ================================================================
01000: =========================hh==BMDSh====h=hSh=h==========h=DBBBhBh
01400: ======hh=hSDD..h=======h=====h==DBBB..BBBBBBBBBh===BBMD....SB=BT
01800: h=h========.Shh===========MDSShh=.h===hBhBMDh=D.h==============S
01c00: ShBBhBh=h========MD....h...ShB.ShBBBh====.h======h=Sh=h=D.h===h=
02000: ===h====...h===hB.B=hhhBh==Dh.h===BSB=...hSTBh==h===h===========
02400: ===h==hBSh=======..Sh===BMD..ShSh.h==hSSSh==================h===
02800: .....h=======Sh==Sh=Sh===Shhh====================h==h=h===......
02c00: ...h======S........Sh==============h===.hhMD..hDShh=Shh.hSh=h==h
03000: h=SS..h==h=.h==SS.......h=h==============h======S........Sh=====
03400: ==..........h==hh===h=hhh==hhhh=hh=====hh=hh=hh=hh=hh=hh=hh==hh=
03800: =.....B..S.......SSh===.hS.ShSh=Sh=Sh=hhh=========h=.......h====
03c00: ===BBB=Shh===========h====...h======h.............h.S...h=======
04000: h==========.h=======......................Sh===S.Sh=============
04400: ===========================h=======h=h====h=======h====..h....h.
04800: .......hSSh=.......S...hS..........Sh=...h.hhhSh=h===========.hh
04c00: =Dhhh===.........h=======Shh==h=======.Sh=h===h=h=SSh...Sh...Shh
05000: ============....h........Sh======h=========h=h=hh=hh=hh=======hh
05400: ===h=Shh==hh==hh==========h==h=====hh=====hh==h==============...
05800: h======S.h=hS......Shh==============h=======.hh=........Sh======
05c00: =h==================.................h.hhh=====..hh=h========...
06000: ....h.....h.....h.....hh========h=======..................h=....
06400: ............h=======h.........Sh=............Sh=........SSh=S...
06800: .....h=........SSh=........S............Sh=Sh=SSh=SBh========BBh
06c00: =B.B=B=..Sh=MD..D...Sh=h=======h=====SSSh==h.hDS.....Mh.hDh=h===
07000: ====ShSh........Sh=h=======SSh=.....Sh=Sh=Sh=h=h=h==h=h=======LL
07400: ....ShSSSh=======Sh=SS....S...DDDDDBB.Sh=======h=h=...B=hBB....S
07800: h==h=======..LB=B...SSh=..................h==============.h==.h=
07c00: h=h======Sh=ShSh=SShSh=h======..hShh=======SSh==hh====..Sh=h====
08000: ===.h=Sh=h========Sh=........SSh=Sh.....SS...............Sh==Dh=
08400: =Dh=..BBL..h=L.h=.h=h===h====h==Sh=h===h====Lh==h=h========.h=h=
08800: ====h===Lh===D..h=......SSSh======hSSh=BB.B=.B=B.D.h=.h=Sh=hh===
08c00: ====.ShB=h===.D.ShShSh==h=...ShSSSh=........Sh=h=B=..DB.h=.....D
09000: ShSh=..h=.S.SSh=h=...hShSh=h======.SShBhhBh..DBhBhh===ShSh==Shh=
09400: ======SS...h=====.DB.BTh===h===T.SSSh=BhD.h==Bh.h===hh=h========
09800: ====h===============Sh=ShhhTSBhh==h===.........h..h==h.h=....Sh=
09c00: =h=h======Shh=h=======Sh=h======SSh=h=hh===hh=hh=hhSh=hh==hh====
0a000: h=====h===hDBB=.Sh=Th===SDBThSh=ShShh===SSh=......h==h==h==h==h=
0a400: h=B=h===DB.BSh==Th===.DBB=T.DB.hSh=h===B.Th===SB=h.B=.SB=hB=h.h=
0a800: ===SD.h==h==h=h=hh=h==B=BBh==DBB..Bh==h===ShB=Bhh============h==
0ac00: ==h==========h====h===========h=============h==hhh==hh==h=h====h
0b000: h===============================================================
0b400: ======================================================h=========
0b800: =========h=hh==h.....................h==================h=======
0bc00: ===========...S.h============================h=hh===============
0c000: ================================================================
0c400: =........................................................h======
0c800: =h==================..B=B=..h==B=BBBBBBSh==B=BBBBBBBhBDBhh===h==
0cc00: =BBBBBSS.....................................h==..h.............
0d000: ...........Sh....B==..SB==h.h==B.Sh.........h========BBh====BSh=
0d400: ==hB=BBh===========.h===SMDS...h==h===========.....D.BBBBSS.h=Sh
0d800: =h================================================B=BShh===Sh=h=
0dc00: ==h=....Sh=.......Sh..Sh..........h=============================
0e000: ==================================h=hh=hh===hh==hh===hh=========
0e400: ===========hh=hh=h.............................Sh...............
0e800: ..........................S.......h============h=========.......
0ec00: .......h.............Sh....h==..............h==..........Sh====.
0f000: ..h=======ShShSh.........................Sh===Sh................
0f400: ..Sh=...Sh==Shh==.....Sh=...Sh=........Sh=..Sh.......Sh=...Sh.Sh
0f800: .Sh=........hh=......h========.Sh.h======h============h==h=h==hh
0fc00: =====hh===h.....................................................
10000: ..........................................................Sh=...
10400: ...Sh=....h.......h===........................................h=
10800: ======..........................................................
10c00: ................................................................
11000: .........................................................h======
11400: ================================================================
11800: ==================..............................................
11c00: ................................................................
12000: .................h=======.......................................
       (3 lines all free)
13000: ................................................h=======h=======
13400: ================================================================
13800: =====================================.................hh........
13c00: ................................................................
14000: ....................................................h=======....
14400: .......................................h==......................
14800: ................................................................
14c00: ......................................Sh===...................S.
15000: ............h...................................................
15400: ................................................................
15800: .....................h=.........................................
15c00: ................................................................
16000: ...h=======..............Sh=....................................
16400: ...................Sh....................................Sh.....
16800: ..................................Sh......................h=====
16c00: ==.................................Sh...........................
17000: ....................Sh....................................Sh....
17400: ...................................h==================..........
17800: ..Sh....................................................hh......
17c00: ............hh..........h=....hh.........h=....h=...hh..........
18000: ..............................................h.................
18400: ................................................................
18800: .............................................h..................
18c00: ................................................................
19000: .........................................................hh.....
19400: ..............................h=................................
       (2 lines all free)
1a000: .........h=====..........................................h======
1a400: h=====h====h=================================h===hh========hh===
1a800: ==========================hh=====hh==========h=ShShShShh========
1ac00: hh====h=h==============h=h==h....h=======h=h=======h=ShShh======
1b000: =====h=h==============================h=h=====hh=====hh=====hh==
1b400: ====hh====hh====hh====hh=======hh====hh====hh==hh====hh==hh=hh=h
1b800: hhhhh===hh===hh====hh======hh====hShShh======h=ShShShShShh==hhhh
1bc00: hh==hShhhhhh=hh==hh=hh=hh==hh=hh=hh==hh==hh=====================
1c000: ================================================================
1c400: ================================================================
1c800: ================================================================
1cc00: ================================================================
1d000: ==========..
>>> import gc
>>> gc.collect()
>>> micropython.mem_info(1)
stack: 752 out of 15360
GC: total: 118976, used: 53504, free: 65472
 No. of 1-blocks: 625, 2-blocks: 181, max blk sz: 264, max free sz: 395
GC memory layout; from 3ffc2f30:
00000: MDShhhMBhDh=Sh=hhh==============================================
00400: ================================================================
00800: ================================================================
00c00: ================================================================
01000: =========================h....MDS.....h=.Sh=h==========h=DBBBh=h
01400: ======h..hS.Dh.h=======h=====h==DBBB.hBBBBBBBBBh===BBMDB....B=..
01800: h=h========.Shh===========MDSSh...h===hBhBMD..D.h==============.
01c00: ShBBhB..h========MD.........hB.ShBBBh====.h======h=Sh=..D.h===h=
02000: ===h====h=.h===hB.B=h..Bh==Dh.h===BSB=...hSTBh==h===h===========
02400: ===h==hBSh=======..Sh===.MD..ShSh.h==hSSSh==================h===
02800: .....h=======Sh==Sh=Sh===Shhh====================h==h=h===......
02c00: ...h======S........Sh==============h===.hhMD..hDShh=Shh.hSh=h==h
03000: h=SS..h==h=.h==SS.......h=...............h======S........Sh=====
03400: ==..........h==hh===h=hhh==hhhh=hh=====hh=hh=hh=hh=hh=hh=hh==h..
03800: ......B..S.......SSh===.hS.ShSh=Sh=Sh=..h=========h=.......h====
03c00: ===BBB=.hh===========........h======h...............S...h=======
04000: h==========.h=======......................Sh===S.Sh=============
04400: ===========================h=======h=h====h=======h====..h....h.
04800: ........SSh=.......S...hS..........Sh=...h.hhhSh=h===========.hh
04c00: =Dhhh===.........h=======Shh==h=======.Sh=h===h=h=SSh...Sh...Shh
05000: ============....h........Sh======h=========h=h=hh=hh=hh=======hh
05400: ===h=Shh==hh==hh==========h==h=====hh=====hh==h==============...
05800: h======S.h=hS......Shh==============h=======.hh=........Sh======
05c00: =h==================.................h.hhh=====.....h========...
06000: .......................h========h=======..................h=....
06400: ............h=======h.........Sh=............Sh=........SSh=....
06800: .....h=........SSh=........S............Sh=Sh=SSh=SBh========BBh
06c00: =B.B=B=..Sh=MD..D...Sh=h=======h=====SSSh==h.hDS.....Mh.hDh=h===
07000: ====ShSh........Sh=h=======SSh=.....Sh=Sh=Sh=h=h=h==h=h=======LL
07400: ....ShSSSh=======Sh=SS........DDDDDBB.Sh=======h=h=...B=hBB....S
07800: h==h=======..LB=B...SSh=..................h==============.h==.h=
07c00: h=h======Sh=ShSh=SShSh=h======..hShh=======SSh==hh====..Sh=h====
08000: ===.h=Sh=h========Sh=........SSh=Sh.....SS......................
08400: ......BBL..h=L.h=.h=h===h====h==Sh=h===h====Lh==h=h========.h=h=
08800: ====h===Lh===D..h=........Sh======.SSh=BB.B=.B=B.D.h=.h=Sh=hh===
08c00: ====.ShB=h===.D.ShShSh==h=...ShSSSh=........Sh=h=B=..DB.h=......
09000: ShSh=.....S.SSh=h=...hShSh=h======.SShBhhBh..DBhBhh===ShSh==Shh=
09400: ======SS...h=====.DB.BTh===h===T.SSSh=BhD.h==Bh.h===hh=h========
09800: ====h===============Sh=ShhhT.Bhh==h===............h==h.h=.....h=
09c00: =h=h======Shh=h=======Sh=h======SSh=h=hh===hh=hh=hhSh=hh==hh====
0a000: h=====h===.DBB=.Sh=Th===.DBThSh=ShShh===SSh=......h==h==h==h==..
0a400: h=B=h===DB.BSh==Th===.DBB=T.DB.hSh=h===B.Th===.B=..B=..B=.B=h.h=
0a800: ===SD.h==h==h=h=hh=h==B=BBh==DBB..Bh==h===..B=Bh.............h==
0ac00: ==................h===========h=============h==hhh==hh==h=h====h
0b000: h===============================================================
0b400: ======================================================h=========
0b800: =========h=hh==h.....................h==================........
0bc00: ................h============================h=hh===============
0c000: ================================================================
0c400: =...............................................................
0c800: .h==================..B=B=..h==B=BBBBBB.h==B=BBBBBBBhBDBhh===h==
0cc00: =BBBBBSS........................................................
0d000: .................B==...B==..h==B............h========BBh====B.h=
0d400: ==.B=BBh===========.h===.MDS......h===========.....D.BBBBSS.h=Sh
0d800: =h================================================B=B..h===Sh=h=
0dc00: ==h=..............................h=============================
0e000: ==================================h=hh=hh===hh==hh===hh=========
0e400: ===========hh=hh=h..............................................
0e800: ..................................h============.................
0ec00: .......h...................h==..............h==.................
0f000: ..h=======ShShSh................................................
0f400: ..Sh=...........................................................
0f800: ..h=..............................h======h============h==h=h==hh
0fc00: =====hh===h.....................................................
10000: ................................................................
10400: ..............................................................h=
10800: ======..........................................................
       (5 lines all free)
12000: .................h=======.......................................
       (3 lines all free)
13000: ................................................h=======........
13400: ................................................................
13800: ......................................................hh........
13c00: ................................................................
14000: ....................................................h=======....
14400: .......................................h==......................
14800: ................................................................
14c00: ......................................Sh===...................S.
15000: ............h...................................................
15400: ................................................................
15800: .....................h=.........................................
15c00: ................................................................
16000: ...h=======..............Sh=....................................
16400: ...................Sh....................................Sh.....
16800: ..................................Sh......................h=====
16c00: ==.................................Sh...........................
17000: ....................Sh....................................Sh....
17400: ...................................h==================..........
17800: ..Sh....................................................hh......
17c00: ............hh..........h=....hh.........h=....h=...hh..........
18000: ..............................................h.................
18400: ................................................................
18800: .............................................h..................
18c00: ................................................................
19000: .........................................................hh.....
19400: ..............................h=................................
       (2 lines all free)
1a000: .........h=====..........................................h======
1a400: h=====h====h=================================h===hh========hh===
1a800: ==========================hh=====hh==========h=ShShShShh========
1ac00: hh====h=h==============h=h==h....h=======h=h=======h=ShShh======
1b000: =====h=h==============================h=h=====hh=====hh=====hh==
1b400: ====hh====hh====hh====hh=======hh====hh====hh==hh====hh==hh=hh=h
1b800: hhhhh===hh===hh====hh======hh====hShShh======h=ShShShShShh==hhhh
1bc00: hh==hShhhhhh=hh==hh=hh=hh==hh=hh=hh==hh==h......................
       (4 lines all free)
1d000: ............
>>> 

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Memory allocation failed, when loading a html page

Post by jimmo » Tue Mar 17, 2020 9:44 am

So your program is holding references to things... e.g. appending to a list on every request or something like that.

mardi2020
Posts: 15
Joined: Fri Mar 06, 2020 8:48 am

Re: Memory allocation failed, when loading a html page

Post by mardi2020 » Tue Mar 17, 2020 10:49 am

jimmo wrote:
Tue Mar 17, 2020 9:44 am
So your program is holding references to things... e.g. appending to a list on every request or something like that.
yes, i have a lot of loops collecting infos and storing the Info in dictionaries.
Is there a better way of allocating the memory?
Is there a possibility of defragmenting memory?

Thanks for your advice

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Memory allocation failed, when loading a html page

Post by jimmo » Tue Mar 17, 2020 12:17 pm

You should watch that talk I linked to (sorry it's a bit all over the place and my voice is terrible but I do answer these questions in it).

The simple answer though is that you need to understand how the allocator works (even just at a basic level):
- It starts off pointing at zero.
- For each allocation it then just moves forward that pointer by as much as it needs.
- When it runs out of memory (or you explicitly call gc.collect() ) it finds all memory currently in use by the program, and marks everything else as free.
- It moves the pointer to the first free address. Now when it's searching, it skips over "used" sections.

So your goal is to ensure that your "long lived" allocations happen when the pointer is near the start of memory (i.e. so the long-lived ones all bunch up together), and that collections happen when your program is actively using the least amount of memory.

The best way I can "codify" this into a simple solution is that your program should manually call "gc.collect()' after each request is handled, but only once as much temporary memory used by that request is no longer referenced.

So something like:

Code: Select all

while True:
  handle_request()

def handle_request():
  do_as_much_work_as_possible()
  gc.collect()
  do_stuff_that_results_in_long_lived_allocations()
  gc.collect()
Note the use of functions here -- when variables go out of scope at the end of functions they are automatically dereferenced, so you don't have to worry about keeping track of them.

The other things you can do is pre-allocate data structures. If you're building up byte data, make a big bytearray at the start, then slice-assign (or memoryview) into the bytearray as you go. Or for lists of objects, preallocated the objects and then just set values on their members rather than creating and appending as you go.

Post Reply