Page 1 of 1

How to use String and free memory with gc.collect()

Posted: Sat Mar 14, 2020 8:43 pm
by mardi2020
Hi,

i implemented an oauth2 flow which is using a lot of strings (and a lot of memory)
After booting i have a memory of about
DEBUG:memorystate:free Heap 103216 bytes and after gc.collect() 106416 bytes

There are 5 steps in sequence. which look similiar like the code snipped down there and my question is - does it help the gc.collect() to set Strings and objects to None - or is not necessary?

At the moment I need to use gc.collect(), which is hidden behind memorystate.memStatus() after each step.
Otherwise the memory would not survive the oauth workflow. Are there "cleaner" ways to archive that?

Feedback appreciated.

Code: Select all

        step1_params = {
            "response_type": "code",
            "client_id": self.oauth_client_id,
            "code_challenge": code_challenge,
            "code_challenge_method": "S256",
            "scope": self.OAUTH_SCOPE,
            "redirect_uri": self.redirect_uri
        }

        step1_url = self.OAUTH_AUTHORIZE_URL

        step1_headers = {
            "Accept-Language": self.accept_lang,
            "X-Requested-With": self.app_name,
            "Accept": "*/*",
            "User-Agent": ANDROID_USERAGENT,
        }

        memorystate.memStatus()
        _LOGGER.debug('step1_url   : %s' % step1_url)
        _LOGGER.debug('step1_header: %s' % step1_headers)
        _LOGGER.debug('step1_params: %s' % step1_params)
        # First we get a 302 redirect to the login server
        # request(method, url, data=None,json=None, headers={}, stream=None, parse_headers=True):
        _LOGGER.info('Starting Request Step1: %s' % step1_url)
        step1_resp = requests.requestExt('GET', step1_url, step1_params, {}, None, None, step1_headers, None, True, True)
        step1_resp_url = step1_resp.url
        session_cookies = step1_resp.cookies
        step1_resp_extractedList = step1_resp.extractTargetTag('<input type="hidden"', '>')
        _LOGGER.info(('Step 1 Statuscode: %s' % str(step1_resp.status_code)))
        if not (step1_resp.status_code == 200):
            _LOGGER.debug(('Response Headers: %s' % str(step1_resp.headers)))
            _LOGGER.debug(('Response Cookies: %s' % str(step1_resp_cookies)))
            _LOGGER.debug(('Response Extract: %s' % str(step1_resp_extractedList)))
            _LOGGER.debug("Error getting Access-Token in Step 1")
        # manual clean up because of a lot string operations
        step_1_params = None
        step_1_url = None
        step1_headers = None
        step1_resp = None
        memorystate.memStatus()

Re: How to use String and free memory with gc.collect()

Posted: Sat Mar 14, 2020 10:07 pm
by deshipu
MicroPython interns all string constants, so once loaded, they never get released.

Re: How to use String and free memory with gc.collect()

Posted: Sun Mar 15, 2020 8:31 am
by mardi2020
Thank you for your reply.
I understand if i supply strings as constants - they don't release memory,

Does setting a variable to None, help gc.collect() when i create these strings during runtime?
Like the objects in this snippet? This is all in the same scope.

Code: Select all

        step1_resp = requests.requestExt('GET', step1_url, step1_params, {}, None, None, step1_headers, None, True, True)
        step1_resp_url = step1_resp.url
        session_cookies = step1_resp.cookies
        step1_resp_extractedList = step1_resp.extractTargetTag('<input type="hidden"', '>')
        # and later...
        step1_resp = None
        step1_resp_extractedList  = None
        gc.collect()
        # continue with step 2, looks similar like step1 but different parameters and operations on the response
        step2_resp = ...

Re: How to use String and free memory with gc.collect()

Posted: Sun Mar 15, 2020 7:38 pm
by deshipu
The strings will be released when there is nothing else referring to them — so just setting one variable to None may not be enough, if you have already passed that string around and something is still using it. Also, there is no need to manually call gc.collect, it will get called automatically sooner or later.

There is, however, a problem with memory fragmentation. If you create that string in memory, and later create something else, then even when the first string has been released, there is now a "hole" in memory the size of that string. It will be used if you create another string that is the same size or smaller, but anything larger will have to be created somewhere else, as it wouldn't fit in that hole. If you repeat that multiple times, all your memory may consist of such holes, and at some point there will be no hole large enough for your string left anymore.