How to use String and free memory with gc.collect()

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Post Reply
mardi2020
Posts: 15
Joined: Fri Mar 06, 2020 8:48 am

How to use String and free memory with gc.collect()

Post by mardi2020 » Sat Mar 14, 2020 8:43 pm

Hi,

i implemented an oauth2 flow which is using a lot of strings (and a lot of memory)
After booting i have a memory of about
DEBUG:memorystate:free Heap 103216 bytes and after gc.collect() 106416 bytes

There are 5 steps in sequence. which look similiar like the code snipped down there and my question is - does it help the gc.collect() to set Strings and objects to None - or is not necessary?

At the moment I need to use gc.collect(), which is hidden behind memorystate.memStatus() after each step.
Otherwise the memory would not survive the oauth workflow. Are there "cleaner" ways to archive that?

Feedback appreciated.

Code: Select all

        step1_params = {
            "response_type": "code",
            "client_id": self.oauth_client_id,
            "code_challenge": code_challenge,
            "code_challenge_method": "S256",
            "scope": self.OAUTH_SCOPE,
            "redirect_uri": self.redirect_uri
        }

        step1_url = self.OAUTH_AUTHORIZE_URL

        step1_headers = {
            "Accept-Language": self.accept_lang,
            "X-Requested-With": self.app_name,
            "Accept": "*/*",
            "User-Agent": ANDROID_USERAGENT,
        }

        memorystate.memStatus()
        _LOGGER.debug('step1_url   : %s' % step1_url)
        _LOGGER.debug('step1_header: %s' % step1_headers)
        _LOGGER.debug('step1_params: %s' % step1_params)
        # First we get a 302 redirect to the login server
        # request(method, url, data=None,json=None, headers={}, stream=None, parse_headers=True):
        _LOGGER.info('Starting Request Step1: %s' % step1_url)
        step1_resp = requests.requestExt('GET', step1_url, step1_params, {}, None, None, step1_headers, None, True, True)
        step1_resp_url = step1_resp.url
        session_cookies = step1_resp.cookies
        step1_resp_extractedList = step1_resp.extractTargetTag('<input type="hidden"', '>')
        _LOGGER.info(('Step 1 Statuscode: %s' % str(step1_resp.status_code)))
        if not (step1_resp.status_code == 200):
            _LOGGER.debug(('Response Headers: %s' % str(step1_resp.headers)))
            _LOGGER.debug(('Response Cookies: %s' % str(step1_resp_cookies)))
            _LOGGER.debug(('Response Extract: %s' % str(step1_resp_extractedList)))
            _LOGGER.debug("Error getting Access-Token in Step 1")
        # manual clean up because of a lot string operations
        step_1_params = None
        step_1_url = None
        step1_headers = None
        step1_resp = None
        memorystate.memStatus()

User avatar
deshipu
Posts: 1365
Joined: Thu May 28, 2015 5:54 pm

Re: How to use String and free memory with gc.collect()

Post by deshipu » Sat Mar 14, 2020 10:07 pm

MicroPython interns all string constants, so once loaded, they never get released.

mardi2020
Posts: 15
Joined: Fri Mar 06, 2020 8:48 am

Re: How to use String and free memory with gc.collect()

Post by mardi2020 » Sun Mar 15, 2020 8:31 am

Thank you for your reply.
I understand if i supply strings as constants - they don't release memory,

Does setting a variable to None, help gc.collect() when i create these strings during runtime?
Like the objects in this snippet? This is all in the same scope.

Code: Select all

        step1_resp = requests.requestExt('GET', step1_url, step1_params, {}, None, None, step1_headers, None, True, True)
        step1_resp_url = step1_resp.url
        session_cookies = step1_resp.cookies
        step1_resp_extractedList = step1_resp.extractTargetTag('<input type="hidden"', '>')
        # and later...
        step1_resp = None
        step1_resp_extractedList  = None
        gc.collect()
        # continue with step 2, looks similar like step1 but different parameters and operations on the response
        step2_resp = ...

User avatar
deshipu
Posts: 1365
Joined: Thu May 28, 2015 5:54 pm

Re: How to use String and free memory with gc.collect()

Post by deshipu » Sun Mar 15, 2020 7:38 pm

The strings will be released when there is nothing else referring to them — so just setting one variable to None may not be enough, if you have already passed that string around and something is still using it. Also, there is no need to manually call gc.collect, it will get called automatically sooner or later.

There is, however, a problem with memory fragmentation. If you create that string in memory, and later create something else, then even when the first string has been released, there is now a "hole" in memory the size of that string. It will be used if you create another string that is the same size or smaller, but anything larger will have to be created somewhere else, as it wouldn't fit in that hole. If you repeat that multiple times, all your memory may consist of such holes, and at some point there will be no hole large enough for your string left anymore.

Post Reply