Potential urequests out of memory?

[hansamann]

I am trying to get a basic https Oauth2 request running on the eps8266, here's my code which you can try out yourself, too, including the id and secret as I just created a new app that I just setup for this purpose:

import urequests
client_id = 'xf2ZD11MDHDdNhdHNs1Am0PAwhFVeqPH'
client_secret = 'hP5FRZ3a238wLNnk'
endpoint = 'https://api.yaas.io/hybris/oauth2/b1/token'
formParams = { 'grant_type' : 'client_credentials', 'scope' : '', 'client_id' : client_id, 'client_secret' : client_secret }
urlencoded = 'grant_type=client_credentials&scope=&client_id=2EGmcRNFwWt57Y4dYxcyFoMIV6RU7NyK&client_secret=8ZXCunmApPdIWVKO'
r = urequests.post(endpoint, data = formParams, headers= {'Content-Type': 'application/x-www-form-urlencoded'})

Once I am connected and run this code, I get this:
Traceback (most recent call last):
File "<stdin>", line 8, in <module>
File "urequests.py", line 98, in post
File "urequests.py", line 56, in request
MemoryError: memory allocation failed, allocating 4256 bytes

Is there anything I can do to make this work? I've tried the urllib.urequest lib, but the issue here is that headers are not supported. so even though I urlencoded the data to be sent in the POST, the server will not accept it as the Content-Type header is missing. Any idea how to make this work at this point?

[pfalcon]

So, now you know why there's no arbitrary headers support - any feature added leaves less and less memory for user applications, and there're a lot of features already.

P.S. It needs detailed investigation why this seemingly simple example runs out of memory. Start with confirming that you run it after a clean reboot.

[deshipu]

It fails whlile trying to parse the response it received, which it puts into memory all at once.
I think it would be better to read the response from the socket gradually -- first only the headers, and the rest only when the user requests it.

[jms]

More generally it would be good to account for RAM on the module.

i.e. How much the module has (96k ?), how much the manufacturer's SDK code consumes and finally micropython itself.

Certainly on mine there is about 16k left and one of the examples tries to read a 4k block over TLS can't because of memory. (Just maybe there's a bug and it's trying 4k x unicode or something.)

I wonder if pointers are or can be made 16-bit given what we've got to work with.

Jon

[deshipu]

Certainly on mine there is about 16k left and one of the examples tries to read a 4k block over TLS can't because of memory. (Just maybe there's a bug and it's trying 4k x unicode or something.)

One thing is how much memory there is in total, another is how much is available in continuous chunk. As things get allocated and released from memory, it gets fragmented and while you can have 16K of memory free in total, it may be all in chunks smaller than 4K.

[jms]

True.

Code: Select all

>>> esp.meminfo()
data  : 0x3ffe8000 ~ 0x3ffe8410, len: 1040
rodata: 0x3ffe8410 ~ 0x3ffe9038, len: 3112
bss   : 0x3ffe9038 ~ 0x3fff6bd0, len: 56216
heap  : 0x3fff6bd0 ~ 0x3fffc000, len: 21552

BSS sticks out somewhat. I shall see if I can understand it a little.

Looking at the symbols in the BSS section in my build and sorting by size only a few things really stick out. There's something called dns_table at 1120 bytes but by far the largest thing is called heap.

Code: Select all

3ffef34f 00007000 b heap
3fff634f 00000001 b lock$5424

Now that's 28k, defined in main.c and appears to be quite distinct from the other 21k heap at the higher address. Anybody know why there appear to be two heaps ?

[pfalcon]

Anybody know why there appear to be two heaps ?

Forum knows for sure (can be asked via its friend named "Search"): http://forum.micropython.org/viewtopic. ... eap#p10590 . (It also knows a lot of other things newcomers may be interested in.)

[pfalcon]

micropython.mem_info() was updated to report maximum allocatable (i.e. currently free) block size, e.g.

Code: Select all

GC: total: 28160, used: 13216, free: 14944
 No. of 1-blocks: 326, 2-blocks: 22, max blk sz: 263, max free sz: 783

Note that the value is in allocation units (16 bytes for esp8266), not bytes. Please include micropython.mem_info() output in future reports of issues like in this topic.

There's also ongoing work to optimize memory allocation in uPy in general and esp8266 port in particular - some already landed, more on TODO.

[ernitron]

Controlling memory of course is fundamental and the esp.memifo() and micropython.mem_info() give some clue about memory allocation. Besides it would be nice if these routines could return some sort of tuple or strings with the same information so the code can also take advantage of it. I use them in a status page output from a basic httpserver that is almost working but as I add some more function or string it just goes out of memory.



Disclaimer: As I am playing/porting my apps from LUA/NodeMCU to uPython I found issues and problems of the new machine that others has already discovered, and I apologize for resuming old topics.

[deshipu]

You can easily get those numbers using gc.mem_alloc() and gc.mem_free() (and subtracting one from the other).