MicroPython Forum (Archive)

I'm using urequests.py to query an API endpoint every minute or so and retrieve JSON data. My script is usually successful on the first couple of tries, but eventually fails with a memory error. Is there something obvious I'm doing wrong? Is it possible the JSON data is being stored in memory somewhere and causing the script to eventually crash? Is there a command I can include at the end of my function to flush everything stored in memory?

Alternatively, is there a better way to retrieve JSON that uses less memory?

Here's a sample of the code (apologies in advance, I'm new to all of this):

import gc
import machine
import urequests
import json
import ht16k33_matrix
import bitmapfont
import time
gc.collect()

...

def myfunction():
array = []
api_data = urequests.get('http://apiwebsite.com/data&format=json').json()
for level2 in api_data["level2"]:
for level3 in level2["level3"]:
for level4 in level3["level4"]:
for level5 in level4["level5"]:
if (level2["Type"] == "Condition") and (level4["SecondType"] == "SecondCondition"):
array.append(int(level5["Characteristic"])/60)
api_data.close()
del api_data
gc.collect()

Hello,

I didn't have a close look to your code but, I'm experiencing the same behavior with API's (google, openweathermap) AND an OLED screen.

As I understand it's a heap problem (something like a lack of "working" memory).

I bought a Wemos Mini pro with 16Mo of memory but the built version of Micropython doesn't handle it.

May be you can ask another machine (like a Raspberry) to poll the JSON API instead of your ESP and transfer it to the ESP through MQTT for example.

Just a look at your code. In 4 nested loops how many times the condition is satisfied and the array growths? To check the problem I would put a stub (just a print?) instead of filling the array. If the problem is the same, we have an issue with urequests, otherwise it should be investigated how many times the array list is filled.

my2cents

[quote="ernitron"]Just a look at your code. In 4 nested loops how many times the condition si satisfied and the array growths?[/quote]

Actually, the script crashes before it attempts to parse the JSON and add objects to the array. I studied it a little more carefully yesterday and I think the JSON response simply exceeds the available heap memory during certain parts of the day (the number of JSON entries fluctuates). The code runs smoothly during periods when the JSON response is the shortest, so I don't think the problem has to do with variables being stored in memory.

As the first poster suggested, I think the easiest solution is to have another machine query the API and sort out the response. This isn't ideal -- I'd prefer to keep the entire operation on a single chip -- but I'm not aware of any way to read the JSON incrementally or filter the response to cut down the size.