Loading json very slow

[leosok]

Hello,

I'm working with MP 1.18 on a ESP32 TTGO-Display Board (with a small TFT) using st7789py_mpy library by russhughes. When my module is loading, I want to show a splash-screen. Inizializing the screen takes around 1,3 seconds, which I find ok.

My Problem is the following code, which loads the configuration for the Display as the Code should be compatible with different modules. Loading a small JSON takes 5 seconds to run - does someone have an idea why? (And how to fix)

Code: Select all

class BoardConfig:
    """
    Loads board config from JSON
    """
    
    CONF_FILE = 'config/board_config.json'

    def __init__(self):
        self.erika_rts = None
        self.erika_cts = None
        self.erika_rx = None
        self.erika_tx= None
        
        self.screen_display_type = None
        self.screen_rst = None
        self.screen_scl = None
        self.screen_sda = None
        
        self.load()

    def load(self):
        """
        Loads config from JSON
        Returns: dict or False (if config_file cannot be opend)
        """
        try:
            with open(self.CONF_FILE, 'r') as f:
                data = json.load(f)
            for k,v in data.items():
                setattr(self,k,v)
        except:
            return False

    def __repr__(self):
        return str(self.__dict__)

[jimmo]

Loading a small JSON takes 5 seconds to run - does someone have an idea why? (And how to fix)

I tried to replicate this but it takes 4ms to run for me (on v1.18 on ESP32).

Code: Select all

bc = BoardConfig()

t1 = time.ticks_ms()
bc.load()
t2 = time.ticks_ms()

print(time.ticks_diff(t2, t1))
print(bc)

prints

Code: Select all

4
{'screen_scl': 9, 'screen_sda': 10, 'erika_tx': 12, 'erika_rts': 1, 'screen_rst': 8, 'screen_display_type': 'color', 'erika_rx': 13, 'erika_cts': 0}

Are you able to narrow down which part is taking the time?

[leosok]

Hi jimmo,
thanks for testing it yourself! I did a lot of printing and finally found the problem. Depending on where I called BoardConfig it took seconds (as described) or milliseconds (as you measured). When called at the very start of boot.py it took long, while higher up in main.py it took just milliseconds. I had the idea, that maybe the first read from the flash took longer. Then I finally realized that this could be due to imports because all the imports in main.py take almost 10 seconds. And this was it:

Code: Select all

# configurator.py
import os 
import json
import uasyncio as asyncio  # pylint: disable=import-error
import mrequests as requests  # pylint: disable=import-error
import machine  # pylint: disable=import-error
from erika import Erika
from utils.network_utils import scan_wlan, get_wlan_strength, do_connect

I did not realize, that only doing this import

Code: Select all

from config import BoardConfig

would import all these modules. Once I put the imports in the functions where they are used, loading the json and returning the class went down to 400ms.

Thx!

[jimmo]

Then I finally realized that this could be due to imports because all the imports in main.py take almost 10 seconds. And this was it:

Yes, import requires a lot of work to load, compile, etc. Freezing some modules can help with this (because they're pre-compiled and will execute directly from ROM), although module initialisation still has to run of course.

[KJM]

It's probably bad etiquette to 'question bomb' someone elses thread but I'm gonna risk it. I tend to compartmentalise my python code using defs. The trouble with this is I end up with an inconveniently long main.py full of mostly defs which see me do a lot of scrolling. I've been pondering what to do about this? I was thinking maybe put some of the stable defs into a lib & import them instead? From jimmo's comment it seems maybe there is going to be an overhead penalty to pay if I do this?

[jimmo]

From jimmo's comment it seems maybe there is going to be an overhead penalty to pay if I do this?

No, there's no penalty specifically to importing modules. More that the time to boot scales with total amount of code.

So having 100 defs in one file, or 10 files of 10 defs each, will be the same.