Memorry allocation issues with the HUZZAH

[binux]

Hello everyone,

I'm currently building a small tool with the huzzah (adafruit feather version) which communicates with an OBD-II adaptor via wifi.
I have a neopixel led strip aswell which is live showing me the information I gather. Now that the project is soon to be finalized I'm running into an annoying issue:

Code: Select all

MemoryError: memory allocation failed, allocating 158 bytes 

The error happens on when I import my self made sub-modules into the main.py. Below I attached the init sequences so you can make yourself a picture of it. The strange thing is, that the allocation error does not always happen. If I reimport the script and reboot, sometimes it works without any issue. But the moment I change or add stuff to any submodule I get this issue again.

I guess that this issue is based on the variables I initialize in the init part of my sub-modules, but I don't know how I could strip them down to reduce their size (Web-Repl is also disabled). Can anyone help me making this more memory effecient? Thanks a lot.

*BTW if any admin could enable BBCode for this post, would be great...

Above error happens either at the import of "display" or at the import of "elm".

main.py init:

Code: Select all

from time import sleep_ms, ticks_ms
import ujson as json
from states import MPyState, ELMState
from display import Display
from elm import ELM
import network
import machine

display.py init:

Code: Select all

from machine import Pin
from machine import Timer
from time import ticks_ms
from neopixel import NeoPixel
from ema import EMA
from states import MPyState
from constants import Car
import urandom
import math
import machine

class Display:
    def __init__(self, numLeds=10, brightnessReadTimeout=10000, pin=0):
        dataPin = Pin(pin, Pin.OUT)
        self.numLeds = numLeds
        self.pixels = [(0,0,0)] * numLeds
        self.display = NeoPixel(dataPin, numLeds)
        self.way = [False] * numLeds
        for i, led in enumerate(self.pixels):
            rand = urandom.getrandbits(8)
            self.pixels[i] = (0,0,rand)
        
        self.state = MPyState.INIT
        self.lastState = MPyState.INIT
        self.isSwitching = False

        self.delayCounter = 0
        self.position = 0
        self.mustWait = False

        self.adc = machine.ADC(0)
        self.ema = EMA()
        self.brightness = 100
        self.brightnessDelay = ticks_ms()
        self.brightnessReadTimeout = brightnessReadTimeout
        self.displayBrightness = 100

        self.value = -255
        self.timer = Timer(-1)
        self.timer.init(period=33, mode=Timer.PERIODIC, callback=self.refresh)
        self.show()

elm.py init:

Code: Select all

from states import BufferState, ELMState
from utime import sleep_ms, ticks_ms
from ema import EMA
import math
import socket

class ELM:
    def __init__(self, adress="192.168.0.10", port=35000):
        self.adress = adress
        self.port = port
        self.elmSocket = socket.socket()

        self.bufferState = BufferState.NONE
        self.elmState = ELMState.INIT

        self.rawBarometricPressure = -1
        self.rawManifoldAbsolutePressure = -1
        self.boostPressure = -1

        self.step = 0

        self.ema = EMA()

        self.waitingTimer = 0
        self.barometricTimer = 0
        self.mapTimer = 0

        self.errors = 0
        print("Initialized ELM at \"{0}\" with port {1}".format(adress,port))

[Roberthh]

The heap size on the ESP8266 is small, which limits the size of code which can be imported directly. Ways around:
a) precompile your code
b) frozen bytecode

a) Precompiling: for that, you use mpy-cross to convert your .py files input a compiled version with extension .mpy. That helps sometimes, if the error occurs during import.
b) Frozen bytecode. That requires you to build your own image. You have to clone the micropython directory, and once you can successfully build and flash the image, add your code to esp8266/modules and rebuild the image. You code will be placed and executed in flash, and RAM is only used for data.

[binux]

The heap size on the ESP8266 is small, which limits the size of code which can be imported directly. Ways around:
a) precompile your code
b) frozen bytecode

a) Precompiling: for that, you use mpy-cross to convert your .py files input a compiled version with extension .mpy. That helps sometimes, if the error occurs during import.
b) Frozen bytecode. That requires you to build your own image. You have to clone the micropython directory, and once you can successfully build and flash the image, add your code to esp8266/modules and rebuild the image. You code will be placed and executed in flash, and RAM is only used for data.

Ok. So the first thing I should try is to precompile. Is there a guide on how to create frozen bytecode?

[Roberthh]

Ok. So the first thing I should try is to precompile. Is there a guide on how to create frozen bytecode?

In any way, you have to set up the build environment for micropython. This is easiest don on a Linux system, although Mac OS x also works. You have to clode the micropython sources from https://github.com/micropython/micropython and follow the instructions in both the to level README.md and esp8266/README.md. Please follow it literally. Most problems occur if you miss a step. You have also to clone the espressif idf, like told in README. Once you are able to re-build the firmware image and flash it to you board, you can add you own code. This is done by placing your code into the directory esp8266/modules and rebuilding the firmware. The directory already exists, and it contains already some of the modules you are using in you code, like neopixel.py.
Once everything is set up, making and flashing an image is merely executing the commands:
make
make deploy

[binux]

Ok. So the first thing I should try is to precompile. Is there a guide on how to create frozen bytecode?

In any way, you have to set up the build environment for micropython. This is easiest don on a Linux system, although Mac OS x also works. You have to clode the micropython sources from https://github.com/micropython/micropython and follow the instructions in both the to level README.md and esp8266/README.md. Please follow it literally. Most problems occur if you miss a step. You have also to clone the espressif idf, like told in README. Once you are able to re-build the firmware image and flash it to you board, you can add you own code. This is done by placing your code into the directory esp8266/modules and rebuilding the firmware. The directory already exists, and it contains already some of the modules you are using in you code, like neopixel.py.
Once everything is set up, making and flashing an image is merely executing the commands:
make
make deploy

Okay I'm currently building it. I later want to allow other people to run my code on their ESP so obviously I will have to provide them the modified firmware. However what came in my mind was: Is it possible to prepopulate the filesystem with my .py files like the "main.py"? I would like to make it as easy as possible for others to get my software running.

[pythoncoder]

I dealt with this problem by modifying _boot.py as follows. In my case the application to run on startup is mqtt.py:

Code: Select all

import gc
gc.threshold((gc.mem_free() + gc.mem_alloc()) // 4)
import uos
from flashbdev import bdev

try:
    if bdev:
        uos.mount(bdev, '/')
except OSError:
    import inisetup
    inisetup.setup()

try:
    uos.stat('/main.py')
except OSError:
    with open("/main.py", "w") as f:
        f.write("""\
import mqtt
""")

gc.collect()

You would normally only need a trivial main.py like mine in the filesystem: every other module (including _boot.py) can be frozen.

[binux]

I dealt with this problem by modifying _boot.py as follows. In my case the application to run on startup is mqtt.py:

Code: Select all

import gc
gc.threshold((gc.mem_free() + gc.mem_alloc()) // 4)
import uos
from flashbdev import bdev

try:
    if bdev:
        uos.mount(bdev, '/')
except OSError:
    import inisetup
    inisetup.setup()

try:
    uos.stat('/main.py')
except OSError:
    with open("/main.py", "w") as f:
        f.write("""\
import mqtt
""")

gc.collect()

You would normally only need a trivial main.py like mine in the filesystem: every other module (including _boot.py) can be frozen.

My issue is I have not only a main.py but also a settings.json where all settings arestored so nobody has to actually fiddle with the code. How can I get this file copied over inside the firmware?

[Roberthh]

Like in the template of @pythoncoder, you can embed into frozen bytecode a python script which creates the files you need in the file system. If that is called for instance "create_templates.py", you would use "import create_templates" instead of the code section which creates main.py (or in addition to...).
Without having digged into further, I have a faint impression that the support of packages made by @pfalcon in one of the last commits also points into the direction you need.

[binux]

Like in the template of @pythoncoder, you can embed into frozen bytecode a python script which creates the files you need in the file system. If that is called for instance "create_templates.py", you would use "import create_templates" instead of the code section which creates main.py (or in addition to...).
Without having digged into further, I have a faint impression that the support of packages made by @pfalcon in one of the last commits also points into the direction you need.

Okay that means I have no other choice than to create the files from scratch inside the boot.py (as you two suggested). I was hoping that the technique was a bit more straight forward like with the modules where you have a folder which correlates to the root of the filesystem. Anyhow thank you for the suggestions.

Maybe I also find a better tool to transfer the files onto the esp. I currently use the ampy package suggested by adafruit but it looks like it didn't get too much love in the past and I had cases where it just didn't work.

[Roberthh]

You can also use webrepl to transfer files, and there are also two variants of an ftp server here: https://github.com/robert-hh/ESP8266-FTP-Server, which you might give a look.