Page 1 of 2

[workaround found] ESP8266 - use of Timer without storing reference leads to Fatal exception 28(LoadProhibitedCause)

Posted: Thu Jan 31, 2019 5:55 pm
by jmi2
Updated:
If Timer is used and it's reference is not saved on place, where it's still reachable, then expect errors.
Such timer is then subject for garbage collection and will be removed. Unfortunatelly underlaying implementation is still trying to reference this memory, so it leads to Fatal exception 28 or 9 or 0 (these 3 i saw during elaboration for reason of error).

initial post:

Hello,

on ESP8266 i hit last days errors like

Fatal exception 28(LoadProhibitedCause):
epc1=0x401014b0, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000008, depc=0x00000000

While trying to figure out, what could be a reason, by removing parts of code etc, i realized, that it's problem, if i have at least two timers.
I started with 3 timers:
1. polling for incomming TCP connections
2. reading GPIO PINs and sometimes sending UDP packet or creating/deleting 3rd timer.
3. this one is dynamically initialized and deinitialized from 2nd one under certain conditions. On trigger it also sends UDP packet.

i'm on v1.9.4-272-g46091b8a on 2018-07-18
as with v1.10-8-g8b7039d7d (stable) and v1.10-23-g1fa8f977f (debug) i cannot get TCP polling working. See other thread.

Combination of all 3 leads to error described above.
Attempt 1: I disabled 1st timer. Then it lead to crashes somewhen after condition created/destroyed and triggered few timers of type 3.
Attempt 2: like attempt 1, but additionnaly timer3 was replaced with triggering from timer2 (timer 2 triggers each 10ms, timer 3 each 500ms,hence i fired 3rd one on each 50th run if flag, that timer should be active is set). This worked fine.
Attempt 3: like attempt 2 + reenabled timer 1. Again Fatal exception 28.
Attempt 4: like 2 + and logic behind timer 1 executed on each 20th execution of timer2 instead. Hence only one timer left. This also works fine. This time i also added jmeter to create 1 req per second to TCP port. No problem at all so far.

so far my outcome is so far, that two timers are problematic, if at least one of them is sending UDP packet.
Going to try to make minimalistic code with two timers, which fails.

Is there any limitations, what could Timer callback do on ESP8266?
As i know, they are soft interrupts, aren't they?
And according to my past tests, execution of one soft timer interrupt is never interrupted by another soft interrupt execution. Was my observation correct?

thank you a lot
best regards
jano

Re: 8266 - use of multiple timers leads to Fatal exception 28(LoadProhibitedCause)

Posted: Fri Feb 01, 2019 3:23 pm
by jmi2
Hi,

i tested on two boards so far.
Came to this as more-less minimal code causing issues. It's not even using GPIO and NETWORK. Only timers and print function.
There are inline comments telling, what changes will lead to "fixing" it, but they are so trivial, that i don't understand, why they should change behavior.

Any ideas?

Code below is failing after 100-300 iterations.

Code: Select all

from machine import Timer

eventSeq = 0

class MyClass(object):
    timer1 = Timer(-1)

    def __init__(self):
        self.timer1.init(period=5, mode=Timer.PERIODIC, callback=self.onTimer) # if longer period used, it fails on same iteration, only takes longer to simulate.

    def timer1Callback(self, timer):
        if self == None: # if removed, then takes randomly 4-6x more timer events until it fails.
            pass

    def onTimer(self, timer):
        global eventSeq
        eventSeq += 1
        print(str(eventSeq)) # if replaced with print(eventSeq) then it works.
        # print(eventSeq)


switch = MyClass()

# timer2 = None

def initTimer2():
    global switch
    print('some dummy message perhaps eating little memory') # if removed, then it randomly takes ~ 10x longer to fail
    # global timer2 # if variable made global (initialization can stay here), it starts working.
    timer2 = Timer(-1)
    timer2.init(period=10000, mode=Timer.PERIODIC, callback=switch.timer1Callback) # if replaced with switch.periodicPinCheck then it works.


initTimer2()


thank you a lot
best regards
jmi

Re: 8266 - use of multiple timers leads to Fatal exception 28(LoadProhibitedCause)

Posted: Fri Feb 01, 2019 3:39 pm
by jmi2
problem with same code exists also on latest v1.10-45-g67689bfd7 on 2019-02-01;
and also on oldest avaiable: esp8266-20170108-v1.8.7.bin

Re: 8266 - use of multiple timers leads to Fatal exception 28(LoadProhibitedCause)

Posted: Fri Feb 01, 2019 4:45 pm
by jmi2
strage is, that error code (0 - IllegalInstructionCause or 9 -LoadStoreAlignmentCause) depends on presense on print command.

Code: Select all

from machine import Timer

class MyClass(object):
    timer1 = Timer(-1)
    eventSeq = 0

    def __init__(self):
        self.timer1.init(period=20, mode=Timer.PERIODIC, callback=self.onTimer) # if longer period used, it fails on same iteration, only takes longer to simulate.

    def onTimer(self, timer):
        self.eventSeq += 1
        print(str(self.eventSeq)) # if replaced with print(eventSeq) then it works.
        # print(self.eventSeq)


switch = MyClass()


def initTimer2():
    print('some dummy message perhaps eating little memory') # if removed, then it randomly takes ~ 10x longer to fail
    # global timer2 # if variable made global (initialization can stay here), it starts working.
    timer2 = Timer(-1)
    timer2.init(period=10000, mode=Timer.ONE_SHOT, callback=None) # if replaced with switch.periodicPinCheck then it works.


initTimer2()
leads to

Code: Select all

190
191
192
Fatal exception 0(IllegalInstructionCause):
epc1=0x4027db0d, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000000, depc=0x00000000

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x40100000, len 30964, room 16
tail 4
chksum 0x02
load 0x3ffe8000, len 1080, room 4
tail 4
chksum 0xc6
load 0x3ffe8440, len 808, room 4
tail 4
chksum 0xa2
csum 0xa2
while same code without printing static text as first statement of initTimer2 function:

Code: Select all

from machine import Timer

class MyClass(object):
    timer1 = Timer(-1)
    eventSeq = 0

    def __init__(self):
        self.timer1.init(period=20, mode=Timer.PERIODIC, callback=self.onTimer) # if longer period used, it fails on same iteration, only takes longer to simulate.

    def onTimer(self, timer):
        self.eventSeq += 1
        print(str(self.eventSeq)) # if replaced with print(eventSeq) then it works.
        # print(self.eventSeq)


switch = MyClass()


def initTimer2():
    # print('some dummy message perhaps eating little memory') # if removed, then it randomly takes ~ 10x longer to fail
    # global timer2 # if variable made global (initialization can stay here), it starts working.
    timer2 = Timer(-1)
    timer2.init(period=10000, mode=Timer.ONE_SHOT, callback=None) # if replaced with switch.periodicPinCheck then it works.


initTimer2()
leads to

Code: Select all

194
195
Fatal exception 9(LoadStoreAlignmentCause):
epc1=0x40101565, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00006dfe, depc=0x00000000

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x40100000, len 30964, room 16
tail 4
chksum 0x02
load 0x3ffe8000, len 1080, room 4
tail 4
chksum 0xc6
load 0x3ffe8440, len 808, room 4
tail 4
chksum 0xa2
csum 0xa2
both from same board on MicroPython v1.10-45-g67689bfd7 on 2019-02-01; ESP module with ESP8266

Re: 8266 - use of multiple timers leads to Fatal exception 28(LoadProhibitedCause)

Posted: Fri Feb 01, 2019 4:55 pm
by jmi2
even simpler case, which is failing on Fatal exception 0(IllegalInstructionCause):

Code: Select all

from machine import Timer

def onTimer1(timer):
    print(str(123))

Timer(-1).init(period=10, mode=Timer.PERIODIC, callback=onTimer1)

Timer(-1).init(period=10000, mode=Timer.ONE_SHOT, callback=None)

Re: 8266 - use of multiple timers leads to Fatal exception 28(LoadProhibitedCause)

Posted: Sat Feb 02, 2019 9:47 am
by pythoncoder
Running timers at the REPL can be confusing. This sample is more representative of a real application and it runs indefinitely here.

Code: Select all

from machine import Timer

def onTimer1(timer):
    print(str(123))

def foo(_):
    print('******* second timer *******')

t1 = Timer(-1)
t1.init(period=10, mode=Timer.PERIODIC, callback=onTimer1)

t2 = Timer(-1)
t2.init(period=10000, mode=Timer.ONE_SHOT, callback=foo)

try:
    while True:
        pass  # Application code here!
except KeyboardInterrupt:
    for t in (t1, t2):
        t.deinit()
Note that you can interrupt it with ctrl-c.

Re: 8266 - use of multiple timers leads to Fatal exception 28(LoadProhibitedCause)

Posted: Sat Feb 02, 2019 6:26 pm
by jmi2
Thanks Peter, after your message, i started to have feeling, that REPL is a problem.

But after some more tests, i have a feeling, that problem is elsewhere.

Actually I suspect, that there is problem with Timers, which are not reachable from user-code anymore and hence are subject for garbage collection.
I guess, when garbage collection runs, they are removed, but layer below python (RTOS or something) probably still tries to access their memory. Hence errors 0, 9, 28 for very similar scenarios.

This would explain, why my tests are working, if i define timer variable as global (module global), but not working if they are in local variable only or not assigned to variable at all.

Update: so far i can only confirm this. I took my real world project, where i experienced these fatal errors and after little adaptations to keep references to timers, and it started to work like a charm.

Is it somewhere documented, or should it be reported as bug, that initialized (running) Timers are causing issue, if all references to them are lost?

Btw, Below is fragment of code, where timers are not referenced and GC is executed after 10s. Once GC runs, it crashes very soon (8 iterations, maybe already in execution queue in moment, when GC is executed):

Code: Select all

from machine import Timer

evt = 0

def onTimer1(timer):
    global evt
    evt += 1
    print(str(evt))

def foo(_):
    print('******* second timer *******')
    import gc
    gc.collect()

# t1 = Timer(-1)
# t1.init(period=10, mode=Timer.PERIODIC, callback=onTimer1)
#
# t2 = Timer(-1)
# t2.init(period=10000, mode=Timer.ONE_SHOT, callback=foo)


Timer(-1).init(period=10, mode=Timer.PERIODIC, callback=onTimer1)
Timer(-1).init(period=10000, mode=Timer.ONE_SHOT, callback=foo)

try:
    while True:
        pass  # Application code here!
except KeyboardInterrupt:
    # t1.deinit()
    # t2.deinit()
    pass

Re: 8266 - use of multiple timers leads to Fatal exception 28(LoadProhibitedCause)

Posted: Sun Feb 03, 2019 2:21 pm
by pythoncoder
The garbage collector works as you describe. It's therefore wise to ensure that there is always a reference to any object you create. This extends beyond timers: there are many cases where the creation of an anonymous object puts it at risk of deletion. Consider this code fragment:

Code: Select all

class HeartBeat:
    def __init__(self):
        loop = asyncio.get_event_loop()
        loop.create_task(self.run())

    async def run(self):
        led = pyb.LED(1)
        while True:
            asyncio.sleep(1)
            led.toggle()

def foo():
    HeartBeat()
When you run foo() a HeartBeat is instantiated but there is no reference to it. I assume it is therefore at risk of GC. I take the precaution of assigning such objects to an object (e.g. a set) with an appropriate lifetime.

Re: 8266 - use of multiple timers leads to Fatal exception 28(LoadProhibitedCause)

Posted: Sun Feb 03, 2019 4:24 pm
by jmi2
As uasyncio is pure python implementation as i know and hence believe, that there won't be any problem.
I expect asyncio will make reference to self.run method, which must hold reference to context and hence to object instance itself. Hence there i don't see a problem.

Problem is, when reference it not existing in python, but still exists in underlaying C++/RTOS code.

Opened ticket for this: https://github.com/micropython/micropython/issues/4458

Re: [workaround found] ESP8266 - use of Timer without storing reference leads to Fatal exception 28(LoadProhibitedCause)

Posted: Mon Feb 04, 2019 2:17 am
by tylersuard
Hello. I'm trying to make it so when a sensor on my board is triggered, a red LED flashes in a pattern. And when a different sensor is triggered, a yellow LED flashes in a different pattern. I'm having a hard time making them both go at the same time, and loop until the sensor is no longer triggered. Help please? Here is my code:

import uasyncio as asyncio
import machine
import time
import os

# Yellow LED: Pin 15
Y_LED = machine.PWM(machine.Pin(15))
Y_LED.freq(300)

# Red LED: Pin 2
R_LED = machine.PWM(machine.Pin(2))
R_LED.freq(300)

H2_1 = machine.Pin(9, machine.Pin.IN, machine.Pin.PULL_UP)

H2_2 = machine.Pin(4, machine.Pin.IN, machine.Pin.PULL_UP)

async def FlashRed():
for i in range (20):
R_LED.duty(1023)
await asyncio.sleep(0.5)
R_LED.duty(0)
await asyncio.sleep(0.5)

async def FlashYellow():
for i in range (10):
Y_LED.duty(1023)
await asyncio.sleep(1)
Y_LED.duty(0)
await asyncio.sleep(1)

loop = asyncio.get_event_loop()

def Check_Button():
while True:
if H2_1.value():
loop.create_task(FlashRed())
loop.run_forever()
if H2_2.value():
loop.create_task(FlashYellow())
loop.run_forever()
time.sleep(0.1)

Check_Button()