[workaround found] ESP8266 - use of Timer without storing reference leads to Fatal exception 28(LoadProhibitedCause)

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
jmi2
Posts: 21
Joined: Mon Oct 01, 2018 9:31 pm

[workaround found] ESP8266 - use of Timer without storing reference leads to Fatal exception 28(LoadProhibitedCause)

Post by jmi2 » Thu Jan 31, 2019 5:55 pm

Updated:
If Timer is used and it's reference is not saved on place, where it's still reachable, then expect errors.
Such timer is then subject for garbage collection and will be removed. Unfortunatelly underlaying implementation is still trying to reference this memory, so it leads to Fatal exception 28 or 9 or 0 (these 3 i saw during elaboration for reason of error).

initial post:

Hello,

on ESP8266 i hit last days errors like

Fatal exception 28(LoadProhibitedCause):
epc1=0x401014b0, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000008, depc=0x00000000

While trying to figure out, what could be a reason, by removing parts of code etc, i realized, that it's problem, if i have at least two timers.
I started with 3 timers:
1. polling for incomming TCP connections
2. reading GPIO PINs and sometimes sending UDP packet or creating/deleting 3rd timer.
3. this one is dynamically initialized and deinitialized from 2nd one under certain conditions. On trigger it also sends UDP packet.

i'm on v1.9.4-272-g46091b8a on 2018-07-18
as with v1.10-8-g8b7039d7d (stable) and v1.10-23-g1fa8f977f (debug) i cannot get TCP polling working. See other thread.

Combination of all 3 leads to error described above.
Attempt 1: I disabled 1st timer. Then it lead to crashes somewhen after condition created/destroyed and triggered few timers of type 3.
Attempt 2: like attempt 1, but additionnaly timer3 was replaced with triggering from timer2 (timer 2 triggers each 10ms, timer 3 each 500ms,hence i fired 3rd one on each 50th run if flag, that timer should be active is set). This worked fine.
Attempt 3: like attempt 2 + reenabled timer 1. Again Fatal exception 28.
Attempt 4: like 2 + and logic behind timer 1 executed on each 20th execution of timer2 instead. Hence only one timer left. This also works fine. This time i also added jmeter to create 1 req per second to TCP port. No problem at all so far.

so far my outcome is so far, that two timers are problematic, if at least one of them is sending UDP packet.
Going to try to make minimalistic code with two timers, which fails.

Is there any limitations, what could Timer callback do on ESP8266?
As i know, they are soft interrupts, aren't they?
And according to my past tests, execution of one soft timer interrupt is never interrupted by another soft interrupt execution. Was my observation correct?

thank you a lot
best regards
jano
Last edited by jmi2 on Sun Feb 03, 2019 6:23 pm, edited 1 time in total.

jmi2
Posts: 21
Joined: Mon Oct 01, 2018 9:31 pm

Re: 8266 - use of multiple timers leads to Fatal exception 28(LoadProhibitedCause)

Post by jmi2 » Fri Feb 01, 2019 3:23 pm

Hi,

i tested on two boards so far.
Came to this as more-less minimal code causing issues. It's not even using GPIO and NETWORK. Only timers and print function.
There are inline comments telling, what changes will lead to "fixing" it, but they are so trivial, that i don't understand, why they should change behavior.

Any ideas?

Code below is failing after 100-300 iterations.

Code: Select all

from machine import Timer

eventSeq = 0

class MyClass(object):
    timer1 = Timer(-1)

    def __init__(self):
        self.timer1.init(period=5, mode=Timer.PERIODIC, callback=self.onTimer) # if longer period used, it fails on same iteration, only takes longer to simulate.

    def timer1Callback(self, timer):
        if self == None: # if removed, then takes randomly 4-6x more timer events until it fails.
            pass

    def onTimer(self, timer):
        global eventSeq
        eventSeq += 1
        print(str(eventSeq)) # if replaced with print(eventSeq) then it works.
        # print(eventSeq)


switch = MyClass()

# timer2 = None

def initTimer2():
    global switch
    print('some dummy message perhaps eating little memory') # if removed, then it randomly takes ~ 10x longer to fail
    # global timer2 # if variable made global (initialization can stay here), it starts working.
    timer2 = Timer(-1)
    timer2.init(period=10000, mode=Timer.PERIODIC, callback=switch.timer1Callback) # if replaced with switch.periodicPinCheck then it works.


initTimer2()


thank you a lot
best regards
jmi
Last edited by jmi2 on Fri Feb 01, 2019 3:40 pm, edited 1 time in total.

jmi2
Posts: 21
Joined: Mon Oct 01, 2018 9:31 pm

Re: 8266 - use of multiple timers leads to Fatal exception 28(LoadProhibitedCause)

Post by jmi2 » Fri Feb 01, 2019 3:39 pm

problem with same code exists also on latest v1.10-45-g67689bfd7 on 2019-02-01;
and also on oldest avaiable: esp8266-20170108-v1.8.7.bin

jmi2
Posts: 21
Joined: Mon Oct 01, 2018 9:31 pm

Re: 8266 - use of multiple timers leads to Fatal exception 28(LoadProhibitedCause)

Post by jmi2 » Fri Feb 01, 2019 4:45 pm

strage is, that error code (0 - IllegalInstructionCause or 9 -LoadStoreAlignmentCause) depends on presense on print command.

Code: Select all

from machine import Timer

class MyClass(object):
    timer1 = Timer(-1)
    eventSeq = 0

    def __init__(self):
        self.timer1.init(period=20, mode=Timer.PERIODIC, callback=self.onTimer) # if longer period used, it fails on same iteration, only takes longer to simulate.

    def onTimer(self, timer):
        self.eventSeq += 1
        print(str(self.eventSeq)) # if replaced with print(eventSeq) then it works.
        # print(self.eventSeq)


switch = MyClass()


def initTimer2():
    print('some dummy message perhaps eating little memory') # if removed, then it randomly takes ~ 10x longer to fail
    # global timer2 # if variable made global (initialization can stay here), it starts working.
    timer2 = Timer(-1)
    timer2.init(period=10000, mode=Timer.ONE_SHOT, callback=None) # if replaced with switch.periodicPinCheck then it works.


initTimer2()
leads to

Code: Select all

190
191
192
Fatal exception 0(IllegalInstructionCause):
epc1=0x4027db0d, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000000, depc=0x00000000

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x40100000, len 30964, room 16
tail 4
chksum 0x02
load 0x3ffe8000, len 1080, room 4
tail 4
chksum 0xc6
load 0x3ffe8440, len 808, room 4
tail 4
chksum 0xa2
csum 0xa2
while same code without printing static text as first statement of initTimer2 function:

Code: Select all

from machine import Timer

class MyClass(object):
    timer1 = Timer(-1)
    eventSeq = 0

    def __init__(self):
        self.timer1.init(period=20, mode=Timer.PERIODIC, callback=self.onTimer) # if longer period used, it fails on same iteration, only takes longer to simulate.

    def onTimer(self, timer):
        self.eventSeq += 1
        print(str(self.eventSeq)) # if replaced with print(eventSeq) then it works.
        # print(self.eventSeq)


switch = MyClass()


def initTimer2():
    # print('some dummy message perhaps eating little memory') # if removed, then it randomly takes ~ 10x longer to fail
    # global timer2 # if variable made global (initialization can stay here), it starts working.
    timer2 = Timer(-1)
    timer2.init(period=10000, mode=Timer.ONE_SHOT, callback=None) # if replaced with switch.periodicPinCheck then it works.


initTimer2()
leads to

Code: Select all

194
195
Fatal exception 9(LoadStoreAlignmentCause):
epc1=0x40101565, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00006dfe, depc=0x00000000

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x40100000, len 30964, room 16
tail 4
chksum 0x02
load 0x3ffe8000, len 1080, room 4
tail 4
chksum 0xc6
load 0x3ffe8440, len 808, room 4
tail 4
chksum 0xa2
csum 0xa2
both from same board on MicroPython v1.10-45-g67689bfd7 on 2019-02-01; ESP module with ESP8266

jmi2
Posts: 21
Joined: Mon Oct 01, 2018 9:31 pm

Re: 8266 - use of multiple timers leads to Fatal exception 28(LoadProhibitedCause)

Post by jmi2 » Fri Feb 01, 2019 4:55 pm

even simpler case, which is failing on Fatal exception 0(IllegalInstructionCause):

Code: Select all

from machine import Timer

def onTimer1(timer):
    print(str(123))

Timer(-1).init(period=10, mode=Timer.PERIODIC, callback=onTimer1)

Timer(-1).init(period=10000, mode=Timer.ONE_SHOT, callback=None)

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: 8266 - use of multiple timers leads to Fatal exception 28(LoadProhibitedCause)

Post by pythoncoder » Sat Feb 02, 2019 9:47 am

Running timers at the REPL can be confusing. This sample is more representative of a real application and it runs indefinitely here.

Code: Select all

from machine import Timer

def onTimer1(timer):
    print(str(123))

def foo(_):
    print('******* second timer *******')

t1 = Timer(-1)
t1.init(period=10, mode=Timer.PERIODIC, callback=onTimer1)

t2 = Timer(-1)
t2.init(period=10000, mode=Timer.ONE_SHOT, callback=foo)

try:
    while True:
        pass  # Application code here!
except KeyboardInterrupt:
    for t in (t1, t2):
        t.deinit()
Note that you can interrupt it with ctrl-c.
Peter Hinch
Index to my micropython libraries.

jmi2
Posts: 21
Joined: Mon Oct 01, 2018 9:31 pm

Re: 8266 - use of multiple timers leads to Fatal exception 28(LoadProhibitedCause)

Post by jmi2 » Sat Feb 02, 2019 6:26 pm

Thanks Peter, after your message, i started to have feeling, that REPL is a problem.

But after some more tests, i have a feeling, that problem is elsewhere.

Actually I suspect, that there is problem with Timers, which are not reachable from user-code anymore and hence are subject for garbage collection.
I guess, when garbage collection runs, they are removed, but layer below python (RTOS or something) probably still tries to access their memory. Hence errors 0, 9, 28 for very similar scenarios.

This would explain, why my tests are working, if i define timer variable as global (module global), but not working if they are in local variable only or not assigned to variable at all.

Update: so far i can only confirm this. I took my real world project, where i experienced these fatal errors and after little adaptations to keep references to timers, and it started to work like a charm.

Is it somewhere documented, or should it be reported as bug, that initialized (running) Timers are causing issue, if all references to them are lost?

Btw, Below is fragment of code, where timers are not referenced and GC is executed after 10s. Once GC runs, it crashes very soon (8 iterations, maybe already in execution queue in moment, when GC is executed):

Code: Select all

from machine import Timer

evt = 0

def onTimer1(timer):
    global evt
    evt += 1
    print(str(evt))

def foo(_):
    print('******* second timer *******')
    import gc
    gc.collect()

# t1 = Timer(-1)
# t1.init(period=10, mode=Timer.PERIODIC, callback=onTimer1)
#
# t2 = Timer(-1)
# t2.init(period=10000, mode=Timer.ONE_SHOT, callback=foo)


Timer(-1).init(period=10, mode=Timer.PERIODIC, callback=onTimer1)
Timer(-1).init(period=10000, mode=Timer.ONE_SHOT, callback=foo)

try:
    while True:
        pass  # Application code here!
except KeyboardInterrupt:
    # t1.deinit()
    # t2.deinit()
    pass

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: 8266 - use of multiple timers leads to Fatal exception 28(LoadProhibitedCause)

Post by pythoncoder » Sun Feb 03, 2019 2:21 pm

The garbage collector works as you describe. It's therefore wise to ensure that there is always a reference to any object you create. This extends beyond timers: there are many cases where the creation of an anonymous object puts it at risk of deletion. Consider this code fragment:

Code: Select all

class HeartBeat:
    def __init__(self):
        loop = asyncio.get_event_loop()
        loop.create_task(self.run())

    async def run(self):
        led = pyb.LED(1)
        while True:
            asyncio.sleep(1)
            led.toggle()

def foo():
    HeartBeat()
When you run foo() a HeartBeat is instantiated but there is no reference to it. I assume it is therefore at risk of GC. I take the precaution of assigning such objects to an object (e.g. a set) with an appropriate lifetime.
Peter Hinch
Index to my micropython libraries.

jmi2
Posts: 21
Joined: Mon Oct 01, 2018 9:31 pm

Re: 8266 - use of multiple timers leads to Fatal exception 28(LoadProhibitedCause)

Post by jmi2 » Sun Feb 03, 2019 4:24 pm

As uasyncio is pure python implementation as i know and hence believe, that there won't be any problem.
I expect asyncio will make reference to self.run method, which must hold reference to context and hence to object instance itself. Hence there i don't see a problem.

Problem is, when reference it not existing in python, but still exists in underlaying C++/RTOS code.

Opened ticket for this: https://github.com/micropython/micropython/issues/4458

tylersuard
Posts: 9
Joined: Mon Jan 21, 2019 4:09 pm

Re: [workaround found] ESP8266 - use of Timer without storing reference leads to Fatal exception 28(LoadProhibitedCause)

Post by tylersuard » Mon Feb 04, 2019 2:17 am

Hello. I'm trying to make it so when a sensor on my board is triggered, a red LED flashes in a pattern. And when a different sensor is triggered, a yellow LED flashes in a different pattern. I'm having a hard time making them both go at the same time, and loop until the sensor is no longer triggered. Help please? Here is my code:

import uasyncio as asyncio
import machine
import time
import os

# Yellow LED: Pin 15
Y_LED = machine.PWM(machine.Pin(15))
Y_LED.freq(300)

# Red LED: Pin 2
R_LED = machine.PWM(machine.Pin(2))
R_LED.freq(300)

H2_1 = machine.Pin(9, machine.Pin.IN, machine.Pin.PULL_UP)

H2_2 = machine.Pin(4, machine.Pin.IN, machine.Pin.PULL_UP)

async def FlashRed():
for i in range (20):
R_LED.duty(1023)
await asyncio.sleep(0.5)
R_LED.duty(0)
await asyncio.sleep(0.5)

async def FlashYellow():
for i in range (10):
Y_LED.duty(1023)
await asyncio.sleep(1)
Y_LED.duty(0)
await asyncio.sleep(1)

loop = asyncio.get_event_loop()

def Check_Button():
while True:
if H2_1.value():
loop.create_task(FlashRed())
loop.run_forever()
if H2_2.value():
loop.create_task(FlashYellow())
loop.run_forever()
time.sleep(0.1)

Check_Button()

Post Reply