[SOLVED] Crashing/freezing every few hours

All ESP8266 boards running MicroPython.
Official boards are the Adafruit Huzzah and Feather boards.
Target audience: MicroPython users with an ESP8266 board.
User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Crashing/freezing every few hours

Post by pythoncoder » Thu Nov 15, 2018 8:12 am

If that fails you could mount the ESP8266 remote from the pump. If necessary have a second relay mounted close to the pump, whose coil is energised by a switched DC level from a small relay switched by the ESP8266. That way all the AC wiring is kept close to the pump. The ESP8266 is powered by power source on a separate AC supply.

That approach minimises the risk of conducted or radiated electromagnetic interference from the pump's AC circuit affecting the ESP8266.
Peter Hinch
Index to my micropython libraries.

mkiotdev
Posts: 10
Joined: Sun Nov 11, 2018 9:12 am

Re: Crashing/freezing every few hours

Post by mkiotdev » Mon Nov 19, 2018 6:52 pm

In general, after few days of testing, I have now, about 1 freezing/crashing per 24h. There is no 'error' in logs (condition that is catched by try as exception). This is also not acceptable, I want the ESP8266 to be able to control heating, that is pumps and mixing valves; two pairs of those (two pumps, two motorized mixing valves).

I do have three phase current; I could find socket to different line; but in general, this is not solution.

I am currently thinking in the lines of external WDT, something like DS1832. The thing is, I need something in SOIC packaging to be able to solder it. The other thing is, apparently DS1832 is more expensive than ESP8266.

DS1832:
https://datasheets.maximintegrated.com/en/ds/DS1832.pdf

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: Crashing/freezing every few hours

Post by jickster » Mon Nov 19, 2018 6:56 pm

mkiotdev wrote:
Mon Nov 19, 2018 6:52 pm
In general, after few days of testing, I have now, about 1 freezing/crashing per 24h. There is no 'error' in logs (condition that is catched by try as exception). This is also not acceptable, I want the ESP8266 to be able to control heating, that is pumps and mixing valves; two pairs of those (two pumps, two motorized mixing valves).

I do have three phase current; I could find socket to different line; but in general, this is not solution.

I am currently thinking in the lines of external WDT, something like DS1832. The thing is, I need something in SOIC packaging to be able to solder it. The other thing is, apparently DS1832 is more expensive than ESP8266.

DS1832:
https://datasheets.maximintegrated.com/en/ds/DS1832.pdf
Are you sure you're catching any gc issues?

Run an infinite while-loop inside a try-catch that merely allocates/deallocates random amount of memory.

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: Crashing/freezing every few hours

Post by Roberthh » Mon Nov 19, 2018 7:22 pm

I am currently thinking in the lines of external WDT, something like DS1832. The thing is, I need something in SOIC packaging to be able to solder it. The other thing is, apparently DS1832 is more expensive than ESP8266.
You could use a NE555 based circuit. Needs more external components, but is also more flexible in dimensioning. It is sold for 15-20 ct. in small quantities. Drawback: Vcc >= 4.5V

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: Crashing/freezing every few hours

Post by kevinkk525 » Mon Nov 19, 2018 9:16 pm

As we figured out, the software watchdog I posted would be enough to reset the frozen board. There's actually no need to put an external watchdog to the board.
My NodeMCUs freeze every 1 to 3 days which is not a problem if the program is designed to handle resets of any kind.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

mkiotdev
Posts: 10
Joined: Sun Nov 11, 2018 9:12 am

Re: Crashing/freezing every few hours

Post by mkiotdev » Fri Nov 30, 2018 4:45 pm

Kevin was right. I have been silent here on forum for last few days, almost 2 weeks. I have been testing. So this is what I have learned.

First 5-6 days, crashing/freezeng continued, and my ESP8266 froze about once a day, and allmost allways in the time when pumps and mixing valves were in full operation, about 30-60 mimutes after turning heating on.

Then, I have implemented simple software WDT. Here is the code:

Code: Select all

## Simple software WDT implementation
wdt_counter = 0

def wdt_callback():
    global wdt_counter
    wdt_counter += 1
    if (wdt_counter >= 10):
        machine.reset()

def wdt_feed():
    global wdt_counter
    wdt_counter = 0

wdt_timer = machine.Timer(-1)
wdt_timer.init(period=1000, mode=machine.Timer.PERIODIC, callback=lambda t:wdt_callback())
## END Simple software WDT implementation
Code was copied to the board on 24th this month; no freezing/crashin since then!

Well, to be more precise, I can say that now I have continued, uninterrupted operation for 5+ days. If board resets itself and works as expected after that, that is OK, fully acceptable. I have noticed few delays in data that board is sending, this could be due to the freezing; but since I have not implemented any logging, I have no way of knowing. Next step is to implement log. I wanted as simple software WDT solution as it can be, so that there is no extra part of code to blamed for any bad effects.

This should be put somewhere in official ESP8266 MicroPython documentation.

It would save me bunch of time and energy. I must say, that I dont understand what exactly happens, but it works. I don't like that, I'm engineer by training, so I would like to understand this situation. How come that code/board freezes, but timer succeeds to do callbacks? I have read in documentation that timers are "hardware" timers, so this "hardware" timer obviousely somehow succeeds to do function calls.

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: Crashing/freezing every few hours

Post by kevinkk525 » Fri Nov 30, 2018 5:53 pm

Good to hear from you again and that you got it under control. The simple wdt looks good.
I experience the same pattern, sometimes 5 days without a WDT reset, sometimes twice a day. Sometimes all microcontrollers freeze on the same day but never at the same time.
Could be something with the power supply, something with wifi and socket operations that freezes the esp-idf code executing the python main loop but still weird that Timers continued to work..
I did not get to the core of that problem even with extensive logging. Would be happy for an answer to your questions about the why too.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

deonis
Posts: 11
Joined: Wed Aug 08, 2018 3:02 am

Re: Crashing/freezing every few hours

Post by deonis » Fri Nov 30, 2018 7:03 pm

One thing I noticed for nodeMCU or esp12 boards if you leave both station and access point enabled you will get those freezes every day. I usually use 1.91 version of micropython for these boards, disable access point, and connect the ESPs to the router. Also, some routers do not like active socket connection and blacklist or disconnect boards, so you need to make sure to put your ESPs in "Trusted Devices". Another thing, as have discussed earlier, your power to ESPs should be very good. This is why I do not like boards with AMS1117 voltage regulators. The HT7333 is much better suited for esp82666. Please see Andreas video for more details: https://www.youtube.com/watch?v=wf_msvWv1jk&vl=en

I had situations when the voltage on esp8266 drops by 0.2-0.3 V and the board resets itself.

cheers,

Denis

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: Crashing/freezing every few hours

Post by kevinkk525 » Fri Nov 30, 2018 7:10 pm

I always disable AP and only use station.
As for power it would still not make any sense that this would result in a board freeze although timers still run and execute interrupt code. But it can of course reset the board, which would not be a problem in my case.
The problem with the routers you are describing should not be an issue if your code catches connection problems and should not freeze it.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Not crashing or freezing!

Post by pythoncoder » Sat Dec 01, 2018 7:55 am

It is possible to write ESP8266 applications which do not crash or freeze. The key is understanding why crashes occur. I wrote this guide to detail my findings.
Peter Hinch
Index to my micropython libraries.

Post Reply