Debugging and crash analysis techniques

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Post Reply
rx8pilot
Posts: 12
Joined: Fri Mar 26, 2021 3:39 am

Debugging and crash analysis techniques

Post by rx8pilot » Fri Mar 26, 2021 3:54 am

New to micropython and loving it so far.

I come from a C world for embedded work and have been noodling in Python for about a year for desktop utilities. As I have made a few projects in microPython on ESP32 (SparkFun ESP32 Thing) - I have discovered that I have no idea how to best learn and approach low-ish debugging that crashes the system.

Initially, I was going to post something about the specific issue I am having, but then I decided it would be more helpful to learn how to figure it out on my own. The crashy code at unpredictable times gives me this....

Code: Select all

Guru Meditation Error: Core  0 panic'ed (LoadProhibited)
. Exception was unhandled.
Core 0 register dump:
PC      : 0x400dd39a  PS      : 0x00060d30  A0      : 0x800dbe3c  A1      : 0x3ffb6480  
A2      : 0x00000000  A3      : 0x3ffb6a64  A4      : 0x00000fff  A5      : 0xffffffff
A6      : 0x00000001  A7      : 0x00000e82  A8      : 0x3f40a064  A9      : 0x3ffb6470  
A10     : 0x00000000  A11     : 0x00000004  A12     : 0x00000000  A13     : 0x3ffbf774
A14     : 0x3ffbf788  A15     : 0x3ffbf788  SAR     : 0x00000006  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x00000000  LBEG    : 0x4000c2e0  LEND    : 0x4000c2f6  LCOUNT  : 0xffffffff

Backtrace: 0x400dd39a:0x3ffb6480 0x400dbe39:0x3ffb64a0 0x400dbe8d:0x3ffb64c0 0x400dcd84:0x3ffb64f0 0x4008dda5:0x3ffb6570 0x400e9305:0x3ffb6590 0x400dfbe8:0x3ffb6630 0x400dbe49:0x3ffb6660 0x400dbeb1:0x3ffb6680 0x400e8f0d:0x3ffb66a0 0x400dfbe8:0x3ffb6740 0x400dbe49:0x3ffb67b0 0x400dbeb1:0x3ffb67d0 0x400e8f0d:0x3ffb67f0 0x400dfbe8:0x3ffb6890 0x400dbe49:0x3ffb68c0 0x400dbe76:0x3ffb68e0 0x400f872a:0x3ffb6900 0x400f8981:0x3ffb69a0 0x400ef907:0x3ffb69c0

Rebooting...
It happens at various places in the code, and totally unpredictable as far as I can tell. Using interrupts and timers along with MQQT module called umqttsimple. Trying to figure out how to best understand the exception output and perhaps some reading or video resources to better educate myself on the lower levels of micropython debugging.

Hoping to eventually use this platform for a commercial product if I can get the stability I have grown accustomed to when designing firmware with C.

Looking forward to any suggestions for good learning resources.

User avatar
russ_h
Posts: 88
Joined: Thu Oct 03, 2019 2:26 am
Contact:

Re: Debugging and crash analysis techniques

Post by russ_h » Mon Mar 29, 2021 9:07 pm

Take a look at https://github.com/tve/esp32-backtrace, it has helped me track down a few issues with my code.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Debugging and crash analysis techniques

Post by pythoncoder » Tue Mar 30, 2021 8:52 am

As a general point I have never attempted to debug those crash traces as they indicate that the MicroPython interpreter or the FreeRTOS runtime have crashed. If you can produce a minimal Python script which exhibits the crash, you have found a bug which can be reported on GitHub. Of course you are free to pursue this further, find a fix, and raise a PR against the firmware.

I think it's worth bearing in mind a few limitations. One is that ESP32's do occasionally reboot. SPIRAM versions are also known to be unstable. That said, non-SPIRAM chips should run for days between reboots, and should not crash. A repeated crash does suggest a problem, but it could be hardware or power supply issues. Standard Python code should not crash.

Finally the official MQTT versions have serious limitations discussed here.
Peter Hinch
Index to my micropython libraries.

rx8pilot
Posts: 12
Joined: Fri Mar 26, 2021 3:39 am

Re: Debugging and crash analysis techniques

Post by rx8pilot » Tue Apr 06, 2021 2:11 am

Interesting notes.....

I think I will be looking closely at the power first. Power electronics and high-speed PDN design is one of my specialties - easy to take a look. The board I am using is a SparkFun Thing which does not appear to make any major efforts in clean power.

Also looking closer at the MQTT limitations linked.

Perhaps just doing WiFi tests alone, push-button tests alone......and then add the MQTT to see if it only crashes with MQTT running.

Old school - code it and try it method.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Debugging and crash analysis techniques

Post by pythoncoder » Tue Apr 06, 2021 6:38 am

Typical power problems arise because ESP32 current consumption can rise to about 0.5A which can cause voltage drops in USB leads. Another cause are lousy wall-warts designed for phone charging rather than continuous running.
Peter Hinch
Index to my micropython libraries.

modulusmath
Posts: 30
Joined: Thu Jun 30, 2022 3:21 am

Re: Debugging and crash analysis techniques

Post by modulusmath » Sat Aug 20, 2022 1:24 am

pythoncoder wrote:
Tue Apr 06, 2021 6:38 am
Typical power problems arise because ESP32 current consumption can rise to about 0.5A which can cause voltage drops in USB leads. Another cause are lousy wall-warts designed for phone charging rather than continuous running.
(To anyone reading this post :D ):

Are there any specific USB cables and/or power supplies that are "known good" in this regard?

I've youtubed this and there's general advice, but links to buy high quality items are like gold for possible problems like this.

I am a big fan of pragmatically using things others know are of quality.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Debugging and crash analysis techniques

Post by pythoncoder » Sun Aug 21, 2022 8:28 am

Official Raspberry Pi power supplies are designed for continuous use and have plenty of capacity.
Peter Hinch
Index to my micropython libraries.

Post Reply