ESP32 reliability

All ESP32 boards running MicroPython.
Target audience: MicroPython users with an ESP32 board.
User avatar
pythoncoder
Posts: 4678
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

ESP32 reliability

Post by pythoncoder » Mon Jul 27, 2020 6:00 pm

I have been known to criticise the ESP32 port on grounds of instability. The firmware has been substantially updated so myself and Kevin Köck decided to re-run our tests to assess its current stability.

We came to the same conclusions. The SPIRAM boards are still prone to crashing. They will sometimes run for a day or two, but in the end they crash.

Non-SPIRAM boards never crashed in our tests, but we both experienced occasional reboots. Our micropython-iot library provides for a software WDT, but we don't enable it in the tests (so that crashing is obvious). We suspect that the reboot is coming from FreeRTOS.

My test is particularly demanding hitting a socket with rapid asynchronous sequences of messages from both endpoints. It has now run for 10 days through 194 WiFi outages without message loss. Two spontaneous reboots occurred. The board is the reference board running a recent daily build. It's deliberately located in an area with poor WiFi connectivity. Formerly I had this test down as "not working" on ESP32 so the firmware is much improved.

Reboots could be a concern in some applications.
Peter Hinch

User avatar
rcolistete
Posts: 328
Joined: Thu Dec 31, 2015 3:12 pm
Location: Brazil
Contact:

Re: ESP32 reliability

Post by rcolistete » Tue Jul 28, 2020 2:40 am

pythoncoder wrote:
Mon Jul 27, 2020 6:00 pm
We came to the same conclusions. The SPIRAM boards are still prone to crashing. They will sometimes run for a day or two, but in the end they crash.

Non-SPIRAM boards never crashed in our tests, but we both experienced occasional reboots. Our
So flashing a MicroPython firmware without SPIRAM, e. g., esp32-idf3-20200728-unstable-v1.12-662-g8da40baa4.bin, on a ESP32 board with SPIRAM can have some advantages :
- faster to import .py/.mpy modules (as internal RAM is faster than SPIRAM);
- better stability.
My "MicroPython Samples". My "MicroPython Firmwares" with many options (double precision, ulab, etc).

User avatar
pythoncoder
Posts: 4678
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: ESP32 reliability

Post by pythoncoder » Tue Jul 28, 2020 5:11 am

That is highly likely but I haven't tested it.
Peter Hinch

User avatar
jimmo
Posts: 2015
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: ESP32 reliability

Post by jimmo » Tue Jul 28, 2020 6:38 am

Thanks Peter, that is very interesting, especially the link to SPIRAM.
pythoncoder wrote:
Mon Jul 27, 2020 6:00 pm
The board is the reference board running a recent daily build.
Which IDF version were you using?

kevinkk525
Posts: 782
Joined: Sat Feb 03, 2018 7:02 pm

Re: ESP32 reliability

Post by kevinkk525 » Tue Jul 28, 2020 9:40 am

I was testing the psram unit with idf4.0
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

User avatar
pythoncoder
Posts: 4678
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: ESP32 reliability

Post by pythoncoder » Wed Jul 29, 2020 5:07 am

My tests were with V3. I plan to repeat the test with V4 over a similar period.
Peter Hinch

User avatar
pythoncoder
Posts: 4678
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: ESP32 reliability

Post by pythoncoder » Mon Aug 03, 2020 6:07 am

I am repeating the test with idf V4 on the reference board with no SPIRAM.

It has been running for four days through 70 WiFi outages and has already clocked up three reboots. There would seem to be no improvement in this regard. I'll let the test run for 10 days to assess whether it has got worse in this regard.
Peter Hinch

User avatar
Mike Teachman
Posts: 94
Joined: Mon Jun 13, 2016 3:19 pm
Location: Victoria, BC, Canada

Re: ESP32 reliability

Post by Mike Teachman » Tue Aug 04, 2020 2:41 pm

Have you ruled out a voltage dip as the cause?

Consider: The resets you observe also fit the pattern of a voltage dip that brings a processor's VCC, flash memory, or SRAM to below spec, leading to a reset. The poor wifi environment in your test may cause the ESP32 to lose the wifi connection, followed by the ESP32 attempting a re-connection to wifi. The ESP32 draws the greatest current (>150mA) during the wifi connection phase. Could this high current draw during re-connection be dipping the voltage to the processor, flash, and SRAM?

If a voltage dip is suspected here are some root causes you might consider:

1. are you powering the dev board from a PC usb port, USB hub, or a dedicated PSU? Most dedicated PSUs provide a "stiff" power supply that minimize voltage drops as the current rises. USB sources are often not very stiff, leading to voltage drops as current increases.

2. are you powering your dev board using a usb cable? if so, what is the wire gauge of the usb cable? many USB cables are cheaply built, using 28AWG wire for the power and ground wires - this can result in a significant voltage drop when current rises.

User avatar
pythoncoder
Posts: 4678
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: ESP32 reliability

Post by pythoncoder » Wed Aug 05, 2020 7:26 am

I take extreme precautions against PSU problems, the whole test setup being isolated from mains power.

The power source is a 12V lead acid battery feeding a switch-mode converter. This provides a regulated 5V which feeds the USB port via a power-only USB lead, so the onboard linear regulator provides additional noise immunity.
Peter Hinch

User avatar
Mike Teachman
Posts: 94
Joined: Mon Jun 13, 2016 3:19 pm
Location: Victoria, BC, Canada

Re: ESP32 reliability

Post by Mike Teachman » Wed Aug 05, 2020 2:32 pm

Using a battery is an effective setup for eliminating noise effects from the AC mains power. But, it might not address the concerns of voltage drops leading into the supply voltage pins of various critical components on the dev board.

In the context of voltage drops I have a couple of other suggestions that might help you to find root cause:
- for the switch mode converter: are you confident that it does not show a voltage drop on the +5V output during the current surge associated with wifi transmission?
- for the USB power-only cable: have you measured the voltage drop across the cable going from the PSU to the dev board? ideally done with an oscilloscope, looking at the time when the ESP32 transmits over wifi
- what is the dropout voltage of the linear regulator on the dev board? often is it over 1 volt, and goes higher as current increases. This is a known contributor to the type of symptoms that you are seeing.

One test you could try uses an oscilloscope to measure the 3.3V output coming out of the dev board. Configure the oscilloscope to capture the 3.3V output when wifi transmits. What is the voltage drop? You can trigger the scope using a digital output initiated from your uPy program when wifi transmits. Or, trigger the scope to capture the +3.3V output at say < 2.5V which is where the ESP32 and the flash memory can start to exhibit problems.

Hope this helps!

Post Reply