ESP32 reliability

[pythoncoder]

I have been known to criticise the ESP32 port on grounds of instability. The firmware has been substantially updated so myself and Kevin Köck decided to re-run our tests to assess its current stability.

We came to the same conclusions. The SPIRAM boards are still prone to crashing. They will sometimes run for a day or two, but in the end they crash.

Non-SPIRAM boards never crashed in our tests, but we both experienced occasional reboots. Our micropython-iot library provides for a software WDT, but we don't enable it in the tests (so that crashing is obvious). We suspect that the reboot is coming from FreeRTOS.

My test is particularly demanding hitting a socket with rapid asynchronous sequences of messages from both endpoints. It has now run for 10 days through 194 WiFi outages without message loss. Two spontaneous reboots occurred. The board is the reference board running a recent daily build. It's deliberately located in an area with poor WiFi connectivity. Formerly I had this test down as "not working" on ESP32 so the firmware is much improved.

Reboots could be a concern in some applications.

[rcolistete]

We came to the same conclusions. The SPIRAM boards are still prone to crashing. They will sometimes run for a day or two, but in the end they crash.

Non-SPIRAM boards never crashed in our tests, but we both experienced occasional reboots. Our

So flashing a MicroPython firmware without SPIRAM, e. g., esp32-idf3-20200728-unstable-v1.12-662-g8da40baa4.bin, on a ESP32 board with SPIRAM can have some advantages :
- faster to import .py/.mpy modules (as internal RAM is faster than SPIRAM);
- better stability.

[pythoncoder]

That is highly likely but I haven't tested it.

[jimmo]

Thanks Peter, that is very interesting, especially the link to SPIRAM.

The board is the reference board running a recent daily build.

Which IDF version were you using?

[kevinkk525]

I was testing the psram unit with idf4.0

[pythoncoder]

My tests were with V3. I plan to repeat the test with V4 over a similar period.

[pythoncoder]

I am repeating the test with idf V4 on the reference board with no SPIRAM.

It has been running for four days through 70 WiFi outages and has already clocked up three reboots. There would seem to be no improvement in this regard. I'll let the test run for 10 days to assess whether it has got worse in this regard.

[Mike Teachman]

Have you ruled out a voltage dip as the cause?

Consider: The resets you observe also fit the pattern of a voltage dip that brings a processor's VCC, flash memory, or SRAM to below spec, leading to a reset. The poor wifi environment in your test may cause the ESP32 to lose the wifi connection, followed by the ESP32 attempting a re-connection to wifi. The ESP32 draws the greatest current (>150mA) during the wifi connection phase. Could this high current draw during re-connection be dipping the voltage to the processor, flash, and SRAM?

If a voltage dip is suspected here are some root causes you might consider:

1. are you powering the dev board from a PC usb port, USB hub, or a dedicated PSU? Most dedicated PSUs provide a "stiff" power supply that minimize voltage drops as the current rises. USB sources are often not very stiff, leading to voltage drops as current increases.

2. are you powering your dev board using a usb cable? if so, what is the wire gauge of the usb cable? many USB cables are cheaply built, using 28AWG wire for the power and ground wires - this can result in a significant voltage drop when current rises.

[pythoncoder]

I take extreme precautions against PSU problems, the whole test setup being isolated from mains power.

The power source is a 12V lead acid battery feeding a switch-mode converter. This provides a regulated 5V which feeds the USB port via a power-only USB lead, so the onboard linear regulator provides additional noise immunity.

[Mike Teachman]

Using a battery is an effective setup for eliminating noise effects from the AC mains power. But, it might not address the concerns of voltage drops leading into the supply voltage pins of various critical components on the dev board.

In the context of voltage drops I have a couple of other suggestions that might help you to find root cause:
- for the switch mode converter: are you confident that it does not show a voltage drop on the +5V output during the current surge associated with wifi transmission?
- for the USB power-only cable: have you measured the voltage drop across the cable going from the PSU to the dev board? ideally done with an oscilloscope, looking at the time when the ESP32 transmits over wifi
- what is the dropout voltage of the linear regulator on the dev board? often is it over 1 volt, and goes higher as current increases. This is a known contributor to the type of symptoms that you are seeing.

One test you could try uses an oscilloscope to measure the 3.3V output coming out of the dev board. Configure the oscilloscope to capture the 3.3V output when wifi transmits. What is the voltage drop? You can trigger the scope using a digital output initiated from your uPy program when wifi transmits. Or, trigger the scope to capture the +3.3V output at say < 2.5V which is where the ESP32 and the flash memory can start to exhibit problems.

Hope this helps!