station.scan() issue on ESP32

[Mystic]

I am using micropython on an ESP32
I am using a G4 dongle to give me a WiFi access point as this ESP32 will eventually run where there is no WiFi cover somewhere in the woods
This is the type of USB powered dongle that you would use with a laptop on the go
Most of the time everything works fine, but sometimes the G4 dongle hangs and I need to power it on/off to get it working again, so there are some issues there .
I plan to take care of such issues by having a relay controlled by my ESP32 that would power the G4dongle off/on when I no longer get connectivity (ie no more MQTT messages from my remote server)
I note that periodically my ESP32 stops working and hangs, so either it has crashed or somthingis in a loop
I have narrowed that down to the station.scan() command, as that is teh last thing that has executed before things stop working.
In order to assess whether it is in a look or just a code issue in the station.scan(),I have wrapped that in a try catch which never triggers, so I have to assume that the station.scan() goes into a loop and/or never returns to my code
I have a message display before and after that try catch so that I can confirm where it all stops working

My code is as follows

... display message before

try:
nets = station.scan()
except OSError as e:
asynchDisplay('Failed sta scan',0)
printText (("Failed sta scan, restart and connect"), "info")
restart_and_reconnect()

... display message after

The above is as far as I can seem to get to diagnose things other than somehow trying to get into the the micropython built in network library

Can anyone shed some light on this issue and help identyify where the problem might be and solution to this ??

Many thanks

[davef]

I use two HUAWEI-E8231 USB dongles for my system. The issue I have is if I lose 5V to the dongles on re-power they need to be connected to a laptop to do the mode-switching, back to modem operation. It appears that you might not be seeing the same issue, but why are they hanging? In a marginal signal area or something wrong with the dongles?

I have quite a bit of experience with ESP32s hanging. Are you running a software WDT? Are you using the system WDT? Which ESP32 dev boards are you using? I do several power supply mods to try to keep them alive when the WiFi connects.

[Mystic]

Hello
Thanks for your response.
I have the ESP32-DevKitC
It works perfectly fine for everything else so far
The are two distinct issues here.
The one is with the USB dongle where it periodically hangs and no longer communicates even though the ESP32 still thinks it is there
The only way to fix that is to power it off and on
I have proven that by leaving the ESP32 alone and just unplug replug the dongle and everything recovers
The other issue is with the station.scan()
When the ESP32 starts up from power on or machine reset, it scans all the SSIDs available , then compares that to a fixed list of SSIDs which I willl allow connection to, then it takes the strongest signal one to connect to
If no suitable SSIDs are found , it call the restart_and_reconnect() routine which basically calls macine.reset() and the whole loop starts again until it finds a suitable SSID to connect to.
Interestingly, when the modem is in that state and the ESP32 has "hung" , if I power off / on just the ESP32, the whole loop above starts up again, and goes round many times with the station.scan() being called successfully everytime and does not fail. (It could be hours though I have no record of that yet )
Eventually, next morning when I check, it is back in that hung state or just still going round in that loop looking for a suitable SSID
I do not use WDTs. I did look into that, but could not see how that could help resolve my issue
My dongle is never connected to my PC . The dongle, and ESP32 are both powered from a rechargeable battery as eventually this whole unit will be deployed in some woods somewhere where there is no WIFi, so the dongle does recover with just a power off/on
I have a small OLED with six lines of display that scrolls through as certain debug points are reached so that I can see the last thing that has happenned
Is there any other information I can give youto help diagnose the issue?
Many thanks

[davef]

I use mostly ESP32 DevKit C Wroom32U. The battery that is supplying both and USB dongles, describe it and how you are getting a clean 5V. When WiFi is messing-about you can get up to 0.5 Amp pulses of current, so if the regulator supplying your 5V is not strong enough then you can have problems. The AMS 3V3 regulator on these boards has been a suspect. I usually slap a 1000uF low-ESR cap on the 5V input and then another on the 3v3 regulator output.

For units that I power off one Li-ion I use a HZ3777 3v3 regulator and a 1000uF on it's output.

Eventually, next morning when I check, it is back in that hung state or just still going round in that loop looking for a suitable SSID

Maybe, a WDT would help. I run a software WDT which you can find on this site, search for:
Simple software WDT implementation
Also, the system WDT should at least help to work out why they are hanging.

Also, I sprinkle lotsa try,except statements through-out the code to try and trap where things might go wrong.

Haven't used scan so can't help there. One thing I do is every time I connect to WiFi is I make sure that both the AP and STA are disconnected and inactive and then connect to and make active the one I want.

So, tell me about your power systems.

[Mystic]

My ESP32 devkitC is also ESP32-WROOM-32U
The battery is the portable type you would use to power a laptop or mobile phone on the move
It is charged via a usb cable and has two USB output power sockets
Whilst testing in my office, I am powering the USB dongle using that battery which is getting kept charged via a USB power adapter and am powering the ESP32 from the same USB power adapter which is a dual USB port one for now. In teh field, they will both be powered from the USB battery pack (probably a 10000 ma one or larger with a solar charger to keep it topped up
I would think that they should all get enough power this way
I understand what you mean about not supplying enough power can cause problems as I have seen that as well before
Yes, adding a 1000 uF cap is always a good idea to smooth out power and add a bit of power buffer for peak loads
I suspect that not enough power is not the issue in my case
I have reviisited the WDT and added that into the mix and retstarted the ESP32 (leaving the USB WiFi dongle in it's hung state as sometimes it can take days or more to hang like that so not an easily repeatable scenario)
I have set it to 5 minutes and set it as the first thing I do in the code, so that I am covered from the start as there is quite a lot of preamble code and waits for various things before the main code loop. Unfortunately, you can only set the WDT once so it has to be a compromise. Let's see what happens now !!
I use try / catch on all my code so as to reduce error situations.
What is your reason for ensuring that both the AP and STA are disconnected and inactive before connecting. In my logic, if anything goes wrong or does not connect, I do a machine reset, so effectively a complete reboot, so AP and STA would not be connected when I start.
I agree that it is a good idea though if in a loop trying to connect to WiFi as one may have connected previously and it is a good idea to ensure they are disconnected
Out of interested , I enable both AP and STA
I use the AP for webrepl so as to allow me to do code updates remotely, though I have not fully explored that mechanism yet, but it does work
I start the AP and webrepl first before starting STA and WiFi

[davef]

What is your reason for ensuring that both the AP and STA are disconnected and inactive before connecting.

On the ESP32 if you have been connected and try to re-connect, using a CTRL-C (soft-reset) you get a WiFi internal error. If you do a hard-reset things seem to work OK. While you are working on the program and only do a soft-reset you may not start the WiFi properly. It only seems to be an issue when connecting as STA but who knows what happens to AP, so I am being cautious.

I gave up trying to work out the strange things that happened when I tried to use WebREPL. My problems seem to reduce just using rshell.

Good luck!

[Mystic]

Yes, A soft reset will randomly cause failure on WiFi restart , so while testing with Thonny connected, it can be annoying, so disconnecting WiFi first will help
As I use on OLED to track what is going on for each module , once I have roughly tested my code connected to Thonny, I run it offline while soaking to see how it stands up.
I still would like to understand why the station.scan() hangs from time to time
When everything works fine and the modem is working as it should, it can go for a week or more without issues
Doing a lot of online research, it does seem like hardware freezes on microprocessors is a thing that just happens from time to time regardless , so I wonder if using the internal WDT will guarantee reliability, though it should improve things . Time will tell
I am not concerned about restarts as long as they happen .. Once I have my devices up in the woods, it would be a real pain to have to visit them to restart them .
I am also looking at getting some sort of external WDT that is fully hardware/electronics (ie not another microprocessor running code again) as that should then provide a very reliably recoverable system

[davef]

If anything causes an exception I log the error to an errors.txt and visit that periodically. Can you get station.scan to raise an error?

I recently built a hardware wdt using a ADM6316 family part with a wdt of 1.6 seconds. Wish I had bought 25 second parts. 2 NPN transistors and 3 resistors. Fortunately, the internal WDT and a S/W WDT have kept things going for days/weeks. But, like you say, if you can recover quickly then maybe that is OK.

[Mystic]

Hi
I can't log errors in station.scan() as that is part of the Micropython image installed on the ESP32
I am not sure how I would get to the code to update that
I do have a try / catch around the station.scan() command to catch any errors, but it does not pick up an error, because if it did, I would have done a machine reset and all would be well , so I am suspectimg that I am possibly getting some form of hardware freeze
I am interested in exploring an external WDT as well
I have seen circuits based on a 555 chip
I will explore the ADM6316 chip as well . Do you know if it caters for both scenarios when the "pulse" to the external WDT is frozen as high or low and does it trigger in both cases resulting in a reboot ?
Thanks

[davef]

I can't log errors in station.scan() as that is part of the Micropython image installed on the ESP32

The ESP32 board is in a place where you can't get to it. And the image is "frozen" or you can't modify it. You are not making this easy

Hardware watchdog. As I said, I wish I had chosen a 25 second part, so that I didn't have to use some many "pats". One thing that is a bit tricky with these parts is the the chip reset line has to be "tri-stated" while you are not supplying a reset pule to it. Fortunately, a ESP32 GPIO seems to do a good job of this as shown in the last few lines.

Code: Select all

#  test for ADM6316CY29ARJZ-R7 WDT chip (1.6 seconds)

from machine import Pin
import utime


pin33 = Pin(33, Pin.OUT)
pin33.off() #  start off

while True:

    if (pin33.value() == 0):
        pin33.value(1)
    elif (pin33.value() == 1):
        pin33.value(0)
    print ('looping')
    utime.sleep(.5)
    old_pin33_value = pin33() #  save current pin33 output state
    pin33.init(mode=pin33.IN) #  change to input so that WDT chip doesn't reset

    utime.sleep(.5)
#    utime.sleep(5) #  long delay to test WDT
    pin33.init(mode=pin33.OUT, value=old_pin33_value) #  change back to output

If you want to build a hardware WDT using these parts I will draw up a schematic.