ENOMEM Error when using UDP Broadcast

All ESP32 boards running MicroPython.
Target audience: MicroPython users with an ESP32 board.
__deets__
Posts: 23
Joined: Sun Aug 20, 2017 4:50 pm

ENOMEM Error when using UDP Broadcast

Post by __deets__ » Sun Jul 29, 2018 8:40 am

Hi,

I encounter OSError ENOMEM when working with an ESP32 board. I've flashed the latest repo state yesterday, to no avail:

Version info:

Code: Select all

MicroPython v1.9.4-410-g11a38d5dc on 2018-07-28; ESP32 module with ESP32
I downsized my code to the following minimal example. It will crash for me after a few seconds, sometimes quicker, sometimes slower. Just invoke main.

Any suggestions on how to debug this? I'm not scared of the actual uP-Implementation, but have not yet debugged memory errors on an embedded target.

Code: Select all

import gc
import array
import machine
import time
import network
import socket

PORT = 5000

KNOWN_NETWORKS = {
    b'SSID': b'password',
}

def ip2bits(ip):
    res = 0
    for part in ip.split("."):
        res <<= 8
        res |= int(part)
    return res


def bits2ip(bits):
    res = []
    for _ in range(4):
        res.append(str(bits & 0xff))
        bits >>= 8
    return ".".join(reversed(res))


def setup_wifi():
    nic = network.WLAN(network.STA_IF)
    nic.active(True)
    networks = nic.scan()
    broadcast_address = None
    for name, *_ in networks:
        if name in KNOWN_NETWORKS:
            nic.connect(name, KNOWN_NETWORKS[name])
            print("Connected to {}".format(name.decode("ascii")))
            ip, netmask, _, _ = nic.ifconfig()
            bca_bits = ip2bits(ip)
            netmask_bits = ip2bits(netmask)
            bca_bits &= netmask_bits
            bca_bits |= ~netmask_bits
            broadcast_address = bits2ip(bca_bits)
    return nic, broadcast_address


def setup_socket(nic):
    sock = socket.socket(
        socket.AF_INET,
        socket.SOCK_DGRAM
    )
    while not nic.isconnected():
        time.sleep(.1)
    sock.settimeout(2.0)
    ip, *_ = nic.ifconfig()
    sock.bind((ip, PORT))
    return sock


def main():
    nic, broadcast_address = setup_wifi()
    s = setup_socket(nic)
    address = (broadcast_address, PORT)
    message = "foobar"
    then = time.time()
    while True:
        print(message, address)
        s.sendto(message, address)
        if time.time() - then > 1:
            print(gc.mem_free())
            gc.collect()
            then += 1

SpotlightKid
Posts: 463
Joined: Wed Apr 08, 2015 5:19 am

Re: ENOMEM Error when using UDP Broadcast

Post by SpotlightKid » Sun Jul 29, 2018 10:52 am

Please post the full error message and traceback (you may need to allocate memory for exception tracebacks to see this). .

__deets__
Posts: 23
Joined: Sun Aug 20, 2017 4:50 pm

Re: ENOMEM Error when using UDP Broadcast

Post by __deets__ » Sun Jul 29, 2018 7:31 pm

There is not really anything more, the problem seems to be in the C-layer. The full traceback, after allocating 1000 bytes of traceback ram:

Code: Select all

  File "main.py", line 8, in <module>
  File "minimal.py", line 69, in main
OSError: [Errno 12] ENOMEM
MicroPython v1.9.4-410-g11a38d5dc on 2018-07-28; ESP32 module with ESP32
The main.py is just

Code: Select all

from minimal import main
import micropython

micropython.alloc_emergency_exception_buf(1000)
main()
The shift in line-number of main.py is just a copyright header I didn't quote here.

minimal.py is the above quoted code.

SpotlightKid
Posts: 463
Joined: Wed Apr 08, 2015 5:19 am

Re: ENOMEM Error when using UDP Broadcast

Post by SpotlightKid » Sun Jul 29, 2018 7:43 pm

With the traceback we can now see at least at which line in minimal.py the exception occurred.

So your sending out UDP messages as fast as possible. Maybe some transfer buffer is continuously re-allocated? What happens if you remove the print() call at the start of the loop?

__deets__
Posts: 23
Joined: Sun Aug 20, 2017 4:50 pm

Re: ENOMEM Error when using UDP Broadcast

Post by __deets__ » Sun Jul 29, 2018 7:50 pm

True, the line in minimal.py was not obvious, sorry for not providing that information.

If the print is removed, the ENOMEM happens more or less instantaneously after setting up the network. So your assumption might be true.

__deets__
Posts: 23
Joined: Sun Aug 20, 2017 4:50 pm

Re: ENOMEM Error when using UDP Broadcast

Post by __deets__ » Sun Jul 29, 2018 7:55 pm

I modified the inner loop:

Code: Select all

def main():
    nic, broadcast_address = setup_wifi()
    s = setup_socket(nic)
    address = (broadcast_address, PORT)
    message = "foobar"
    then = time.time()
    count = 0
    while True:
        #print(message, address)
        try:
            s.sendto(message, address)
            count += 1
        except OSError:
            print(count)
            raise
        if time.time() - then > 1:
            print(gc.mem_free())
            gc.collect()
            then += 1
The exception usually occurs around 33, sometimes 35 sendtos.

If I put back in the print, the packets sent varies much more, 100, 235, 1200...

__deets__
Posts: 23
Joined: Sun Aug 20, 2017 4:50 pm

Re: ENOMEM Error when using UDP Broadcast

Post by __deets__ » Sun Jul 29, 2018 8:01 pm

And pulling up the gc to loop-level instead of just doing it every second, no print:

Code: Select all

    while True:
        #print(message, address)
        try:
            s.sendto(message, address)
            count += 1
        except OSError:
            print(count)
            raise
        # if time.time() - then > 1:
        #     print(gc.mem_free())
        gc.collect()
        # then += 1
Here again the numbers are all over the place, from 30 to >7000, running for several seconds.

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: ENOMEM Error when using UDP Broadcast

Post by jickster » Wed Aug 08, 2018 7:10 pm

__deets__ wrote:
Sun Jul 29, 2018 8:01 pm
And pulling up the gc to loop-level instead of just doing it every second, no print:

Code: Select all

    while True:
        #print(message, address)
        try:
            s.sendto(message, address)
            count += 1
        except OSError:
            print(count)
            raise
        # if time.time() - then > 1:
        #     print(gc.mem_free())
        gc.collect()
        # then += 1
Here again the numbers are all over the place, from 30 to >7000, running for several seconds.
If you think it's related to gc, enable all gc diagnostics by setting these to 1:
MICROPY_DEBUG_VERBOSE, EXTENSIVE_HEAP_PROFILING.

__deets__
Posts: 23
Joined: Sun Aug 20, 2017 4:50 pm

Re: ENOMEM Error when using UDP Broadcast

Post by __deets__ » Fri Aug 17, 2018 1:53 pm

I do not think so. I've done that (I opened an issue in github and got the same advice). The reason I'm confident this isn't the problem: I've modified the source of sendto to *not* perform the actual lwip_sendto_r call. Then I do *not* get a ENOMEM, suggesting that the leak is not within uPy/gc, but somewhere in the lower rungs of the ESP SDK or something. I've searched the ESP SDK issues but also didn't find anything.

However adding the flags you suggest when building led to a non-working system for me, see my gh issue:

https://github.com/micropython/micropython/issues/4029

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: ENOMEM Error when using UDP Broadcast

Post by jickster » Fri Aug 17, 2018 5:25 pm

__deets__ wrote:
Fri Aug 17, 2018 1:53 pm
However adding the flags you suggest when building led to a non-working system for me, see my gh issue:

https://github.com/micropython/micropython/issues/4029
They're not flags you pass to `make` but macros defined in the code in `gc.c`.
If enabling those macros results in a non-working system then there indeed is a huge problem.

Post Reply