Page 1 of 2

ENOMEM Error when using UDP Broadcast

Posted: Sun Jul 29, 2018 8:40 am
by __deets__
Hi,

I encounter OSError ENOMEM when working with an ESP32 board. I've flashed the latest repo state yesterday, to no avail:

Version info:

Code: Select all

MicroPython v1.9.4-410-g11a38d5dc on 2018-07-28; ESP32 module with ESP32
I downsized my code to the following minimal example. It will crash for me after a few seconds, sometimes quicker, sometimes slower. Just invoke main.

Any suggestions on how to debug this? I'm not scared of the actual uP-Implementation, but have not yet debugged memory errors on an embedded target.

Code: Select all

import gc
import array
import machine
import time
import network
import socket

PORT = 5000

KNOWN_NETWORKS = {
    b'SSID': b'password',
}

def ip2bits(ip):
    res = 0
    for part in ip.split("."):
        res <<= 8
        res |= int(part)
    return res


def bits2ip(bits):
    res = []
    for _ in range(4):
        res.append(str(bits & 0xff))
        bits >>= 8
    return ".".join(reversed(res))


def setup_wifi():
    nic = network.WLAN(network.STA_IF)
    nic.active(True)
    networks = nic.scan()
    broadcast_address = None
    for name, *_ in networks:
        if name in KNOWN_NETWORKS:
            nic.connect(name, KNOWN_NETWORKS[name])
            print("Connected to {}".format(name.decode("ascii")))
            ip, netmask, _, _ = nic.ifconfig()
            bca_bits = ip2bits(ip)
            netmask_bits = ip2bits(netmask)
            bca_bits &= netmask_bits
            bca_bits |= ~netmask_bits
            broadcast_address = bits2ip(bca_bits)
    return nic, broadcast_address


def setup_socket(nic):
    sock = socket.socket(
        socket.AF_INET,
        socket.SOCK_DGRAM
    )
    while not nic.isconnected():
        time.sleep(.1)
    sock.settimeout(2.0)
    ip, *_ = nic.ifconfig()
    sock.bind((ip, PORT))
    return sock


def main():
    nic, broadcast_address = setup_wifi()
    s = setup_socket(nic)
    address = (broadcast_address, PORT)
    message = "foobar"
    then = time.time()
    while True:
        print(message, address)
        s.sendto(message, address)
        if time.time() - then > 1:
            print(gc.mem_free())
            gc.collect()
            then += 1

Re: ENOMEM Error when using UDP Broadcast

Posted: Sun Jul 29, 2018 10:52 am
by SpotlightKid
Please post the full error message and traceback (you may need to allocate memory for exception tracebacks to see this). .

Re: ENOMEM Error when using UDP Broadcast

Posted: Sun Jul 29, 2018 7:31 pm
by __deets__
There is not really anything more, the problem seems to be in the C-layer. The full traceback, after allocating 1000 bytes of traceback ram:

Code: Select all

  File "main.py", line 8, in <module>
  File "minimal.py", line 69, in main
OSError: [Errno 12] ENOMEM
MicroPython v1.9.4-410-g11a38d5dc on 2018-07-28; ESP32 module with ESP32
The main.py is just

Code: Select all

from minimal import main
import micropython

micropython.alloc_emergency_exception_buf(1000)
main()
The shift in line-number of main.py is just a copyright header I didn't quote here.

minimal.py is the above quoted code.

Re: ENOMEM Error when using UDP Broadcast

Posted: Sun Jul 29, 2018 7:43 pm
by SpotlightKid
With the traceback we can now see at least at which line in minimal.py the exception occurred.

So your sending out UDP messages as fast as possible. Maybe some transfer buffer is continuously re-allocated? What happens if you remove the print() call at the start of the loop?

Re: ENOMEM Error when using UDP Broadcast

Posted: Sun Jul 29, 2018 7:50 pm
by __deets__
True, the line in minimal.py was not obvious, sorry for not providing that information.

If the print is removed, the ENOMEM happens more or less instantaneously after setting up the network. So your assumption might be true.

Re: ENOMEM Error when using UDP Broadcast

Posted: Sun Jul 29, 2018 7:55 pm
by __deets__
I modified the inner loop:

Code: Select all

def main():
    nic, broadcast_address = setup_wifi()
    s = setup_socket(nic)
    address = (broadcast_address, PORT)
    message = "foobar"
    then = time.time()
    count = 0
    while True:
        #print(message, address)
        try:
            s.sendto(message, address)
            count += 1
        except OSError:
            print(count)
            raise
        if time.time() - then > 1:
            print(gc.mem_free())
            gc.collect()
            then += 1
The exception usually occurs around 33, sometimes 35 sendtos.

If I put back in the print, the packets sent varies much more, 100, 235, 1200...

Re: ENOMEM Error when using UDP Broadcast

Posted: Sun Jul 29, 2018 8:01 pm
by __deets__
And pulling up the gc to loop-level instead of just doing it every second, no print:

Code: Select all

    while True:
        #print(message, address)
        try:
            s.sendto(message, address)
            count += 1
        except OSError:
            print(count)
            raise
        # if time.time() - then > 1:
        #     print(gc.mem_free())
        gc.collect()
        # then += 1
Here again the numbers are all over the place, from 30 to >7000, running for several seconds.

Re: ENOMEM Error when using UDP Broadcast

Posted: Wed Aug 08, 2018 7:10 pm
by jickster
__deets__ wrote:
Sun Jul 29, 2018 8:01 pm
And pulling up the gc to loop-level instead of just doing it every second, no print:

Code: Select all

    while True:
        #print(message, address)
        try:
            s.sendto(message, address)
            count += 1
        except OSError:
            print(count)
            raise
        # if time.time() - then > 1:
        #     print(gc.mem_free())
        gc.collect()
        # then += 1
Here again the numbers are all over the place, from 30 to >7000, running for several seconds.
If you think it's related to gc, enable all gc diagnostics by setting these to 1:
MICROPY_DEBUG_VERBOSE, EXTENSIVE_HEAP_PROFILING.

Re: ENOMEM Error when using UDP Broadcast

Posted: Fri Aug 17, 2018 1:53 pm
by __deets__
I do not think so. I've done that (I opened an issue in github and got the same advice). The reason I'm confident this isn't the problem: I've modified the source of sendto to *not* perform the actual lwip_sendto_r call. Then I do *not* get a ENOMEM, suggesting that the leak is not within uPy/gc, but somewhere in the lower rungs of the ESP SDK or something. I've searched the ESP SDK issues but also didn't find anything.

However adding the flags you suggest when building led to a non-working system for me, see my gh issue:

https://github.com/micropython/micropython/issues/4029

Re: ENOMEM Error when using UDP Broadcast

Posted: Fri Aug 17, 2018 5:25 pm
by jickster
__deets__ wrote:
Fri Aug 17, 2018 1:53 pm
However adding the flags you suggest when building led to a non-working system for me, see my gh issue:

https://github.com/micropython/micropython/issues/4029
They're not flags you pass to `make` but macros defined in the code in `gc.c`.
If enabling those macros results in a non-working system then there indeed is a huge problem.