Is there a non-recoverable memory leak in the uSSL implementation on the ESP32?

[oesb]

Is there a memory leak in the uSSL implementation on the ESP32? I wrote a simple socket server with uasyncio and micropython 1.10 which connects using uSSL. However, the free memory drops each time it connects and does not recover it until the ESP32 is rebooted. I used a bytearray as a buffer for the key and cert certificates so they don’t use memory from free mem and it appears that the free mem drop occurs during the opening of the SSL connection and it never recovers to the original level. The output below shows two connections and the associated two non-recoverable free memory drops.

OUTPUT:

ESP32 time up: 17, Hall Value: 129, Temperature: 110, Free Memory: 84640
ESP32 time up: 18, Hall Value: 133, Temperature: 110, Free Memory: 84640
ESP32 time up: 19, Hall Value: 132, Temperature: 110, Free Memory: 84640

INFO: AsyncServer: client connection on ('192.168.1.72', 60724)
DEBUG: AsyncServer: free_mem after key read: 84432
DEBUG: AsyncServer: free_mem after cert read: 84432

DEBUG: AsyncServer: free_mem after SSL conn: 79472

INFO: AsyncServer: client connection on ('192.168.1.72', 60724) succeeded
DEBUG: read: starting read while loop
DEBUG: read: free_mem: 80816
DEBUG: read: response sent
INFO: AsyncServer: End loop, connection closed
DEBUG: AsyncServer: free_mem: 80672

ESP32 time up: 23, Hall Value: 131, Temperature: 110, Free Memory: 80752
ESP32 time up: 24, Hall Value: 131, Temperature: 110, Free Memory: 80752
ESP32 time up: 25, Hall Value: 134, Temperature: 110, Free Memory: 80752

INFO: AsyncServer: client connection on ('192.168.1.72', 60727)
DEBUG: AsyncServer: free_mem after key read: 80592
DEBUG: AsyncServer: free_mem after cert read: 80624

DEBUG: AsyncServer: free_mem after SSL conn: 77152

INFO: AsyncServer: client connection on ('192.168.1.72', 60727) succeeded
DEBUG: read: starting read while loop
DEBUG: read: free_mem: 78624
DEBUG: read: response sent
INFO: AsyncServer: End loop, connection closed
DEBUG: AsyncServer: free_mem: 78752

ESP32 time up: 29, Hall Value: 127, Temperature: 110, Free Memory: 78928
ESP32 time up: 30, Hall Value: 131, Temperature: 110, Free Memory: 78928
ESP32 time up: 31, Hall Value: 131, Temperature: 110, Free Memory: 79024


The code has gc.collect() before each readout of free_mem.

Has anyone else encountered this? Is there a known solution? I have only been programming for a few months and, well, I am stuck.

[pythoncoder]

A common cause of this is failure to close sockets. Ensure that every path through the code (including exception handling) closes unused sockets.

[oesb]

I am fairly certain I covered all the socket closures. I am not certain if I linked in uasyncio correctly though.


def listen(self):
log = logging.getLogger('AsyncServer')
log.info('Server Started')
self.sock.listen(3)
while True:
gc.collect()
log.debug('free_mem: {}'.format(gc.mem_free()))
yield uasyncio.IORead(self.sock)
if self.one_conn is False:
conn, addr = self.sock.accept()
self.one_conn = True
conn.setblocking(False)
print('socket: ', conn, addr)
log.info('client connection on {}'.format(addr))
try:
try:
with open('ssl/server.key', 'rb') as f:
f.readinto(buf_key)
except Exception as e:
log.error('Error in key file open: {}'.format(e))
try:
gc.collect()
log.debug('free_mem after key read: {}'.format(gc.mem_free()))
with open('ssl/server.pem', 'rb') as f:
f.readinto(buf_cert)
except Exception as e:
log.error('Error in pem file open: {}'.format(e))
gc.collect()
log.debug('free_mem after cert read: {}'.format(gc.mem_free()))
try:
conn.setblocking(True)
conn.settimeout(4)
conn = ssl.wrap_socket(conn, server_side=True, key=bytes(buf_key), cert=bytes(buf_cert))
conn.setblocking(False)
gc.collect()
log.debug('free_mem after SSL conn: {}'.format(gc.mem_free()))
except Exception as e:
self.one_conn = False
log.error('Error in SSL: {}'.format(e))
conn.close()
if self.one_conn is True:
log.info('client connection on {} succeeded'.format(addr))
extra = {"peername": addr}
gc.collect()
yield from libserver_async.Message(uasyncio.StreamReader(conn), uasyncio.StreamWriter(conn, extra),
self.one_conn, addr, self.pin_num, self.mac).read_it()
except Exception as exc:
log.error('Connection {} Error: {}'.format(addr, exc))
self.one_conn = False
conn.close()
conn.close()
self.one_conn = False
log.info('End loop, connection closed')
gc.collect()
conn.close()
self.one_conn = False
log.error('Broken loop close')
gc.collect()


Thank you for your help, it is appreciated.

[pythoncoder]

I can't claim to understand your code: it isn't the way I write servers or use uasyncio. That doesn't mean it's necessarily wrong but this would be my approach. I'm not a networking guru but I do have some knowledge of uasyncio.

Firstly I'd get it running without TLS, to separate SSL/TLS issues from uasyncio issues. For security test by running the server on a local machine with local clients.

The normal way to use uasyncio is to use blocking sockets throughout. Nonblocking sockets can be used, but require special handling. They are not compatible with IORead/IOWrite. The notes below assume blocking sockets.

I would have one continuously running listen coro. This would have a single server socket and use select.ipoll to reduce blocking time to 1ms. Whenever this returned a connection socket conn it would spawn a coro with

Code: Select all

loop.create_task(self.run_conn(conn))

The run_conn method would use the IORead/IOWrite methods. Note that under the hood uasyncio uses select.ipoll to ensure blocking time is minimised.

[oesb]

Is there an example of the coding style you suggest hiding somewhere? I am fairly certian I understand what you are suggesting, but I am new to programming and I still find example code very useful for ensuring comprehension.

Thank you very much for your help!

[pythoncoder]

There is a uasyncio tutorial in my async repo but this lacks specific examples of client and server code. I plan to update it soon. There are examples on the web, however Python asyncio syntax went through a number of twists and turns before settling on the Python 3.5 syntax used in MicroPython. One of the aims of my tutorial is to encourage users to standardise on this syntax so you might want to adapt web examples.

[oesb]

I found the solution to the memory leak in the uansyncio __init__.py. I am not sure why, but by placing the unwrapped socket onto the uasyncio poller and reading and writing from the uSSL wrapped socket my continuous memory leak disappered. I still have a 4 kb chunk taken on the initial connection though. Is that the SSL impementation creating a buffer?

Out of curiosity why do you suggest polling the server listening socket with ipoll() instead of placing the server socket on the uasyncio poller with IORead?

Thank you very much for your help!

[pythoncoder]

A server socket returns a socket instance: I don't think the IORead mechanism can do this.

I've posted some code in my uasyncio repo in the client_server directory. Testing is not fully complete and it is subject to change. The tutorial is not yet updated to reflect it. But it does illustrate the fact that you can build a client and server using blocking sockets and yet have nonblocking systems.

I ran the server on an ESP8266 and ran multiple clients on a PC under the Unix build. The server flashes the ESP8266 LED at high speed to illustrate the fact that the code is nonblocking even though it uses blocking sockets throughout. It copes with interrupting clients or server with ctrl-C. The server LED continues to flash while clients come and go.

Note that simplistic demo code like this is not resilient in the face of WiFi outages. Handling these reliably is remarkably difficult and does necessitate replacing the IORead mechanism with nonblocking sockets (and much else). If this is of concern to you I can point you to work I have done on this, but the two available solutions do place restrictions on your system design.

[oesb]

Thank you very much for that! It is greatly appreciated! I had pretty much the same thing thanks to your description.

I will definitly take a delve into your github on resiliant servers!

On that note... Thank you for putting together all of your documentation and code, they are a fantastic resourse and I have learned a lot from them!

[pythoncoder]

One point for the tutorial is that socket.getaddrinfo blocks. The time is minimal in the example code with an explicit address, but if a DNS lookup is required it can be substantial. This is a known limitation.

[EDIT]
If you successfully adapt this approach to use TLS/SSL I would be very interested to see the code.