Low level details documentation

[SimonR]

Hi all!

Decided to make this post in the off chance someone else has happen to know about a solution for my fairly niche requirements. These are not questions about how to use some specific library.

Questions first:
Micropython ports can differ a bit. Especially when it come to low level details.

• Is there any good reading material regarding the lower level stuff of MicroPython and the system it run on?

• More specifically; I looking for a MicroPython system that has the TCP_KEEPIDLE socket option or an equivalent. Is there one?

I realise this is very port-dependent but I have found it to be hard to find good information about what low level options are available on any given port. When possible I have tried to see what is included the chipset's firmware. So far no luck, thoughI have tried to not dive too deep into the issue.

Background
I'm working on a project programmed in Python where an embedded computer program another piece of hardware over a TCP socket and lightly digest what is coming back from it before sending the data to a more capable machine. The problem is that the target machine is hard coded to drop any connection that has been silent for more than 2 seconds, which is not very long. This is a problem as the communication is expected to be silent for up to several hours in a real scenario.

The current way I have had to solve this issue is to set the TCP_KEEPIDLE socket option to 1 second. This option is not readily available on all systems. Notably it has been a part of Linux for a long time and exist on Windows 10 revisions younger than 2 years. It does not exist on MacOS nor is available on Android for example.

As the Python program itself doesn't really do much it would be very interesting to see if I could make it run on a MicroPython board. For it to be successful in this instance there need to be a way to keep the communication open.

[kevinkk525]

Isn't there any other way to keep the connection open? TCP_KEEPIDLE isn't implemented in any port (as far as I know).
For mqtt and custom communication libraries we use keepalive pings, typically just empty messages but would your server be able to handle those?

[jimmo]

Hi,

To your specific question: Most ports use LWIP for their TCP stack. either directly (e.g. STM32), or via an arch-specific SDK (e.g. ESP32 via IDF).

LWIP supports TCP_KEEPIDLE, but only when built with "#define LWIP_TCP_KEEPALIVE 1". For STM32, you could set this in ports/stm32/lwip_inc/lwipopts.h. The ESP32 IDF appears to already set this to 1 in components/lwip/port/esp32/include/lwipopts.h.

So...worth trying -- if you're on STM32, build your own firmware with the above change and try using setsockopt. ESP32 might just work already?

Code: Select all

TCP_KEEPALIVE = const(0x03)
s.setsockopt(SOL_SOCKET, TCP_KEEPALIVE, 1) # in seconds

To the general question... Yes this is definitely an area for improvement. I'm currently tackling the machine docs to try and address the port variantions (but making slow progress). The network docs are much the same. This example (TCP_KEEPALIVE) is interesting because there are so many layers to making the documentation able to handle a (relatively obscure) question like this. But just knowing which ports use LWIP would be a good start!