Wifi bridge for RPI Pico

RP2040 based microcontroller boards running MicroPython.
Target audience: MicroPython users with an RP2040 boards.
This does not include conventional Linux-based Raspberry Pi boards.
kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: Wifi bridge for RPI Pico

Post by kevinkk525 » Sun Jan 31, 2021 11:23 am

Well I don't know why they chose SPI but maybe that allows it to work with other hardware too? Regardless, it's a working ESP32 project that could be utilized as is.


You are right about it being a blocking socket. But I think it should be easy to implement a nonblocking socket using the existing implementation. Isn't the main difference just how you handle the read() method? The current implementation blocks until a timeout and seems to return b'' if nothing was received. With a nonblocking socket it'd need to return b'' if the connection is broken and None if there is no data in the buffer.
Having checked that, what I'm missing is some handling of broken sockets. This seems to be completely missing making me wonder if it automatically reconnects broken sockets on the esp32? Guess I need to dig into the C++ code of the nina-fw.. Not passing on the broken socket state to the micropython application would be a problem..
Or I need to join some curcuitpython/adafruit forum and ask there.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Wifi bridge for RPI Pico

Post by pythoncoder » Sun Jan 31, 2021 1:11 pm

I'll be interested to hear how you get on. Also how resilient their existing solution may be. Given our experiences you might want to test resilience with their standard code before making changes ;)
Peter Hinch
Index to my micropython libraries.

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: Wifi bridge for RPI Pico

Post by kevinkk525 » Sun Jan 31, 2021 2:10 pm

You're right.. I should test Circuitpython first and only make minimal changes to the existing code to support nonblocking sockets if the blocking sockets work.
But I should probably first confirm how they handle socket errors because hiding socket errors from the Pico could make some libraries problematic..
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Wifi bridge for RPI Pico

Post by pythoncoder » Tue Feb 02, 2021 9:46 am

I'm interested in how you plan to handle resilience. I see two options:
  • Build a resilient socket. Such an object would not throw exceptions but, in the event of a WiFi outage would pause until the outage had cleared. It would offer guaranteed (eventual) communication. Like a full-featured version of the object in our iot project.
  • Build a socket that emulates a BSD socket with its various exception states.
Clearly the level of resilience provided by the Adafruit code dictates how much work is required in either option.

A resilient socket would, of course, benefit all protocols, not just MQTT.

Whichever approach you use, mqtt_as will require significant rework to run via a bridge. At the moment it is based on the assumption that WiFi is handled and monitored locally.

Regarding the existing MQTT bridge project.

The API differences with mqtt_as should be easy to code around. For example, consider publish. When the host issues the synchronous publish, return is immediate. The message is queued for transmission to the ESP8266 which then handles the protocol. If you want a qos==1 publication to pause until complete you would write a coro which issued publish, then waited on an Event. The Event would be triggered by a status handler on receipt of the PUBOK status (see pb_status.py).

I still plan to make this bridge V3 compatible and cross-platform; I don't expect this to be difficult and it would make an interesting first project for the Pico. And it will offer an interim solution.

I admire the ambition of your socket project. If you can make this work asynchronously and in a resilient way you will have made a significant contribution to MicroPython. :D
Peter Hinch
Index to my micropython libraries.

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: Wifi bridge for RPI Pico

Post by kevinkk525 » Tue Feb 02, 2021 4:34 pm

I don't plan on building a resilient socket, it's better if the application handles wifi outages and broken sockets correctly as if the wifi was controlled on the device directly.
pythoncoder wrote:
Tue Feb 02, 2021 9:46 am
Clearly the level of resilience provided by the Adafruit code dictates how much work is required in either option.
I studied the C++ code for the esp32 and it looks good to me and I expect errors to be passed on but my understanding of the C++ isn't very deep and I have to properly test it with circuitpython on the pico, haven't done anything yet.
pythoncoder wrote:
Tue Feb 02, 2021 9:46 am
Whichever approach you use, mqtt_as will require significant rework to run via a bridge. At the moment it is based on the assumption that WiFi is handled and monitored locally.
On the contrary, the goal is to not need any modification at all and provide a socket and network class that work as a drop-in replacement.
Adafruit seems to use a standard request library that works with any kind of socket, so this is the goal for mqtt_as too, which should be quite achievable because mqtt_as only uses standard methos with non-blocking sockets.
On the Pico there is no network module so a drop-in would be easy and the goal is to also offer a simple drop in for the socket module.
That way any program can work without any modifications as if the WIFI was connected locally. Not sure I'll get there since a socket is a complex object but at least basic functionality should be drop-in compatible.
pythoncoder wrote:
Tue Feb 02, 2021 9:46 am
If you want a qos==1 publication to pause until complete you would write a coro which issued publish, then waited on an Event. The Event would be triggered by a status handler on receipt of the PUBOK status (see pb_status.py).
In my case it's not quite as easy because I also work with timeouts and cancellations of publications, which means I'd also write code to remove unsent publications from the queue again. But yes, it is possible to work around those issues but I'm not sure it's worth the effort.
pythoncoder wrote:
Tue Feb 02, 2021 9:46 am
I admire the ambition of your socket project. If you can make this work asynchronously and in a resilient way you will have made a significant contribution to MicroPython.
Well now I have to do it :D Thanks for the pep talk :lol:
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: Wifi bridge for RPI Pico

Post by kevinkk525 » Wed Feb 03, 2021 11:48 am

Funny story: Thought I could try that CP Project on 2 esp32s so I spent 2 hours trying to flash Circuitpython onto my esp32 with no success until I realized, they only support esp32-s2! So no luck for me.. I don't have any esp32-s2.. only the old ones.. Guess I should have used the Pico from the start :D

Other problem though: I can't even get the Nina-FW https://learn.adafruit.com/upgrading-es ... -one-board working. I flash it but no output indicating that it works but their Airlift actually uses the same esp32 chip so it should work.. I feel lost :D

Code: Select all

ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:832
ho 0 tail 12 room 4
load:0x40078000,len:9764
load:0x40080400,len:5700
entry 0x4008069c

I'll ask on the Circuitpython forum for help..
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Wifi bridge for RPI Pico

Post by pythoncoder » Wed Feb 03, 2021 12:10 pm

Oh dear :(

Thanks for the explanation re sockets. It sounds good.

There is a potential resilience issue which results from their use of SPI. In our development of resilient MQTT solutions, we never had to worry about message corruption. This is because, though TCP/IP can't guarantee message delivery over WiFi, it does guarantee integrity.

Regarding bridges, the design of my SynCom protocol ensures that invalid messages can't be delivered. This does not apply to SPI. If the slave crashes or reboots, the master will continue to clock in garbage. I think CRC or some other means of guaranteeing integrity will be required on the interface. I haven't trawled through the Adafruit code: I wonder if they have paid attention to it? We both know ESP32's keel over from time to time.

Tests I performed on my bridge were to press the reset button on the ESP8266 while it was running to simulate a spontaneous reboot. I also simulate a crash by hitting ctrl-c at the ESP8266 REPL.
Peter Hinch
Index to my micropython libraries.

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: Wifi bridge for RPI Pico

Post by kevinkk525 » Wed Feb 03, 2021 1:45 pm

Hmm yeah you might be right about needing CRC to be really sure we don't receive garbage. I think this could be easily implemented if the rest works as expected.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: Wifi bridge for RPI Pico

Post by kevinkk525 » Thu Feb 04, 2021 9:01 pm

So a short update of my progress so far:

- Documentation of both projects from Adarfruit are (in my opinion) quite far from perfect, had to dig deep into the code to even find the correct pins (because adafruit only uses their pin labels on their products, which are quite different), looked at schematics etc.
- The documented test sketch for Arduino ESP32 didn't quite work and made me waste many hours thinking my communication between the boards is somehow broken
- Turns out everything was fine. I realized that when I ported some basic functions to micropython to see if the SPI communication would work correctly. Result: It does! I can now speak to the Adafruit WIFI bridge firmware, check the WLAN status, (dis)connect and get the current time. I could have tested that using circuitpython but that only works on the PICO and I haven't soldered the heads onto the pico yet, so can't use it yet.. Only funny thing: Only SoftSPI works, both hardware SPI implementations don't work even though they are supposed to be possible on other pins too but no.. Doesn't matter much though, the esp32 is only for testing at the moment, nobody will use an esp32 to provide wlan to another esp32 :lol:
- Porting the project to micropython seems quite doable but their use of properties instead of functions is getting on my nerves :lol: It kind of feels like working with visual basic and not python but ok..
- For my more ambitious goal of creating a drop-in replacement for the network module: There are quite a lot of functions that work differently, are named differently and have a different return value. So here I'll need some more work on compatibility but that should work just fine, it'll jus take time.
- I can't say anything about resilience yet but I assume it'll do just fine. I will test the cicruitpython examples on the PICO when I have soldered the heads onto it but testing the resilience there might be tricky because circuitpython is different and has no uasyncio.

I'd say let's get this project started :D I already forked their repository and will be porting it to micropython. Help will be appreciated and any input as well. For now the main goal would be to get the core functionality ported to micropython and in a second stage the abstract socket so we can test resilience. 3rd step would be a drop-in compatible network module and the 4th step would be adding all extra functions like controlling the esp32 pins from the pico (who cares :?: ). The Adafruit WIFI bridge project also supports using Bluetooth over UART but that's obviously not part of the circuitpython SPI project (because it's over UART) and I have no ambition of getting that working either. You can read about BLE support here: https://learn.adafruit.com/adafruit-air ... python-ble

Please let me know what you think.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Wifi bridge for RPI Pico

Post by pythoncoder » Fri Feb 05, 2021 9:25 am

I'm puzzled you've had problems with ESP32 hard SPI, but the docs are a little confusing. You have to state pins, even if they are default. I haven't tried non-default pins. I have used this invocation successfully:

Code: Select all

spi = SPI(1, 10_000_000, sck=Pin(14), mosi=Pin(13), miso=Pin(12))
I wouldn't bother emulating everything Adafruit does. Controlling pins on the ESP32 sounds pointless when the Pico has plenty of pins. Bluetooth sounds like another project entirely. If you can provide a socket object that supports blocking and nonblocking modes and copes with all the error conditions I'll be very impressed.

Perhaps there is no point in testing the Adafruit code for resilience. It sounds as if you may need to modify the ESP32 firmware to support CRC. Your handling of the SPI protocol will need to be cause the socket to behave in a reasonable way if the CRC fails, if the ESP times out due to a crash or if the ESP spontaneously reboots. There may be a need to physically reboot the ESP if it fails to respond. My MQTT bridge does this: if the communications protocol fails it resets the ESP8266 and re-synchronises.

It may be possible to treat the creation of a socket as a separate exercise from that of ensuring reliability. However at the outset I suggest you consider how the socket should behave if communications suddenly go down and restart.
Peter Hinch
Index to my micropython libraries.

Post Reply