Wifi bridge for RPI Pico
-
- Posts: 969
- Joined: Sat Feb 03, 2018 7:02 pm
Re: Wifi bridge for RPI Pico
Well I don't know why they chose SPI but maybe that allows it to work with other hardware too? Regardless, it's a working ESP32 project that could be utilized as is.
You are right about it being a blocking socket. But I think it should be easy to implement a nonblocking socket using the existing implementation. Isn't the main difference just how you handle the read() method? The current implementation blocks until a timeout and seems to return b'' if nothing was received. With a nonblocking socket it'd need to return b'' if the connection is broken and None if there is no data in the buffer.
Having checked that, what I'm missing is some handling of broken sockets. This seems to be completely missing making me wonder if it automatically reconnects broken sockets on the esp32? Guess I need to dig into the C++ code of the nina-fw.. Not passing on the broken socket state to the micropython application would be a problem..
Or I need to join some curcuitpython/adafruit forum and ask there.
You are right about it being a blocking socket. But I think it should be easy to implement a nonblocking socket using the existing implementation. Isn't the main difference just how you handle the read() method? The current implementation blocks until a timeout and seems to return b'' if nothing was received. With a nonblocking socket it'd need to return b'' if the connection is broken and None if there is no data in the buffer.
Having checked that, what I'm missing is some handling of broken sockets. This seems to be completely missing making me wonder if it automatically reconnects broken sockets on the esp32? Guess I need to dig into the C++ code of the nina-fw.. Not passing on the broken socket state to the micropython application would be a problem..
Or I need to join some curcuitpython/adafruit forum and ask there.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: Wifi bridge for RPI Pico
I'll be interested to hear how you get on. Also how resilient their existing solution may be. Given our experiences you might want to test resilience with their standard code before making changes
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
-
- Posts: 969
- Joined: Sat Feb 03, 2018 7:02 pm
Re: Wifi bridge for RPI Pico
You're right.. I should test Circuitpython first and only make minimal changes to the existing code to support nonblocking sockets if the blocking sockets work.
But I should probably first confirm how they handle socket errors because hiding socket errors from the Pico could make some libraries problematic..
But I should probably first confirm how they handle socket errors because hiding socket errors from the Pico could make some libraries problematic..
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: Wifi bridge for RPI Pico
I'm interested in how you plan to handle resilience. I see two options:
A resilient socket would, of course, benefit all protocols, not just MQTT.
Whichever approach you use, mqtt_as will require significant rework to run via a bridge. At the moment it is based on the assumption that WiFi is handled and monitored locally.
Regarding the existing MQTT bridge project.
The API differences with mqtt_as should be easy to code around. For example, consider publish. When the host issues the synchronous publish, return is immediate. The message is queued for transmission to the ESP8266 which then handles the protocol. If you want a qos==1 publication to pause until complete you would write a coro which issued publish, then waited on an Event. The Event would be triggered by a status handler on receipt of the PUBOK status (see pb_status.py).
I still plan to make this bridge V3 compatible and cross-platform; I don't expect this to be difficult and it would make an interesting first project for the Pico. And it will offer an interim solution.
I admire the ambition of your socket project. If you can make this work asynchronously and in a resilient way you will have made a significant contribution to MicroPython.
- Build a resilient socket. Such an object would not throw exceptions but, in the event of a WiFi outage would pause until the outage had cleared. It would offer guaranteed (eventual) communication. Like a full-featured version of the object in our iot project.
- Build a socket that emulates a BSD socket with its various exception states.
A resilient socket would, of course, benefit all protocols, not just MQTT.
Whichever approach you use, mqtt_as will require significant rework to run via a bridge. At the moment it is based on the assumption that WiFi is handled and monitored locally.
Regarding the existing MQTT bridge project.
The API differences with mqtt_as should be easy to code around. For example, consider publish. When the host issues the synchronous publish, return is immediate. The message is queued for transmission to the ESP8266 which then handles the protocol. If you want a qos==1 publication to pause until complete you would write a coro which issued publish, then waited on an Event. The Event would be triggered by a status handler on receipt of the PUBOK status (see pb_status.py).
I still plan to make this bridge V3 compatible and cross-platform; I don't expect this to be difficult and it would make an interesting first project for the Pico. And it will offer an interim solution.
I admire the ambition of your socket project. If you can make this work asynchronously and in a resilient way you will have made a significant contribution to MicroPython.
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
-
- Posts: 969
- Joined: Sat Feb 03, 2018 7:02 pm
Re: Wifi bridge for RPI Pico
I don't plan on building a resilient socket, it's better if the application handles wifi outages and broken sockets correctly as if the wifi was controlled on the device directly.
Adafruit seems to use a standard request library that works with any kind of socket, so this is the goal for mqtt_as too, which should be quite achievable because mqtt_as only uses standard methos with non-blocking sockets.
On the Pico there is no network module so a drop-in would be easy and the goal is to also offer a simple drop in for the socket module.
That way any program can work without any modifications as if the WIFI was connected locally. Not sure I'll get there since a socket is a complex object but at least basic functionality should be drop-in compatible.
I studied the C++ code for the esp32 and it looks good to me and I expect errors to be passed on but my understanding of the C++ isn't very deep and I have to properly test it with circuitpython on the pico, haven't done anything yet.pythoncoder wrote: ↑Tue Feb 02, 2021 9:46 amClearly the level of resilience provided by the Adafruit code dictates how much work is required in either option.
On the contrary, the goal is to not need any modification at all and provide a socket and network class that work as a drop-in replacement.pythoncoder wrote: ↑Tue Feb 02, 2021 9:46 amWhichever approach you use, mqtt_as will require significant rework to run via a bridge. At the moment it is based on the assumption that WiFi is handled and monitored locally.
Adafruit seems to use a standard request library that works with any kind of socket, so this is the goal for mqtt_as too, which should be quite achievable because mqtt_as only uses standard methos with non-blocking sockets.
On the Pico there is no network module so a drop-in would be easy and the goal is to also offer a simple drop in for the socket module.
That way any program can work without any modifications as if the WIFI was connected locally. Not sure I'll get there since a socket is a complex object but at least basic functionality should be drop-in compatible.
In my case it's not quite as easy because I also work with timeouts and cancellations of publications, which means I'd also write code to remove unsent publications from the queue again. But yes, it is possible to work around those issues but I'm not sure it's worth the effort.pythoncoder wrote: ↑Tue Feb 02, 2021 9:46 amIf you want a qos==1 publication to pause until complete you would write a coro which issued publish, then waited on an Event. The Event would be triggered by a status handler on receipt of the PUBOK status (see pb_status.py).
Well now I have to do it Thanks for the pep talkpythoncoder wrote: ↑Tue Feb 02, 2021 9:46 amI admire the ambition of your socket project. If you can make this work asynchronously and in a resilient way you will have made a significant contribution to MicroPython.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
-
- Posts: 969
- Joined: Sat Feb 03, 2018 7:02 pm
Re: Wifi bridge for RPI Pico
Funny story: Thought I could try that CP Project on 2 esp32s so I spent 2 hours trying to flash Circuitpython onto my esp32 with no success until I realized, they only support esp32-s2! So no luck for me.. I don't have any esp32-s2.. only the old ones.. Guess I should have used the Pico from the start
Other problem though: I can't even get the Nina-FW https://learn.adafruit.com/upgrading-es ... -one-board working. I flash it but no output indicating that it works but their Airlift actually uses the same esp32 chip so it should work.. I feel lost
I'll ask on the Circuitpython forum for help..
Other problem though: I can't even get the Nina-FW https://learn.adafruit.com/upgrading-es ... -one-board working. I flash it but no output indicating that it works but their Airlift actually uses the same esp32 chip so it should work.. I feel lost
Code: Select all
ets Jun 8 2016 00:22:57
rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:832
ho 0 tail 12 room 4
load:0x40078000,len:9764
load:0x40080400,len:5700
entry 0x4008069c
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: Wifi bridge for RPI Pico
Oh dear
Thanks for the explanation re sockets. It sounds good.
There is a potential resilience issue which results from their use of SPI. In our development of resilient MQTT solutions, we never had to worry about message corruption. This is because, though TCP/IP can't guarantee message delivery over WiFi, it does guarantee integrity.
Regarding bridges, the design of my SynCom protocol ensures that invalid messages can't be delivered. This does not apply to SPI. If the slave crashes or reboots, the master will continue to clock in garbage. I think CRC or some other means of guaranteeing integrity will be required on the interface. I haven't trawled through the Adafruit code: I wonder if they have paid attention to it? We both know ESP32's keel over from time to time.
Tests I performed on my bridge were to press the reset button on the ESP8266 while it was running to simulate a spontaneous reboot. I also simulate a crash by hitting ctrl-c at the ESP8266 REPL.
Thanks for the explanation re sockets. It sounds good.
There is a potential resilience issue which results from their use of SPI. In our development of resilient MQTT solutions, we never had to worry about message corruption. This is because, though TCP/IP can't guarantee message delivery over WiFi, it does guarantee integrity.
Regarding bridges, the design of my SynCom protocol ensures that invalid messages can't be delivered. This does not apply to SPI. If the slave crashes or reboots, the master will continue to clock in garbage. I think CRC or some other means of guaranteeing integrity will be required on the interface. I haven't trawled through the Adafruit code: I wonder if they have paid attention to it? We both know ESP32's keel over from time to time.
Tests I performed on my bridge were to press the reset button on the ESP8266 while it was running to simulate a spontaneous reboot. I also simulate a crash by hitting ctrl-c at the ESP8266 REPL.
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
-
- Posts: 969
- Joined: Sat Feb 03, 2018 7:02 pm
Re: Wifi bridge for RPI Pico
Hmm yeah you might be right about needing CRC to be really sure we don't receive garbage. I think this could be easily implemented if the rest works as expected.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
-
- Posts: 969
- Joined: Sat Feb 03, 2018 7:02 pm
Re: Wifi bridge for RPI Pico
So a short update of my progress so far:
- Documentation of both projects from Adarfruit are (in my opinion) quite far from perfect, had to dig deep into the code to even find the correct pins (because adafruit only uses their pin labels on their products, which are quite different), looked at schematics etc.
- The documented test sketch for Arduino ESP32 didn't quite work and made me waste many hours thinking my communication between the boards is somehow broken
- Turns out everything was fine. I realized that when I ported some basic functions to micropython to see if the SPI communication would work correctly. Result: It does! I can now speak to the Adafruit WIFI bridge firmware, check the WLAN status, (dis)connect and get the current time. I could have tested that using circuitpython but that only works on the PICO and I haven't soldered the heads onto the pico yet, so can't use it yet.. Only funny thing: Only SoftSPI works, both hardware SPI implementations don't work even though they are supposed to be possible on other pins too but no.. Doesn't matter much though, the esp32 is only for testing at the moment, nobody will use an esp32 to provide wlan to another esp32
- Porting the project to micropython seems quite doable but their use of properties instead of functions is getting on my nerves It kind of feels like working with visual basic and not python but ok..
- For my more ambitious goal of creating a drop-in replacement for the network module: There are quite a lot of functions that work differently, are named differently and have a different return value. So here I'll need some more work on compatibility but that should work just fine, it'll jus take time.
- I can't say anything about resilience yet but I assume it'll do just fine. I will test the cicruitpython examples on the PICO when I have soldered the heads onto it but testing the resilience there might be tricky because circuitpython is different and has no uasyncio.
I'd say let's get this project started I already forked their repository and will be porting it to micropython. Help will be appreciated and any input as well. For now the main goal would be to get the core functionality ported to micropython and in a second stage the abstract socket so we can test resilience. 3rd step would be a drop-in compatible network module and the 4th step would be adding all extra functions like controlling the esp32 pins from the pico (who cares ). The Adafruit WIFI bridge project also supports using Bluetooth over UART but that's obviously not part of the circuitpython SPI project (because it's over UART) and I have no ambition of getting that working either. You can read about BLE support here: https://learn.adafruit.com/adafruit-air ... python-ble
Please let me know what you think.
- Documentation of both projects from Adarfruit are (in my opinion) quite far from perfect, had to dig deep into the code to even find the correct pins (because adafruit only uses their pin labels on their products, which are quite different), looked at schematics etc.
- The documented test sketch for Arduino ESP32 didn't quite work and made me waste many hours thinking my communication between the boards is somehow broken
- Turns out everything was fine. I realized that when I ported some basic functions to micropython to see if the SPI communication would work correctly. Result: It does! I can now speak to the Adafruit WIFI bridge firmware, check the WLAN status, (dis)connect and get the current time. I could have tested that using circuitpython but that only works on the PICO and I haven't soldered the heads onto the pico yet, so can't use it yet.. Only funny thing: Only SoftSPI works, both hardware SPI implementations don't work even though they are supposed to be possible on other pins too but no.. Doesn't matter much though, the esp32 is only for testing at the moment, nobody will use an esp32 to provide wlan to another esp32
- Porting the project to micropython seems quite doable but their use of properties instead of functions is getting on my nerves It kind of feels like working with visual basic and not python but ok..
- For my more ambitious goal of creating a drop-in replacement for the network module: There are quite a lot of functions that work differently, are named differently and have a different return value. So here I'll need some more work on compatibility but that should work just fine, it'll jus take time.
- I can't say anything about resilience yet but I assume it'll do just fine. I will test the cicruitpython examples on the PICO when I have soldered the heads onto it but testing the resilience there might be tricky because circuitpython is different and has no uasyncio.
I'd say let's get this project started I already forked their repository and will be porting it to micropython. Help will be appreciated and any input as well. For now the main goal would be to get the core functionality ported to micropython and in a second stage the abstract socket so we can test resilience. 3rd step would be a drop-in compatible network module and the 4th step would be adding all extra functions like controlling the esp32 pins from the pico (who cares ). The Adafruit WIFI bridge project also supports using Bluetooth over UART but that's obviously not part of the circuitpython SPI project (because it's over UART) and I have no ambition of getting that working either. You can read about BLE support here: https://learn.adafruit.com/adafruit-air ... python-ble
Please let me know what you think.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: Wifi bridge for RPI Pico
I'm puzzled you've had problems with ESP32 hard SPI, but the docs are a little confusing. You have to state pins, even if they are default. I haven't tried non-default pins. I have used this invocation successfully:
I wouldn't bother emulating everything Adafruit does. Controlling pins on the ESP32 sounds pointless when the Pico has plenty of pins. Bluetooth sounds like another project entirely. If you can provide a socket object that supports blocking and nonblocking modes and copes with all the error conditions I'll be very impressed.
Perhaps there is no point in testing the Adafruit code for resilience. It sounds as if you may need to modify the ESP32 firmware to support CRC. Your handling of the SPI protocol will need to be cause the socket to behave in a reasonable way if the CRC fails, if the ESP times out due to a crash or if the ESP spontaneously reboots. There may be a need to physically reboot the ESP if it fails to respond. My MQTT bridge does this: if the communications protocol fails it resets the ESP8266 and re-synchronises.
It may be possible to treat the creation of a socket as a separate exercise from that of ensuring reliability. However at the outset I suggest you consider how the socket should behave if communications suddenly go down and restart.
Code: Select all
spi = SPI(1, 10_000_000, sck=Pin(14), mosi=Pin(13), miso=Pin(12))
Perhaps there is no point in testing the Adafruit code for resilience. It sounds as if you may need to modify the ESP32 firmware to support CRC. Your handling of the SPI protocol will need to be cause the socket to behave in a reasonable way if the CRC fails, if the ESP times out due to a crash or if the ESP spontaneously reboots. There may be a need to physically reboot the ESP if it fails to respond. My MQTT bridge does this: if the communications protocol fails it resets the ESP8266 and re-synchronises.
It may be possible to treat the creation of a socket as a separate exercise from that of ensuring reliability. However at the outset I suggest you consider how the socket should behave if communications suddenly go down and restart.
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.