Use micropython to do dead simple "over-the-air" code updates

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: Use micropython to do dead simple "over-the-air" code updates

Post by pfalcon » Thu Oct 11, 2018 9:42 am

When you come to that, there's a port to Zephyr RTOS. (Use forum search to find info about it and any other matters.)
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

User avatar
WhiteHare
Posts: 129
Joined: Thu Oct 04, 2018 4:00 am

Re: Use micropython to do dead simple "over-the-air" code updates

Post by WhiteHare » Sat Oct 27, 2018 2:55 pm

I have my first pass on nRF52840 OTA updates (using Nordic's proprietary radio mode) now working with micropython. So, proof of concept works. :D I feel I should improve the code a bit before posting it though. For instance, I should probably add a hash function like MD5 or SHA256 to ensure the entire transmitted update is absolutely correct and devoid of any transmission/reception errors before making it go live.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Use micropython to do dead simple "over-the-air" code updates

Post by pythoncoder » Mon Oct 29, 2018 8:00 am

I'm not familiar with their proprietary radio, but their NRF24l01+ performs checksumming and automatic retransmission. So you can be confident that any packet you receive will be correct.

Where a message comprises multiple packets you need to check that every packet has been received before attempting to reassemble the message: any radio system can drop packets due to interference or moving out of range. This can be as simple as sending the number of expected packets, with each packet having a packet id.
Peter Hinch
Index to my micropython libraries.

User avatar
WhiteHare
Posts: 129
Joined: Thu Oct 04, 2018 4:00 am

Re: Use micropython to do dead simple "over-the-air" code updates

Post by WhiteHare » Mon Oct 29, 2018 12:10 pm

pythoncoder wrote:
Mon Oct 29, 2018 8:00 am
I'm not familiar with their proprietary radio, but their NRF24l01+ performs checksumming and automatic retransmission. So you can be confident that any packet you receive will be correct.

Where a message comprises multiple packets you need to check that every packet has been received before attempting to reassemble the message: any radio system can drop packets due to interference or moving out of range. This can be as simple as sending the number of expected packets, with each packet having a packet id.
You're exactly right. I'm having it use a 3-byte checksum, and I (so far) haven't noticed any corrupted packets, only some missing packets if the file being transmitted isn't small.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Communication protocols

Post by pythoncoder » Tue Oct 30, 2018 11:12 am

So you need a communication protocol. The simplest is for the recipient to require the entire message to be retransmitted if any packet is missing. But the probability of retransmission is then proportional to message length: with long messages in the presence of noise repeated retransmissions mean it can take a long time for the message to get through. That approach really only works with messages comprising a very few packets.

To fix this each packet needs a sequential ID: the receiver then detects a missing packet and requests retransmission of that packet only, which then needs to be slotted in place amongst the received packets. The receiver also needs to know the ID of the last packet in case that one is dropped. Lastly you need a timeout in case the link has failed.

Communications protocols are interesting, and fairly hard to make completely bomb-proof.
Peter Hinch
Index to my micropython libraries.

User avatar
WhiteHare
Posts: 129
Joined: Thu Oct 04, 2018 4:00 am

Re: Use micropython to do dead simple "over-the-air" code updates

Post by WhiteHare » Tue Oct 30, 2018 3:15 pm

Thanks for the suggestions.

There are doubtless many ways to make this better and more bomb proof and faster, but as a very simplified first attempt what I did was transmit each payload 3 times in a row. Each payload contains an identifier, so any received packets that are redundant are just discarded. If for some reason none of the three packets is received, then it will wait and you must re-transmit the file. The receiver would then pick up where it left off when it receives a payload with the expected identifier. i.e. it always monotonically marches toward completion. At the end, as added insurance, it computes an SHA-256 hash on the received file and compares it against the one it receives. If they match, then all is good and the file was successfully received.

Here's what it looks like on the receiving side for a short demo file, with diagnostics turned on:

Code: Select all

MicroPython v1.9.4-651-g0f6f86ca4-dirty on 2018-10-28; PCA10056 with NRF52840
Type "help()" for more information.
>>> start()
rxRadio version 6.000
Starting...
My address is 0xAADEADBEEF
1 Payload received:  1         # This program for a
4 Payload received:  2         n nRF52840 or nRF528
7 Payload received:  3         32 receives a string
10 Payload received:  4          on the radio and pr
13 Payload received:  5         ints the string

16 Payload received:  6         # Start the program
19 Payload received:  7         by typing:

22 Payload received:  8         # start()

25 Payload received:  9         # at the REPL >>> pr
28 Payload received:  10        ompt.

31 Payload received:  11        $$$$$$$$
34 Payload received:  12        89c0d3e1de5a2b68a1e4
37 Payload received:  13        49e2c839cbe7a5e2f184
40 Payload received:  14        1040549e5203e4301876
43 Payload received:  15        412b
46 Payload received:  16        $!$!$!$!


Here is the received file:
# This program for an nRF52840 or nRF52832 receives a string on the radio and prints the string
# Start the program by typing:
# start()
# at the REPL >>> prompt.

Received SHA-256 hash is:   89c0d3e1de5a2b68a1e449e2c839cbe7a5e2f1841040549e5203e4301876412b
Computed SHA-256 hash of received file is  89c0d3e1de5a2b68a1e449e2c839cbe7a5e2f1841040549e5203e4301876412b
Sucess!  Hash values match.  File successfully received.

User avatar
WhiteHare
Posts: 129
Joined: Thu Oct 04, 2018 4:00 am

Re: Use micropython to do dead simple "over-the-air" code updates

Post by WhiteHare » Tue Oct 30, 2018 9:29 pm

I posted working OTA micropython code on github: https://github.com/rabbithat/NRF52840_M ... TA_Updates for both the transmitter node and the receiver node (i.e. the node whose code is to be updated over the air).

In short, you copy the updated receiver code to a file called "update.txt" on the transmitter board. Then you type

Code: Select all

>>> transmit()
at the REPL prompt.

This is how it now looks on the receiving side after the update file is received:

Code: Select all

Received SHA-256 hash is:   f9c6ae33f9b261b91b6127bbd17db7fbce8571469a2c3219ff24d4a0711e303a
Computed SHA-256 hash of received file is  f9c6ae33f9b261b91b6127bbd17db7fbce8571469a2c3219ff24d4a0711e303a
Sucess!  Hash values match.  File successfully received.
Starting copy of  update.txt  onto  main.py
Finished copying  update.txt  onto  main.py
Rebooting....
MicroPython v1.9.4-651-g0f6f86ca4-dirty on 2018-10-28; PCA10056 with NRF52840
Type "help()" for more information.
>>>
Enjoy!

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Use micropython to do dead simple "over-the-air" code updates

Post by pythoncoder » Wed Oct 31, 2018 7:47 am

Great work! Getting the proprietary radio working must have taken a lot of effort. I now feel very tempted to get a couple of those modules.

If I may be permitted a few comments there are ways to make the code more efficient. Each constant uses 8 bytes of RAM: this is because they can be imported from another module. If you prepend the name with '_' Python disallows imports so the MicroPython compiler does not allocate the RAM.

Copying to a buffer can be done more efficiently using a memoryview:

Code: Select all

mv = memoryview(radioBuffer)

def copyStringToRadioBuffer(s):
    l = len(s)
    mv[: l] = s.encode('utf8')[: l]
    mv[l] = 0
Further hints on optimisation were in Damien's talk at PyCon AU notably a recommendation to use short variable names, and to re-use them, to save RAM. This is, of course, contrary to accepted practice so the choice is between clarity and RAM efficiency.
Peter Hinch
Index to my micropython libraries.

User avatar
WhiteHare
Posts: 129
Joined: Thu Oct 04, 2018 4:00 am

Re: Use micropython to do dead simple "over-the-air" code updates

Post by WhiteHare » Wed Oct 31, 2018 4:31 pm

pythoncoder wrote:
Wed Oct 31, 2018 7:47 am
Great work! Getting the proprietary radio working must have taken a lot of effort. I now feel very tempted to get a couple of those modules.

If I may be permitted a few comments there are ways to make the code more efficient. Each constant uses 8 bytes of RAM: this is because they can be imported from another module. If you prepend the name with '_' Python disallows imports so the MicroPython compiler does not allocate the RAM.

Copying to a buffer can be done more efficiently using a memoryview:

Code: Select all

mv = memoryview(radioBuffer)

def copyStringToRadioBuffer(s):
    l = len(s)
    mv[: l] = s.encode('utf8')[: l]
    mv[l] = 0
Further hints on optimisation were in Damien's talk at PyCon AU notably a recommendation to use short variable names, and to re-use them, to save RAM. This is, of course, contrary to accepted practice so the choice is between clarity and RAM efficiency.
Thanks! :D Concrete suggestions on how to improve the code are always welcome. 8-)

I implemented your "_" and memoryview suggestionsand just now posted it to github: https://github.com/rabbithat/NRF52840_M ... TA_Updates I also gave you an attribution for it in the header comments on the files.

I think for clarity's sake I'm going to stick with long variable names, at least for now. With 256K of RAM on the nRF52840, I have more than enough headroom remaining. However, for the long-term it's a good suggestion, and I may look into whittling them down at a later date after the code has been further improved and stabilized.

User avatar
WhiteHare
Posts: 129
Joined: Thu Oct 04, 2018 4:00 am

Re: Use micropython to do dead simple "over-the-air" code updates

Post by WhiteHare » Wed Oct 31, 2018 11:37 pm

pythoncoder wrote:
Wed Oct 31, 2018 7:47 am
Copying to a buffer can be done more efficiently using a memoryview:

Code: Select all

mv = memoryview(radioBuffer)

def copyStringToRadioBuffer(s):
    l = len(s)
    mv[: l] = s.encode('utf8')[: l]
    mv[l] = 0
Whoops! Your code produced an error message:

Code: Select all


Hello.  I am the transmitter node.
My address is 0xAAFEEDBEEF
The target address is 0xAADEADBEEF
Put update code in a file named 'update.txt'
Type 'transmit()' at the REPL propt to begin OTA update transmission.
MicroPython v1.9.4-651-g0f6f86ca4-dirty on 2018-10-27; PCA10056 with NRF52840
Type "help()" for more information.
>>> transmit()
Starting...
Ready to transmit.
Traceback (most recent call last):
  File "<stdin>", in <module>
  File "main.py", in transmit
  File "main.py", in sendShortStrings
  File "main.py", in copyStringToRadioBuffer
TypeError: 'memoryview' object doesn't support item assignment
>>>
The current version (with your code) is on the github server: https://github.com/rabbithat/NRF52840_M ... TA_Updates

Let me know if you can spot the error. If not, I can revert the file (txRadio_v008.py) to the earlier code.

Post Reply