Downloading 1GB of data with MicroPython ESP8266 port

[pfalcon]

Recently, there's growing discussion on how to develop reliable, long-running applications in MicroPython for ESP8266. Writing reliable applications is a separate science from writing just applications, and not specific to MicroPython or ESP8266. Just consider that developing a reliable, always-running application in "big" Python for a standard desktop/server is pretty different from writing a quick app which you start manually, which runs for couple of seconds or minutes and then terminates. For MicroPython and ESP8266 specifically, the reliability depends at least on several factors:

• Reliability of MicroPython itself. We have an extensive testsuite with more than 90% code coverage, which can run on a device itself. All these traits are much more than you usually find in other embedded languages, or just embedded projects. And yet the issues are known too, the biggest being memory fragmentation. It's not specific to MicroPython, but to any language, or application, using dynamic memory allocation. Python as a language goes farther than most scripting languages in alleviating this problem my supporting in-place buffer operations, and in MicroPython, we go even further by exploring additional ways of no-allocation operations.
• Reliability of a wireless network. This itself depends on many things, which roughly can be broke down to: a) how well your particular ESP8266 board is designed regarding RF part (antenna, RF part screening, etc.); b) how well your router/access point works; c) how well ESP8266 and your router work together - they can work pretty well on their own, and yet "not like" each other; d) what kind of WiFi signal propagation and interference you get. All these parameters are hard to quantify. Everyone has a different router. You can't rely that a particular board model will always work well - you particular examplar may have soldering problem in RF part or degraded component. And signal interference picture just changes every second and most people have no idea what it's like at all.
• Reliability of ESP8266 chip. ESP8266 literally popped out from nowhere, from a company nobody knew, and using obscure architecture. Nobody knows how well it *can* work. Maybe there're hardware bugs and after executing a billion of instructions CPU crashes. Sounds not very plausible, but maybe there's no enough frequency interference protection and CPU may crash with some probability when WiFi signal is active. Even if CPU is ok (it likely is of course, but try to prove that), WiFi firmware has many issues. It's known that it has dozens of "asserts" which cause ESP8266 to hang (and then hopefully to be reset by WDT). So, ESP8266 itself is only as much reliable as you can expect from two-dollar thingy.

Summing up the above, we're committed to make MicroPython itself as reliable as possible. We're also try and will try to do the same for ESP8266, but given that its firmware is closed-source binary blob, the progress is slow, and there's only so much we can do. So, as the matter of fact, your ESP8266 will reboot. And if you want to write reliable application, you should deal with that (and MicroPython offers all needed for that - you can run your app on boot, you can save state to file system or non-volatile memory, you can cause reboot yourself if you detected that something went wrong, etc.) Between the above, it's our learning together with the community how to write software reliably working under such conditions. For example, it's already clear that any networking code should use timeouts when working over non-reliable medium like WiFi, or it risks to get stuck waiting to reply which may never arrive.

With all that in mind, everyone wonders - how reliable ESP8266 running MicroPython can be? Well I have mine running for some 15 or so hours, and have been downloading more than 1GB of data over this time. There's a detailed log below. What it specifically does is downloading a pretty large file (23MB) and checking its SHA1 checksum. As you can see, there's no checksum failures. However, some attempts to download fail, but the script just retries (such partial downloads aren't counted towards 1GB figure, so actually there're more bytes pumped thru, 1GB is fully successfully downloaded total amount):

Code: Select all

Size : 24267216
Speed: 23541.09 bytes/s
Elapsed: 1030.845
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 23M
Size : 24267216
Speed: 25311.88 bytes/s
Elapsed: 958.728
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 46M
Size : 24267216
Speed: 24325.42 bytes/s
Elapsed: 997.6069
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 69M
Size : 24267216
Speed: 24243.5 bytes/s
Elapsed: 1000.978
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 92M
Traceback (most recent call last):
  File "<stdin>", line 70, in main
  File "<stdin>", line 51, in dl
  File "<stdin>", line 36, in dl
OSError: [Errno 110] ETIMEDOUT
ESP bufs: tx:7 rx:7
Size : 24267216
Speed: 24045.56 bytes/s
Elapsed: 1009.218
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 115M
Size : 24267216
Speed: 22934.12 bytes/s
Elapsed: 1058.127
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 138M
Size : 24267216
Speed: 25274.63 bytes/s
Elapsed: 960.1408
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 162M
Size : 24267216
Speed: 25673.66 bytes/s
Elapsed: 945.218
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 185M
Traceback (most recent call last):
  File "<stdin>", line 70, in main
  File "<stdin>", line 51, in dl
  File "<stdin>", line 36, in dl
OSError: [Errno 110] ETIMEDOUT
ESP bufs: tx:8 rx:7
Size : 24267216
Speed: 23118.81 bytes/s
Elapsed: 1049.674
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 208M
Size : 24267216
Speed: 24269.18 bytes/s
Elapsed: 999.9189
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 231M
Size : 24267216
Speed: 24255.59 bytes/s
Elapsed: 1000.479
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 254M
Size : 24267216
Speed: 24148.48 bytes/s
Elapsed: 1004.917
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 277M
Traceback (most recent call last):
  File "<stdin>", line 70, in main
  File "<stdin>", line 51, in dl
  File "<stdin>", line 36, in dl
OSError: [Errno 110] ETIMEDOUT
ESP bufs: tx:8 rx:7
Size : 24267216
Speed: 25642.33 bytes/s
Elapsed: 946.3728
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 300M
Size : 24267216
Speed: 23244.39 bytes/s
Elapsed: 1044.003
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 324M
Size : 24267216
Speed: 24581.73 bytes/s
Elapsed: 987.2048
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 347M
Traceback (most recent call last):
  File "<stdin>", line 70, in main
  File "<stdin>", line 51, in dl
  File "<stdin>", line 36, in dl
OSError: [Errno 110] ETIMEDOUT
ESP bufs: tx:7 rx:7
Traceback (most recent call last):
  File "<stdin>", line 70, in main
  File "<stdin>", line 51, in dl
  File "<stdin>", line 36, in dl
OSError: [Errno 110] ETIMEDOUT
ESP bufs: tx:8 rx:7
Size : 24267216
Speed: 23450.12 bytes/s
Elapsed: 1034.844
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 370M
Size : 24267216
Speed: 24156.14 bytes/s
Elapsed: 1004.598
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 393M
Size : 24267216
Speed: 22709.67 bytes/s
Elapsed: 1068.585
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 416M
Size : 24267216
Speed: 23077.93 bytes/s
Elapsed: 1051.533
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 439M
Traceback (most recent call last):
  File "<stdin>", line 70, in main
  File "<stdin>", line 51, in dl
  File "<stdin>", line 36, in dl
OSError: [Errno 110] ETIMEDOUT
ESP bufs: tx:8 rx:7
Size : 24267216
Speed: 23226.08 bytes/s
Elapsed: 1044.826
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 462M
Size : 24267216
Speed: 24413.42 bytes/s
Elapsed: 994.0109
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 486M
Size : 24267216
Speed: 24258.38 bytes/s
Elapsed: 1000.364
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 509M
Size : 24267216
Speed: 23165.12 bytes/s
Elapsed: 1047.575
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 532M
Traceback (most recent call last):
  File "<stdin>", line 70, in main
  File "<stdin>", line 51, in dl
  File "<stdin>", line 36, in dl
OSError: [Errno 110] ETIMEDOUT
ESP bufs: tx:8 rx:7
Size : 24267216
Speed: 24408.46 bytes/s
Elapsed: 994.2129
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 555M
Size : 24267216
Speed: 24790.73 bytes/s
Elapsed: 978.8818
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 578M
Size : 24267216
Speed: 24598.1 bytes/s
Elapsed: 986.5479
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 601M
Size : 24267216
Speed: 23867.01 bytes/s
Elapsed: 1016.768
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 624M
Traceback (most recent call last):
  File "<stdin>", line 70, in main
  File "<stdin>", line 51, in dl
  File "<stdin>", line 36, in dl
OSError: [Errno 110] ETIMEDOUT
ESP bufs: tx:6 rx:6
Size : 24267216
Speed: 23957.37 bytes/s
Elapsed: 1012.933
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 648M
Size : 24267216
Speed: 24363.49 bytes/s
Elapsed: 996.0478
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 671M
Size : 24267216
Speed: 25075.26 bytes/s
Elapsed: 967.7749
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 694M
Size : 24267216
Speed: 23274.35 bytes/s
Elapsed: 1042.659
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 717M
Traceback (most recent call last):
  File "<stdin>", line 70, in main
  File "<stdin>", line 51, in dl
  File "<stdin>", line 36, in dl
OSError: [Errno 110] ETIMEDOUT
ESP bufs: tx:7 rx:7
Size : 24267216
Speed: 24284.19 bytes/s
Elapsed: 999.301
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 740M
Size : 24267216
Speed: 23966.59 bytes/s
Elapsed: 1012.543
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 763M
Size : 24267216
Speed: 25720.49 bytes/s
Elapsed: 943.4968
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 786M
Size : 24267216
Speed: 22991.51 bytes/s
Elapsed: 1055.486
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 810M
Traceback (most recent call last):
  File "<stdin>", line 70, in main
  File "<stdin>", line 51, in dl
  File "<stdin>", line 36, in dl
OSError: [Errno 110] ETIMEDOUT
ESP bufs: tx:8 rx:7
Size : 24267216
Speed: 24140.11 bytes/s
Elapsed: 1005.265
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 833M
Size : 24267216
Speed: 24635.71 bytes/s
Elapsed: 985.042
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 856M
Size : 24267216
Speed: 23011.41 bytes/s
Elapsed: 1054.573
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 879M
Size : 24267216
Speed: 23369.26 bytes/s
Elapsed: 1038.424
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 902M
Traceback (most recent call last):
  File "<stdin>", line 70, in main
  File "<stdin>", line 51, in dl
  File "<stdin>", line 36, in dl
OSError: [Errno 110] ETIMEDOUT
ESP bufs: tx:8 rx:6
Size : 24267216
Speed: 24192.55 bytes/s
Elapsed: 1003.086
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 925M
Size : 24267216
Speed: 25541.69 bytes/s
Elapsed: 950.1018
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 948M
Size : 24267216
Speed: 24125.42 bytes/s
Elapsed: 1005.877
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 972M
Size : 24267216
Speed: 25575.28 bytes/s
Elapsed: 948.854
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 995M
Traceback (most recent call last):
  File "<stdin>", line 70, in main
  File "<stdin>", line 51, in dl
  File "<stdin>", line 36, in dl
OSError: [Errno 110] ETIMEDOUT
ESP bufs: tx:8 rx:7
Size : 24267216
Speed: 16359.03 bytes/s
Elapsed: 1483.414
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 1018M
Size : 24267216
Speed: 15363.08 bytes/s
Elapsed: 1579.58
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 1041M
Size : 24267216
Speed: 20283.23 bytes/s
Elapsed: 1196.418
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 1064M
Traceback (most recent call last):
  File "<stdin>", line 70, in main
  File "<stdin>", line 51, in dl
  File "<stdin>", line 36, in dl
OSError: [Errno 110] ETIMEDOUT
ESP bufs: tx:7 rx:7
Size : 24267216
Speed: 24370.77 bytes/s
Elapsed: 995.751
SHA1 : 9602e52394ac820c3121bb394ad92520e1a7bc0a
Total downloaded so far: 1087M
16858K

[pfalcon]

The test script is available as https://github.com/pfalcon/micropython- ... test_dl.py

Now, how I arrived at being able to download 1GB in one go. It wasn't easy. Let's start with saying that when testing ESP8266 reliability, you should avoid connecting via your main access point - that adds another unknown into the testing. Connect directly to ESP8266's own AP. Of course, if you need Internet access, you have to use your Internet AP. Just use it well. My work desk is located ~8-10m from an AP, thru 1 wall and a complex wall configuration (junction of few walls). I also live in a typical "WiFi hell", that means that # of APs I can scan around starts at 2 dozens and can go up. Note that in such configuration, I can use WebREPL when connected via external AP ("can use" means "can use", not that WebREPL can go for months, weeks, or days without dropping). But from previous experience with ESP8266 I know that there's no talk about downloading not gigabytes, but mere gigabytes in such conditions - speed will be dismal and connection will drop again and again. So, I don't even try.

I move for module to be within 1-2m from AP, across a brick wall (can't get closer than that). What I do next is what nobody else seems to be ever doing - I measure WiFi signal level. I don't measure it once, I measure it over period of time. To that end, I wrote a simple script: https://github.com/pfalcon/micropython- ... rssi_watch . I'm surprised nobody wrote it before (Or did I miss it? I can't reply in time, but I watch forum.) So, here's result of it running in those 2-meter-from-AP conditions for 2 mins:

Code: Select all

02:03 min: -53 cur: -51 max: -48 missed: 0 failed: 0

Note that in some more seconds it becomes:

Code: Select all

02:20 min: -72 cur: -72 max: -48 missed: 0 failed: 0

Does it mean that I get interference once in 2 minutes? No, it's always there (suppose, every few seconds), just scans happen relatively rarely, so don't coincide for some time. The script above of course should be extended to show average RSSI and perhaps dispersion. Patches welcome. (When writing them, you of course keep in mind that the script should work even on WiPy, so don't use floats.)

So, now we know under which conditions ESP8266 can download 1GB of data: if signal (RSSI) is at sustained -50 level and "dips" happen no more frequently than once in few minutes as detected by scan.

Can it work with lower signal? Dunno, if you're interested, try it and share your experiences. However, if you're interested in downloading 1GB, get adequate level of the signal. Without good signal nothing will work, sorry. Physics.

(to be continued)

[markxr]

That is very interesting.

I have noticed that the new build of Micropython with the new firmware runs my "stumbler"-type application much more robustly than the previous one.

There were some cases where it would crash inside scan(), which I could never figure out. Now it appears that scan() crashes seldom or never. Sometimes there may be too many accesspoints (or not enough memory) to give a reasonable result, but it just raises OSError instead of crashing.

I'm not 100% sure this is what happens because I can't easily test it - it happens when the device is "in the field" and can't be easily debugged.

The reliability of the ESP-12 modules Flash is something else, I think I've bricked two so far

[ernitron]

Thank you very much for sharing your thoughts and benchmark.

When I was approaching micropython my first issue was the reliability (maybe you remember). You asked to me what I meant by that and pointed out the different levels of stability as you further clarify here. I have to say that version 1.8.5 has proved to be much more stable (my deployed applications last days without rebooting) and I acknowledge that you and the core team outperformed what I considered a minimal standard for reliability. Again thank all the micropython dev team and also this great community from which I learned a lot.

I use esp8266 and micropython as a proof of concept for some applications that could and eventually will go industrial. I strongly agree that the intrinsic unreliability of the device should be taken into account. This requires a shift in programming in the same way as the limited resources of the device must be taken into account.

I already discussed in this forum about a feature that should be included in the firmware like rebooting in some circumstances like out of memory and even code exceptions. I will not get back to that but my point here is that both the development framework (micropython and runtime VM) and the applications should have a different approach with 2dollars devices when it comes to reliability. ESP8266 is not the first and will not be the last one of the target device we will work on. We know they can crash, we know that wifi, hardware and especially applications will be more unstable than big, full-stack devices. That's why I think micropython and VM should support reliability by design in applications.

The support probably should comprise statefull mechanisms for easily recovering unexpected crashes and reboots and more resilient networking code and memory management implemented in the VM rather than in the applications. A big challenge and probably beyond scope, I know.

I guess (but maybe I am wrong) your point is a sort of "limited liability" of micropython but nobody (not me for sure) is asking anything more than a robust environment. And you, the core developers, have already achieved that with the actual implementation.

My point is that this project is not just to demonstrate that python can be successfully ported on awkward limited devices. It has already been proved beyond expectations! I think this project, the core developers and the community should address what it really means to write real world applications for industrial and maybe consumer worlds with acceptable reliability (managed risk) in low-price scarce-resources devices to make developers and end users positive that an IoT world will not be a nightmare of insecure, fragile and ultimately unusable things.

Sorry for the (too) long post.

[deshipu]

For example, it's already clear that any networking code should use timeouts when working over non-reliable medium like WiFi, or it risks to get stuck waiting to reply which may never arrive.

I wonder how all the Linux applications running on laptops and smartphones get away with not doing that. Perhaps those sockets are somehow different?

[Lysenko]

I wonder how all the Linux applications running on laptops and smartphones get away with not doing that. Perhaps those sockets are somehow different?

They're different in that they have a full socket.setsockopt implementation that supports SO_RECVTIMEOUT and they're running under a pre-emptive multi-threaded environment. That's before you get into completion callbacks and message loops like node.js, NGINX or twisted etc. At least some of the MP socket code only implements SO_REUSEADDR and errors out everything else.

[bitninja]

...a feature that should be included in the firmware like rebooting in some circumstances like out of memory and even code exceptions.

I have had similar thoughts when working on an application... it is very troublesome to have an application quit without any kind of indicator that it is not running anymore. If you can implement some kind of "heartbeat" that indicates your app is still running, that is one way... but this is not always desired.

I think MicroPython is very robust at this point, but I think it needs some tweaks to make crash recovery more controlled.

[Damien]

I had a go with @pfalcon's test_dl.py script but didn't get very good results: after running for a few hours I only managed 1 full download of the 24mb file (SHA passed). Other tests either had ETIMEOUT or only downloaded part of the file and failed the SHA. It ran at about 6k/s which is pretty slow and is likely a problem with the local wifi that I used for testing.

But there were no problems with the actual software/firmware/hardware. It kept trying to download and would have continued longer if I let it. I have had much better success (and speeds) in the past with other wifi networks.

[Roberthh]

I gave it a try this morning. Even if the speed is fast (~72/s), I very ofte run ito a lock situation, where it hangs forever.

Device: Huzzah Feather. Log:

Code: Select all

MicroPython v1.8.5-121-g5c3d75c on 2016-11-06; ESP module with ESP8266
Type "help()" for more information.
>>> import test_dl
Traceback (most recent call last):
  File "test_dl.py", line 70, in main
  File "test_dl.py", line 51, in dl
  File "test_dl.py", line 36, in dl
OSError: [Errno 110] ETIMEDOUT
ESP bufs: tx:0 rx:7
Traceback (most recent call last):
  File "test_dl.py", line 70, in main
  File "test_dl.py", line 16, in dl
OSError: -2
ESP bufs: tx:0 rx:7
Traceback (most recent call last):
  File "test_dl.py", line 70, in main
  File "test_dl.py", line 16, in dl
OSError: -2
ESP bufs: tx:0 rx:7

And then the Error -2 continues forever, and I need a reset to recover. The file itself can be downloaded.
I have a test on two devices, one Huzzah and one Wemos.
Pass: 3
Fail: 11

[Roberthh]

Update: It looks as if the problem with my test was between keyboard and chair. The test code activated station mode, which I do by default in main.py. And setting station mode twice confuses either the esp8266 or my access point. Now, the test runs much better, although at sometimes I still get the TIMEOUT situation, and the device does not recover.