Downloading 1GB of data with MicroPython ESP8266 port

[pfalcon]

I had a go with @pfalcon's test_dl.py script but didn't get very good results: after running for a few hours I only managed 1 full download of the 24mb file (SHA passed). Other tests either had ETIMEOUT or only downloaded part of the file and failed the SHA. It ran at about 6k/s which is pretty slow and is likely a problem with the local wifi that I used for testing.

Yes, that's why I started the message with discussing non-reliability factors against which we fight, and there're many. And why there's a second "reserved" message, it's supposed to contain description how I arrived at being able to have downloaded 1GB of data in one session without crashes. That's actually the whole point of this write up: that without sensible and reproducible setup, it makes no sense to test it or expect to work in the long run - it won't, there're too many factors playing against esp8266 and its user. As a sneak peek, it took me 3 days and a dozen of tries (each few or multi-hour) to arrive there. So, ESP8266 *can* download 1GB in one go. Can it do worse than that? Oh, very, very easily!

[pfalcon]

The test code activated station mode, which I do by default in main.py.

Yes, sorry, that's the problem with it - I have all these test scripts, but they need to be cleaned up for public consumption. Activating STA was in particular an artifact of me trying to do "fault injection", like suddenly shutdown STA while download runs and see what comes out (it didn't crash, as in "completely"). I now updated test by the link above to comment it out. I also increased socket timeout to 10s. You see, even 5s is long enough for real-time things like MQTT, but apparently, too short for "batch" stuff, like big downloads. (But changing timeout may change results considerably - in any direction.)

And setting station mode twice confuses either the esp8266 or my access point.

That would be strange and bad. I would submit a good, detailed bugreport on that.

[pfalcon]

I posted some new content at the "reserved" message above, http://forum.micropython.org/viewtopic. ... 348#p15348

[Roberthh]

Hi @pfalcon, just a few results.
- Setting station mode twice does not seem the problem, only how. My AP takes a while to connect. If I add a loop waiting to be connected it works all the time.
- Achieved download sizes vary a lot, between 6kBytes and 480 MBytes.
- On one of the devices I had already set the timeout to 30 seconds, which caused WDT to trap.
What puzzles me is, that once the device it gets into timeout, it does not recover and sticks at the statement:

Code: Select all

    ai = socket.getaddrinfo(host, 80)

It stick in getting to that statement by timeout/retry over and over, until the device eventually resets. Before that, only reset will bring it to a clean condition again.
I'm using the home AP., not a dedicated one. Signal level is between -50 and -60 db. There are typically about 10 WiFi networks in reach.

[Roberthh]

Just for fun, I started that test again yesterday evening, and it still runs on one device, now at over 6GB. It still gets timeouts, but now recovers from timeout. The run lengths between timeouts are between ~100M and ~1.2G. Changes to previous set-up:

a) Version 1.8.6-2 of MicroPython
b) AP_IF disabled
c) a gc.collect() call in the loop receiving the file.

The device used is a Huzzah feather.
A second device (Wemos D1 mini) w/o change a) ran into reset after about 250m, but with a recovered timeouts in between. I restarted it now with version 1.8.6-2. Let's see how far it gets.

[Roberthh]

So, after 60 hours and reaching 15GB I stopped the test with the huzzah feather device. I guess it would have continued like that. It had timeout errors, but recovered from these. The average run length between timeout error was about 500MBytes.

Also interesting are the test with the Wemos D1, which showed to be less robust. These devices seem to have a lower quality USB plug, because that get's warm during use. Switching off the AP interface reduced the power consumption and thus the temperature, but still this is a bad sign.
So I took one Wemos D1 and powered it by the 5V/Gnd inputs directly from a (very old) lab power supply, not connected to USB. That gives also the internal regulator a little bit larger range for regulation. The control output runs through webrepl.

This devices runs stable, reaching about 5 GB data now.
First lesson learned for Wemos D1: If you need stable operation, do not supply it through the USB Plug.

I have attached the schematics of Wemos D1 mini, which I found on the net, just for interest.
It shows a voltage divider between the module's A0 pin and the ESP module's analog input, meaning that the DC range on A0 is 3.3 V, different from NODEMCU and Huzzah Feather.

wemos-d1-mini-shematics.zip

(90.73 KiB) Downloaded 314 times

[pfalcon]

Great work, Roberthh, thanks for taking it further and virtually confirming that ESP8266 can sustain, under high network load, extended run-times, subject to favorable all other conditions. So, my general experience and notes for when I ran the test myself (and which I yet need to write down in this thread) are largely match yours. There're some incorrect concepts however, let me go over those first:

On one of the devices I had already set the timeout to 30 seconds, which caused WDT to trap.

I don't know if you meant to word it like that or not, but setting timeout 30s didn't cause WDT to trap. Not more than your getting out of house can cause rain to start. The latter may happen any time, 2 times in 2 days, and each of us had that dozens of times throughout of life, but those are all coincidences. If we want to get to the bottom of it, it's important to use non-confusing language and terminology, especially that there maybe readers ho did't take probability theory courses last 20 years. Speaking of probability theory, there's always a (small) chance that setting timeout could cause WDT to trap - it's software after all, the land of colored ponies and unicorns. But saying "setting timeout to 30 seconds caused WDT to trap" doesn't help anyone still. Please provide a reproducible testcase in a github ticket. This is the whole point of this thread - Let's be precise (or we're chasing, blindfolded, a cat in a dark room which may be not even there).

[pfalcon]

c) a gc.collect() call in the loop receiving the file.

I purposedly didn't do that, nor used .readinto(). The whole idea of that test_dl.py os to show that MicroPython on its side can handle it even without that, and to see what would be needed on the side close to ESP8266 (that's, operation timeouts, exception handling, etc.)

[pfalcon]

Also interesting are the test with the Wemos D1, which showed to be less robust. These devices seem to have a lower quality USB plug, because that get's warm during use.

This, and the rest of your last message is very insightful, thanks much for going thus far. But I'd like to get back to "be precise" mantra. USB plug gets warm - how do you know, tell us how you found that out. And I'd like to say that I would personally consider the only "scientific" method is to look at the entire board via infrared camera (imager). Also, did you measure power consumption of the board when this happens?

I understand that all this takes times and resources in general, and of course I wouldn't expect all that done at once, but if you could be more specific of what you've done so far, that would be helpful. That will other people to compare their results with yours and continue your experiment.

Thanks again for doing the above experiments and sharing them with us!

[pfalcon]

So, after 60 hours and reaching 15GB I stopped the test with the huzzah feather device.

I'm glad that to hear that about Adafruit Huzzah device. That's of course why it was the recommended device for MicroPython during the Kickstarter campaign, and remains such. We expected it to be of high quality and have good support, and I'm glad there're facts to put behind the expectations.