Avoid flash wearing by altering behavior originally introduced by Espressif (the inventor of the ESP8266)

All ESP8266 boards running MicroPython.
Official boards are the Adafruit Huzzah and Feather boards.
Target audience: MicroPython users with an ESP8266 board.
Post Reply

Would you like to make good-intended, but backward-incompatible changes to WiFi setup?

Poll ended at Sun Jan 08, 2017 7:55 am

Yes, I think this change is good, even though it's backward-incompatible and will confuse some users. We'll get thru it.
36
100%
No, I don't think this change is sound technically or that making backward-incompatible changes of such scale is good.
0
No votes
 
Total votes: 36

User avatar
kfricke
Posts: 342
Joined: Mon May 05, 2014 9:13 am
Location: Germany

Avoid flash wearing by altering behavior originally introduced by Espressif (the inventor of the ESP8266)

Post by kfricke » Tue Nov 08, 2016 11:59 pm

This is a long post, but please let me introduce you to a issue i have with the MicroPython ESP port.
TL;DR - If you do not care about the details, please read the summary below and last three paragraphs at the bottom of this post.


Summary of the nature of the changes proposed (summarized by Paul):
  • This is backwards-incompatible change to the behavior of MicroPython regarding WiFi setup, i.e. functionality each user starts with their usage of MicroPython on ESP8266.
  • Not only this changes MicroPython behavior, this changes "normal" ESP8266 behavior, how everyone who had some experience with ESP8266 got used to.
  • However, in general, this change is good, for example it makes ESP8266 port of MicroPython more compatible with the behavior of other ports.
  • It also helps to avoid wearing FlashROM, which may be a problem for some modules, or some usages, but generally not a problem if you use WiFi setup functions as intended (once, for setup, not to call every second).
A few weeks ago I did stumble upon a pull request #2510 on Github ("esp8266: modnetwork: add optional save_to_flash parameter to connect" - https://github.com/micropython/micropython/pull/2510).
This pull request points out the fact that the "WLAN.connect(<ssid>, <wlan-secret>)" method on the ESP8266 platform does write a special flash segment of the EEPROM, which does hold the flash for the ESP8266 microcontroller. The intent of that pull request is to change the call to WLAN.connnect(...) in the way to introduce an optional argument to avoid the saving to flash.

The flash segment holding the network configuration of the ESP8266 microcontroller is made of an fixed size of 16KB and positioned "at the end of the flash".
This is written in the "NON-OS SDK API Reference" by Espressif. That document describes the "vendor API" often also called "propriatary binary blob" used by the MicroPython ESP8266 port to implement network and other hardware features.

In the discussion of the pull request Paul did mentioned that this is "the natural way" on the ESP8266 from the beginning. Of course he is correct about this fact and everyone who has used Paul's fantastic work we all rely upon, the "ESP Open SDK" (https://github.com/pfalcon/esp-open-sdk) does know that this as is the default behavior for nearly every ESP8266 out in the wild.
There are of course quite a few thousands of persons out there, who are aware of this. Let's say about everyone who does software development on this microcontroller.
So he has a good point on not to change this well-known behavior. And also being the maintainer of this port he of course has the final decision on this topic.

My additional examination made me aware of the fact that not only the method "WLAN.connect(...)" does write the flash segment, but also every writing call to WLAN.active(<state>).

This is bad. Why you should ask of course! I will tell you why:

I have 5 ESP8266 based sensor nodes in my home monitoring different environmental states (temperature, humidity and since last week air pressure). Of course they are now implemented in MicroPython. Each sensor node does wake up every 5 minutes to send sensor data to my MQTT broker. From there I do the usual stuff (feeding this into home automation stack and an InfluxDB for statistics and so on...).

My problem is that 4 of my sensor nodes are battery based and therefore I do need to reduce power consumption whenever possible. At the end of the day the ESP8266 is quite power hungry and not well suited for real low power usage. That of course is the burden of using WLAN for data transmission.

Back to my problem with writing the flash segment: I do deactivate the wireless interface right at boot to save power and also to reduce the impact of RF signals to my environmental sensors. This is done using the following steps:
  1. Call "WLAN.active(False)" to bring down the wireless station interface
  2. Read the sensor(s)
  3. Bringing up the wireless station interface by calling "WLAN.active(True)"
  4. Connect top my WLAN by calling "WLAN.connect(...)"
  5. Send data to the MQTT broker and go into deep sleep for 5 minutes.
Each of my sensors does the above steps 288 times a day, about 8640 times a month and 103.680 times over a whole year.
If you count the "flash writing" methods mentioned above there are 3 writes to the EEPROM in my program on each run every 5 minutes.

So the flash EEPROM of my sensor nodes will die someday after 4 months, because the EEPROM only has more than 100.000 flash and erase cycles (It is a Winbond W25Q16BV 16mb chip).

Of course one could argue several ways:
  • "Let the station interface stay on all the time then!"
  • "Use longer deep-sleep durations!"
  • "Dude, you have a security issue by keeping your WLAN credentials in plaintext! Oops, everyone has stored them in flash anyways, forget this argument!"
  • "Read sensors often but send data once every 20 minutes only!"
  • ...
All of this is correct and could avoid me wearing out my flash chip at the current rate, but not avoid it at all.

But in the world of the C-API (the initially mentioned vendor SDK) we have a cure for this problem: The vendor does provide methods which are avoiding the write to flash. These methods got introduced during the early 1.0er versions of the ESP vendor API.
So I could implement my sensor node in C (e.g. using the easy Arduino oprt for the ESP) and use them right away.
Hell, no! I did not back all official MicroPython kickstarter campaigns to fall back to C for my code.
So I did file a pull request nearly two weeks ago (https://github.com/micropython/micropython/pull/2564). This does alter the behavior of the WLAN.connect(...) and WLAN.active(...) method by not writing to flash. Additionally adding a method to the "esp" module called "save_config()", doing the obvious left-over steps to write the current network configurations to the flash chip. This must be called manually whenever one wants to store network settings.
No changes to the existing API, but adding a new method doing the previously undocumented things when wanted.
Of course this breaks with the well-known behavior on the ESP platform!

And guess how often my sensor nodes would need to write the flash segment: "Not even once... Never ever!"

Paul did argue me down in my pull request then and I did hold tight. Until yesterday night, when he did encourage me to write this story on the forum. What are your thoughts on this topic? Please discuss my pull request here!

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: Avoid flash wearing by altering behavior originally introduced by Espressif (the inventor of the ESP8266)

Post by Roberthh » Wed Nov 09, 2016 7:52 am

Hello @kfricke. I can see your point of beeing anxious, that unneeded writes to flash will shorten the lifetime of a device substantially. And I also agree that your application of the esp8266 is not exotic. So how about a compromise in:
The device boots up with the standard behaviour. The call save_config() woudl have a flag, which modifies the behaviour of wwiring to flash, like save_config(True) would activate the flash writes, dave_config(False) stop that. The default boot-up mode would be True. So, before you device connects, you could issue a save_config(False). If that is never called, everything proceeds as ususal.
I do not know how much the esp-sdk has to be changed for that, but I read from your post, that you already looked into it.

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: Avoid flash wearing by altering behavior originally introduced by Espressif (the inventor of the ESP8266)

Post by pfalcon » Wed Nov 09, 2016 7:59 am

kfricke, thanks for the post, though it's no exactly written how I'd picture it. I also suggested to add a poll. I'm not sure if only moderators can setup a poll, so let me help with it. Also please let me add a summary of the nature of the changes you propose.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: Avoid flash wearing by altering behavior originally introduced by Espressif (the inventor of the ESP8266)

Post by pfalcon » Wed Nov 09, 2016 8:20 am

"Dude, you have a security issue by keeping your WLAN credentials in plaintext! Oops, everyone has stored them in flash anyways, forget this argument!"
And yet previously, it was ESP SDK which did that (and fairly speaking, there's no way around that), you now try to force MicroPython (its users) to do that. Well, you actually thought out a way how to avoid doing, and still use ESP SDK's capabilities on that, so it should be fine, once many people test that it works as expected.

Generally, the biggest problem is that it's backward-incompatible change to something each user starts their work with. I'm in favor to make such change - eventually, in due time, and I'm not sure about the exact API changes towards that. As few users brought up this issue and push it forward now, not in "due time", we have this thread to get everyone ready (it will take some time anyway).
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

User avatar
kfricke
Posts: 342
Joined: Mon May 05, 2014 9:13 am
Location: Germany

Re: Avoid flash wearing by altering behavior originally introduced by Espressif (the inventor of the ESP8266)

Post by kfricke » Wed Nov 09, 2016 8:55 am

pfalcon wrote:kfricke, thanks for the post, though it's no exactly written how I'd picture it. I also suggested to add a poll. I'm not sure if only moderators can setup a poll, so let me help with it.
I did try to do this myself, but it is indeed seems to be restricted to moderators.
Also please let me add a summary of the nature of the changes you propose.
Thanks a lot for this as well!
Last edited by kfricke on Wed Nov 09, 2016 9:00 am, edited 1 time in total.

torwag
Posts: 220
Joined: Fri Dec 13, 2013 9:25 am

Re: Avoid flash wearing by altering behavior originally introduced by Espressif (the inventor of the ESP8266)

Post by torwag » Wed Nov 09, 2016 9:00 am

Hi,
However, in general, this change is good, for example it makes ESP8266 port of MicroPython more compatible with the behavior of other ports.
From a pure Micropython users point of view, I was first surprised to see that the wlan config is kind of persistent. I guess this happens to a lot of users which come from other platforms. Usually, a hard reset is a hard reset. Unlike it is clearly indicated by code or docs that one performs a persistent write operation. That this write operation is kind of hidden in the esp SDK architecture might mislead new esp users in the same way as it might confuse "old" esp users that this feature is gone.

I read the entire pull-request. And argument on both sides seems to be valid. I wonder, do we have or might have similar challenges with other ports or in other areas of persisting writing (e.g. RTC)? If so, might it be an option to introduce some sort of lock/persistent mechanism to the machine class? That is, if call machine.persistent.lock('wifi') writes and reads will go towards RAM instead of flash. If machine.persistent.unlock('wifi') is called writes will go to the designated flash. Sure, there might be the need for something like machine.persistent.status('wifi').
This scheme might be used by other ports and other memory access operations in general. I know this solution would be a rather big thing and many details need clarification. E.g. RAM-size vs. Flash-size, there would be the need to parse each write and read command which addresses flash writes through a function to redirect it (which possibly slows down the process), etc. Thus, I believe this solution is only valid, if there is a higher demand for locking and unlocking memory areas. It most likely make no sense to add all this for a single use scenario.
Anyhow, it would be something, which would be equal for all possible ports and among different functionalities of a SoC.
BTW. this kind of locking unlocking mechanism is not unusual in the embedded world.

User avatar
kfricke
Posts: 342
Joined: Mon May 05, 2014 9:13 am
Location: Germany

Re: Avoid flash wearing by altering behavior originally introduced by Espressif (the inventor of the ESP8266)

Post by kfricke » Wed Nov 09, 2016 9:14 am

pfalcon wrote:
"Dude, you have a security issue by keeping your WLAN credentials in plaintext! Oops, everyone has stored them in flash anyways, forget this argument!"
And yet previously, it was ESP SDK which did that (and fairly speaking, there's no way around that), ...
I does not matter who did threw the first stone. One fact is that the ESP SDK has a way around storing WLAN credentials (and all other interface configurations) to flash by providing all the relevant methods with an alternative '_current' suffixed method, those do avoid flash writes and keeps the configuration non-persistent (volatile) in RAM only. This feature is missing in MicroPython previous to the suggested changes.

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: Avoid flash wearing by altering behavior originally introduced by Espressif (the inventor of the ESP8266)

Post by pfalcon » Wed Nov 09, 2016 9:15 am

From a pure Micropython users point of view, I was first surprised to see that the wlan config is kind of persistent.
I said this in pull request, will repeat here again: How your desktop computer works? Do you re-enter AP password each time you power it on? How your phone works? You enter AP password each time you press power button? Then what you're actually surprised about? ESP8266 with MicroPython is a little personal computer, and works in a way comfortable to user, that's it.

Myself, as a developer, I'd prefer that ESP8266 credentials were not persistent. But as a user, I love that persistence feature very much, it's very practical! A lot of users is used to it, and a lot of those users won't read any of documentation updates, and there will be a blizzard of reports "My esp8266 doesn't remember my AP, help!!!111"
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Avoid flash wearing by altering behavior originally introduced by Espressif (the inventor of the ESP8266)

Post by dhylands » Wed Nov 09, 2016 10:15 am

My desktop persists the SSID/password, but it doesn't re-persist it every time it connects, only when the password actually changes.

User avatar
deshipu
Posts: 1388
Joined: Thu May 28, 2015 5:54 pm

Re: Avoid flash wearing by altering behavior originally introduced by Espressif (the inventor of the ESP8266)

Post by deshipu » Wed Nov 09, 2016 11:04 am

@pfalcon we already know your opinion, thank you, you expressed it multiple times already in all those comments to pull requests. Can you please let the other forum members express their opinion now, as you suggested, so that we can learn where the community stands on this?

Post Reply