SmartHome with sonoff, esp8266, mqtt and micropython

[torwag]

Hi,

the start of all this you can find here
http://forum.micropython.org/viewtopic.php?f=16&t=2474
a rather cheap wifi relay box, with an esp8266 core.
This thread will cover the findings and ideas how to make a full fledged smart-home device out of it running:

• micropython (otherwise we would not post here )
  MQTT as protocol

This thread might also cover other topics which are remotely related, e.g. the mqtt broker, visualizers, schedulers, hardware-hacks, etc.

[torwag]

Yesterday I had problems to get the mqtt.publish part working.
Strangely, whenever I called the function as
(brain dump)

Code: Select all

c.publish(TOPIC, b"manual_press")

I got the following error message:
TypeError: function takes 2 positional arguments but 3 were given
if I remove one of the arguments,

Code: Select all

c.publish(b"manual_press")

I receive:
TypeError: function takes 3 positional arguments but 2 were given

This seems a bit funny to me. Any idea?

[torwag]

Furthermore,

for the fast switching scheme I planned to use a global variable and a timer and an interrupt, which would reset the variable after x sec. However, a complete scheduler would be even better and solve many problems at other parts.
@pythoncoder, Peter could you make some statements about the differences between your scheduler and uasyncio.

Yet another idea I had was to make sure that all devices essentially run the same code. I do not want to run around plug in a serial adapter or use webreply or any other method to update my units. Furthermore, it would be a mess to keep all those files somehow synced.
I thought of a MQTT topic like /home/autoconfig/ which each new device starts to subscribe too as it starts up for the first time. Within this, each device requests a config and the server will create a proper config like subscribed topics, possible payloads, etc. and send it over, where it get locally stored. The server itself can issue a reconfig command to send in a new configuration. That would allow me to write a small python program on the server, which deals with all the configuration. I would like to hear wether this is feasible and if something like this had been done before.

Finally, I plan to use an extra wifi-network for the nodes with a hidden SSID, just because I want to avoid that many of those little nodes start to pollute my normal wifi network.

@kfricke, Sure I would be very glad if we could work together on this and share ideas. Unfortunately, my time is always very limited, that means my progress will be rather slow, hope this is not a problem for you

[kfricke]

...
This seems a bit funny to me. Any idea?

Not really. IIRC i had a similar issue where i ended up realising a mixture of named and unnamed arguments, but your seems different.

Yet another idea I had was to make sure that all devices essentially run the same code. I do not want to run around plug in a serial adapter or use webreply or any other method to update my units. Furthermore, it would be a mess to keep all those files somehow synced.
I thought of a MQTT topic like /home/autoconfig/ which each new device starts to subscribe too as it starts up for the first time. Within this, each device requests a config and the server will create a proper config like subscribed topics, possible payloads, etc. and send it over, where it get locally stored. The server itself can issue a reconfig command to send in a new configuration. That would allow me to write a small python program on the server, which deals with all the configuration. I would like to hear wether this is feasible and if something like this had been done before.

I use to create code that is reusable accross my sensor-nodes. They simply publish to a topic called 'raw_sensors/<sensor-id>', where <sensor-id> is any unique sensor- or hardware-ID. The inventory is kept on the server side ("Home Assistent" in my case). (edit:) Moving sensors can be handled in the configuration here and is reasonable in my oppinion. (/edited)
There i do also forward the data into my InfluxDB for further visualization.

I will follow this path with my actuators as well (KISS).

Maybe that is reasonable for devices like displays or the like, where you want to keep local configuration near to nothing, but still display depending on a timezone or other things. But even there I would simply publish the whole customized content to a "dumb" display to keep the code the same on all devices of one model/series.

Finally, I plan to use an extra wifi-network for the nodes with a hidden SSID, just because I want to avoid that many of those little nodes start to pollute my normal wifi network.

Sound over-engineered to me. I will stick with my single WLAN until i need to put stuff in quarantine or a DMZ. The toys i do flash my own are trusted by me. And even if you open up a new SSID, the sensor will still pollute your WLAN frequency!

@kfricke, Sure I would be very glad if we could work together on this and share ideas. Unfortunately, my time is always very limited, that means my progress will be rather slow, hope this is not a problem for you

Sound familiar to me. But in the beginning I do appreciate brainstorming and exchanging ideas.

I do plan to publish my final work back to the community on Github. Maybe we can start a collection of MicroPython home automation "bricks" there?

There are several reasons, why one should stick to a scheduler-based approach then. I will try to understand the basics of the async scheduler the next days.
Besides setting up a working directory for the project and collecting the different MicroPython libraries, testing basic umqtt client on the Unix port, i have not really started anything mention-able.

As you mentioned "visualizers" in your initial post. This is the my most viewed page in my smart home. The sensor nodes on the left are of course ESP8266 MicroPython nodes publishing to a Mosquitto MQTT broker, where my Home-Assistant fetches the values from. Home-Assistant also transforms the payload and pushes it onto my InfluxDB, where I use Grafana to paint graphs of them.

Grafana+HomeAssistant.png (132.95 KiB) Viewed 13316 times

(yes, i know we need to air our house asap. We own a passive house and i need to switch on the automatic venting system again. This is my direct motivation for this visulization, btw )

[torwag]

I use to create code that is reusable accross my sensor-nodes. They simply publish to a topic called 'raw_sensors/<sensor-id>', where <sensor-id> is any unique sensor- or hardware-ID. The inventory is kept on the server side ("Home Assistent" in my case). (edit:) Moving sensors can be handled in the configuration here and is reasonable in my oppinion. (/edited)
There i do also forward the data into my InfluxDB for further visualization.
I will follow this path with my actuators as well (KISS).

Ohhh I am all for KISS and I think our ideas are rather similar. I would like to get a generic piece of code running on the nodes and create a way to configure them via MQTT. E.g. One could set the measurement time interval and averaging on sensor nodes, or additional delay times or something like auto-power-off on actuator nodes. My idea here was to create as much flexibility as possible to "reprogram/reconfigure" the nodes without touching the micropython code. However, further thinking makes me wonder if that is really a good way. It would require to implement a set of commands or even a very simple programming language send over MQTT to the nodes. Albeit, there is already a micropython running on them. Maybe it is better to think of something like automatic OTA-updates?! I like to keep the housekeeping centralized as well to make maintenance easier, however, the power of running micropython on each node allows us to think about more autonomy of each node.

My initial idea what to do with a micropython-based home control:
In principle there are two dominating approaches in the smart home world:

1. Dump nodes and a centralized sever, which defines all the logic, rules and does all the work. Nodes simply receive and send data directly addressed to them, without any further processing. That is what 99% of the commercial smart home market stuff is doing.
2. Decentralized systems (this is what large building services management system often use e.g. KNX) whereas each node reads on a bus system and makes decision by itself how to proceed with the data coming from the bus, or simply send data to the bus.

Whereas 1 is relatively simple and for the home market commercially of bigger interest (most companies try to sell you some sort of cloud service with monthly fees and try to create a gated environment), version 2 is much more robust. Even if parts of the network fails, other parts can still communicate with each other. However, it is usually much more expensive as it requires enough processing power in each and every node.

I believe a MQTT + Micropython gives us the best of both worlds. The MQTT broker is still a centralized service. However, it is rather simple, one could replace/move it within minutes without much reconfiguration. It is also rather resource friendly and can run on the wifi router itself, a Rpi or anything else.
Micropython will allow us, to create easily smart nodes, and the rules/logic could be implemented in the node itself. That is each node reads the MQTT topics and acts accordingly to it.
A simple example: A light should turn on if a light-sensor indicate darkness and a switch is pressed. It should turn off after 10 min. Conventionally, this rule would have been defined in the centralized system.
Read status light-sensor
Read status button
If light-sensor is low and button is pressed send command to actuator
Actuator switches the light on
After 10 min, send actuator to switch the light off.
The problem here, if the actuator is not longer reachable e.g. after the light has been switched on, it will not go off anymore, or one has to deal with complex error handling etc.

With MQTT and micropython the node itself could take care of that rule by itself.
Read last status light-sensor via MQTT
Read last status switch via MQTT
If light-sensor is low and switch is on, pull the relay and hence turn on the light (here I am not sure, see above, if that rule could not be a MQTT payload as well)
Set timer with value from MQTT
Turn off light as timer finishes.
All this would happen in the actuator node and is rather simple to do in micropython.
I would love to hear opinions for such a system.

Finally, I plan to use an extra wifi-network for the nodes with a hidden SSID, just because I want to avoid that many of those little nodes start to pollute my normal wifi network.

Sound over-engineered to me. I will stick with my single WLAN until i need to put stuff in quarantine or a DMZ. The toys i do flash my own are trusted by me. And even if you open up a new SSID, the sensor will still pollute your WLAN frequency!

Well I should have been more precise here, I meant to say I go with an extra (hidden) SSID. Simply to achieve a clean and easy way to manage those nodes. I configured it in a way, that none of the nodes can directly access or being accessible from outside. The only connection here is the server which runs the mqtt broker and the home-assistant software. I have some tools which do network scans and I do not want to see many of strangely named nodes within the list of the usual network clients. It might also add some sort of safety.

Home-Assistant also transforms the payload and pushes it onto my InfluxDB, where I use Grafana to paint graphs of them.Grafana+HomeAssistant.png
(yes, i know we need to air our house asap. We own a passive house and i need to switch on the automatic venting system again. This is my direct motivation for this visulization, btw )

I'm new to Home-Assistant and never heard of Grafana, I will follow up on this. Are you satisfied with Home-Assistant? I still have problems to understand the concept. BTW. as I saw your graphs, in which part of Germany do you live. If it is close to Aachen, we might can chat about this vis-a-vis

[kfricke]

...

I'm new to Home-Assistant and never heard of Grafana, I will follow up on this.

Grafana is just "The leading tool for querying and visualizing time series and metrics" as they describe on their homepage.

Are you satisfied with Home-Assistant? I still have problems to understand the concept.

I did set it up ony a few months back. Personally i am also not crazy in love with the whole home automation stuff. Just got in touch with it while connecting Philips Hue light bulbs with my home network and wanting to control them without the vendor tools.
At the beginning i got confused with the different entity ids in the internal inventory of Home Assistant. Especially locating the entities for use in the configuration file was not very self-explanatory (maybe i should have put more focus on reading their documentation). What i got used to was checking the known entities in the web UI at the URL path '/dev-state'. There you can find all entities by the correct name (light bulbs, streaming boxes and sensor nodes in my case).
As already mentioned, basically i did start using Home Assistant as a MQTT -> InfluxDB bridge and am now discovering all its other use-cases.

BTW. as I saw your graphs, in which part of Germany do you live. If it is close to Aachen, we might can chat about this vis-a-vis

I do live close to Hannover, sry!

[dhylands]

To control my Hue lights, I use a python script, which in turn uses this library: https://github.com/quentinsf/qhue and then use crontab (under linux) to invoke it periodically.

[kfricke]

To control my Hue lights, I use a python script, which in turn uses this library: https://github.com/quentinsf/qhue and then use crontab (under linux) to invoke it periodically.

Home Assistant by itself does support the Philips Hue lights. It just works like a charm here too.

[pythoncoder]

@torwag To answer your question about my scheduler, uasyncio is the official way to implement cooperative multi-threading and arguably should be used where possible. However its current implementation has a timing granularity of one second. Though fine for web applications, it's too slow for many purposes including user interfaces. Mine offers a faster solution using an established technique of using generators to implement "micro threads". It has classes for specific devices such as switches with debouncing and pushbuttons also supporting double-clicks and long presses.

[torwag]

Mine offers a faster solution using an established technique of using generators to implement "micro threads". It has classes for specific devices such as switches with debouncing and pushbuttons also supporting double-clicks and long presses.

Thanks for the info. I will give both a test and see how it works out. One second might be rather long for our case. Is there also any metric about resources? Maybe one is lighter then the other?

I still struggle to get mqtt.publish working. Still have no clue why I get this strange error message