Pico W Micro Python MQTT of data to a RP4B

[beetle]

Your global var only gets set when the on_message function is called (when a mqtt message is received)
Your program whips through the setting up of the mqtt stuf and find a print statement to print a global variable is knows nothing about until a mqtt message arrives. So it complains and rightly too!

[Rissy]

Of course now that the "if topics " lines are successfully being dipped into, I've now tried writing these individual message payloads into individual .txt files. This is now successful too (thank you very much beetle!)

However, to get my main code to have to open and read these constantly updating .txt files is a bit clumsy/nasty in my opinion.

I'd much rather just take the values as global variables by calling from this script from my main script.

Remembering of course the last time i tried this, my main script got completely taken over by my MQTT script!?

[Rissy]

Your global var only gets set when the on_message function is called (when a mqtt message is received)
Your program whips through the setting up of the mqtt stuf and find a print statement to print a global variable is knows nothing about until a mqtt message arrives. So it complains and rightly too!

Ok, so how do i do it then?

It seems to be a chicken and egg scenario!

[Rissy]

Right , i tried this (at the bottom and inside the if statements). No cigar. Any help?

Code: Select all

#!/usr/bin/env python
#!/bin/bash
#MQTT_PW_Vars4.py

import time
import paho.mqtt.client as mqtt

#initial screen message with delay attached before running program
print("---------------------------------------------------------")
print("---Pico W Receiving Program is loading, please wait...---")
print("---------------------------------------------------------")
time.sleep(2)

broker = '192.168.0.51'
port = 1883
mqttclient = "PiLogger"
client = mqtt.Client(mqttclient)

#topic1 = "PicoW/OS_Temp"
#topic2 = "PicoW/OS_Hum"
#topic3 = "PicoW/Pico_Temp"

#topic1 = 'PicoW/OS_Temp'
#topic2 = 'PicoW/OS_Hum'
#topic3 = 'PicoW/Pico_Temp'

topic1 = 'OS_Temp'
topic2 = 'OS_Hum'
topic3 = 'Pico_Temp'

def on_connect(client, userdata, flags, rc):
    if rc == 0:
        print(f"Connected to MQTT Broker with result: {rc}")
    else:
        print("Failed to connect to Broker, return code = ", rc)

def on_disconnect(client, userdata, rc):
    if rc != 0:
        print("Unexpected disconnection!")

msg1set = False
msg2set = False
msg3set = False

def on_message(client, userdata, message):
    #topics = message.topic.split('/') # topics is now a list containing the topic segments
    #topics = message.topic.split() # topics is now a list containing the topic segments
    topics = message.topic
    #print(topic[1])
    payload = message.payload.decode("utf-8")
    #print(payload)
    if topics == topic1:
        global OS_Temp
        print(f"Received: {(message.payload.decode('utf-8'))} from Topic: {message.topic}")
        GVAR = payload
        OS_Temp = GVAR
        PWTfile = open('PicoTempFile.txt', 'w')
        PWTfile.write(OS_Temp)
        PWTfile.close()
        msg1set=True
        return msg1set
    if topics == topic2:
        global OS_Hum
        print(f"Received: {(message.payload.decode('utf-8'))} from Topic: {message.topic}")
        GVAR = payload
        OS_Hum = GVAR
        PWHfile = open('PicoHumFile.txt', 'w')
        PWHfile.write(OS_Hum)
        PWHfile.close()
        msg2set=True
        return msg2set
    if topics == topic3:
        global Pico_Temp
        print(f"Received: {(message.payload.decode('utf-8'))} from Topic: {message.topic}")
        GVAR = payload
        Pico_Temp = GVAR
        PicoFile = open('PicoFile.txt', 'w')
        PicoFile.write(Pico_Temp)
        #PicoFile.write("\n")
        PicoFile.close()
        msg3set=True
        return msg3set

client.connect(broker, port)
client.on_connect = on_connect
client.on_disconnect = on_disconnect
time.sleep(3)
client.subscribe(topic1)
client.subscribe(topic2)
client.subscribe(topic3)
if msg1set == True:
    print("Global OST set: ", OS_Temp)
if msg2set == True:
    print("Global OSH set: ", OS_Hum)
if msg3set == True:
    print("Global PT set: ", Pico_Temp)
client.on_message = on_message
time.sleep(2)
client.loop_forever()

[beetle]

"Right , i tried this (at the bottom and inside the if statements). No cigar. Any help"

Not even a quick drag on a ciggy for that amendment You are assuming, are you not, that your if the global var is set to True then a print will occur, but the code that sets it to True is, again, only in your on_message function that is not called until a message is received. I think you are assuming your if statements are continually being read. Is this because your mqtt is set to loop forever ? I think this only keeps the mqtt code, which I think runs in a separate thread, running forever, but your main code is just run through the once. If you think about it if all your code is continually looping then you would be trying to set up the mqtt stuff again and again. So you must create an endless loop to poll the if statements to see if the current state of the variables will allow a print() to be made, not just a onetime check.

Your code amendments take you no further than your original with the exception that no attempt is now made to perform the print statement as your code charges through the mqtt setup stuff, then it reaches your if statements, the condition of the variables is False, no print() is done and then your code stops The if statements are never checked again.

Whilst its good to learn python by using some example code, if you do this without getting a few fundamentals under your belt then frustration will be the result. My suggestion, and its meant to be helpful, is to take a little time out to get to grips with a bit more of the basics before ploughing on. I dont think it will take you long, but a weeks worth of slogging at some basic tutorials will see you waltzing though this sort of project to a very different tune. (can you waltz to 'happy days are here again' - well you will probably be bursting into song at the very least )

May I suggst a couple of tutorial resources.
You could try the following beginner youtube tutorials. Get through the fist 9 of this playlist and you will soon be humming a happy tune.
The first episode is here:
https://www.youtube.com/watch?v=YYXdXT2 ... FUiLCjGgY7

also the following is a good on-line read to get to grips with python:
https://thepythoncodingbook.com

[Rissy]

"Right , i tried this (at the bottom and inside the if statements). No cigar. Any help"

Not even a quick drag on a ciggy for that amendment You are assuming, are you not, that your if the global var is set to True then a print will occur, but the code that sets it to True is, again, only in your on_message function that is not called until a message is received. I think you are assuming your if statements are continually being read. Is this because your mqtt is set to loop forever ? I think this only keeps the mqtt code, which I think runs in a separate thread, running forever, but your main code is just run through the once. If you think about it if all your code is continually looping then you would be trying to set up the mqtt stuff again and again. So you must create an endless loop to poll the if statements to see if the current state of the variables will allow a print() to be made, not just a onetime check.

Your code amendments take you no further than your original with the exception that no attempt is now made to perform the print statement as your code charges through the mqtt setup stuff, then it reaches your if statements, the condition of the variables is False, no print() is done and then your code stops The if statements are never checked again.

Whilst its good to learn python by using some example code, if you do this without getting a few fundamentals under your belt then frustration will be the result. My suggestion, and its meant to be helpful, is to take a little time out to get to grips with a bit more of the basics before ploughing on. I dont think it will take you long, but a weeks worth of slogging at some basic tutorials will see you waltzing though this sort of project to a very different tune. (can you waltz to 'happy days are here again' - well you will probably be bursting into song at the very least )

May I suggst a couple of tutorial resources.
You could try the following beginner youtube tutorials. Get through the fist 9 of this playlist and you will soon be humming a happy tune.
The first episode is here:
https://www.youtube.com/watch?v=YYXdXT2 ... FUiLCjGgY7

also the following is a good on-line read to get to grips with python:
https://thepythoncodingbook.com

Thanks for the feedback and your time and patience with me. I appreciate you spending your time writing this feedback to me, i really do.

I only started playing with a Raspberry Pi for the first time around 4 weeks ago.

I've only been playing with my Pico W for a little over a week.

I've spent a LOT of time reading, watching videos, and hacking away testing and creating files etc.

I think i've come a long way in a short space of time.

Clearly there are seemingly more complex things i've yet to learn and understand which i'd considerd "easy" until today. (I've made global variables all over the place up to now, and they all work. I just don't know why I couldn't get those ones to work at all!?)

I've been in a bit of a rush to get this project up and running. I'm monitoring loft aspects of my house dew to potential condesation issues, so I've really had a lot of pressure on me for the past month.

I've got my program(s) all running now. I've left it using writing in and reading from .txt files until I learn what the hell is going on with these GVs. At least now I have a working solution. It might not be as slick as I wanted, but it works.

[beetle]

Good to hear you have a running solution and I think you have done very well for such a short time working with Python.

For when you may feel ready to up your python game I give you a couple of programs, one incomplete example to run on your PicoW that sends mqtt data relating to sensor readings and one program that will run on the Raspberry Pi to receive the messages, append the data received to a CSV file (handy for loading the data into a spreadsheet) and has an example of printing the data to the terminal screen. Probably you would later like to expand this to print info to one of the small displays one can get for the rpi. (this rpi program example shows an extra mqtt subscription for, for example, linking in yet another PicoW perhaps reading an outside temperature sensor, mentioned here incase my code confuses)

The program example you could put in your PicoW:

Code: Select all

import json
import time

# 1. put in your code for connecting to your MQTT broker - 
# # assume 'client' is the connection hence client.publish(...) below

# 2. Periodically read your sensors and populate your varialbes with their values 
# - eg you end up with something like
temp = 22.4
temp2 = 23.5
hum = 66.44

# 3. Just afer each period read of sensors note the current date and time
# for later putting the data into a CSV file on the rpi.
read_time = time.localtime()
r_time = list(read_time)
if r_time[1] < 10:
    r_time[1] = "0" + str(r_time[1])
rD = int(str(r_time[0]) + str(r_time[1]) + str(r_time[2])) #the date in YYYYMMDD format, handy for sorting
rT = str(r_time[3]) + ":" + str(r_time[4])  #the time in Hours:Minutes format

# Serialise the data to publish in the json format
allvaluesJ = json.dumps([rD,rT,temp,temp2,hum])
# print this to have a peek at what you are about to publish (if you desire)
print(allvaluesJ)

# Now publish to data to you broker
client.publish('PicoW/Loft',allvaluesJ)

# now go back to periodically reading your sensors

and a program to run on the RPI

Code: Select all

import time
from paho.mqtt import client as mqtt
import json
import csv

Broker = "10.0.1.141"  #put in your MQTT broker IP address


# --------------- Functions to handle MQTT messages received --
def Loft(payload):
    # json back to python list
    data = json.loads(payload)
    # print what we have received for amusement
    print(data)
    # append the data to a loft_data.csv file
    with open('Loft_Data.csv', mode = 'a') as f:
        write = csv.writer(f)
        write.writerow(data)
    # print the date, time and humidity value
    print('Humidity at ', data[0], ' ',data[1], ' is ',data[4])    
    

def Outside(payload):
    print(payload)


# -------------------- MQTT  ----------------------------------
def mqtt_on_connection(client, userdata, flags, rc):
    if rc == 0:
        client.connected_flag = True  # set flag
        # print("mqtt_on_connection reports: connected OK")
        # subscribe to mqtt topics

        client.subscribe('PicoW/Loft')
        client.subscribe('PicoW/Outside')
        print("MQTT broker is connected ")
    else:
        print("Bad connection Returned code=", rc)


def on_disconnect(client, userdata, rc):
    # print("disconnecting reason  "  +str(rc))
    client.connected_flag = False
    print('MQTT is DISCONNECTING')


def mqtt_connect(broker):
    try:
        # set mqtt
        mqtt.Client.connected_flag = False  # create flag in class
        client = mqtt.Client()
        client.on_connect = mqtt_on_connection
        client.loop_start()
        print("Connecting to broker ", broker)
        client.connect(broker)
        while not client.connected_flag:  # wait in loop
            print("Waiting for mqtt broker connection")
            time.sleep(2)

        return client
    except:
        print("error initiating MQTT")
        # raise error and quit program


# mqtt callback function
def mqtt_on_message(mqttc, userdats, message):
    topics = message.topic
    payload = str(message.payload.decode("utf-8"))
    if topics == 'PicoW/Loft':
        Loft(payload)
    elif topics == 'PicoW/Outside':
        Outside(payload)
    else:
        print('message received from a subscrition, but not handled')


# start mqtt
client1 = mqtt_connect(Broker)
# assign callback function
client1.on_message = mqtt_on_message
client1.on_disconnect = on_disconnect

# ------------------ STARTUP ---------------------------------
# program will run forever, use Control C in the terminal to stop it running
while True:
    pass

[Rissy]

Good to hear you have a running solution and I think you have done very well for such a short time working with Python.

For when you may feel ready to up your python game I give you a couple of programs, one incomplete example to run on your PicoW that sends mqtt data relating to sensor readings and one program that will run on the Raspberry Pi to receive the messages, append the data received to a CSV file (handy for loading the data into a spreadsheet) and has an example of printing the data to the terminal screen. Probably you would later like to expand this to print info to one of the small displays one can get for the rpi. (this rpi program example shows an extra mqtt subscription for, for example, linking in yet another PicoW perhaps reading an outside temperature sensor, mentioned here incase my code confuses)

The program example you could put in your PicoW:

Code: Select all

import json
import time

# 1. put in your code for connecting to your MQTT broker - 
# # assume 'client' is the connection hence client.publish(...) below

# 2. Periodically read your sensors and populate your varialbes with their values 
# - eg you end up with something like
temp = 22.4
temp2 = 23.5
hum = 66.44

# 3. Just afer each period read of sensors note the current date and time
# for later putting the data into a CSV file on the rpi.
read_time = time.localtime()
r_time = list(read_time)
if r_time[1] < 10:
    r_time[1] = "0" + str(r_time[1])
rD = int(str(r_time[0]) + str(r_time[1]) + str(r_time[2])) #the date in YYYYMMDD format, handy for sorting
rT = str(r_time[3]) + ":" + str(r_time[4])  #the time in Hours:Minutes format

# Serialise the data to publish in the json format
allvaluesJ = json.dumps([rD,rT,temp,temp2,hum])
# print this to have a peek at what you are about to publish (if you desire)
print(allvaluesJ)

# Now publish to data to you broker
client.publish('PicoW/Loft',allvaluesJ)

# now go back to periodically reading your sensors

and a program to run on the RPI

Code: Select all

import time
from paho.mqtt import client as mqtt
import json
import csv

Broker = "10.0.1.141"  #put in your MQTT broker IP address


# --------------- Functions to handle MQTT messages received --
def Loft(payload):
    # json back to python list
    data = json.loads(payload)
    # print what we have received for amusement
    print(data)
    # append the data to a loft_data.csv file
    with open('Loft_Data.csv', mode = 'a') as f:
        write = csv.writer(f)
        write.writerow(data)
    # print the date, time and humidity value
    print('Humidity at ', data[0], ' ',data[1], ' is ',data[4])    
    

def Outside(payload):
    print(payload)


# -------------------- MQTT  ----------------------------------
def mqtt_on_connection(client, userdata, flags, rc):
    if rc == 0:
        client.connected_flag = True  # set flag
        # print("mqtt_on_connection reports: connected OK")
        # subscribe to mqtt topics

        client.subscribe('PicoW/Loft')
        client.subscribe('PicoW/Outside')
        print("MQTT broker is connected ")
    else:
        print("Bad connection Returned code=", rc)


def on_disconnect(client, userdata, rc):
    # print("disconnecting reason  "  +str(rc))
    client.connected_flag = False
    print('MQTT is DISCONNECTING')


def mqtt_connect(broker):
    try:
        # set mqtt
        mqtt.Client.connected_flag = False  # create flag in class
        client = mqtt.Client()
        client.on_connect = mqtt_on_connection
        client.loop_start()
        print("Connecting to broker ", broker)
        client.connect(broker)
        while not client.connected_flag:  # wait in loop
            print("Waiting for mqtt broker connection")
            time.sleep(2)

        return client
    except:
        print("error initiating MQTT")
        # raise error and quit program


# mqtt callback function
def mqtt_on_message(mqttc, userdats, message):
    topics = message.topic
    payload = str(message.payload.decode("utf-8"))
    if topics == 'PicoW/Loft':
        Loft(payload)
    elif topics == 'PicoW/Outside':
        Outside(payload)
    else:
        print('message received from a subscrition, but not handled')


# start mqtt
client1 = mqtt_connect(Broker)
# assign callback function
client1.on_message = mqtt_on_message
client1.on_disconnect = on_disconnect

# ------------------ STARTUP ---------------------------------
# program will run forever, use Control C in the terminal to stop it running
while True:
    pass

Thank you very much for this beetle. I'll have a read through the programs you've offered up to me and see what they're doing by comparison to my own.

My current setup does basically what you're suggesting me to "up my game with".

My setup is as follows:
Pico W and Raspberry Pi 4B both have serial I2C devices connected to them and of course the Pico W is MQTTing its data to the RP via Wi-Fi. RP is collecting its own data directly from its own sensors as well as receiving the Pico W data via MQTT.
Each sensor has its own dedicated monitoring terminal running on the screen of the RP as well as a terminal monitoring the incoming MQTT traffic from the Pico W.
As well as each terminal displaying realtime measurements at 15 min periods, every time the latest values are displayed, they're also sent to a spreadsheet in a titled and columned designed layout to make sense of, and produce a graph from, appending a new line each time a new sample is recorded.
For every sample of data every 15 mins, it's assessed against an incrementally stored historical MIN and MAX value from an appropriately stored file and has an incremetally stored sample number appended on to it too, also stored and updated to a file. This means if i have to stop my programs running at any time for any reason, or if I have a power cut; not only does everything load back up again upon return of power, I also don't lose my obtained MIN/MAX or sample number as these are some of the first things read in, before then being updated automatically where appropriate.
It's lovely watching it do its thing, and looking at the graphs produced in post analysis. I'm quite proud of how much I've learned and achieved in such a short space of time. Of course it wasn't without help from some of you lovely people, and of course a shout out to people who've prewritten driver files for the sensors i'm using. If i'd had to do that myself, it would have been YEARS for me to get a system up and running, not days/a week in each case.

I first concentrated on the RP loft sensors first, but then of course thermodynamics dictates that I need to compare these readings with the outside temperature/humidity to get a proper assessment of the dew point calculations, so that's why i then appended the Pico W on to my growning project. It's going to be taking the outside values once i get that setup through the wall of the house (hopefully next weekend). Until then, the Pico W is going through a "soak test" right now, just taking measurements from the lounge and i test the stability of the MQTT connection.

All this and I still can't work out how to get those damned global variables out of my functions in my MQTT receive program. I really wish you could give me the hint/tip i'm clearly in need of.

I watched about 10 videos from your suggested YouTube link, including no.8 which i think is the one you were most advising me to watch and take in. Very good videos indeed. I've now subscribed to his channel and i'll be updating my knowledge through his exercises etc, so thank you for that indeed. Unfortunately, that still doesn't lend to me understanding how i get the GVs out. So that's why i've had to leave it running with both programs sharing use of .txt files instead.
Perhaps i'm missing, or not understanding the flow that Python takes when running programs, and unfortunately nobody seems to tell you anywhere. I thought they ran from top to bottom and repeated over and over, but clearly not, as otherwise my GVs would have been "discovered" by the time they're called at the now hashed out parts of my program due to activating the corresponding function before that. Obviously not. So now i'm confused at how Python works. This also leads me to further confusion at how the hell i've managed to achieve everything else i've done in the past month so successfully!?

I came downstrairs after the first night of my new Pico W MQTT setup running through the night and found that for whatever reason, my RP MQTT server had stopped due to an NTP time sync issue. I then realised even after fixing this within about an hour of yet more time spent with it, that i then had to pull the plug of my Pico W to get it to reinitialise its MQTT publishing script again as it had hung without access to the MQTT server most of the night. I'm now wondering if someone can advise me on how to get the Pico W to self reboot automatically on loss of the MQTT server until such time as the connection is re-estabilished. Any help there?

It's funny you should mention little local displays as I was thinking today that this might be quite a nice edition to be able to get a more immediate appreciation of live values local to each sensor. I'm also interested in taking a look into this Influx DB program and Grafana too so I can simply look up a webpage and see my trended data in a polished looking fashion instead of using spreadsheets.

I've now ordered another Pico W, another RP4B and another sensor from the PiHut so that I can further advance my skills whilst leaving my working system alone to do its thing. So i'm definitely not finished here with you guys, not one bit.

I'm quite tired now, and suffering insomnia with all of the intense learning i've been doing over the past month, so i think my progress will have to slow down somewhat though, i'm starting to feel a bit ill from it all. Still interested and keen, just tired. Plus the missus has been very patient with me over the past month whilst i've been buried in what she just sees as bits of junk. So now I have to give her some time again now. Other jobs etc, as well as loving attention.

Thanks again beetle.

[beetle]

Rissy, I probably did not explain so clearly about the global variables in your program so try this.
Your program starts and sets up a connection to your mqtt broker and makes subscriptions. These instruction go over your wifi and eventually you receive a message relating to your subscriptions. At that point your on-message function and its code springs into action. Before the message received point in time you have just set up what will happen as and when a message is received. Having set up the possibility of what will eventually happen (creating your globals etc), and in computer terms this messaging comms will take a while. The network comms is much much slower then your pico processor which has, the meanwhile, shot passed the setting up of your function on to the bit of your code where you want to print the global variables you function will, eventually, create. Of course these global variable have not yet been created when your program first tries to use them, a few microsecond later maybe, but not yet.

Having said all that, as I hope you can see from my example mqtt receive message program, you don't need any global variables at all, and they are best avoided if you can as if your program grows to cater for all manner of sesnor reads etc. eventually you will trip yourself up to too many globals.

Sound like a good meaty project you have and I expect you will be refactoring your code for a year or two so I would not rush at it. But then I dont rush at much anyway. Now for another cup of tea. Maybe a nap

[Rissy]

Rissy, I probably did not explain so clearly about the global variables in your program so try this.
Your program starts and sets up a connection to your mqtt broker and makes subscriptions. These instruction go over your wifi and eventually you receive a message relating to your subscriptions. At that point your on-message function and its code springs into action. Before the message received point in time you have just set up what will happen as and when a message is received. Having set up the possibility of what will eventually happen (creating your globals etc), and in computer terms this messaging comms will take a while. The network comms is much much slower then your pico processor which has, the meanwhile, shot passed the setting up of your function on to the bit of your code where you want to print the global variables you function will, eventually, create. Of course these global variable have not yet been created when your program first tries to use them, a few microsecond later maybe, but not yet.

Having said all that, as I hope you can see from my example mqtt receive message program, you don't need any global variables at all, and they are best avoided if you can as if your program grows to cater for all manner of sesnor reads etc. eventually you will trip yourself up to too many globals.

Sound like a good meaty project you have and I expect you will be refactoring your code for a year or two so I would not rush at it. But then I dont rush at much anyway. Now for another cup of tea. Maybe a nap

Without the use of global variables in my MQTT subscription code (MQTT_PW_Vars), how can I get OS_Temp, OS_Hum and Pico_Temp out of my

Code: Select all

on_message

function by using

Code: Select all

from MQTT_PW_Vars import OS_Temp as OS_Temp

etc. rather than depositing the values into a text file (like i'm doing now) and then reading in the same value from the same text file from my main data logging code (PicoW_Vars)?