Pico W Micro Python MQTT of data to a RP4B

[Rissy]

Is that it? Do i have it? (Please just be straight with me now, I'm having a hard enough time trying to get this into my thick skull)

To be straight, then no I don't think so, but I find it very hard to make sense of your sudo code, it quite boggles my eyes

Sheesh. I'm just not getting this, am i. Either that, or I have, but you can't understand my beginners coding technique.

I think what I'll have to do is make some basic global variable passing code scripts on my new RP4B once i get that up and running, and try them out.
I want to get my head around this old lady before I move on to a younger model who'll run circles around me so that I can remove the need for using .txt files like i am now; BEFORE I start thinking like someone else more experienced than I am right now.

I need to understand things in my own way first before I try shifting that to a different train of thinking. I'm already looking at my own code from last week and losing my head on that, and I wrote it! I think my brain is tired to be honest.

[Rissy]

I found this! Not sure why I never found this before now!?

https://instructobit.com/tutorial/108/H ... -in-Python

[beetle]

Rissy, very good, and a good example of sharing global variables with a 'globals' module as per one of my snippets to be sure.

But, and I hope this does not do you head in, the question is really not how to use global variables, but rather how to share data between your modules.

But first do have a good play with the globals as you suggest, make some of your own code snipped files and I'm sure that if you just concentrate on that aspect, initially without trying to amend your existing code, the results will soon prove to be of benefit.

However, after that then you should go on to see what other ways are available to you in python. I have given an example of functions returning a value (can be values). And my final example which you may like is to consider is using a python dictionary to store the data and pass it around your modules. Why show you this?. Well this example (and indeed functions that return values) ultimately make for the most readable and robust code.

module2.py:

Code: Select all

# define a dictionary to store values
sensor_data = {
    "temperature" : 0.0,
    "humidity" : 0.0
}
# define a fuction that read sensors and store it in a dictionary.
def set_me_var():
    # so stuff to read sensors and set a varible to a value
    sensor_data["temperature"] = 20.8
    sensor_data["humidity"] = 96.999
    

module_main.py:

Code: Select all

from  module2 import sensor_data, set_me_var

print('Sensor data before reading')
print('all my sensor data: ', sensor_data)

set_me_var()

print('Sensor data after reading')
print('all my sensor data: ', sensor_data)
print('Just the temperature')
print(sensor_data["temperature"])

Result is:
Sensor data before reading
all my sensor data: {'temperature': 0.0, 'humidity': 0.0}
Sensor data after reading
all my sensor data: {'temperature': 20.8, 'humidity': 96.999}
Just the temperature
20.8

I will leave you in peace now to contemplate your globals and wait to see if you pick up on using function that return values or dictionary items if you get to see their utility. I hope you will eventually. I have not by any means shown an exhaustive list of the ways to share data between your modules (for example classes) but I think you have quite enough info to eventually get your programs into python gems.

[Rissy]

Rissy, very good, and a good example of sharing global variables with a 'globals' module as per one of my snippets to be sure.

But, and I hope this does not do you head in, the question is really not how to use global variables, but rather how to share data between your modules.

But first do have a good play with the globals as you suggest, make some of your own code snipped files and I'm sure that if you just concentrate on that aspect, initially without trying to amend your existing code, the results will soon prove to be of benefit.

However, after that then you should go on to see what other ways are available to you in python. I have given an example of functions returning a value (can be values). And my final example which you may like is to consider is using a python dictionary to store the data and pass it around your modules. Why show you this?. Well this example (and indeed functions that return values) ultimately make for the most readable and robust code.

module2.py:

Code: Select all

# define a dictionary to store values
sensor_data = {
    "temperature" : 0.0,
    "humidity" : 0.0
}
# define a fuction that read sensors and store it in a dictionary.
def set_me_var():
    # so stuff to read sensors and set a varible to a value
    sensor_data["temperature"] = 20.8
    sensor_data["humidity"] = 96.999
    

module_main.py:

Code: Select all

from  module2 import sensor_data, set_me_var

print('Sensor data before reading')
print('all my sensor data: ', sensor_data)

set_me_var()

print('Sensor data after reading')
print('all my sensor data: ', sensor_data)
print('Just the temperature')
print(sensor_data["temperature"])

Result is:
Sensor data before reading
all my sensor data: {'temperature': 0.0, 'humidity': 0.0}
Sensor data after reading
all my sensor data: {'temperature': 20.8, 'humidity': 96.999}
Just the temperature
20.8

I will leave you in peace now to contemplate your globals and wait to see if you pick up on using function that return values or dictionary items if you get to see their utility. I hope you will eventually. I have not by any means shown an exhaustive list of the ways to share data between your modules (for example classes) but I think you have quite enough info to eventually get your programs into python gems.

Thanks beetle,

I'm a bit quiet for a little bit, because I'm now spending some time getting my second RP4B and my second Pico W up and running, but taking notes along the way for what I'm doing so that I have notes on reproducing what I have even on the first one.

I'm going to make up some test code using a random number generator in place of a sensor and get some proof of global variable passing. Then my next task is to try to write the code for each of the Pico W and the RPB4, but in reverse function of what they are right now so that I can put the ultimately cheaper of the two in the loft and the more expensive one in the main body of the house rather than the other way around as they are now.

This means I'll have to get a couple of BME280 sensors on I2C working with the Pico W and the SHT-30 working with the RP4B. Then of course MQTT between the two. I'm sure i'll have fun with that! It feels like going back to the beginning again!

[beetle]

I'm sure i'll have fun with that! It feels like going back to the beginning again!

I'm sure you will have fun too. I get a good measure of satisfaction when I get to improve my version 1's (and v2's). Theres always room for improvements

Actually you have motivated me to so something very similar in that I want to monitor the humidity of the loft void above my conservatory ceiling. I replaced the glass roof with a proper sloping roof designed as a warm roof, but then went on to put in lower flat ceiling creating a void. I contemplated putting in ceiling vents to allow air circulation but did not actually do so. I have been popping my head through a ceiling hatch to have a visual look to see if I have any condensation issues from time to time, but I should really pop a humidity sensor up there.

So now motivated thank to your posting, I've been giving a bit of consideration as to the design. I will power the microcontroller board for the loft via a battery so a board with lower power consumption in sleep mode is indicated. A reading every half an hour would seem sufficient.

I will try out a rpi pico, but I also have an ESP32 TinyPico board and will test that as well to see which makes the battery last the longest. Of course either board will be running micropython and the code for each will be very similar. I already have a dedicated mqtt broker running on an RPi4 which also runs a mysql database server as well. I have another rpi4 that runs mqtt client programs to receive data from various sensors, display the data, and store it in the database for later analysis.

The rpi pico or the TinyPico will be using the excellent micropython async mqtt library by Peter Hinch, but as this uses async its probably not for beginners just starting out as its take a bit of getting used to programming the async way. When I have programmed my test program you will be welcome to have a copy, but you will be best advised to stick with your current micropython library.

The client code that runs on the rpi4 will just be another subscription to my mqtt programs that receives the data from the other sensors, so it will do a lot of stuff you would not need. However I did give an example of a rpi4 mqtt client program earlier in this thread.

A final though for you is that I think you should keep your rpi4 program as just one module and not to split it up into several files. (of course import all the python modules you need)

As programs get larger and for use of re-using your chunks of code in other projects then its a good practice to create several modules. But considering the size of the program you are producing this may just add to the complexity especially for a beginner.

If you put all the mqtt operations into a separate module then consider what you want to happen when a message arrives. This separate module could just plonk the data into a file, but if you populate global variables and then use these variable in the code in your main file, then you have to consider how your code, in the main module, will be alerted to the fact a global variable has been updated with a new value.

I'm in the habit of having a 'mqtt connect' module for easy import into other projects. In my main program I call the connection function in this module, pass it the subsections I want it to make (as a list) and receive back a connection object. In the main program I create the on-message function and assign my connection object to that function e.g. myclient.on_message = on_message. So my main program can be directly altered when an mqtt message is received. I hope I making sense

But for beginner I would recommend you simply put all you code into one main file. Probably best not to complicate your programming at this stage. As you can see if you examine the rpi4 mqtt code snippet I gave, theres not many lines of code required for both the mqtt stuff and a function or 2 to handle incoming messages, save the data to a file and print the data. It's a good starter example.

I'm working on some other stuff and it will be a few weeks before I have my completed test micropython code running but I will post it on here in case it proves to be of any interest to you when its ready However I expect you will have your own version2 programs running by that time

[Rissy]

Excellent stuff. I'm glad I can in some way have a positive influence on you

Now back to the negative influence (sorry).

I reproduced the web-link example of passing global variables which i showed you in a previous post. That worked, and I understood it no problem.
Then i tried to use that same methodology in my three scripts(modules).

It's very simple. I just want to run two programs at once. PicoW_Vars.py and MQTT_PW_Vars.py

This is so that PicoW_Vars.py prints out the results of MQTT_PW_Vars.py just once (for a start).
This would be achieved using global variables passing through Globals.py.

It's that simple. Nothing more right now, and nothing less.

I've tried various things to make it work, and nothing does, and now i have a circular reference error due to the passing of global variables.

Can you please take a look and understand what i am trying to do, and then tell me where i'm going wrong. This is literally the hardest thing in the world to achieve!

MQTT_PW_Vars.py

Code: Select all

#!/usr/bin/env python
#!/bin/bash
#MQTT_PW_Vars.py

import time
import paho.mqtt.client as mqtt
import Globals

broker = '192.168.0.51'
port = 1883
mqttclient = "PiLogger2"
client = mqtt.Client(mqttclient)

topic1 = 'OS_Temp'
topic2 = 'OS_Hum'
topic3 = 'Pico_Temp'

def on_connect(client, userdata, flags, rc):
    if rc == 0:
        print(f"Connected to MQTT Broker with result: {rc}")
    else:
        print("Failed to connect to Broker, return code = ", rc)

def on_disconnect(client, userdata, rc):
    if rc != 0:
        print("Unexpected disconnection!")

def on_message(client, userdata, message):
    topics = message.topic
    payload = message.payload.decode("utf-8")
    
    if topics == topic1:
        Globals.initialise()
        global temp
        print(f"Received: {(message.payload.decode('utf-8'))} from Topic: {message.topic}")
        Globals.GVAR = message.payload
        OS_Temp = str(Globals.GVAR)
        temp = float(Globals.GVAR)
        
    if topics == topic2:
        Globals.initialise()
        global hum
        print(f"Received: {(message.payload.decode('utf-8'))} from Topic: {message.topic}")
        Globals.GVAR = message.payload
        OS_Hum = str(Globals.GVAR)
        hum = float(Globals.GVAR)
        
    if topics == topic3:
        Globals.initialise()
        global pico
        print(f"Received: {(message.payload.decode('utf-8'))} from Topic: {message.topic}")
        Globals.GVAR = message.payload
        Pico_Temp = str(Globals.GVAR)
        pico = float(Globals.GVAR)
        
client.connect(broker, port)
client.on_connect = on_connect
client.on_disconnect = on_disconnect
time.sleep(3)
client.subscribe(topic1)
client.subscribe(topic2)
client.subscribe(topic3)
client.on_message = on_message
time.sleep(10)
client.loop_forever()

Globals.py

Code: Select all

#!/usr/bin/env python
#!/bin/bash
#Global_Variables.py

#Following used in MQTT_PW_Vars.py

from MQTT_PW_Vars import temp, hum, pico

def initialise(): 
    global GVAR
    global OS_Temp
    global OS_Hum
    global Pico_Temp
    #global temp
    #global hum
    #global pico
    GVAR = 0
    OS_Temp = '0'
    OS_Hum = '0'
    Pico_Temp = '0'
    #temp = 0
    #hum = 0
    #pico = 0

#Following used in PicoW_Vars.py

def getvalues(temp, hum, pico):
    global PW_Temp
    global PW_Hum
    global Pi_Temp
    PW_Temp = temp
    PW_Hum = hum
    Pi_Temp = pico

PicoW_Vars.py

Code: Select all

#!/usr/bin/env python
#!/bin/bash
#PicoW_Vars.py

#Required Imports

import Globals

if __name__ == "__main__":
    #MQTT_PW_Vars.on_message()
    Globals.getvalues()
    OS_Temp = getvalues.PW_Temp
    OS_Hum = getvalues.PW_Hum
    Pico_Temp = getvalues.Pi_Temp
    print("temperature: ", OS_Temp)
    print("humidity: ", OS_Hum)
    print("Pico temp: ", Pico_Temp)

[beetle]

It's very simple. I just want to run two programs at once. PicoW_Vars.py and MQTT_PW_Vars.py

Sorry Rissy, the penny has finally dropped. 2 separate programs running at once and not a main module importing the code from another module. You cannot pass data between 2 entirely separately running programs with the global variables technique It is possible to share data between two separate programs with the multiprocessing module with pipes, queues and whatever, but thats rather advanced stuff. Don't consider that route until you are a most competent python programmer.

You will have to amend your thoughts on how to achieve what you want in a different way. On way easy way with two separate programs that need the same data from an outside mqtt source is for both have the mqtt client function in their code. That way both programs could subscribe to the same topic, thus both getting data from the same publishing source. These separately running programs could also publish and subscribe to a shared topic thus they could be in communication with each other.

Another way would be to construct just one program that does all you need. In fact I'm rather (very) sure that this is all you really need, though use may be made of two separate python scripts, but one would import the other. To assist you in how your program (ok, could be programs) should be constructed you need to put what you are trying to achieve in plain English (not code) for my old brain to truly understand what you are trying to achieve. Note the steps you are trying to achieve in the following way. (i.e. not too wordy just some simple statements)

the Pico
1. read the temperature and humidity from a SHT30 sensor attached via i2c
2. publish the temperature and humidity data via MQTT

The Rpi4
1. receives the Pico temperature and humidity from MQTT
2. read temperature sensors attached to the RPi4 via i2c
3 append all the sensor readings to a to a file.
4 print the data to a terminal screen
5 show the current sensor readings on a dinky screen attached to the rpi4 via SPI

The above is probably wide of the mark, but you get the idea. From the specification it will be much easier to make suggestions.

[Rissy]

I think you’ve pretty much got it to be fair, except I’m not using an LCD screen at all (yet), and I’m using my third (actually running) script to send the logged data to a separate spreadsheet entirely from my other program collecting readings from two bme280 sensors in my loft.
(So basically I’m generating two spreadsheets; one for the loft and the other for outside.

I think you’ve pretty much answered my question then. It can’t be done, or it’s too hard to do without being a python genius.

The fact I’m getting the result I want by using interposing text files instead, although appearing to be a bit clunky in its premise, is clearly a clever way of getting around what would otherwise be a much more involved task with no better payoff.

So I think the answer is to stick with my working solution and concentrate on shifting around code ownership to swap the positions of my raspberry pi and my pico w now.

[Rissy]

Just out of interest, where in the world are you beetle?

[beetle]

Rissy, yes, I could have mentioned another way to send data between to separately running programs is to use the file system, clunky indeed but it works as you have found (btw. pickle or json is good for this sort of thing, but I leave it for you google them if desired)

But if my specification notes are indeed all that you are trying to achieve then you only need one program running on the Pico to read and publish the data, and one program on the Rpi that reads sensor data, reads mqtt data, and creates two spreadsheet files. For example you could create a function that appends data to a spreadsheet file that has the filename to use and a list[] or CSV string (or some other structure) containing the data passed as arguments. Thus this one function could be used to create as many spreadsheet files as you wish. Keep your code DRY. (Dont Repeat Yourself)

I think you are complicating your life by having two separate programs, so if you are considering a version 2 then I recommend you give this your consideration. But theres more demands on one's time that beautiful code... At least you are no longer beating your head on the proverbial brick wall.