Connecting to AWS with MQTT
-
- Posts: 9
- Joined: Wed Aug 15, 2018 8:31 pm
Connecting to AWS with MQTT
Hi,
I have been struggling with this for last 2 days. I have a ESP32 with micro python ruining. I created AWS account and set all the cert and polices. I tested the certificate through MQTT.FX (works for for both subscribe and publish)
Now I am trying to connect my ESP32 to AWS but no luck. I have tried everything possible.
I uploaded all the certificate file to root directory of my ESP32 and using main2.py file with following code:
import machine
from network import WLAN
import network
import time
from umqtt.simple import MQTTClient
DISCONNECTED = 0
CONNECTING = 1
CONNECTED = 2
HOST = "a21sigud7911d7.iot.us-west-2.amazonaws.com"
TOPIC = "myThingName"
state = DISCONNECTED
connection = None
wlan = network.WLAN(network.STA_IF)
wlan.active(True)
def pub_msg(msg):
global connection
connection.publish(topic=TOPIC, msg=msg, qos=0)
print('Sending: ' + msg)
def run():
global state
global connection
while True:
while state != CONNECTED:
try:
state = CONNECTING
connection = MQTTClient(client_id=TOPIC, server=HOST, port=8883, keepalive=10000, ssl=True, ssl_params={"certfile":"/49c84a8c4a-certificate.pem.crt", "keyfile":"/49c84a8c4a-private.pem.key", "ca_certs":"/root.pem"})
connection.connect()
state = CONNECTED
except:
print('Could not establish MQTT connection')
time.sleep(0.5)
continue
print('MQTT LIVE!')
while state == CONNECTED:
msg = '{"device_id":"some_id", "data":"some_data"}'
pub_msg(msg)
time.sleep(2.0)
while True:
if 1==1:
print('Emerge' +" was found!")
wlan.connect('Emerge', '123456789')
while not wlan.isconnected():
machine.idle()
print('Connected to '+ 'Emerge')
run()
break
AND OT PUT AT THE END IS:
"Could not establish MQTT connection"
I am not sure if it is reading all 3 certificate files right OR do i need to put them somewhere else!!!
anyone knows what the issue might be?
Thanks in advance
I have been struggling with this for last 2 days. I have a ESP32 with micro python ruining. I created AWS account and set all the cert and polices. I tested the certificate through MQTT.FX (works for for both subscribe and publish)
Now I am trying to connect my ESP32 to AWS but no luck. I have tried everything possible.
I uploaded all the certificate file to root directory of my ESP32 and using main2.py file with following code:
import machine
from network import WLAN
import network
import time
from umqtt.simple import MQTTClient
DISCONNECTED = 0
CONNECTING = 1
CONNECTED = 2
HOST = "a21sigud7911d7.iot.us-west-2.amazonaws.com"
TOPIC = "myThingName"
state = DISCONNECTED
connection = None
wlan = network.WLAN(network.STA_IF)
wlan.active(True)
def pub_msg(msg):
global connection
connection.publish(topic=TOPIC, msg=msg, qos=0)
print('Sending: ' + msg)
def run():
global state
global connection
while True:
while state != CONNECTED:
try:
state = CONNECTING
connection = MQTTClient(client_id=TOPIC, server=HOST, port=8883, keepalive=10000, ssl=True, ssl_params={"certfile":"/49c84a8c4a-certificate.pem.crt", "keyfile":"/49c84a8c4a-private.pem.key", "ca_certs":"/root.pem"})
connection.connect()
state = CONNECTED
except:
print('Could not establish MQTT connection')
time.sleep(0.5)
continue
print('MQTT LIVE!')
while state == CONNECTED:
msg = '{"device_id":"some_id", "data":"some_data"}'
pub_msg(msg)
time.sleep(2.0)
while True:
if 1==1:
print('Emerge' +" was found!")
wlan.connect('Emerge', '123456789')
while not wlan.isconnected():
machine.idle()
print('Connected to '+ 'Emerge')
run()
break
AND OT PUT AT THE END IS:
"Could not establish MQTT connection"
I am not sure if it is reading all 3 certificate files right OR do i need to put them somewhere else!!!
anyone knows what the issue might be?
Thanks in advance
-
- Posts: 9
- Joined: Wed Aug 15, 2018 8:31 pm
Re: Connecting to AWS with MQTT
Just an update, after print out the exception i get this:
I (22290) network: CONNECTED
I (22950) event: sta ip: 192.168.111.91, mask: 255.255.255.0, gw: 192.168.111.1
I (22950) network: GOT_IP
Could not establish MQTT connection list index out of range
Could not establish MQTT connection extra keyword arguments given
Could not establish MQTT connection 23
Could not establish MQTT connection 23
I (22290) network: CONNECTED
I (22950) event: sta ip: 192.168.111.91, mask: 255.255.255.0, gw: 192.168.111.1
I (22950) network: GOT_IP
Could not establish MQTT connection list index out of range
Could not establish MQTT connection extra keyword arguments given
Could not establish MQTT connection 23
Could not establish MQTT connection 23
-
- Posts: 9
- Joined: Wed Aug 15, 2018 8:31 pm
Re: Connecting to AWS with MQTT
Well I guess no answer to this issue?
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: Connecting to AWS with MQTT
This question comes up repeatedly (search for TLS in this forum). Last year I tried to get MQTT working with TLS and failed. Firmware support has improved since then and it is allegedly possible to use certificates. A definitive answer, with a code sample, would be good.
Peter Hinch
Index to my micropython libraries.
Index to my micropython libraries.
Re: Connecting to AWS with MQTT
we‘ve been using wipy3 devices to connect to AWS IoT, basically following a tutorial from pycom doc webpage (https://docs.pycom.io/chapter/tutorials/all/aws - sadly doesn‘t seem to exist anymore )
Anyway you might get some ideas by visiting their git repo
https://github.com/pycom/aws-pycom?files=1
As @pythoncoder pointed out it would be nice to have step by step tutorials for common cloud sites using official fw builds (no forks).
Anyway you might get some ideas by visiting their git repo
https://github.com/pycom/aws-pycom?files=1
As @pythoncoder pointed out it would be nice to have step by step tutorials for common cloud sites using official fw builds (no forks).
-
- Posts: 9
- Joined: Wed Aug 15, 2018 8:31 pm
Re: Connecting to AWS with MQTT
I racked my brain for a few days but finally got it to work. It turned out to be a missing parameters in ssl_params list:
I added "server_side":False and removed Ca root (I am not sure why it does not need this variable) and it start publishing and subscribing to the AWS. Here is my code:
CACERT_PATH = "/flash/root.pem" // apparently this wont be used!!!
KEY_PATH = "/flash/49c84a8c4a-private.key"
CERT_PATH = "/flash/49c84a8c4a-certificate.pem"
with open(KEY_PATH, 'r') as f:
key1 = f.read()
with open(CACERT_PATH, 'r') as f:
key2 = f.read()
with open(CERT_PATH, 'r') as f:
cert1 = f.read()
client = MQTTClient(client_id="My_topic/klm", server="a21sigud7911d7.iot.us-west-2.amazonaws.com", port=8883, keepalive=4000, ssl=True, ssl_params={ "key":key1, "cert":cert1, "server_side":False })
Just to let everyone knows in case they run into same issue.
Thanks
I added "server_side":False and removed Ca root (I am not sure why it does not need this variable) and it start publishing and subscribing to the AWS. Here is my code:
CACERT_PATH = "/flash/root.pem" // apparently this wont be used!!!
KEY_PATH = "/flash/49c84a8c4a-private.key"
CERT_PATH = "/flash/49c84a8c4a-certificate.pem"
with open(KEY_PATH, 'r') as f:
key1 = f.read()
with open(CACERT_PATH, 'r') as f:
key2 = f.read()
with open(CERT_PATH, 'r') as f:
cert1 = f.read()
client = MQTTClient(client_id="My_topic/klm", server="a21sigud7911d7.iot.us-west-2.amazonaws.com", port=8883, keepalive=4000, ssl=True, ssl_params={ "key":key1, "cert":cert1, "server_side":False })
Just to let everyone knows in case they run into same issue.
Thanks
- pythoncoder
- Posts: 5956
- Joined: Fri Jul 18, 2014 8:01 am
- Location: UK
- Contact:
Re: Connecting to AWS with MQTT
I have pretty much done exactly as you have stated in your example (james_km69)
KEY_PATH = "62c4233362-private.pem.key"
CERT_PATH = "62c4233362-certificate.pem.crt "
with open(KEY_PATH, 'r') as f:
key1 = f.read()
print(key1)
with open(CERT_PATH, 'r') as f:
cert1 = f.read()
print(cert1)
client = MQTTClient(client_id="EntranceRFID", server="XXXXXXXXXXX.eu-central-1.amazonaws.com", port=8883, keepalive=4000, ssl=True, ssl_params={ "key":key1, "cert":cert1, "server_side":False })
## (XXXXXXXXXXX = my AWS endpoint).
When running key1 and cer1t is loaded (and I do a printout in REPL for these as you can see above).
But when it is time for the client.connect()
I get
File "umqtt/simple.py", line 61, in connect
ValueError: invalid key.
I am not sure what I have done wrong. The certificate "62c4233362..." is activated and I have connected a "Thing" (EntranceRFID) to it in AWS IoT.
Any advice?
KEY_PATH = "62c4233362-private.pem.key"
CERT_PATH = "62c4233362-certificate.pem.crt "
with open(KEY_PATH, 'r') as f:
key1 = f.read()
print(key1)
with open(CERT_PATH, 'r') as f:
cert1 = f.read()
print(cert1)
client = MQTTClient(client_id="EntranceRFID", server="XXXXXXXXXXX.eu-central-1.amazonaws.com", port=8883, keepalive=4000, ssl=True, ssl_params={ "key":key1, "cert":cert1, "server_side":False })
## (XXXXXXXXXXX = my AWS endpoint).
When running key1 and cer1t is loaded (and I do a printout in REPL for these as you can see above).
But when it is time for the client.connect()
I get
File "umqtt/simple.py", line 61, in connect
ValueError: invalid key.
I am not sure what I have done wrong. The certificate "62c4233362..." is activated and I have connected a "Thing" (EntranceRFID) to it in AWS IoT.
Any advice?
-
- Posts: 1
- Joined: Sun May 05, 2019 7:28 am
Re: Connecting to AWS with MQTT
I was getting the same issue as DrKarma, slightly different error however:
>>> client.connect()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "simple.py", line 97, in connect
IndexError: bytes index out of range
If anyone else gets this problem, check your AWS thing's permissions. The default policy created when you onboard a device through the AWS wizard doesn't appear to allow connections. The default is:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"iot:Publish",
"iot:Receive"
],
"Resource": [
"arn:aws:iot:ap-southeast-2:xxx:topic/sdk/test/java",
"arn:aws:iot:ap-southeast-2:xxx:topic/sdk/test/Python",
"arn:aws:iot:ap-southeast-2:xxx:topic/topic_1",
"arn:aws:iot:ap-southeast-2:xxx:topic/topic_2"
]
},
{
"Effect": "Allow",
"Action": [
"iot:Subscribe"
],
"Resource": [
"arn:aws:iot:ap-southeast-2:xxx:topicfilter/sdk/test/java",
"arn:aws:iot:ap-southeast-2:xxx:topicfilter/sdk/test/Python",
"arn:aws:iot:ap-southeast-2:xxx:topicfilter/topic_1",
"arn:aws:iot:ap-southeast-2:xxx:topicfilter/topic_2"
]
},
{
"Effect": "Allow",
"Action": [
"iot:Connect"
],
"Resource": [
"arn:aws:iot:ap-southeast-2:xxx:client/sdk-java",
"arn:aws:iot:ap-southeast-2:xxx:client/basicPubSub",
"arn:aws:iot:ap-southeast-2:xxx:client/sdk-nodejs-*"
]
}
]
}
Replace this with a very basic:
{
"Statement": [
{
"Effect": "Allow",
"Action": "iot:*",
"Resource": "*"
}
],
"Version": "2012-10-17"
}
and your device will connect!!! I wasted HOURS before working this out, so hope this saves someone else some trouble
>>> client.connect()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "simple.py", line 97, in connect
IndexError: bytes index out of range
If anyone else gets this problem, check your AWS thing's permissions. The default policy created when you onboard a device through the AWS wizard doesn't appear to allow connections. The default is:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"iot:Publish",
"iot:Receive"
],
"Resource": [
"arn:aws:iot:ap-southeast-2:xxx:topic/sdk/test/java",
"arn:aws:iot:ap-southeast-2:xxx:topic/sdk/test/Python",
"arn:aws:iot:ap-southeast-2:xxx:topic/topic_1",
"arn:aws:iot:ap-southeast-2:xxx:topic/topic_2"
]
},
{
"Effect": "Allow",
"Action": [
"iot:Subscribe"
],
"Resource": [
"arn:aws:iot:ap-southeast-2:xxx:topicfilter/sdk/test/java",
"arn:aws:iot:ap-southeast-2:xxx:topicfilter/sdk/test/Python",
"arn:aws:iot:ap-southeast-2:xxx:topicfilter/topic_1",
"arn:aws:iot:ap-southeast-2:xxx:topicfilter/topic_2"
]
},
{
"Effect": "Allow",
"Action": [
"iot:Connect"
],
"Resource": [
"arn:aws:iot:ap-southeast-2:xxx:client/sdk-java",
"arn:aws:iot:ap-southeast-2:xxx:client/basicPubSub",
"arn:aws:iot:ap-southeast-2:xxx:client/sdk-nodejs-*"
]
}
]
}
Replace this with a very basic:
{
"Statement": [
{
"Effect": "Allow",
"Action": "iot:*",
"Resource": "*"
}
],
"Version": "2012-10-17"
}
and your device will connect!!! I wasted HOURS before working this out, so hope this saves someone else some trouble
Re: Connecting to AWS with MQTT
To summarize james_km69's example, it looks like this works for publishing MQTT messages with SSL:
But as others have mentioned, this code does not verify the certificate's chain of trust, which could leave your device open to man-in-the-middle attacks. I couldn't figure out how to verify a CA from a cursory look at the 'modussl_mbedtls' module, but I could be missing something and that functionality is supported by the underlying mbedtls C libraries.
And if you're having trouble connecting to an AWS endpoint with an 'invalid key' error, it might be worth double-checking that the certificate was successfully activated after it was provisioned. If you click on your Thing in the AWS dashboard, you can see the certificates associated with it by clicking 'Security' in the left sidebar. If you click on the certificate that you want to use, it will open another dashboard with an 'Actions' menu in the upper-right that lets you activate or deactivate the certificate. If the 'activate' button is greyed-out, the certificate is already active.
Code: Select all
import machine
from network import WLAN
import network
from umqtt.simple import MQTTClient
# Setup WiFi connection.
wlan = network.WLAN( network.STA_IF )
wlan.active( True )
wlan.connect( "WiFi_ssid", "WiFi_password" )
while not wlan.isconnected():
machine.idle()
# Connect to MQTT broker.
mqtt = MQTTClient( "test_mqtt_client_id",
"[...your MQTT endpoint...]",
port = 8883,
keepalive = 10000,
ssl = True,
ssl_params = {
"cert": "-----BEGIN[...the rest of your certificate...]",
"key": "-----BEGIN[...the rest your private key...]",
} )
mqtt.connect()
# Publish a test MQTT message.
mqtt.publish( topic = 'test', msg = 'hello world', qos = 0 )
And if you're having trouble connecting to an AWS endpoint with an 'invalid key' error, it might be worth double-checking that the certificate was successfully activated after it was provisioned. If you click on your Thing in the AWS dashboard, you can see the certificates associated with it by clicking 'Security' in the left sidebar. If you click on the certificate that you want to use, it will open another dashboard with an 'Actions' menu in the upper-right that lets you activate or deactivate the certificate. If the 'activate' button is greyed-out, the certificate is already active.
Remember, the "S" in "IoT" stands for "Security".