Connecting to AWS with MQTT

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
james_km69
Posts: 9
Joined: Wed Aug 15, 2018 8:31 pm

Connecting to AWS with MQTT

Post by james_km69 » Sun Aug 19, 2018 4:33 pm

Hi,
I have been struggling with this for last 2 days. I have a ESP32 with micro python ruining. I created AWS account and set all the cert and polices. I tested the certificate through MQTT.FX (works for for both subscribe and publish)
Now I am trying to connect my ESP32 to AWS but no luck. I have tried everything possible.
I uploaded all the certificate file to root directory of my ESP32 and using main2.py file with following code:

import machine
from network import WLAN
import network
import time
from umqtt.simple import MQTTClient

DISCONNECTED = 0
CONNECTING = 1
CONNECTED = 2
HOST = "a21sigud7911d7.iot.us-west-2.amazonaws.com"
TOPIC = "myThingName"
state = DISCONNECTED
connection = None

wlan = network.WLAN(network.STA_IF)
wlan.active(True)
def pub_msg(msg):
global connection
connection.publish(topic=TOPIC, msg=msg, qos=0)
print('Sending: ' + msg)

def run():
global state
global connection

while True:
while state != CONNECTED:
try:
state = CONNECTING
connection = MQTTClient(client_id=TOPIC, server=HOST, port=8883, keepalive=10000, ssl=True, ssl_params={"certfile":"/49c84a8c4a-certificate.pem.crt", "keyfile":"/49c84a8c4a-private.pem.key", "ca_certs":"/root.pem"})
connection.connect()
state = CONNECTED
except:
print('Could not establish MQTT connection')
time.sleep(0.5)
continue

print('MQTT LIVE!')

while state == CONNECTED:
msg = '{"device_id":"some_id", "data":"some_data"}'
pub_msg(msg)
time.sleep(2.0)


while True:
if 1==1:
print('Emerge' +" was found!")
wlan.connect('Emerge', '123456789')
while not wlan.isconnected():
machine.idle()
print('Connected to '+ 'Emerge')
run()
break

AND OT PUT AT THE END IS:
"Could not establish MQTT connection"
I am not sure if it is reading all 3 certificate files right OR do i need to put them somewhere else!!!

anyone knows what the issue might be?
Thanks in advance

james_km69
Posts: 9
Joined: Wed Aug 15, 2018 8:31 pm

Re: Connecting to AWS with MQTT

Post by james_km69 » Sun Aug 19, 2018 5:01 pm

Just an update, after print out the exception i get this:

I (22290) network: CONNECTED
I (22950) event: sta ip: 192.168.111.91, mask: 255.255.255.0, gw: 192.168.111.1
I (22950) network: GOT_IP
Could not establish MQTT connection list index out of range
Could not establish MQTT connection extra keyword arguments given
Could not establish MQTT connection 23
Could not establish MQTT connection 23

james_km69
Posts: 9
Joined: Wed Aug 15, 2018 8:31 pm

Re: Connecting to AWS with MQTT

Post by james_km69 » Mon Aug 20, 2018 8:46 pm

Well I guess no answer to this issue?

User avatar
pythoncoder
Posts: 3725
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Connecting to AWS with MQTT

Post by pythoncoder » Thu Aug 23, 2018 7:34 am

This question comes up repeatedly (search for TLS in this forum). Last year I tried to get MQTT working with TLS and failed. Firmware support has improved since then and it is allegedly possible to use certificates. A definitive answer, with a code sample, would be good.
Peter Hinch

User avatar
marfis
Posts: 193
Joined: Fri Oct 31, 2014 10:29 am
Location: Zurich / Switzerland

Re: Connecting to AWS with MQTT

Post by marfis » Thu Aug 23, 2018 11:46 am

we‘ve been using wipy3 devices to connect to AWS IoT, basically following a tutorial from pycom doc webpage (https://docs.pycom.io/chapter/tutorials/all/aws - sadly doesn‘t seem to exist anymore :( )

Anyway you might get some ideas by visiting their git repo
https://github.com/pycom/aws-pycom?files=1

As @pythoncoder pointed out it would be nice to have step by step tutorials for common cloud sites using official fw builds (no forks).

james_km69
Posts: 9
Joined: Wed Aug 15, 2018 8:31 pm

Re: Connecting to AWS with MQTT

Post by james_km69 » Thu Aug 23, 2018 12:02 pm

I racked my brain for a few days but finally got it to work. It turned out to be a missing parameters in ssl_params list:
I added "server_side":False and removed Ca root (I am not sure why it does not need this variable) and it start publishing and subscribing to the AWS. Here is my code:

CACERT_PATH = "/flash/root.pem" // apparently this wont be used!!!
KEY_PATH = "/flash/49c84a8c4a-private.key"
CERT_PATH = "/flash/49c84a8c4a-certificate.pem"
with open(KEY_PATH, 'r') as f:
key1 = f.read()

with open(CACERT_PATH, 'r') as f:
key2 = f.read()

with open(CERT_PATH, 'r') as f:
cert1 = f.read()

client = MQTTClient(client_id="My_topic/klm", server="a21sigud7911d7.iot.us-west-2.amazonaws.com", port=8883, keepalive=4000, ssl=True, ssl_params={ "key":key1, "cert":cert1, "server_side":False })

Just to let everyone knows in case they run into same issue.
Thanks

User avatar
pythoncoder
Posts: 3725
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Connecting to AWS with MQTT

Post by pythoncoder » Fri Aug 24, 2018 7:50 am

Very useful :D
Peter Hinch

DrKarma
Posts: 1
Joined: Tue Dec 25, 2018 5:08 pm

Re: Connecting to AWS with MQTT

Post by DrKarma » Thu Jan 03, 2019 11:33 pm

I have pretty much done exactly as you have stated in your example (james_km69)


KEY_PATH = "62c4233362-private.pem.key"
CERT_PATH = "62c4233362-certificate.pem.crt "
with open(KEY_PATH, 'r') as f:
key1 = f.read()
print(key1)

with open(CERT_PATH, 'r') as f:
cert1 = f.read()
print(cert1)

client = MQTTClient(client_id="EntranceRFID", server="XXXXXXXXXXX.eu-central-1.amazonaws.com", port=8883, keepalive=4000, ssl=True, ssl_params={ "key":key1, "cert":cert1, "server_side":False })
## (XXXXXXXXXXX = my AWS endpoint).

When running key1 and cer1t is loaded (and I do a printout in REPL for these as you can see above).

But when it is time for the client.connect()
I get
File "umqtt/simple.py", line 61, in connect
ValueError: invalid key.


I am not sure what I have done wrong. The certificate "62c4233362..." is activated and I have connected a "Thing" (EntranceRFID) to it in AWS IoT.

Any advice?

Outback Bob
Posts: 1
Joined: Sun May 05, 2019 7:28 am

Re: Connecting to AWS with MQTT

Post by Outback Bob » Sun May 05, 2019 7:39 am

I was getting the same issue as DrKarma, slightly different error however:

>>> client.connect()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "simple.py", line 97, in connect
IndexError: bytes index out of range

If anyone else gets this problem, check your AWS thing's permissions. The default policy created when you onboard a device through the AWS wizard doesn't appear to allow connections. The default is:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"iot:Publish",
"iot:Receive"
],
"Resource": [
"arn:aws:iot:ap-southeast-2:xxx:topic/sdk/test/java",
"arn:aws:iot:ap-southeast-2:xxx:topic/sdk/test/Python",
"arn:aws:iot:ap-southeast-2:xxx:topic/topic_1",
"arn:aws:iot:ap-southeast-2:xxx:topic/topic_2"
]
},
{
"Effect": "Allow",
"Action": [
"iot:Subscribe"
],
"Resource": [
"arn:aws:iot:ap-southeast-2:xxx:topicfilter/sdk/test/java",
"arn:aws:iot:ap-southeast-2:xxx:topicfilter/sdk/test/Python",
"arn:aws:iot:ap-southeast-2:xxx:topicfilter/topic_1",
"arn:aws:iot:ap-southeast-2:xxx:topicfilter/topic_2"
]
},
{
"Effect": "Allow",
"Action": [
"iot:Connect"
],
"Resource": [
"arn:aws:iot:ap-southeast-2:xxx:client/sdk-java",
"arn:aws:iot:ap-southeast-2:xxx:client/basicPubSub",
"arn:aws:iot:ap-southeast-2:xxx:client/sdk-nodejs-*"
]
}
]
}


Replace this with a very basic:
{
"Statement": [
{
"Effect": "Allow",
"Action": "iot:*",
"Resource": "*"
}
],
"Version": "2012-10-17"
}

and your device will connect!!! I wasted HOURS before working this out, so hope this saves someone else some trouble :)

WRR
Posts: 7
Joined: Wed Jan 09, 2019 8:33 pm

Re: Connecting to AWS with MQTT

Post by WRR » Mon May 06, 2019 6:40 pm

To summarize james_km69's example, it looks like this works for publishing MQTT messages with SSL:

Code: Select all


import machine
from network import WLAN
import network
from umqtt.simple import MQTTClient

# Setup WiFi connection.
wlan = network.WLAN( network.STA_IF )
wlan.active( True )
wlan.connect( "WiFi_ssid", "WiFi_password" )
while not wlan.isconnected():
  machine.idle()

# Connect to MQTT broker.
mqtt = MQTTClient( "test_mqtt_client_id",
                   "[...your MQTT endpoint...]",
                   port = 8883,
                   keepalive = 10000,
                   ssl = True,
                   ssl_params = {
                     "cert": "-----BEGIN[...the rest of your certificate...]",
                     "key": "-----BEGIN[...the rest your private key...]",
                   } )
mqtt.connect()

# Publish a test MQTT message.
mqtt.publish( topic = 'test', msg = 'hello world', qos = 0 )

But as others have mentioned, this code does not verify the certificate's chain of trust, which could leave your device open to man-in-the-middle attacks. I couldn't figure out how to verify a CA from a cursory look at the 'modussl_mbedtls' module, but I could be missing something and that functionality is supported by the underlying mbedtls C libraries.

And if you're having trouble connecting to an AWS endpoint with an 'invalid key' error, it might be worth double-checking that the certificate was successfully activated after it was provisioned. If you click on your Thing in the AWS dashboard, you can see the certificates associated with it by clicking 'Security' in the left sidebar. If you click on the certificate that you want to use, it will open another dashboard with an 'Actions' menu in the upper-right that lets you activate or deactivate the certificate. If the 'activate' button is greyed-out, the certificate is already active.
Remember, the "S" in "IoT" stands for "Security".

Post Reply