MQTT using transparent mode on Cellular modem (SIM7000g)

[kinno]

Good morning friends!

I am working through a proof of concept idea for adapting MQTT libraries to use the transparent mode of my cellular modem.

Topology:
ESP32 connected to Cellular modem (SIM7000g) via UART 1.

Then modem goes into transparent mode after connecting to my MQTT server. This means ALL data sent over serial (UART 1) goes directly to the open TCP connection. AND all data received from the server comes back through the UART.

I am having a tough time modifying the simple.mqtt package to do this. My first attempt was to try to trick the simple.mqtt package by naming my UART serial connection 'sock' and just running the package, some small calibrations needed to be made because the connection was already made to the TCP port. But no worky.. I mean that would have been an absolute miracle if it worked. I know serial and sockets are similar, but obviously not the exact same. Worth a try anyways.

I thought this could be a great brain teaser for anyone interested.

My next steps are to strip out the MQTT functions and build an even slimmer client over UART only looking at functions after the connection has been established.

I am also going to try to use JUST the MQTT functions of the modem (SIMCOM7000g has AT commands for MQTT functionality and I have that working). But, I am not sure best practices in this situation in order to process commands sent to my device.

I look forward to some discussions. Ask any questions you wish.

Thank you everyone for your time.

[kevinkk525]

It might be possible if you replace the socket import in mqtt with a custom socket implementation that reads/writes and buffers the UART.

However this won't be as easy as it sounds and I wouldn't recommend the mqtt.simple library after proof-of-concept succeeded since the resilience of that library is not good and with a cellular modem connection you probably get a lot of outages, delays and partial transmissions.
If you get transparent mode running and a good UART to socket bridge then you can have a look at mqtt_as https://github.com/peterhinch/micropython-mqtt as a resilient mqtt library.
However since that library also controls the wifi, you'd have to replace the wifi code too..

I'd certainly be interested in how it turns out


If your modem supports AT commands for MQTT then this is probably the easiest and (depending on their implementation) most resilient approach with the least amount of work.

[kinno]

@kevinkk525

After some more fiddling today I think you are right with the idea of scripting the AT commands and using the Simcom 7000g's MQTT protocol. I am asking for a whole bunch of trouble by trying to use libraries that are not designed for this type of use.

I am going to be doing more research into the idea of transparent mode and how it is usually utilized. I will report any interesting finding back here in order to keep the conversation going. I am sure other people are looking for similar solutions.

Thanks for your prompt reply!

[kinno]

So, I have thought of an idea in order to use the transparent mode of my cellular modem on UART 1.

I am looking for feedback on this general idea as I am having trouble sorting through the higher level conceptualization of this.

I am looking to make a sort of proxy server on the ESP32 that handles the socket to UART conversion.

The process would look like this:

1. Set up socket that listens on local host with port 1883.

Code: Select all

server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(('', 1883))
server.listen(1)

2. have umqtt.simple connect to this socket.

Code: Select all

mqtt_server = 'localhost'
mqtt_port = '1883'
client_id = "003"
topic_pub = "Channel_LTE"

def connect_and_subscribe():
    print('Trying to connect to MQTT')
    global client_id, mqtt_server, topic_sub
    client = MQTTClient(client_id, mqtt_server)
    client.connect()
    print('Connected to %s MQTT broker.' % mqtt_server)
    return client

3. Once connected have the server pass the information received on the socket to the UART and then send information receive don UART back through the socket.

Code: Select all

conn, addr = server.accept()
payload = conn.recv(1024)
        print('Sending = %s' % str(payload))
        simcom.write(payload)
        at(chr(26))
        time.sleep(1)
        read = simcom.read()

I know this is super simple but I am trying to proof of concept this.

My problem right now is once I wait for conn, addr = server.accept() the code stops as it is waiting for connections. I am thinking asyncio may fix this and this is where I am heading now to try to find some code for async sockets.

If anyone has ideas on this please share!!

Thanks again everyone.

[kevinkk525]

I would try to implement a socket class as a drop-in replacement, so it provides read(), write(), connect() and close().
Then you have one coroutine running all the time that reads from the UART into a (ring)buffer.
Read() could then read from the buffer, write directly writes to the uart. not sure how close and connect are handled over uart.