MicroPython Forum (Archive)

So I spent a few months whenever free time presented itself both learning about MQTT and how to get the latest version of SSL/TLS (1.2) and latest mqtt standard fully working with micropython on an esp32 with a raspberry pi as both the broker running mosquitto, and functioning as a client data-logger using paho mqtt in python....you know, a pretty standard setup that plenty of micropython tutorials are already out for, except save the TLS part
*(more accurately: got it working as completely as possible wrt this moment in the evolution of micropython firmware. See endnote.)

I believe a lot of people are interested in this in general, and for me, getting the TLS part going was a rather tedious & time-consuming learning curve compared to other things that I'm more personally more drawn to (you know using micropython to learn about all of the usual suspects of beginner's geekery with microcontrollers: controlling sensors, lights, accelerometers, etc) but I found the payoff to be well worth effort as a foundation for data collection, for both home automation, and to generate some real data for my statistical education.

I'd like to contribute something useful to micropython in whatever ways I can as a newcomer to all things microcontrollers, because, if I understand correctly--and to probably put it only partially--you developers are working really hard as volunteers to make programming microcontrollers accessible to the interested public in general, because many of us might not have the optimal combinations of the underlying technical know-hows (and time and energy to acquire them) and this a raison d'etre of micropython and why the python language was chosen as the medium of that accessibility.

Which obviously is what makes micropython quite freakin awesome.

So I was thinking it'd be nice to spare others in my non-expert positions from this tedium--getting as full of an understanding as possible of what's required to get the whole TLS deal working in general, becoming aware of and fixing multiple problems going on in Raspbian that were not obvious to me at the time, and finally realizing that if one wants to use client certificates, the way one does so with the python version of paho and way one does so with micropython is apparently not exactly the same, although the syntax options share the same names (not a criticism), and finally there are nuances with the mqtt broker configuration file options that can conflict/override, but this isn't always obvious.

So how to share it?

While I realize that setting up a mqtt broker on a raspberry pi is not within the scope of the micropython Q&A areas, I just wanted to put this out there, in case it's of any use:

The best idea I could come up with would be if there were some kind of 'micropython projects' wiki or some name for a collection of tutorials that is a centralized knowledge-base that all users could consult and contribute to, but that is also strengthened by some sort of peer-review-inspired process where I, the beginning user, knows that I don't have to be totally 100% certain that I'm correct, wise, etc about what I'm putting out there for for others to use because one of the core developers or perhaps a set of advanced users designated by them could make any technical corrections, and/or suggest recommendations and optimizations.

And as someone who is learning, the user could trust and learn from the source more than one could with some guy's blog that may or may not be a very good source of knowledge. Since time is everyone's limited resource, that cuts way down on tons of google searching in hopes of finding a blog tutorial or tidbits of perquisite knowledge that one needs to have in order to accomplish what he or she is out to do.

So perhaps the wiki or whatever would sort of resemble Adafruit's tutorials where anyone can submit a tutorial, and ones that they feel are good and functional get posted. But a difference could perhaps be that it doesn't have a slickly packaged appearance, which takes time to put together.

This might save the developers time as well, because it seems like it would reduce redundant 'how do I' questions, and it would greatly expand the number of volunteers that could aid micropython's documentation development and outreach, and it seems that that would allow the developers to have more time for doing the real work that makes it all tick.

Otherwise, while I could put it on a personal blog or maybe someone else's, but to me, not only would that be less useful, but I also find a good number of blogs to be rather distasteful and shameless self-promotions, because for one thing, it gives the appearance that I have authoritative take on something that's actually an amateur's implementation of a small subset of a lot of hard 'real' work done by the founder and developers, and all of that seems rather narcissistic.

Wouldn't it be cool to tap the general user-base as an enthusiastic resource and create a centralized source of learning material?




*(endnote: Okay well fully working to the extent that it's been developed so far, and when the client verification of the broker aspect of the process is implemented, then the coding I think will largely remain the same. But other than that, the broker verifies the client, and everything is encrypted, including the mqtt topics. I hope to manually implement something that the client can do to verify the broker, such as add a cryptographic string to the mqtt topics or something like that)

I suggest posting tutorial material on GitHub and putting a link to it in this forum and in the MicroPython wiki. Others can review it and raise PR's (pull requests) to indicate errors or offer improvements. This is how we handle this at the moment.

For example this repo has a tutorial on the uasyncio module.

Thanks for the reply.

Yeah I'm aware of what's usually done now with the forum+github, and that's certainly useful, of course.

While it was slightly buffoonish of me to have never noticed the 'wiki' link at the bottom of the homepage, I guess I'd still interpret that section to be only generally reserved for the micropython aspect of my own commonplace project, and I wouldn't want to be presumptive and assume that it's okay to also put there the necessary configuration of both the mqtt broker and a couple of other aspects of Raspbian that need to be 'fixed' with workarounds first...I would have expected to be told that that's beyond the intended scope of the wiki....
Am I wrong about that?

But I'll take you up on your suggestion to use github first, and post a link in the forum so that people can make suggestions to improve it.

I've actually looked at your async mqtt client scripts a few times in the recent past, and on my to-do list is still to see if what I think I did right for SSL works with it and let you know (since you mentioned on there you had some frustrations with SSL/TLS as well and wanted feedback)
I haven't gotten a chance yet to understand when it's more advantageous to use async vs sync with a mqtt client though, and when to block and when not to block...most tutorials don't go into that so much...

And Peter, thanks for sharing your async mqtt client work. It looks quite nice, and the documentation is nice too.

And I thought I would also point out that every github repository has a wiki associated with it as well.

carver wrote: ↑

Mon Mar 19, 2018 4:04 pm

I haven't gotten a chance yet to understand when it's more advantageous to use async vs sync with a mqtt client though, and when to block and when not to block...most tutorials don't go into that so much...

You might like to look at the beginner section of my uasyncio tutorial referenced here. In essence using an asynchronous driver provides an illusion of concurrency by switching between different tasks. It enables your code to do other things while the MQTT code is busy, perhaps while trying to reconnect a failed wireless link. It also enables the driver to do things as a background task. An example is keeping the broker connection alive by issuing periodic pings, which the async driver does entirely transparently if no communication has occurred for a period.

Most non-trivial firmware applications use some kind of asynchronous programming framework. Blocking device drivers are a complete pain as they prevent anything else from occurring for the duration.

As for the MicroPython Wiki I was suggesting it as a location for a link to your GitHub tutorial. I doubt anyone will object to a pointer to something as useful as using a Raspberry Pi as an MQTT broker with TLS.

I sent a Private Message a while ago to the Global Moderators making a proposal for new Forum Category, I include it below as it has a similar drive (I think) to your request @carver

You may have seen me hanging around the forum for several months, its a great resource.
I am a very, very late starter at programming and you would consider me an absolute beginner at Micropython. As a result I have gained a huge amount from tutorials and from-the-bottom-up basic explanations.
Links to many great advice/explanations/tutorials can be found by searching within the forum, but that is a bit hit and miss. Most of the forum discussion is (rightly) very technical and not really aimed at the beginner.

Could you consider the addition of a new category (or at least perhaps a sticky thread) for Micropython beginners; with the aim of providing beginners a list of basic tutorials. Clearly this is a subset of News & General Information.
I understand that the purpose of the forum is not to provide links out to elsewhere, however, by doing this only for absolute beginners, it will hopefully reduce the number of basic questions asked (usually by me!).

Forum users could either post good tutorial examples or beginners may ask if tutorials exist for a particular topic. I have plenty of great links to start the ball rolling!
thanks for listening,
Phil

there was a positive response from a couple of moderators and a suggestion to start with a sticky on one of the forum categories. However, this idea has not developed past this point.

I am sure that many of the regular users of this forum get frustrated by repeated beginner questions, so I think this may help reduce the flow a little.