Threading questions

All ESP32 boards running MicroPython.
Target audience: MicroPython users with an ESP32 board.
Post Reply
kgschlosser
Posts: 13
Joined: Fri Apr 10, 2020 6:18 pm

Threading questions

Post by kgschlosser » Thu Sep 03, 2020 3:47 pm

I have a project where I am running multiple strips of LED's. I would have run a single strip except the strip takes about 4.5 seconds to refresh. This is not going to work for me. So what I did was cut the strip into 10 smaller strips. smaller number of pixels = faster refresh time. I can deal with writing the pixels correctly in code.

There are situations where more then one of these strips will need to be written to and if i write them consecutively I end up back where I started again with the slow refresh rates. So what I want to do is I want to create a thread for each of the strips to take care of the writing of the pixel data. This would work without a hitch except for the lack of an Event class like what is in the threading module in CPython. If i use a no-op loop this actually turns out to be a spinning wheel and it does impact the performance of any thread that may be working.

I am going to be using input pins to trigger what strips get lit. I wanted to use the interrupt callback for the pins to activate a strip. I cannot use locks because I have no way to acquire the lock initially from the thread that is going to be making the callback for the interrupt request. only the thread that acquired a lock can release it. so locks aren't going to work no Event class either. are there any other options? I know there is asynchio my concern there is it's a behemoth and frankly overkill for something s simple as this should be. I need a way to stall a thread where I can release the stall from another thread without having the stalled thread spin it's wheels.

Any suggestions are greatly appreciated.

User avatar
pythoncoder
Posts: 4810
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Threading questions

Post by pythoncoder » Fri Sep 04, 2020 6:33 am

It would help if you clarified the type of LED strip you're using. NeoPixels are timing critical so it's possible that the driver won't work reliably in a threaded environment.

You describe uasyncio as a behemoth. It really isn't and it's built in to the latest release of firmware. There is a learning curve to get familiar with all its features but basic usage is easy. Regardless of whether you use threading or uasyncio you need to know what is causing your slow updates. If it's a piece of code with critical timing which requires sole access to the CPU it may not be possible to use concurrency to speed things up.
Peter Hinch

kgschlosser
Posts: 13
Joined: Fri Apr 10, 2020 6:18 pm

Re: Threading questions

Post by kgschlosser » Mon Sep 07, 2020 6:01 am

The slow updates are because the LED's are slow. The neopixel library is for ws2812 and equivalent LED strips. each bit takes 1.25us to send. I have 288 pixels and 4 LED's per pixel (RGBW). so that 1152 LEDs, 8 bits per LED for a total of 9216 bits. and with a timing of 1.25us per bit that's 11520us to update the LED strip a single time. so if I wanted to move the lit led down the strip it would take 3317760us (3.318 seconds). That is slow. This is the nature of the LED strips. APA102 strips are a heap faster but have been discontinued as of 2018. so locating "real" ones is next to impossible.

Now using threads is a non issue so long as I can stop a thread from spinning it's wheels when there are no updates needed to an LED strip. If I have to use a while loop checking a flag which is considered to be a no op then I end up having timing issues. I have tested this with 6 threads and the while loops cause timing problems. If I stall a thread using a lock object the timing problems go away. if all of the threads are writing to the LED strips at the same time I do not have timing issues. The timing issues only appear when I have no way to stall a thread without having it spin it's wheels.

I also have changed my design up a little bit. I need to have 6 threads plus the main thread in order to get to my goal. Now I was reading the documentation for the _thread module in CPython and it states that a lock object can be released from a thread that does not have the lock acquired. I am really hoping that this is the case with MicroPython as well. Because I have been able to come up with something that may work using lock objects.

In CPython you have the threading.Event class that can be used to stall a thread and has the ability to be released from any other thread. It also has an optional duration that can be used to set how long to stall for. It would be much easier to accomplish my goal using this mechanism instead of having to deal with lock objects to stall a thread.

This is how I have currently written the code (untested)

main thread creates lock objects for each of the threads.
main thread acquires each of the locks.
threads are created
in the loop for each thread the first thing that is done is the lock for that thread is acquired. because the main thread has not released the locks this stops the thread from running.
main thread reads the pins. if a pin has been activated it then makes any needed changes and then releases the lock for the thread that is responsible for writing the data to a string.
once the data has been written by the thread it loops back around and acquires the lock again. but because it was not released a second time the thread gets stalled until the main thread calls release again.

The whole thing above hinges on whether or not MicroPython's _thread.allocate_lock() functions the same as CPythons and a lock can be released by another thread and not the thread that acquired the lock.

as far as using asynchio, I do not know to much about asynchio and what the actual purpose for it is. It does exactly the same thing that can be done with threads and I cannot find where there is any kind of a benefit to it over threads other than it may be easier to use for a person that is not familiar with how to properly implement threading. the end result is more resources used than what would be needed if using threads. This is where my behemoth statement comes in.

kgschlosser
Posts: 13
Joined: Fri Apr 10, 2020 6:18 pm

Re: Threading questions

Post by kgschlosser » Mon Sep 07, 2020 6:11 am

Attached is a copy of the code I have written to handle the LED's. The number of LED's has changed because of my poor soldering skills :lol:, also because of space requirements with the additional wiring as I had to use some pretty decent size wire because of the 10 foot run and wanting to keep the voltage drop under 1%. These LED's will burn if you set the value of each pixel to 255. So I needed to make a heat sink for them and the wiring has to be inside that heat sink along with the LED's.

so now I am down to 254 LEDS.
Attachments
main.zip
(3.59 KiB) Downloaded 29 times

User avatar
pythoncoder
Posts: 4810
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Threading questions

Post by pythoncoder » Mon Sep 07, 2020 8:34 am

While I've used cooperative multi-tasking on other platforms I have no experience of MicroPython threading. I can't help with your detailed queries - hopefully someone will emerge who can.

Re uasyncio vs _thread, the threading module is heavy on resources and isn't provided on platforms with minimal RAM. By contrast uasyncio is lightweight.

There are more generic reasons for preferring cooperative multi tasking: if you're interested you might like to read my view. The issues are perhaps more eloquently expressed in this article why threads are bad. However you are evidently knowledgeable about threaded code so this may not be new to you.
Peter Hinch

kgschlosser
Posts: 13
Joined: Fri Apr 10, 2020 6:18 pm

Re: Threading questions

Post by kgschlosser » Mon Sep 07, 2020 4:57 pm

OK I can see the argument for using asynchio vs threading. asynchio is better used when writing code that is going to be used by others and the original author does not know exactly how it is going to be used. This is not the case with my program. I am the one using it and so there for I am going to know when I need to pause a thread so it can share state with other threads. when writing a framework for say some kind of a web service knowing when to pause is not going to be known because this may be different for each user.

In my use case I am not doing it for the purposes of "sharing a state" I am doing it so that 2 states can run concurrently. While I do share some state data between the threads it is only done for the sole purpose of not having a thread that simply sits there and spins it's wheels.

I had written an extremely complex binding to openzwave using python. It had to be really complex because of how openzwave was written. This binding used x threadworkers where x was defined by the number of zwave nodes (devices) the user had. it was extremely difficult to get the bugs out of the system because it was hard to understand what what each thread was doing at any given point in time... So I had spent some time writing this logging/debugging program https://github.com/kdschlosser/angry_debugger to be able to track what a given thread is doing. It allows a user to track the data as it travels through their program while it is extremely expensive to use that logging program and it is not something that should be used in a production application It has helped me on many occasions to track down collisions between threads.

Threading is a hard concept to fully grasp, and to write code that is efficient using threads is extremely difficult to do. To write threaded code where the end use is not going to be known is next to impossible to do.

my whole problem is because of the speed of the chipset of the LED's, and this stems from the LEDs being the only thing that is readily available. I am going to test my code either today or tomorrow to see if it will work. if it does work then I am going to be in good shape. if it does not then I am going to have to locate some form of a faster LED chipset to use.

Post Reply