LCD UI - M5Stack and the ILI9341

[mattyt]

Hi folks,

I have a need to render some simple UI widgets on an LCD. I'm looking to use an M5Stack which has a 320x240 LCD, driven by an ILI9341 and there are three buttons on the face of the device. The kind of interface I need to build would involve using the buttons to navigate a menu and select options - after selection a chart or progress bar etc may be rendered.

I'm aware that a few basic drivers exist for this chip. The ILI9340/ILI9341 TFT Display forum thread is useful and contains some links: Deshipu has created a driver and Jeffm took it a little further by giving it better font support based on pythoncoder's work.

OutOfTheBots also has a driver that can display images and text.

Loboris' fork contains an apparently fast driver with some decent primitives support (circles, rounded rectangles etc). I would prefer to stay upstream if possible however...

And finally Peter Hinch has something close to what I'm looking for, micropython-gui. Except that it only works with different hardware and is designed to operate with a touch screen.

Are there any other libraries I'm missing?

I was hoping to find a widget library that was independent of the hardware - perhaps built on top of framebuf. Does such a thing exist? Would there be interest in such a library?

[OutoftheBOTS_]

I was originally intending to make a GUI or a ili9341 with touch due to cheap price. I did make a number of bit of prototype code.

The original code was on an ESP8266 so it didn't have much RAM to play with so I used the flash to store all the images (both back ground image and foreground widgets) in 16bit BMP files and text fonts in my own binary format. This way when I wanted to restore the background after remove foreground widgets or text I little math could seek the needed place in back ground BMP and reload the needed window to restore the back ground.

I did also make a transparent back ground text that could be scaled and the back ground could be restored see https://www.youtube.com/watch?v=4OeWaat3jZs

After I switched to ESP32 with psRAM I was now intending to do the same with psRAM instead of flash but Laboris mentioned he was working on same thing so I stopped and was waiting to see what he achieved as his code normally is much superior to mine.

[OutoftheBOTS_]

Loboris' fork contains an apparently fast driver with some decent primitives support (circles, rounded rectangles etc). I would prefer to stay upstream if possible however...

Laboris SPI TFT driver is the by far the fastest SPI driver I have seen. It would be worth while seeing if it can be merged upstream into main port of ESP32

[mattyt]

The original code was on an ESP8266 so it didn't have much RAM to play with so I used the flash to store all the images (both back ground image and foreground widgets) in 16bit BMP files and text fonts in my own binary format. This way when I wanted to restore the background after remove foreground widgets or text I little math could seek the needed place in back ground BMP and reload the needed window to restore the back ground.

I did also make a transparent back ground text that could be scaled and the back ground could be restored see https://www.youtube.com/watch?v=4OeWaat3jZs

Nice work, looks great! Especially impressive to get that working on an ESP8266.

After I switched to ESP32 with psRAM I was now intending to do the same with psRAM instead of flash but Laboris mentioned he was working on same thing so I stopped and was waiting to see what he achieved as his code normally is much superior to mine.

Loboris certainly is prolific! And the ESP32 is a lot more capable in this area.

Laboris SPI TFT driver is the by far the fastest SPI driver I have seen. It would be worth while seeing if it can be merged upstream into main port of ESP32

I have wanted to pull a fair bit of Loboris' work upstream but there are a few more constraints there - particularly around needing to support multiple ports. His TFT code for example; it may be fast and full of features but perhaps the architecture is inappropriate for upstream MicroPython - I'm not at all negative about Loboris' work, it's fantastic!, but he can move a lot faster than upstream. Take a look at moddisplay_tft; note that all LCD's are defined in this file. If I understand this right then I'm not particularly comfortable with that design choice because it will use extra memory on ports that may not have it - and it adds to friction when adding new display drivers. That said his lower-level control of the LCD drivers is excellent - it'll just take some effort to extract it nicely.

But pulling this code upstream is certainly a strong option...

Still, driving the LCD is only part of the problem. I'm really looking for higher-level widget primitives: MenuGroup, ProgressBar etc that can let me render such widgets on to a region of a framebuf for the driver to display them.

I then also need a state machine and a sprinkling of asyncio to tie it all together...

[It's probably best left for another post but I could sure use your help pulling other parts of Boris' work upstream... ]

[OutoftheBOTS_]

That said his lower-level control of the LCD drivers is excellent - it'll just take some effort to extract it nicely.

But pulling this code upstream is certainly a strong option...

Still, driving the LCD is only part of the problem. I'm really looking for higher-level widget primitives: MenuGroup, ProgressBar etc that can let me render such widgets on to a region of a framebuf for the driver to display them.

I have been thinking about this thread a bit over last couple of days because long term I am wanting a nice GUI for my MP projects.

The big thing I started to think about is that high level code builds upon the low level code. Sitting down and spending time to write high level code based upon less efficient low level code isn't a good investment in your time because at a later stage if you want to improve the low level code it can mean a total rewrite of the high level code.

It is better to have the best low level code to start with and build upon that.

Do be aware that Laboris just posted an update to his display module to include added support for FTDI EVE (Embedded Video Engine), FT80x and FT81x and this provides all the widgits built in on the display see https://www.youtube.com/watch?v=SfL8NtfFfio and here is the screen he was developing upon https://www.hotmcu.com/5-graphical-lcd- ... p-286.html

I am a very new programmer just started programming on Lego mind storms less than 2 years ago then after that RPi python then MP, I am now teaching myself C on the STM32 as there is a lot info that I can youtube to learn from.

So far I haven't looked at any of the MP firmware code as it has been over my head but the idea of learning C was to be able to understand and modify the MP firmware.

[pythoncoder]

...I was hoping to find a widget library that was independent of the hardware - perhaps built on top of framebuf. Does such a thing exist? Would there be interest in such a library?

I have been doing just that here. This is far from finished, is subject to change, and is not yet properly documented...

The idea is to take elements from my GUI for the official display and port them to displays whose drivers are subclassed from framebuf. Since these don't have touch overlays there is no point in supporting data entry widgets. It's also worth bearing in mind that only small displays are suitable for this treatment owing to the RAM usage of the framebuf instance. Consequently the widgets have needed some redesign for small displays.

However it does enable extremely simple display drivers to be written: see this example. With that trivial driver you get graphics primitives via framebuf, colour text rendering in multiple fonts via my CWriter class, and (forthcoming) scalable GUI widgets via nano-gui. I plan to write similar drivers for other small colour OLED displays.

The Label, Meter and LED classes have been implemented and tested on SSD1331 colour OLED and SSD1306 monochrome OLED displays using the official driver.

[deshipu]

With such a large display, you are likely to have problems with memory if you use the naive approach of just drawing your ui on the display. However, if you can redesign your UI so that it can be represented as a tiled grid of similar images and some sprites, you could make use of my "stage" library: https://hackaday.io/project/80626

I have it working on the M5 stack, though I have only tested it with a simple "bouncing ball" demo so far on it.

There is some work underway to use a similar approach for displaying graphics in CircuitPython done by Adafruit. I think the general idea of using tiles and sprites to save memory and help you organize your interface is very promising.

[OutoftheBOTS_]

It's also worth bearing in mind that only small displays are suitable for this treatment owing to the RAM usage of the framebuf instance.

You can use the idea that used for my ESP8266 ili9341 driver. Instead of using a framebuffer in RAM I just used the flash to store the image in a 16 bit BMP file then instead of coping it from RAM to SPI I copied it from flash to SPI.

Especially since in order to make a frame buffer it is normally stored in flash then loaded from flash to RAM then sent to SPI, I just cut out the middle man the frame buffer and went from flash to SPI

[mattyt]

I have been thinking about this thread a bit over last couple of days because long term I am wanting a nice GUI for my MP projects.

It seems to me it's a common use case...

The big thing I started to think about is that high level code builds upon the low level code.
...
It is better to have the best low level code to start with and build upon that.

I think it's more about getting the abstractions right. I'm not fussed if I don't have the fastest SPI driver today as long as I get the interface right so I can drop in a faster one tomorrow...

Do be aware that Laboris just posted an update to his display module to include added support for FTDI EVE (Embedded Video Engine),

These devices are quite cool but not what I'm interested in. The EVE chips handle all the graphics rendering; I actually want to do that in the microcontroller. Microcontrollers - at least some - are big enough and fast enough now and the benefit is flexibility; you can make your rendering system do whatever you want.

I am a very new programmer just started programming on Lego mind storms less than 2 years ago then after that RPi python then MP, I am now teaching myself C on the STM32 as there is a lot info that I can youtube to learn from.

I am astonished by how far you've come. Really inspiring!

I have been doing just that here. This is far from finished, is subject to change, and is not yet properly documented...

Of course you have.

The idea is to take elements from my GUI for the official display and port them to displays whose drivers are subclassed from framebuf. Since these don't have touch overlays there is no point in supporting data entry widgets.

This looks very much like what I had envisaged.

Though I still think data entry widgets should be supported even without a touch interface - they could be set through other means such as buttons (up/down to slide values) or even by a remote web interface.

It's also worth bearing in mind that only small displays are suitable for this treatment owing to the RAM usage of the framebuf instance. Consequently the widgets have needed some redesign for small displays.

I agree that memory use is not insignificant with this design. Even with only 8bit colour (the ILI9341 supports 24bit) the 320x240 pixel framebuf consumes 75kB which only leaves 27kB free on my stock ESP32. On a PSRAM model we'll be in better shape...

We could make some optimisations here; If we separated the driver from the framebuf - and allow the user to pass in a framebuf with a starting row when blitting - we could save some memory. This could be used, for example, to render more static parts of the display - headings or icons that are infrequently updated. Maybe?

However it does enable extremely simple display drivers to be written: see this example. With that trivial driver you get graphics primitives via framebuf, colour text rendering in multiple fonts via my CWriter class, and (forthcoming) scalable GUI widgets via nano-gui. I plan to write similar drivers for other small colour OLED displays.

I can see how to do it for the ILI9341, I'll give it a shot...

The Label, Meter and LED classes have been implemented and tested on SSD1331 colour OLED and SSD1306 monochrome OLED displays using the official driver.

This is all great, I think this will work for my use! Thanks Peter!

[mattyt]

With such a large display, you are likely to have problems with memory if you use the naive approach of just drawing your ui on the display. However, if you can redesign your UI so that it can be represented as a tiled grid of similar images and some sprites, you could make use of my "stage" library: https://hackaday.io/project/80626

For sure, memory use is a key problem to a large single framebuf. But I think only a subset of problems map well to grids.

There is some work underway to use a similar approach for displaying graphics in CircuitPython done by Adafruit. I think the general idea of using tiles and sprites to save memory and help you organize your interface is very promising.

I agree and look forward to what you and Adafruit come up with!

But in my case - where I want to display a full-screen chart or present a large number of options for a user to scroll through - I'm not sure the savings in memory would be worth the complexity in managing the state.

A half-way model may be to use a fraction of a full framebuf - make it only 1/x the height of the display. Then render and transfer it in x passes... Alas the widgets will need extra complexity to render at offsets.