Suggested standard approach to font handling

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Suggested standard approach to font handling

Post by pythoncoder » Thu Nov 17, 2016 4:58 pm

This has been referenced on the Github discussion on the framebuffer class. For those not following this I've written a proposal for a hopefully standard way of handling font files. This comprises a Python3 utility for converting industry standard font files to Python sourcecode. The latter may be imported in the normal way or frozen as bytecode to conserve RAM.

The Python font files may be used with a variety of device drivers. It has been tested with drivers using the framebuffer class, ones with buffers based on bytearray objects, and one for a display device having a hardware frame buffer. It is documented here https://github.com/peterhinch/micropyth ... -to-py.git. It supports vertical and horizontal mapping and bit reversal.

Test code is available for the framebuffer based SSD1306 https://github.com/peterhinch/micropyth ... er/SSD1306 and also for the SSD1963 based TFT GUI https://github.com/peterhinch/micropyth ... ter/tft.py. The latter large display has a hardware frame buffer.
Peter Hinch
Index to my micropython libraries.

bitninja
Posts: 165
Joined: Thu Sep 15, 2016 4:09 pm
Location: Spring, Texas

Re: Suggested standard approach to font handling

Post by bitninja » Thu Nov 17, 2016 6:47 pm

I've look at your project and have successfully run it on my ESP8266 devices. I think you did a good job with saving memory by encoding the font files as frozen byte code.

My question is... can the same techniques be used to implement bitmap graphics for the display. I have a current system that reads .txt files on the file system that encode the images in HEX. It is one image per TXT file.

What I would like is the images coded as frozen byte code into a larger image, that contains individual images. With an index to find an individual image in the larger one. Similar to how characters are indexed within the font data.

What do you think?

User avatar
wminarik
Posts: 7
Joined: Thu Oct 27, 2016 12:03 am
Location: Montréal, Canada

Re: Suggested standard approach to font handling

Post by wminarik » Fri Nov 18, 2016 3:20 am

How different is your Python font file format from the format used by Teensy & Adafruit for the ILI9341 font files (in C), if this is a quick question to address? The reason to ask is that there may be fonts already optimized for small screen readability in bit-mapped format, that might just need the wrapper changed...

https://github.com/PaulStoffregen/ILI9341_t3
https://github.com/PaulStoffregen/ILI9341_fonts

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Suggested standard approach to font handling

Post by pythoncoder » Fri Nov 18, 2016 7:23 am

It would be straightforward to write a Python utility to convert a C font file to my Python source format. In fact this was my original approach (albeit with different file formats) when I wrote the e-paper driver and collaborated with @Roberthh on the touch GUI: we used a piece of proprietary software to convert a font to C source, and wrote Python utilities to convert the C to Python. There is nothing to stop the writer of a driver for a specific piece of hardware adopting my file format, using font_to_py.py for large fonts and writing such a converter for the small binary fonts.

However the general problem is a bit more involved. In discussions on the framebuffer module it was agreed that it should support horizontal and vertical mapping. This could be done with suitable bit twiddling, as could bit reversal. But there is no standard format for C font files: some have an index, some don't and so on. So I fear an attempt at a general solution would constantly face demands to accommodate another, different, set of C files.

I'll think some more about this, but it may be that ad hoc solutions for specific sets of files are the best way forward.

Another concern is copyright. font_to_py.py is merely a file format converter. IANAL but it's surely up to the user to ensure that the font files they convert are licensed for their intended use or distribution. If you or I wrote a utility for a specific set of files, would we run into potential legal issues?
Peter Hinch
Index to my micropython libraries.

v923z
Posts: 168
Joined: Mon Dec 28, 2015 6:19 pm

Re: Suggested standard approach to font handling

Post by v923z » Mon Jan 02, 2017 1:02 pm

I was wondering, how much memory overhead a dictionary adds. Inspired by Peter's font conversion tool (somehow, I couldn't get freetype working under anaconda, so I have to make a trip to a command line tool), I saved the font maps in a dictionary. The details can be found under https://github.com/v923z/micropython/bl ... onts.ipynb. The advantage of this approach is that one can work with a non-contiguous set of characters, and that the dictionary is human-readable, while the bytearray is not. However, if the dictionary adds too much to the memory footprint, then this is of no use. Is there a way to compare the memory consumption of the two methods?

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Suggested standard approach to font handling

Post by dhylands » Mon Jan 02, 2017 5:11 pm

I've found that the simplest way is to write some code something like this:

Code: Select all

import gc
import micropython

def foo(i):
    gc.collect()
    a = gc.mem_free()
    b = bytearray(i)
    c = gc.mem_free()
    print('bytearray({}) took {} bytes'.format(i, a - c))

for i in range(64):
    foo(i)
Note that doing the same thing on the REPL won't give the same results (since when you're using the REPL, memory gets allocated to store historical command lines, etc), so using an actual function seems to give the best results.

v923z
Posts: 168
Joined: Mon Dec 28, 2015 6:19 pm

Re: Suggested standard approach to font handling

Post by v923z » Mon Jan 02, 2017 6:51 pm

Thanks Dave, this is a useful piece of code.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Suggested standard approach to font handling

Post by pythoncoder » Tue Jan 03, 2017 9:05 am

v923z wrote:I was wondering, how much memory overhead a dictionary adds...
That's an interesting technique: I didn't realise convert could do that. How are you dealing with the issue of alignment?

My code aims to produce, given a nominal font size, a set of bitmaps with the vertical alignment pre-computed. This is for performance reasons: the aim is that rendering should be fast and simple. The alternative approach is to store the metrics with each glyph and figure out the positioning at render time. While this is probably OK for small displays, it may be too slow for large ones capable of displaying substantial amounts of text.

In order to pre-compute the alignment of each glyph I found it necessary to iterate through the entire character set to establish the metrics of the given font size before doing another pass to create the bitmaps. (There is actually another issue which can require more than one iteration but I won't complicate the discussion with that).

While I appreciate the flexibility of using a dict, again there may be a performance issue: a dict lookup is likely to be slower than handling two levels of indirection, although I haven't tested this.

As for RAM use, this can be fixed with frozen bytecode. My solution produces bytes objects which are immutable. This means that they can be accessed in place, consuming almost no RAM. A dict is mutable, so even if you freeze the bytecode, the runtime will pull the dict into RAM (because Python allows you to change its contents).
Peter Hinch
Index to my micropython libraries.

User avatar
deshipu
Posts: 1388
Joined: Thu May 28, 2015 5:54 pm

Re: Suggested standard approach to font handling

Post by deshipu » Tue Jan 03, 2017 1:07 pm

There are frozendicts out there.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Suggested standard approach to font handling

Post by pythoncoder » Tue Jan 03, 2017 1:25 pm

deshipu wrote:There are frozendicts out there.
To be of use in this context it would need to be implemented in a way that ensured that the MicroPython runtime didn't pull the dict into RAM. As far as I know this hasn't been done. As I understand it even tuples get pulled into RAM, bytes instances being the only way to store data in flash.
Peter Hinch
Index to my micropython libraries.

Post Reply