Suggested standard approach to font handling

C programming, build, interpreter/VM.
Target audience: MicroPython Developers.
User avatar
dhylands
Posts: 3212
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Suggested standard approach to font handling

Post by dhylands » Tue Jan 03, 2017 4:00 pm

I'm pretty sure that the Pin dicts and arrays are not pulled into RAM. This would require that the dicts and arrays be created statically using C declarations rather than code (since anything created with python code will necessarily have to be from RAM).

v923z
Posts: 69
Joined: Mon Dec 28, 2015 6:19 pm

Re: Suggested standard approach to font handling

Post by v923z » Tue Jan 03, 2017 9:33 pm

pythoncoder wrote:
v923z wrote:I was wondering, how much memory overhead a dictionary adds...
That's an interesting technique: I didn't realise convert could do that. How are you dealing with the issue of alignment?
I am not sure I see what you mean: if it is the vertical alignment of fonts (i.e., the baseline of characters), then the bitmap produced by convert is automatically aligned.
pythoncoder wrote: My code aims to produce, given a nominal font size, a set of bitmaps with the vertical alignment pre-computed. This is for performance reasons: the aim is that rendering should be fast and simple. The alternative approach is to store the metrics with each glyph and figure out the positioning at render time. While this is probably OK for small displays, it may be too slow for large ones capable of displaying substantial amounts of text.
I believe, what I have is pretty similar to yours: each character is in a field of fixed height, and its position inside that field is stored in the bytearray itself, so no extra computation is required at run time. I don't think it would make too much sense to chop off the white spaces below and above the characters, and store the size of that as extra. If you look at the output of draw_string in the notebook, you can see that the characters are properly aligned.
pythoncoder wrote:While I appreciate the flexibility of using a dict, again there may be a performance issue: a dict lookup is likely to be slower than handling two levels of indirection, although I haven't tested this.
This is absolutely true, though I would mention that the rendering itself is orders of magnitude slower than the lookup, so it doesn't really matter.
pythoncoder wrote:As for RAM use, this can be fixed with frozen bytecode. My solution produces bytes objects which are immutable. This means that they can be accessed in place, consuming almost no RAM. A dict is mutable, so even if you freeze the bytecode, the runtime will pull the dict into RAM (because Python allows you to change its contents).
I have figured that the dictionary is rather expensive. If I take 32-pt FreeSans fonts for 80 standard characters, I consume something like 30 kB of RAM, while the actual data is only 7.3kB, and the frozen bytecode takes up something like 20 kB.
One has to add, though, that here I am talking about the costs of carrying the whole class, which contains the dictionary, plus the height function. However, the class can't probably be so expensive. I don't think that there is too much difference between

Code: Select all

def height():
    return 32
    
font = {'1': b'\x00....'}
and

Code: Select all

class Font(object):
    def __init__(self):
        self.font = {'1': b'\x00....'}

    def height():
        return 32
while the first form becomes somewhat problematic, if one has to use two kinds of fonts at the same time: you'll never know whose height the function height() returns...

User avatar
pythoncoder
Posts: 3538
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Suggested standard approach to font handling

Post by pythoncoder » Wed Jan 04, 2017 10:06 am

v923z wrote:...if it is the vertical alignment of fonts (i.e., the baseline of characters), then the bitmap produced by convert is automatically aligned.
Apologies, I hadn't appreciated that. In any event, the issue of how you get from a font to a bitmap is secondary. The important issue as far as I'm concerned is how you present the data on a device with limited resources.
v923z wrote:I believe, what I have is pretty similar to yours: each character is in a field of fixed height, and its position inside that field is stored in the bytearray itself, so no extra computation is required at run time. I don't think it would make too much sense to chop off the white spaces below and above the characters, and store the size of that as extra. If you look at the output of draw_string in the notebook, you can see that the characters are properly aligned.
Agreed.
v923z wrote:This is absolutely true, though I would mention that the rendering itself is orders of magnitude slower than the lookup, so it doesn't really matter.
Agreed.
v923z wrote:I have figured that the dictionary is rather expensive. If I take 32-pt FreeSans fonts for 80 standard characters, I consume something like 30 kB of RAM, while the actual data is only 7.3kB, and the frozen bytecode takes up something like 20 kB.
One has to add, though, that here I am talking about the costs of carrying the whole class, which contains the dictionary, plus the height function. However, the class can't probably be so expensive. I don't think that there is too much difference between

Code: Select all

def height():
    return 32
    
font = {'1': b'\x00....'}
and

Code: Select all

class Font(object):
    def __init__(self):
        self.font = {'1': b'\x00....'}

    def height():
        return 32
while the first form becomes somewhat problematic, if one has to use two kinds of fonts at the same time: you'll never know whose height the function height() returns...
The first form works if you store one font per module. Python modules are in many ways similar to classes. Assume font1.py and font2.py use my approach. The following will work:

Code: Select all

import font1, font2
h1 = font1.height()
h2= font2.height()
The key technical problem is this. Bitmapped fonts comprise substantial amounts of constant data. MicroPython targets such as the ESP8266 have very limited amounts of free RAM. On such hardware this data is best stored in Flash, so a solution aiming to be a standard should offer this as an option.

MicroPython offers just one* efficient* way to do this: as precompiled Python source files containing bytes objects stored as frozen bytecode. The incremental cost of importing a font stored in that way is (from memory) about 150 bytes.

*As I understand it.
* You can store them in a random access file but glyph-wise retrieval is slow.
Peter Hinch

v923z
Posts: 69
Joined: Mon Dec 28, 2015 6:19 pm

Re: Suggested standard approach to font handling

Post by v923z » Wed Jan 04, 2017 3:39 pm

pythoncoder wrote:Python modules are in many ways similar to classes. Assume font1.py and font2.py use my approach. The following will work:

Code: Select all

import font1, font2
h1 = font1.height()
h2= font2.height()
Indeed. I totally overlooked this. :oops:
pythoncoder wrote: The key technical problem is this. Bitmapped fonts comprise substantial amounts of constant data. MicroPython targets such as the ESP8266 have very limited amounts of free RAM. On such hardware this data is best stored in Flash, so a solution aiming to be a standard should offer this as an option.

MicroPython offers just one* efficient* way to do this: as precompiled Python source files containing bytes objects stored as frozen bytecode. The incremental cost of importing a font stored in that way is (from memory) about 150 bytes.
Well, you have pretty much convinced me. As I said above, according to a couple of tests that I ran here, loading the font by means of a dictionary is insanely expensive.

User avatar
pythoncoder
Posts: 3538
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

font_to_py.py now supports extended ASCII

Post by pythoncoder » Fri Jan 13, 2017 7:54 am

In response to a suggestion from @Roberthh and comments here and on GitHub I've updated this utility. It now offers command line arguments to define the range of characters to include in a font file. It defaults to standard ASCII (32-126) but you can specify an arbitrary set of consecutive characters in the range 0-255. You can also specify the character to be output if the user attempts to display a character not in the set (default "?"). Docs here https://github.com/peterhinch/micropyth ... -to-py.git.

Since the Python font module includes the code to retrieve the glyph, no changes to application code are required.

The module has two new functions enabling the character range to be retrieved.
Peter Hinch

newb
Posts: 28
Joined: Wed Jan 10, 2018 8:19 pm
Location: Bulgaria

Re: Suggested standard approach to font handling

Post by newb » Mon Aug 27, 2018 7:33 pm

Is there any chance you can add an option to micropython-font-to-py to select charset from utf-8 or another charset ascii codepage (eg 855)?
I need to use Cyrillic fonts and I'm stuck with "????"s :)

User avatar
pythoncoder
Posts: 3538
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Alternative charsets. Limited charsets.

Post by pythoncoder » Tue Aug 28, 2018 6:55 am

It's a good idea but alas I lack expertise on foreign language support. The utility uses freetype which is quite a complex Python library. If you, or anyone else, has some knowledge of how to use it in this way I'd be glad to collaborate.

I guess to even make a start, I'd need to find a ttf or otf file which contained glyphs for the Cyrillic character set.

On another topic, font_to_py.py now supports creating Python font files having a limited set of characters. This limits the size of output files where you want only a few characters in a large size. For example, for a digital clock you might use the option:
-c 0123456789:
Peter Hinch

User avatar
pythoncoder
Posts: 3538
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Solution?

Post by pythoncoder » Tue Aug 28, 2018 9:10 am

I have had some success with this, but I have no confidence that I understand how this actually works ;)

I created a font file with

Code: Select all

[adminpete@axolotl]: /mnt/qnap2/data/Projects/MicroPython/micropython-font-to-py
$ font_to_py.py FreeSans.ttf 20 test.py -c ЀЁЂЃЄ
Used the following code to check rendering

Code: Select all

import font_test
font_test.test_font('test', 'ЀЁЂ')
with this outcome

Code: Select all

>>> import rats6
.............##..##....................
...##........##..##....................
....##.......##..##....................
.......................................
#########...#########...########.......
#########...#########...########.......
##..........##.............##..........
##..........##.............##..........
##..........##.............##..........
##..........##.............#########...
#########...#########......##########..
#########...#########......##......##..
##..........##.............##......##..
##..........##.............##......##..
##..........##.............##......##..
##..........##.............##......##..
#########...#########......##......##..
#########...#########......##......##..
...................................##..
.................................####..
.................................###...
>>> 
[EDIT]
I've posted an update which accepts an optional arg -k filespec or --charset_file filespec. This allows it to read the character set from a file. The repo now has a file 'cyrillic_subset` and the command

Code: Select all

font_to_py.py FreeSans.ttf 20 test.py -k cyrillic_subset
seems to do the job.

I'd be grateful if you could test this with a file containing the complete character set, and report the results.
Peter Hinch

newb
Posts: 28
Joined: Wed Jan 10, 2018 8:19 pm
Location: Bulgaria

Re: Suggested standard approach to font handling

Post by newb » Tue Aug 28, 2018 2:42 pm

Cool! I'll test it in the next few days and post the result here.
Thanks for the prompt reaction on this. I was playing with an OLED display connected to Nodemcu and I wanted to output messages in my own language :)

newb
Posts: 28
Joined: Wed Jan 10, 2018 8:19 pm
Location: Bulgaria

Re: Suggested standard approach to font handling

Post by newb » Tue Aug 28, 2018 4:17 pm

So it seems that you have made quite few changes since I used the Writer class :) I used your example code fro writer class from the git repo. I've changed the pin setup in ssd1306_setup.py to match my Pins setup. Have you tested the code on esp8266? I'm getting this error:

Traceback (most recent call last):
File "<stdin>", line 3, in <module>
MemoryError: memory allocation failed, allocating 3573 bytes

where line 3 is the import of the font file. This fails also with the default freesans20 font.
Does it has anything to do with the 1mb rom on esp8266?

The old version (2-3 months ago) of the writer class worked ok for me.

[EDIT] Got it working with writer_minimal.py. Obviously a memory issue :D I had to add space and return in the char subset file as no spaces were showing. See pictures below.

I also edited your cyrillic_subset file as it lacked half of the alphabet :) It includes the full Cyrillic alphabet (Russian and Bulgarian glyphs, numbers and basic punctuation) Here it is:

АБВГДЕЁЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯабвгдеёжзийклмнопрстуфхцчшщъыьэюя1234567890.?!:-\


IMG_2770.JPG
No spaces
IMG_2770.JPG (65.04 KiB) Viewed 957 times
IMG_2772.JPG
Added spaces
IMG_2772.JPG (57.88 KiB) Viewed 957 times

Post Reply