How to reduce the memory consumption of the object definition

[MCHobby]

Hi everybody,

I have the following object definition (Arial_14 stored inside arial_14.py) which consume 8Kb of ram ( calculated with gc.free_mem() & gc.alloc_mem() ) when I do import it.
So when I do "from arial_14 import Arial_14" then 8 Kb of RAM are used.
Since Arial_14 represent "static data", I can use a decoding algorithm to access the information, that would allow me to reduce the memory usage/consumption.

I would like to know if some of you could have good suggestion to reduce the memory consumption of the object (compressing or hack or ??).
Any suggestion would be welcome.

Cheers,
Dominique

Code: Select all

Arial_14 = {
'width' :0x0D,
'height':0x0E,
32:(0x4000,),
33:(0x4BFC,),
34:(0x403c,0x4000,0x403c),
35:(0x4120,0x4f20,0x41f0,0x412c,0x4f20,0x41f0,0x412c,0x4120),
36:(0x4430,0x4848,0x4844,0x5ffe,0x4884,0x4484,0x4308),
37:(0x4038,0x4044,0x4844,0x4638,0x4180,0x4060,0x4718,0x4884,0x4880,0x4700),
38:(0x4700,0x48b8,0x4844,0x48c4,0x4924,0x4618,0x4500,0x4800),
39:(0x403c,),
40:(0x4ff8,0x580c,0x6002),
41:(0x6002,0x580c,0x4ff8),
42:(0x4008,0x4028,0x401c,0x4028,0x4008),
43:(0x4080,0x4080,0x4080,0x47f0,0x4080,0x4080,0x4080),
44:(0x7800,),
45:(0x4100,0x4100,0x4100,0x4100),
46:(0x4800,),
47:(0x4c00,0x4380,0x4070,0x400c),
48:(0x47f8,0x4804,0x4804,0x4804,0x4804,0x47f8),
49:(0x4010,0x4008,0x4ffc),
50:(0x4818,0x4c04,0x4a04,0x4904,0x4884,0x4878),
51:(0x4618,0x4804,0x4844,0x4844,0x4844,0x47b8),
52:(0x4300,0x4280,0x4260,0x4210,0x4208,0x4ffc,0x4200),
53:(0x4670,0x482c,0x4824,0x4824,0x4c24,0x43c4),
54:(0x47f0,0x4888,0x4844,0x4844,0x4844,0x4788),
55:(0x4004,0x4004,0x4e04,0x41c4,0x4034,0x400c),
56:(0x47b8,0x4844,0x4844,0x4844,0x4844,0x47b8),
57:(0x4478,0x4884,0x4884,0x4884,0x4444,0x43f8),
58:(0x4810,),
59:(0x7810,),
60:(0x4080,0x4140,0x4140,0x4220,0x4220,0x4410),
61:(0x4120,0x4120,0x4120,0x4120,0x4120,0x4120),
62:(0x4410,0x4220,0x4220,0x4140,0x4140,0x4080),
63:(0x4018,0x4004,0x4b04,0x4084,0x4044,0x4038),
64:(0x47c0,0x5830,0x6008,0x67c4,0x6422,0x6412,0x6412,0x6212,0x67a2,0x6472,0x6804,0x6618,0x51e0),
65:(0x4c00,0x4300,0x41c0,0x4138,0x4104,0x4138,0x41c0,0x4300,0x4c00),
66:(0x4ffc,0x4844,0x4844,0x4844,0x4844,0x4844,0x47b8),
67:(0x43f0,0x4408,0x4804,0x4804,0x4804,0x4804,0x4408,0x4210),
68:(0x4ffc,0x4804,0x4804,0x4804,0x4804,0x4804,0x4408,0x43f0),
69:(0x4ffc,0x4844,0x4844,0x4844,0x4844,0x4844,0x4844),
70:(0x4ffc,0x4044,0x4044,0x4044,0x4044,0x4044,0x4004),
71:(0x43f0,0x4408,0x4804,0x4804,0x4804,0x4884,0x4884,0x4488,0x4390),
72:(0x4ffc,0x4040,0x4040,0x4040,0x4040,0x4040,0x4ffc),
73:(0x4ffc,),
74:(0x4600,0x4800,0x4800,0x4800,0x47fc),
75:(0x4ffc,0x4100,0x4080,0x4040,0x40a0,0x4310,0x4408,0x4804),
76:(0x4ffc,0x4800,0x4800,0x4800,0x4800,0x4800,0x4800),
77:(0x4ffc,0x4018,0x4060,0x4380,0x4c00,0x4380,0x4060,0x4018,0x4ffc),
78:(0x4ffc,0x4008,0x4030,0x40c0,0x4300,0x4400,0x4ffc),
79:(0x43f0,0x4408,0x4804,0x4804,0x4804,0x4804,0x4804,0x4408,0x43f0),
80:(0x4ffc,0x4084,0x4084,0x4084,0x4084,0x4084,0x4078),
81:(0x43f0,0x4408,0x4804,0x4804,0x4804,0x4a04,0x4404,0x4e08,0x49f0),
82:(0x4ffc,0x4084,0x4084,0x4084,0x4184,0x4284,0x4484,0x4878),
83:(0x4638,0x4844,0x4844,0x4844,0x4884,0x4884,0x4718),
84:(0x4004,0x4004,0x4004,0x4ffc,0x4004,0x4004,0x4004),
85:(0x43fc,0x4400,0x4800,0x4800,0x4800,0x4400,0x43fc),
86:(0x400c,0x4030,0x40c0,0x4300,0x4c00,0x4300,0x40c0,0x4030,0x400c),
87:(0x400c,0x4070,0x4380,0x4c00,0x4380,0x4078,0x4004,0x4078,0x4380,0x4c00,0x4380,0x4070,0x400c),
88:(0x4804,0x4618,0x4120,0x40c0,0x40c0,0x4320,0x4618,0x4804),
89:(0x4004,0x4008,0x4030,0x4040,0x4f80,0x4040,0x4030,0x4008,0x4004),
90:(0x4800,0x4c04,0x4b04,0x4884,0x4844,0x4834,0x480c,0x4804),
91:(0x7ffe,0x6002),
92:(0x400c,0x4070,0x4380,0x4c00),
93:(0x6002,0x7ffe),
94:(0x4040,0x4038,0x4004,0x4038,0x4040),
95:(0x6000,0x6000,0x6000,0x6000,0x6000,0x6000,0x6000,0x6000),
96:(0x4004,0x4008),
97:(0x4620,0x4910,0x4890,0x4890,0x4490,0x4fe0),
98:(0x4ffc,0x4420,0x4810,0x4810,0x4810,0x47e0),
99:(0x47e0,0x4810,0x4810,0x4810,0x4420),
100:(0x47e0,0x4810,0x4810,0x4810,0x4420,0x4ffc),
101:(0x47e0,0x4890,0x4890,0x4890,0x4890,0x44e0),
102:(0x4010,0x4ff8,0x4014,0x4014),
103:(0x63f0,0x6408,0x6408,0x6408,0x6210,0x7ff8),
104:(0x4ffc,0x4020,0x4010,0x4010,0x4010,0x4fe0),
105:(0x4ff4,),
106:(0x6000,0x7ff4),
107:(0x4ffc,0x4100,0x4080,0x4140,0x4620,0x4810),
108:(0x4ffc,),
109:(0x4ff0,0x4020,0x4010,0x4010,0x4fe0,0x4020,0x4010,0x4010,0x4fe0),
110:(0x4ff0,0x4020,0x4010,0x4010,0x4010,0x4fe0),
111:(0x47e0,0x4810,0x4810,0x4810,0x4810,0x47e0),
112:(0x7ff8,0x4420,0x4810,0x4810,0x4810,0x47e0),
113:(0x47e0,0x4810,0x4810,0x4810,0x4420,0x7ff8),
114:(0x4ff0,0x4020,0x4010,0x4010),
115:(0x4460,0x4890,0x4890,0x4890,0x4720),
116:(0x4010,0x4ffc,0x4810,0x4810),
117:(0x47f0,0x4800,0x4800,0x4800,0x4400,0x4ff0),
118:(0x4030,0x40c0,0x4300,0x4c00,0x4300,0x40c0,0x4030),
119:(0x4030,0x43c0,0x4c00,0x43c0,0x4030,0x43c0,0x4c00,0x43c0,0x4030),
120:(0x4810,0x4660,0x4180,0x4180,0x4660,0x4810),
121:(0x4030,0x6060,0x6380,0x7c00,0x4300,0x40c0,0x4030),
122:(0x4810,0x4c10,0x4b10,0x48d0,0x4830,0x4810),
123:(0x4080,0x5f7c,0x6002),
124:(0x7ffe,),
125:(0x6002,0x5f7c,0x4080),
126:(0x4080,0x4040,0x4040,0x40c0,0x4080,0x4080,0x4040),
127:(0x4ff8,0x4808,0x4808,0x4808,0x4808,0x4808,0x4ff8)
}

[chuckbook]

This is obviously a font description (at least 96 printable chars).
If the char matrix is 14x13 ~2184 bytes are needed to encode a full pattern font in a simple array.
A directory encoded array of 16 bit words will need 650 values.

[pythoncoder]

A solution is in the pipeline: persistent bytecode. This will enable you to create a Python module which declares (say) bytes objects containing constant data. The Python code is cross-compiled to Python bytecode which can then be compiled into a firmware build. The data is stored in Flash memory as part of the firmware image and can be accessed like any other module, but with minimal RAM usage. I have tested this and it works, but to date persistent bytecode hasn't made its way into the official release https://github.com/micropython/micropython/pull/1811.

[PinkInk]

That will be an interesting/useful feature. I'm working on a micro SNMP library and have realised that I can't really achieve everything I want to without representing at least mib-2 as a tree.

I'm working on the assumption that nested python objects will be unwieldy for something that'd only ever be used as part of something running on an embedded device, so am considering packing the structure into a bytearray wrapped in a class to parse it.

Being able to move at least the bytearray into flash would be a nice benefit.

[pythoncoder]

Bear in mind that data stored in Flash is immutable. To avoid RAM use you'll need a bytes object rather than a bytearray. If you declare mutable variables, compile the module, and store it as persistent bytecode, at runtime the variables will presumably be put in RAM. I've only tested immutable data so I'm guessing here, but it strikes me that must work this way because of the physical limitations of the hardware.

[MCHobby]

On Arduino, we do have a way to grab back data from the Flash memory. This indeed requires to compile the Arduino sketck + upload to make the data available in the Flash (but this is a quite easy and short process)
I would be nice to have a way to store some data into the Flash without recompiling the PyBoard Firmware and upload a new Firmware on the board.

I would be nice to being able to load some file (with immutable data, or python precompiled byte code) stored in the Flash Drive (or SDcard) into the Python Flash memory... so it would be made available to the MicroPython interpreter without using the whole compilation chain and PyBoard flashing.

I do understand that dealing with the Pyboard's allocated Flash could be quite tricky... but this would definitively be a great enhancement for storing static data like astronomic data, font definition (my use case), etc

[MCHobby]

Bear in mind that data stored in Flash is immutable. To avoid RAM use you'll need a bytes object rather than a bytearray. If you declare mutable variables, compile the module, and store it as persistent bytecode, at runtime the variables will presumably be put in RAM. I've only tested immutable data so I'm guessing here, but it strikes me that must work this way because of the physical limitations of the hardware.

Do you mean that when defining my structure into a bytes object in my Python code, I would spare RAM?

[pythoncoder]

@MCHobby persistent bytecode is now mainstream so trying it is straightforward http://forum.micropython.org/viewtopic.php?f=6&t=1776.

I use it for font storage. I have a Python module which declares large bytes objects. Compiled as persistent bytecode I can import it and access the bytes objects, yet its RAM usage is minimal because the data is physically located in Flash.

[MCHobby]

So if I do understand properly, there is not other way than compiling a tuned "uPy firmware with Persistant Byte code" to reduce RAM consumption.
It is a nice feature but probably over the knowledge of most of the makers.

Is there other way (good practice) to reduce the memory consumption?
Some thread reading/articles to recommend?

[pythoncoder]

I've used three techniques with fonts.

• Persistent bytecode.
• With proportional fonts don't store trailing whitespace. Use an index table to determine the address of the character data.
• Use a random access file on disk, accessing data one character at a time. Slow.

I have my doubts about the suitability of compression for this type of data, but if anyone knows better or has other ideas I'd be glad to hear them.