MicroPython Forum (Archive)

Posted: **Sun Apr 10, 2016 5:08 pm**

Hi everybody,

I have the following object definition (Arial_14 stored inside arial_14.py) which consume 8Kb of ram ( calculated with gc.free_mem() & gc.alloc_mem() ) when I do import it.
So when I do "from arial_14 import Arial_14" then 8 Kb of RAM are used.
Since Arial_14 represent "static data", I can use a decoding algorithm to access the information, that would allow me to reduce the memory usage/consumption.

I would like to know if some of you could have good suggestion to reduce the memory consumption of the object (compressing or hack or ??).
Any suggestion would be welcome.

Cheers,
Dominique

Code: Select all

Arial_14 = {
'width' :0x0D,
'height':0x0E,
32:(0x4000,),
33:(0x4BFC,),
34:(0x403c,0x4000,0x403c),
35:(0x4120,0x4f20,0x41f0,0x412c,0x4f20,0x41f0,0x412c,0x4120),
36:(0x4430,0x4848,0x4844,0x5ffe,0x4884,0x4484,0x4308),
37:(0x4038,0x4044,0x4844,0x4638,0x4180,0x4060,0x4718,0x4884,0x4880,0x4700),
38:(0x4700,0x48b8,0x4844,0x48c4,0x4924,0x4618,0x4500,0x4800),
39:(0x403c,),
40:(0x4ff8,0x580c,0x6002),
41:(0x6002,0x580c,0x4ff8),
42:(0x4008,0x4028,0x401c,0x4028,0x4008),
43:(0x4080,0x4080,0x4080,0x47f0,0x4080,0x4080,0x4080),
44:(0x7800,),
45:(0x4100,0x4100,0x4100,0x4100),
46:(0x4800,),
47:(0x4c00,0x4380,0x4070,0x400c),
48:(0x47f8,0x4804,0x4804,0x4804,0x4804,0x47f8),
49:(0x4010,0x4008,0x4ffc),
50:(0x4818,0x4c04,0x4a04,0x4904,0x4884,0x4878),
51:(0x4618,0x4804,0x4844,0x4844,0x4844,0x47b8),
52:(0x4300,0x4280,0x4260,0x4210,0x4208,0x4ffc,0x4200),
53:(0x4670,0x482c,0x4824,0x4824,0x4c24,0x43c4),
54:(0x47f0,0x4888,0x4844,0x4844,0x4844,0x4788),
55:(0x4004,0x4004,0x4e04,0x41c4,0x4034,0x400c),
56:(0x47b8,0x4844,0x4844,0x4844,0x4844,0x47b8),
57:(0x4478,0x4884,0x4884,0x4884,0x4444,0x43f8),
58:(0x4810,),
59:(0x7810,),
60:(0x4080,0x4140,0x4140,0x4220,0x4220,0x4410),
61:(0x4120,0x4120,0x4120,0x4120,0x4120,0x4120),
62:(0x4410,0x4220,0x4220,0x4140,0x4140,0x4080),
63:(0x4018,0x4004,0x4b04,0x4084,0x4044,0x4038),
64:(0x47c0,0x5830,0x6008,0x67c4,0x6422,0x6412,0x6412,0x6212,0x67a2,0x6472,0x6804,0x6618,0x51e0),
65:(0x4c00,0x4300,0x41c0,0x4138,0x4104,0x4138,0x41c0,0x4300,0x4c00),
66:(0x4ffc,0x4844,0x4844,0x4844,0x4844,0x4844,0x47b8),
67:(0x43f0,0x4408,0x4804,0x4804,0x4804,0x4804,0x4408,0x4210),
68:(0x4ffc,0x4804,0x4804,0x4804,0x4804,0x4804,0x4408,0x43f0),
69:(0x4ffc,0x4844,0x4844,0x4844,0x4844,0x4844,0x4844),
70:(0x4ffc,0x4044,0x4044,0x4044,0x4044,0x4044,0x4004),
71:(0x43f0,0x4408,0x4804,0x4804,0x4804,0x4884,0x4884,0x4488,0x4390),
72:(0x4ffc,0x4040,0x4040,0x4040,0x4040,0x4040,0x4ffc),
73:(0x4ffc,),
74:(0x4600,0x4800,0x4800,0x4800,0x47fc),
75:(0x4ffc,0x4100,0x4080,0x4040,0x40a0,0x4310,0x4408,0x4804),
76:(0x4ffc,0x4800,0x4800,0x4800,0x4800,0x4800,0x4800),
77:(0x4ffc,0x4018,0x4060,0x4380,0x4c00,0x4380,0x4060,0x4018,0x4ffc),
78:(0x4ffc,0x4008,0x4030,0x40c0,0x4300,0x4400,0x4ffc),
79:(0x43f0,0x4408,0x4804,0x4804,0x4804,0x4804,0x4804,0x4408,0x43f0),
80:(0x4ffc,0x4084,0x4084,0x4084,0x4084,0x4084,0x4078),
81:(0x43f0,0x4408,0x4804,0x4804,0x4804,0x4a04,0x4404,0x4e08,0x49f0),
82:(0x4ffc,0x4084,0x4084,0x4084,0x4184,0x4284,0x4484,0x4878),
83:(0x4638,0x4844,0x4844,0x4844,0x4884,0x4884,0x4718),
84:(0x4004,0x4004,0x4004,0x4ffc,0x4004,0x4004,0x4004),
85:(0x43fc,0x4400,0x4800,0x4800,0x4800,0x4400,0x43fc),
86:(0x400c,0x4030,0x40c0,0x4300,0x4c00,0x4300,0x40c0,0x4030,0x400c),
87:(0x400c,0x4070,0x4380,0x4c00,0x4380,0x4078,0x4004,0x4078,0x4380,0x4c00,0x4380,0x4070,0x400c),
88:(0x4804,0x4618,0x4120,0x40c0,0x40c0,0x4320,0x4618,0x4804),
89:(0x4004,0x4008,0x4030,0x4040,0x4f80,0x4040,0x4030,0x4008,0x4004),
90:(0x4800,0x4c04,0x4b04,0x4884,0x4844,0x4834,0x480c,0x4804),
91:(0x7ffe,0x6002),
92:(0x400c,0x4070,0x4380,0x4c00),
93:(0x6002,0x7ffe),
94:(0x4040,0x4038,0x4004,0x4038,0x4040),
95:(0x6000,0x6000,0x6000,0x6000,0x6000,0x6000,0x6000,0x6000),
96:(0x4004,0x4008),
97:(0x4620,0x4910,0x4890,0x4890,0x4490,0x4fe0),
98:(0x4ffc,0x4420,0x4810,0x4810,0x4810,0x47e0),
99:(0x47e0,0x4810,0x4810,0x4810,0x4420),
100:(0x47e0,0x4810,0x4810,0x4810,0x4420,0x4ffc),
101:(0x47e0,0x4890,0x4890,0x4890,0x4890,0x44e0),
102:(0x4010,0x4ff8,0x4014,0x4014),
103:(0x63f0,0x6408,0x6408,0x6408,0x6210,0x7ff8),
104:(0x4ffc,0x4020,0x4010,0x4010,0x4010,0x4fe0),
105:(0x4ff4,),
106:(0x6000,0x7ff4),
107:(0x4ffc,0x4100,0x4080,0x4140,0x4620,0x4810),
108:(0x4ffc,),
109:(0x4ff0,0x4020,0x4010,0x4010,0x4fe0,0x4020,0x4010,0x4010,0x4fe0),
110:(0x4ff0,0x4020,0x4010,0x4010,0x4010,0x4fe0),
111:(0x47e0,0x4810,0x4810,0x4810,0x4810,0x47e0),
112:(0x7ff8,0x4420,0x4810,0x4810,0x4810,0x47e0),
113:(0x47e0,0x4810,0x4810,0x4810,0x4420,0x7ff8),
114:(0x4ff0,0x4020,0x4010,0x4010),
115:(0x4460,0x4890,0x4890,0x4890,0x4720),
116:(0x4010,0x4ffc,0x4810,0x4810),
117:(0x47f0,0x4800,0x4800,0x4800,0x4400,0x4ff0),
118:(0x4030,0x40c0,0x4300,0x4c00,0x4300,0x40c0,0x4030),
119:(0x4030,0x43c0,0x4c00,0x43c0,0x4030,0x43c0,0x4c00,0x43c0,0x4030),
120:(0x4810,0x4660,0x4180,0x4180,0x4660,0x4810),
121:(0x4030,0x6060,0x6380,0x7c00,0x4300,0x40c0,0x4030),
122:(0x4810,0x4c10,0x4b10,0x48d0,0x4830,0x4810),
123:(0x4080,0x5f7c,0x6002),
124:(0x7ffe,),
125:(0x6002,0x5f7c,0x4080),
126:(0x4080,0x4040,0x4040,0x40c0,0x4080,0x4080,0x4040),
127:(0x4ff8,0x4808,0x4808,0x4808,0x4808,0x4808,0x4ff8)
}

Posted: **Mon Apr 11, 2016 8:15 am**

This is obviously a font description (at least 96 printable chars).
If the char matrix is 14x13 ~2184 bytes are needed to encode a full pattern font in a simple array.
A directory encoded array of 16 bit words will need 650 values.

Posted: **Tue Apr 12, 2016 6:46 am**

A solution is in the pipeline: persistent bytecode. This will enable you to create a Python module which declares (say) bytes objects containing constant data. The Python code is cross-compiled to Python bytecode which can then be compiled into a firmware build. The data is stored in Flash memory as part of the firmware image and can be accessed like any other module, but with minimal RAM usage. I have tested this and it works, but to date persistent bytecode hasn't made its way into the official release https://github.com/micropython/micropython/pull/1811.

Posted: **Wed Apr 13, 2016 11:03 am**

That will be an interesting/useful feature. I'm working on a micro SNMP library and have realised that I can't really achieve everything I want to without representing at least mib-2 as a tree.

I'm working on the assumption that nested python objects will be unwieldy for something that'd only ever be used as part of something running on an embedded device, so am considering packing the structure into a bytearray wrapped in a class to parse it.

Being able to move at least the bytearray into flash would be a nice benefit.

Posted: **Thu Apr 14, 2016 6:49 am**

Bear in mind that data stored in Flash is immutable. To avoid RAM use you'll need a bytes object rather than a bytearray. If you declare mutable variables, compile the module, and store it as persistent bytecode, at runtime the variables will presumably be put in RAM. I've only tested immutable data so I'm guessing here, but it strikes me that must work this way because of the physical limitations of the hardware.

Posted: **Fri Apr 15, 2016 6:42 am**

On Arduino, we do have a way to grab back data from the Flash memory. This indeed requires to compile the Arduino sketck + upload to make the data available in the Flash (but this is a quite easy and short process)
I would be nice to have a way to store some data into the Flash without recompiling the PyBoard Firmware and upload a new Firmware on the board.

I would be nice to being able to load some file (with immutable data, or python precompiled byte code) stored in the Flash Drive (or SDcard) into the Python Flash memory... so it would be made available to the MicroPython interpreter without using the whole compilation chain and PyBoard flashing.

I do understand that dealing with the Pyboard's allocated Flash could be quite tricky... but this would definitively be a great enhancement for storing static data like astronomic data, font definition (my use case), etc

Posted: **Fri Apr 15, 2016 6:44 am**

pythoncoder wrote:Bear in mind that data stored in Flash is immutable. To avoid RAM use you'll need a bytes object rather than a bytearray. If you declare mutable variables, compile the module, and store it as persistent bytecode, at runtime the variables will presumably be put in RAM. I've only tested immutable data so I'm guessing here, but it strikes me that must work this way because of the physical limitations of the hardware.

Do you mean that when defining my structure into a bytes object in my Python code, I would spare RAM?

Posted: **Fri Apr 15, 2016 6:57 am**

@MCHobby persistent bytecode is now mainstream so trying it is straightforward http://forum.micropython.org/viewtopic.php?f=6&t=1776.

I use it for font storage. I have a Python module which declares large bytes objects. Compiled as persistent bytecode I can import it and access the bytes objects, yet its RAM usage is minimal because the data is physically located in Flash.

Posted: **Sat Apr 16, 2016 7:13 am**

So if I do understand properly, there is not other way than compiling a tuned "uPy firmware with Persistant Byte code" to reduce RAM consumption.
It is a nice feature but probably over the knowledge of most of the makers.

Is there other way (good practice) to reduce the memory consumption?
Some thread reading/articles to recommend?

Posted: **Sun Apr 17, 2016 9:54 am**

I've used three techniques with fonts.

Persistent bytecode.
With proportional fonts don't store trailing whitespace. Use an index table to determine the address of the character data.
Use a random access file on disk, accessing data one character at a time. Slow.

I have my doubts about the suitability of compression for this type of data, but if anyone knows better or has other ideas I'd be glad to hear them.

MicroPython Forum (Archive)

How to reduce the memory consumption of the object definition

How to reduce the memory consumption of the object definition

Re: How to reduce the memory consumption of that objet definition

Re: How to reduce the memory consumption of that object definition

Re: How to reduce the memory consumption of that objet definition

Re: How to reduce the memory consumption of that objet definition

Re: How to reduce the memory consumption of that objet definition

Re: How to reduce the memory consumption of that objet definition

Re: How to reduce the memory consumption of that objet definition

Re: How to reduce the memory consumption of the object definition

Re: How to reduce the memory consumption of the object definition