Looking for a way to use a memory consuming array

[schafon]

Hello,
I'm working with MicroPython for serval weeks now.
My board is the ESP32.
My goal is to get calibrated values from the ADC and in order to do so, I converted an ADC calibration script that someone has written in C++ for the Arduino to CPython module.
The script returns (prints) and array consist of 4095 ints, and when I try to load this array(copy the output and creating a python variable with this array) the ESP immediately with memory allocation error.
Now I need some kind of way to get the array without loading it using python.

My question is, how can I create a file and save this array to somewhere where it will be readable by the CPython module script? Where will it be saved?
Maybe there is another solution that I can use that you can think of?


Thanks

[safetyfactorman]

viewtopic.php?t=3899

[schafon]

viewtopic.php?t=3899

Thanks for the answer but this is not the solution I'm looking for.
I'm developing a product and need to reduce the costs

[Roberthh]

If speed does not matter, you can save it to a file. You may also add this table as a python module into frozen bytecode, like @pythoncoder and other do with font tables (see https://github.com/peterhinch/micropython-font-to-py). The drawback is, that you would have to create a specific image for each device. 4096 number does not seem a too large object. So the data module I mentioned above may also fit into RAM if pre-compiled.
But are you sure that you want to use the ESP32 ADC? besides the non-linearity, which you can compensate with that table, it is also very noisy. As a result of that, you have either use a large amount of filtering or accept a lower effective resolution. The latter would also allow you to use a smaller calibration table.

[schafon]

If speed does not matter, you can save it to a file. You may also add this table as a python module into frozen bytecode, like @pythoncoder and other do with font tables (see https://github.com/peterhinch/micropython-font-to-py). The drawback is, that you would have to create a specific image for each device. 4096 number does not seem a too large object. So the data module I mentioned above may also fit into RAM if pre-compiled.
But are you sure that you want to use the ESP32 ADC? besides the non-linearity, which you can compensate with that table, it is also very noisy. As a result of that, you have either use a large amount of filtering or accept a lower effective resolution. The latter would also allow you to use a smaller calibration table.

Thank you for your answer.
The frozen bytecode cannot be used since the calibration table needs to be created after the firmware was flashed to the device.

About the ADC noise, I took it into consideration and I've implemented capacitors near the ADC inputs, it's not perfect but it would work.

So is there a way to write a file using the python C module and read it using the same C module? Or even a C module that can read a file from the file system and convert it into an array?

[Roberthh]

You can use the usual python file operations to write and read files, and you can use the python array module for dealing with arrays. you would need an array of 4096 16 bit integers, or a bytearray of 8192 bytes. Packing 2 calibration values into 3 bytes would reduce the array size to 6144. Due to the memory organization of python, it is wise to allocate that array as soon as possible in your script.

[pythoncoder]

The ADC on the ESP32 is only capable of 12 bit resolution, so its output is constrained to be an integer between 0 and 4095. A calibration array of 4096 integers sounds like a simple lookup table (LUT) which is very RAM intensive.

I would investigate the possibility of a more efficient algorithm. Why not use a much smaller LUT with linear, quadratic or cubic interpolation to produce the intermediate values?

[schafon]

It's worth checking this out, BTW, the ADC reading is for a temperature sensors that can have an error of up to 3 degrees.

[Roberthh]

In that case, linearisation may not be worth the effort. Using the attenuation of 6dBm, the ADC has the best characteristics. And the a simple linear compensation if the form slope_corr * adc.read() + offset_corr should be sufficient. Or you go for a lower resolution, e.g. 9 bit, which shortens the lookup table. But that all depends on the temperature range you want to cover. 3° in a range of 0-100° is one thing, 3° in a range of -200 - 1000° something else. But even then, 9 or 10 bit should be fine.
See also the analysis of Bernd Boser https://github.com/bboser/IoT49/blob/ma ... alog_io.md, and the information on the espressif forum. Some links there, which you may have found:
https://www.esp32.com/viewtopic.php?f=1 ... ADC#p47676
https://www.esp32.com/viewtopic.php?f=1 ... ear#p52264
https://www.esp32.com/viewtopic.php?f=1 ... near#p4558

[schafon]

The temperature range I'm aiming for is 25 - 250 degrees c.
I'm currently using 100K NTC resistor and 4.7K as a series resistor, where the NTC resistor is connected to Vin.
My firsts tests show that I'm using the full range of the ADC input ( up to 3.1 V if I remember correctly) so the ATTN_6DB can't be used here.
I will go over the attached posts you added, but in the meanwhile, can you elaborate on the "slope_corr * adc.read() + offset_corr"?

* The script used in the first post you sent me is the script that I use, this is the script I converted to a C module.
Thanks