Ever expanding heap

[sspaman]

Hi,

I am using a Adafruit ESP8266 feather huzzah running micropython 1.9.4. I have some code that has been executing flawlessly for months. But I decided to add some averaging to smooth some of the noise on the data. Now as the code executes the heap slowly fills up until the following error occurs:

MemoryError: memory allocation failed, allocating 8192 bytes

I added this, micropython.mem_info(1), to observe the heap filling up until error.

I have also read through this:

http://docs.micropython.org/en/latest/r ... ained.html

I think the answer is in there but I haven't solved it yet. I've added ample gc.collect() but it doesn't help since the heap is filling with '=' tail blocks.

Any ideas?

Regards,
Chris

PS - could someone remind me how to post code snippets or point me to the bbcode guide. (search does not find the guide)

[jimmo]

Hi,

Could you post as much code as you can, or especially just the bit you changed. I'm intrigued that it said it was allocated 8192 bytes which seems like quite a lot?

If you paste the code in, select it, then press the "< / >" button it'll add the formatting. Or you can do it manually with " [ code ] " / " [ / code ]". (Without the spaces). It's possible that you need to do a certain number of posts before the forum software lets you use the bbcode markup though?

The two reasons you could be seeing the issue:
- Your code is accidentally holding onto references to previous allocations. e.g. referencing the whole bytearray when you only need a tiny slice of it.
- The pattern of allocations and deallocations is leading to heap fragmentation. The classic way this happens is if your code repeatedly does an operation which results in a few small allocations to calculate a single result, which is appended to a list. This results in that list pointing to a bunch of tiny allocations evenly spread throughout the heap. So you might have a large number of bytes free but any allocation larger than that gap will fail.

[sspaman]

Hi Jimmo,

Thanks for taking a look at this.

Here is the offending part of the code. As far as I can tell the problem is caused by the for loop in the first publish. I can adjust the number of iterations before crash by adjusting the number in the range(x). The routines called by the loop are below.

Code: Select all

    ph_list = [7, 7, 7, 7, 7, 7, 7, 7, 7, 7]
    probe_avg = []
    while True:
        try:
            # Publish
            if accum_time >= PUBLISH_PERIOD_IN_SEC:
                # Publish average water pH level
                for i in range(50):
                    gc.collect()
                    ph_measured = prph.read_ph(mcp.read(1))
                    probe_avg.append(ph_measured)
                ph_water = sum(probe_avg)/len(probe_avg)
                print('pH Measured = {}'.format(ph_water))
                # Calculate rolling average
                ph_list = prph.rotate(ph_list, 1)
                ph_list[len(ph_list)-1] = ph_water
                ph_avg = sum(ph_list)/len(ph_list)
                print('pH list = {}'.format(ph_list))
                print('Publish:  Water pH = {}'.format(ph_avg))
                client.publish(mqtt_PubFD1,
                               bytes(str(ph_avg), 'utf-8'),
                               qos=0)
                # Publish water EC level
                prph.pwm_init(5, True)  # Init pwm
                ec_water = prph.read_ec(mcp.read(7))
                prph.pwm_init(5, False)  # Deinit pwm
                print('Publish:  Water EC = {}'.format(ec_water))
                client.publish(mqtt_PubFD8,
                               bytes(str(ec_water), 'utf-8'),
                               qos=0)
                # Publish water temperature in C or F
                tc_water, tf_water = prph.tmp36(mcp.read(0))
                print('Publish:  Water Temp = {} c'.format(tf_water))
                client.publish(mqtt_PubFD2,
                               bytes(str(tf_water), 'utf-8'),
                               qos=0)
                # Publish outdoor temperature in C or F
                t2320, h2320 = prph.am2320(4)
                tf2320 = (float(t2320) * 1.8)+32
                print('Publish:  Outdoor Temp = {} c'.format(t2320))
                print('Publish:  Outdoor Temp = {} f'.format(tf2320))
                client.publish(mqtt_PubFD3,
                               bytes(str(tf2320), 'utf-8'),
                               qos=0)
                # Publish outdoor humidity in %
                print('Publish:  Outdoor Humidity = {} %'.format(h2320))
                client.publish(mqtt_PubFD4,
                               bytes(str(h2320), 'utf-8'),
                               qos=0)
                # Publish water level in inches
                water_level = prph.read_level(mcp.read(2))
                print('Publish:  Water Level = {} inches'.format(water_level))
                client.publish(mqtt_PubFD6,
                               bytes(str(water_level), 'utf-8'),
                               qos=0)
                # Publish Battery Voltage
                v_batt = prph.batt_mon(mcp.read(3))
                print('Publish:  Battery Voltage = {} V'.format(v_batt))
                client.publish(mqtt_PubFD5,
                               bytes(str(v_batt), 'utf-8'),
                               qos=0)
                # Publish number of publishes since last reset
                print('Publish:  No. of Pubs since reset {}'.format(pubs))
                client.publish(mqtt_PubFD7,
                               bytes(str(pubs), 'utf-8'),
                               qos=0)
                print('*******')
                import micropython
                gc.collect()
                gc.threshold(gc.mem_free() // 4 + gc.mem_alloc())
                print('Garbage collect free: {} allocated: {}'.format(gc.mem_free(), gc.mem_alloc()))
                micropython.mem_info(1)
                accum_time = 0
                pubs += 1

            # Subscribe.  Non-blocking check for a new message.
            client.check_msg()

            time.sleep(SUBSCRIBE_CHECK_PERIOD_IN_SEC)
            accum_time += SUBSCRIBE_CHECK_PERIOD_IN_SEC
            # print('Accumulated Time = {}'.format(accum_time))
        except KeyboardInterrupt:
            print('Ctrl-C pressed...exiting')
            client.disconnect()
            sys.exit()

Below is the prph.read_ph part of prph.read_ph(mcp.read(1))

Code: Select all

def read_ph(adc):
     v = .00322 * adc
     ph = (v-0.854373)/0.056877  # ph = (v - B)/M
     utime.sleep_ms(100)
     return float(round(ph, 1))

And finally the mcp.read(1) is reading the spi MCP3008 ADC using this.

Code: Select all

from machine import Pin


class MCP3008:
    def __init__(self, clk=14, mosi=13, miso=12, cs=15):
        self._clk = Pin(clk, Pin.OUT)
        self._mosi = Pin(mosi, Pin.OUT)
        self._miso = Pin(miso, Pin.IN)
        self._cs = Pin(cs, Pin.OUT)

    def read(self, channel):
        # """ Reads an analog value from the given channel (0-7) and returns it. """
        if channel not in range(0, 8):
            raise ValueError("channel must be 0-7")

        self._cs(1)  # negative edge
        self._cs(0)
        self._clk(0)

        send_cmd = channel
        send_cmd |= 0b00011000  # 0x18 (start bit + single/ended)

        # send bits (only 5 bits considered)
        for i in range(5):
            self._mosi(bool(send_cmd & 0x10))  # check bit on index 4
            self._clk(1)  # negative edge
            self._clk(0)
            send_cmd <<= 1

        # receive value from MCP
        v = 0
        for i in range(11):
            self._clk(1)  # negative edge
            self._clk(0)
            v <<= 1
            if self._miso():
                v |= 0x01

        return v

If it's useful I can post the heap output progression.

Best regards,
Chris

[jimmo]

What stops probe_avg from growing without bound?

[sspaman]

Nothing. Thanks for pointing that out. I think I know what to do.

[sspaman]

Thanks for the help Jimmo. I have fixed the problem and the design is running like a champ again.