How to build string in pre-allocated bytearray

[wimpie]

Background: an ESP8266 running micropython v1.16, is used to measure the voltage and the frequency of the mains power. The frequency is measured using an ISR, which measures the time it takes to receive 250 pulses. The main loop has been changed from sending the measurements of the voltage and the frequency each in a separate MQTT message to building a single string containing a JSON-encoded version of a dictionary with measurements. The string is build piecewise, without the use of module json.
Since that version is in use, a frequency of 49.8 [Hz] is regularly reported (about once an hour), which might be caused by effectively missing one interrupt.

I suspect that garbage collection (gc) is causing the measurements of a low mains frequency. Therefore, automatic gc was disabled, and replaced by a periodic (immediately after sending a message to the MQTT broker) run of the gc. Still a low frequency is reported a few times per hour. Small variations on this way of working did not result in improvements.

Two relevant portions of the script are shown below. The first one is the ISR, the second one the main part of the task to publish the measurements, which is invoked once every 300 seconds.

Code: Select all

  def isr( self, pin ):
#   assert pin is self.pin, 'Unexpected interrupt'
    tocp      = time.ticks_us()         # Time of current pulse
    self.topp = tocp                    # Update pulse statistics
    self.puls+= 1
    if self.puls < self.pulses:
      return                            # Wait till enough pulses seen

    self.lopt= time.ticks_diff( tocp, self.tofp )
    self.tofp= tocp                     # Set up for next measurement
    self.puls= 0
  #
    try:
      micropython.schedule( self.refupd, None )
    except RuntimeError:
      error_count+= 1

Instance variable self.refupd points to a method in the same class / object which updates the sensor object which holds the frequency, using instance variables self.pulses and self.lopt. This update is using floating point arithmetic.

Code: Select all

  if station.isconnected():
    pld= _mqtt_pld_hdr                # Payload header
    for sensor in sensors:
      rslt= sensor[1].Poll()
      pld+= '"{}":{},'.format( sensor[1].Name,
                               sensor[2].format(*rslt) ) 
    pld+= '"count":{}}}'.format( report_count ) 
    try:
      client.publish( _mqtt_topic, pld.encode() ) 
    except OSError:
      print( 'ERROR: MQTT broker connection failed.')
      Restart()
  else:
    print( 'ERROR: Wifi connection broken' ) 
    Restart()
 #
  report_count+= 1 
  gc.collect()                        # Clean up heap

The next step, still assuming that gc is the culprit, would be an attempt to avoid the need for gc, by building the string piecewise in a pre-allocated byte array. Is this possible?

[wimpie]

A partial solution seems to be the use of Python module io, class BytesIO (see https://docs.micropython.org/en/latest/library/io.html). The size of the buffer to be allocated can be specified. However, I found no way to clear the buffer other than releasing it using method close and allocate a new buffer.

The current implementation of the main loop is:

Code: Select all

    if station.isconnected():
      pld= io.BytesIO( 511 )          # Allocate buffer
      pld.write( _mqtt_pld_hdr )      # Payload header
      for sensor in sensors:
        rslt= sensor[1].Poll()
        pld.write( b'"{}":{},'.format( sensor[1].Name,
                                       sensor[2].format(*rslt) ) ) 
      pld.write( b'"count":{}}}'.format(report_count) ) 
      try:
        client.publish( _mqtt_topic, pld.getvalue() ) 
        pld.close()                   # Release buffer
      except OSError:
        print( 'ERROR: MQTT broker connection failed.')
        Restart()
    else:
      print( 'ERROR: Wifi connection broken' )
      Restart()
 #
    report_count+= 1
    gc.collect()                     # Clean up heap

Method write is used to extend the text string piece by piece. While the previous version showed a measurement matching a missed interrupt (due to garbage collection) at least once per hour, the current version showed only one measurement in 20 hours of 49.8 [Hz]. Thus the frequency of the low-frequency-measurements has gone down considerably, indicating that garbage collection is indeed the cause of the missed interrupts.