Question about allocating memory in interrupts

All ESP8266 boards running MicroPython.
Official boards are the Adafruit Huzzah and Feather boards.
Target audience: MicroPython users with an ESP8266 board.
slzatz
Posts: 92
Joined: Mon Feb 09, 2015 1:09 am

Question about allocating memory in interrupts

Post by slzatz » Sun Oct 09, 2016 8:16 pm

Hardware is an esp8266 with latest firmware.

The following works but I am confused because I thought you couldn't allocate memory in an interrupt (including a timer callback, which this is) so just trying to understand why this doesn't cause a memory error). The line is an mqtt check on a topic that either brings back a tuple of the topic and message or None.

Code: Select all

def callback(t):
  ...
  b = umc.check_msg() #<- why is this OK in the callback
  ...
I thought I'd have to create a bytearray b outside the callback and somehow read the mqtt message into that bytearray but this works without doing that so clearly I am misunderstanding something fundamental about what can and can't be done in an interrupt and am looking for any insight. Thanks.

Steve

markxr
Posts: 62
Joined: Wed Jun 01, 2016 3:41 pm

Re: Question about allocating memory in interrupts

Post by markxr » Sun Oct 09, 2016 10:30 pm

If the function check_msg simply returns a reference to an already-allocated buffer, then it won't need to allocate, right?

Mark

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Question about allocating memory in interrupts

Post by pythoncoder » Mon Oct 10, 2016 7:03 am

Assuming you're using the MQTT library, you'll have written a callback function and assigned it to the client with its set_callback() method. This runs when subscription message is received; it does not run in an interrupt context and can allocate memory. I'm guessing that you instantiate the tuple in that callback. As @markxr indicated, if your timer callback returns a reference to that tuple, no memory allocation is taking place.
Peter Hinch
Index to my micropython libraries.

slzatz
Posts: 92
Joined: Mon Feb 09, 2015 1:09 am

Re: Question about allocating memory in interrupts

Post by slzatz » Mon Oct 10, 2016 3:10 pm

check_msg is doing the following when the interrupt is triggered.

Code: Select all

def check_msg(self):
        self.sock.setblocking(False)
        res = self.sock.read(1)
        if res is None:
            return None

        #if res[0] >> 4 !=3: #more general but not handling QoS > 0 right now
        if res[0] in (48,49):
            self.sock.setblocking(True)
            sz = self.sock.recv(1)[0]
            if sz > 127:
                sz1 = self.sock.recv(1)[0]
                #sz = sz1*128 + sz - 128
                sz+= (sz1<<7) - 128

            z = self.sock.recv(sz)
            # topic length is first two bytes of variable header
            topic_len = z[:2]
            topic_len = (topic_len[0] << 8) | topic_len[1]
            topic = z[2:topic_len+2]
            msg = z[topic_len+2:]
            return (topic, msg)

        elif res[0]>>4==13:
            self.sock.setblocking(True)
            s = self.sock.recv(1) # second byte of pingresp should be 0
            return 13

        else:
            return res
Admittedly, the code is not a work of art but the tuple with the topic and message is being generated during the interrupt when there is a message that has not been received yet.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Question about allocating memory in interrupts

Post by pythoncoder » Mon Oct 10, 2016 4:28 pm

It's hard to comment without seeing all the code. Why have you rewritten check_msg()? If that code is being called in an interrupt context I can see problems aside from memory allocation, notably using blocking sockets. These can block for long periods. Interrupt handlers should be designed to execute in a minimal, deterministic time.

One way to use the MQTT library is with some form of cooperative multi-threading (uasyncio or otherwise), with a thread polling check_msg() and the callback doing the work when a subscription is received. This avoids the need for interrupts. MQTT is too slow a protocol to need interrupts in my view.
Peter Hinch
Index to my micropython libraries.

slzatz
Posts: 92
Joined: Mon Feb 09, 2015 1:09 am

Re: Question about allocating memory in interrupts

Post by slzatz » Mon Oct 10, 2016 9:21 pm

@pythoncoder - thanks for your comments. I had modified an earlier version of the mqtt library before it was released and since it has been working in a number of applications I never upgraded to the released version. I haven't had an issue with blocking although the timing in my applications is generally not critical -- I need to periodically break out of a while loop and the timer seemed to be a natural way to do that. The entire code for the mqtt_client is at https://gist.github.com/slzatz/fed5460f ... 4df9a446a5

One more question: the full timer callback is as follows and works not only with the check_msg line but also includes assigning a float, which, again somewhat surprisingly to me, doesn't cause a memory issue.

Code: Select all

def callback(t):
  global MAX_BRIGHT
  b = umc.check_msg()
  print("b =",b)
  if b:
    np.fill((0,0,0))
    np[0] = (100,0,0)
    np.write()

    try:
      MAX_BRIGHT = float(b[1].decode('ascii'))
    except ValueError as e:
      print("Value couldn't be converted to float")
      
tim.init(period=10000, mode=Timer.PERIODIC, callback=callback)
 

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Question about allocating memory in interrupts

Post by pythoncoder » Tue Oct 11, 2016 9:12 am

Are you using timer ID -1? This is a virtual timer which uses the underlying RTOS. I can only assume that its callbacks are "soft interrupts" not subject to the constraints of a true hardware interrupt. I'll raise a query on the IRC channel and try to find out.
Peter Hinch
Index to my micropython libraries.

slzatz
Posts: 92
Joined: Mon Feb 09, 2015 1:09 am

Re: Question about allocating memory in interrupts

Post by slzatz » Tue Oct 11, 2016 11:11 am

Are you using timer ID -1?
Yes. If a soft interrupt like timer ID -1 doesn't have the same memory contraints as a hardware interrupt, would be great to get that documented so that there isn't an assumption that all interrupts have the same memory restrictions (unless I missed it).

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Question about allocating memory in interrupts

Post by pythoncoder » Tue Oct 11, 2016 11:18 am

Indeed it should be documented, it's an observation of considerable significance. When I get confirmation that it's correct and that there aren't any hidden gotchas I'll offer an update.
Peter Hinch
Index to my micropython libraries.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Question about allocating memory in interrupts

Post by pythoncoder » Tue Oct 18, 2016 5:42 am

I now have a response from Damien. It seems that the capabilities of callbacks from ESP8266 virtual timers are somewhat undefined. This is because the underlying RTOS of the ESP8266 is something of a "black box". His best advice is to treat the callback as per other interrupt handlers and avoid memory allocation because there is no guarantee that it will be reliable.

The longer term plan is to support soft IRQ's which aim to provide a stable solution to this problem.
Peter Hinch
Index to my micropython libraries.

Post Reply