Page 1 of 2
Question about allocating memory in interrupts
Posted: Sun Oct 09, 2016 8:16 pm
by slzatz
Hardware is an esp8266 with latest firmware.
The following works but I am confused because I thought you couldn't allocate memory in an interrupt (including a timer callback, which this is) so just trying to understand why this doesn't cause a memory error). The line is an mqtt check on a topic that either brings back a tuple of the topic and message or None.
Code: Select all
def callback(t):
...
b = umc.check_msg() #<- why is this OK in the callback
...
I thought I'd have to create a bytearray b outside the callback and somehow read the mqtt message into that bytearray but this works without doing that so clearly I am misunderstanding something fundamental about what can and can't be done in an interrupt and am looking for any insight. Thanks.
Steve
Re: Question about allocating memory in interrupts
Posted: Sun Oct 09, 2016 10:30 pm
by markxr
If the function check_msg simply returns a reference to an already-allocated buffer, then it won't need to allocate, right?
Mark
Re: Question about allocating memory in interrupts
Posted: Mon Oct 10, 2016 7:03 am
by pythoncoder
Assuming you're using the MQTT library, you'll have written a callback function and assigned it to the client with its set_callback() method. This runs when subscription message is received; it does not run in an interrupt context and can allocate memory. I'm guessing that you instantiate the tuple in that callback. As @markxr indicated, if your timer callback returns a reference to that tuple, no memory allocation is taking place.
Re: Question about allocating memory in interrupts
Posted: Mon Oct 10, 2016 3:10 pm
by slzatz
check_msg is doing the following when the interrupt is triggered.
Code: Select all
def check_msg(self):
self.sock.setblocking(False)
res = self.sock.read(1)
if res is None:
return None
#if res[0] >> 4 !=3: #more general but not handling QoS > 0 right now
if res[0] in (48,49):
self.sock.setblocking(True)
sz = self.sock.recv(1)[0]
if sz > 127:
sz1 = self.sock.recv(1)[0]
#sz = sz1*128 + sz - 128
sz+= (sz1<<7) - 128
z = self.sock.recv(sz)
# topic length is first two bytes of variable header
topic_len = z[:2]
topic_len = (topic_len[0] << 8) | topic_len[1]
topic = z[2:topic_len+2]
msg = z[topic_len+2:]
return (topic, msg)
elif res[0]>>4==13:
self.sock.setblocking(True)
s = self.sock.recv(1) # second byte of pingresp should be 0
return 13
else:
return res
Admittedly, the code is not a work of art but the tuple with the topic and message is being generated during the interrupt when there is a message that has not been received yet.
Re: Question about allocating memory in interrupts
Posted: Mon Oct 10, 2016 4:28 pm
by pythoncoder
It's hard to comment without seeing all the code. Why have you rewritten check_msg()? If that code is being called in an interrupt context I can see problems aside from memory allocation, notably using blocking sockets. These can block for long periods. Interrupt handlers should be designed to execute in a minimal, deterministic time.
One way to use the MQTT library is with some form of cooperative multi-threading (uasyncio or otherwise), with a thread polling check_msg() and the callback doing the work when a subscription is received. This avoids the need for interrupts. MQTT is too slow a protocol to need interrupts in my view.
Re: Question about allocating memory in interrupts
Posted: Mon Oct 10, 2016 9:21 pm
by slzatz
@pythoncoder - thanks for your comments. I had modified an earlier version of the mqtt library before it was released and since it has been working in a number of applications I never upgraded to the released version. I haven't had an issue with blocking although the timing in my applications is generally not critical -- I need to periodically break out of a while loop and the timer seemed to be a natural way to do that. The entire code for the mqtt_client is at
https://gist.github.com/slzatz/fed5460f ... 4df9a446a5
One more question: the full timer callback is as follows and works not only with the check_msg line but also includes assigning a float, which, again somewhat surprisingly to me, doesn't cause a memory issue.
Code: Select all
def callback(t):
global MAX_BRIGHT
b = umc.check_msg()
print("b =",b)
if b:
np.fill((0,0,0))
np[0] = (100,0,0)
np.write()
try:
MAX_BRIGHT = float(b[1].decode('ascii'))
except ValueError as e:
print("Value couldn't be converted to float")
tim.init(period=10000, mode=Timer.PERIODIC, callback=callback)
Re: Question about allocating memory in interrupts
Posted: Tue Oct 11, 2016 9:12 am
by pythoncoder
Are you using timer ID -1? This is a virtual timer which uses the underlying RTOS. I can only assume that its callbacks are "soft interrupts" not subject to the constraints of a true hardware interrupt. I'll raise a query on the IRC channel and try to find out.
Re: Question about allocating memory in interrupts
Posted: Tue Oct 11, 2016 11:11 am
by slzatz
Are you using timer ID -1?
Yes. If a soft interrupt like timer ID -1 doesn't have the same memory contraints as a hardware interrupt, would be great to get that documented so that there isn't an assumption that all interrupts have the same memory restrictions (unless I missed it).
Re: Question about allocating memory in interrupts
Posted: Tue Oct 11, 2016 11:18 am
by pythoncoder
Indeed it should be documented, it's an observation of considerable significance. When I get confirmation that it's correct and that there aren't any hidden gotchas I'll offer an update.
Re: Question about allocating memory in interrupts
Posted: Tue Oct 18, 2016 5:42 am
by pythoncoder
I now have a response from Damien. It seems that the capabilities of callbacks from ESP8266 virtual timers are somewhat undefined. This is because the underlying RTOS of the ESP8266 is something of a "black box". His best advice is to treat the callback as per other interrupt handlers and avoid memory allocation because there is no guarantee that it will be reliable.
The longer term plan is to support soft IRQ's which aim to provide a stable solution to this problem.