Does the PyBoard DAC.write_timed() point to user memory without bumping the refcount?

The official pyboard running MicroPython.
This is the reference design and main target board for MicroPython.
You can buy one at the store.
Target audience: Users with a pyboard.
Post Reply
grodstein
Posts: 13
Joined: Tue Dec 01, 2020 6:23 pm

Does the PyBoard DAC.write_timed() point to user memory without bumping the refcount?

Post by grodstein » Sat Apr 24, 2021 4:15 pm

I've been playing with the DAC on my PyBoard v1.1 and running into some issues that lead me to suspect a bug.
  • I allocate a DAC and then set up a sawtooth. First, buf = array.array ('B', range(256)). Then dac.write_timed(buf, freq, mode=pyb.DAC.CIRCULAR). A scope confirms that the DAC produces the correct sawtooth.
  • Overwrite some elements of buf; e.g., buf[128:256]=0. The scope confirms that the waveform changes appropriately. Conclusion: dac.write_timed() is referencing my original buffer rather than copying the data into safe storage somewhere
  • Write a function that encapsulates the two lines above; i.e., create a buffer and give it to dac.write_timed(). The scope shows a nice sawtooth at first. Then a few seconds later, it becomes random noise.
  • Declare the buffer to be a global variable. The sawtooth no longer disintegrates into noise.
My suspicion: dac.write_timed() is written in C, and references the incoming buffer without telling Python about it. I.e., it doesn't increment the buffer's refcount. Then when my function exits, buf's refcount=0 and at some point the Python garbage collector reuses buffer memory for something else and my nice sawtooth vanishes. Calling buf a global variable prevents the garbage collector from reclaiming it.

Any other explanations? I would be happy to add the full code if anyone is interested. A few potentially-relevant bits:
  • I'm on a PyBoard v1.1 with firmware v1.12. I would burn newer firmware, but haven't quite figured out how yet.
  • The rest of the software does have an event timer with an interrupt loop, which then calls micropython.schedule() to read an ADC.
Thanks,
/Joel

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Does the PyBoard DAC.write_timed() point to user memory without bumping the refcount?

Post by pythoncoder » Sat Apr 24, 2021 6:01 pm

This could be an issue of variable scope. If the buffer is a local variable which has gone out of scope, this is what will happen as the buffer would have existed on the stack.
Peter Hinch
Index to my micropython libraries.

grodstein
Posts: 13
Joined: Tue Dec 01, 2020 6:23 pm

Re: Does the PyBoard DAC.write_timed() point to user memory without bumping the refcount?

Post by grodstein » Sun Apr 25, 2021 12:02 am

Thanks, Peter, that's a good hypothesis -- but I'm not quite sure I fully believe it. Here's why.

The array object (as returned by array.array()) supports a .append method(). I.e., the actual storage can be extended dynamically. Thus, it's hard to see how that storage could be allocated on the stack as opposed to the heap.

Still (and I don't know the details of how MicroPython implements the array type), I would guess that it's similar to a C++ vector; a small fixed-size enclosing object with a pointer to the actual storage. Perhaps the enclosing object is stored on the stack? If so, that would be consistent with your hypothesis.

However, the description of MicroPython memory management at https://docs.micropython.org/en/latest/ ... rymgt.html does not seem to suggest that model. It says
The value of a small integer is stored directly in the mp_obj_t and will be allocated in-place, not on the heap or elsewhere. As such, creation of small integers does not affect the heap. Similarly for interned strings that already have their textual data stored elsewhere, and immediate values like None, False and True. Everything else which is a concrete object is allocated on the heap
So my guess is still that the array is stored on the heap, and that DAC.write_timed is storing a pointer to it without incrementing the refcount. Any other explanations to suggest?

/Joel

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Does the PyBoard DAC.write_timed() point to user memory without bumping the refcount?

Post by pythoncoder » Sun Apr 25, 2021 6:21 am

Elements of an array are stored in contiguous locations. Incidentally MicroPython memory mnagement does not use reference counting - see the docs.

It's best to step outside the complexities of memory management and consider basic Python scoping rules. If I write

Code: Select all

def foo():
    arr = array.array('I', range(10))
    # do something, e.g. set up a write_timed with arr
def bar():
    foo()
    # arr will be out of scope
bar()
This can be expected to fail, probably in the way you describe. If arr were global it would work.
Peter Hinch
Index to my micropython libraries.

grodstein
Posts: 13
Joined: Tue Dec 01, 2020 6:23 pm

Re: Does the PyBoard DAC.write_timed() point to user memory without bumping the refcount?

Post by grodstein » Sun Apr 25, 2021 10:24 am

Hi Peter,

thanks -- that's good information about MicroPython using mark_and_sweep. Given that, the post probably should be retitled "Does the PyBoard DAC.write_timed() point to user memory without telling the mark-and-sweep list of top-level nodes about it?"

In the end, perhaps this is somewhat moot, since I do have a workaround. I would argue, though, that this is a "gotcha" that is unique to MicroPython and has no obvious CPython equivalent. In your example in the previous post, I agree that foo() will be out of scope once bar() has finished calling it foo(). However, in your example (and in any non-MicroPython case I can think of), this will be irrelevant -- since foo() is finished and is out of scope, any object is creates is indeed no longer relevant.

But DAC.write_timed() is sort of special and different. You call it, it returns, and it's still using your memory!. That's the amazing and wonderful thing about its DAC.CIRCULAR parameter. In your example, if your line of code
# do something, e.g. set up a write_timed with arr
were pretty much anything other than write_timed (..., DAC.CIRCULAR), it would work fine. "Normal" Python is structured precisely to not have to worry about things like this. But write_timed (..., DAC.CIRCULAR) is special in this regard, and I would argue it's kind of broken the agreement that Python typically has about not needing to worry about object lifetimes.

So I would argue that this is unreasonable behavior on the part of write_timed(). If it's not easy to fix by having write_timed (..., DAC.CIRCULAR) tell the MicroPython garbage collector about itself, then at least it could use a mention in the docs. But in any case, thanks for the info about mark-and-sweep :-)

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Does the PyBoard DAC.write_timed() point to user memory without bumping the refcount?

Post by pythoncoder » Sun Apr 25, 2021 5:49 pm

This is one of a small set of special cases where you need to think about object lifetimes. It is basically a concurrency issue where a background task is accessing a resource which the foreground task has allowed to go out of scope.
Peter Hinch
Index to my micropython libraries.

grodstein
Posts: 13
Joined: Tue Dec 01, 2020 6:23 pm

Re: Does the PyBoard DAC.write_timed() point to user memory without bumping the refcount?

Post by grodstein » Sun Apr 25, 2021 9:50 pm

> This is one of a small set of special cases where you need to think about object lifetimes. It is basically a concurrency issue where a background task is accessing a resource which the foreground task has allowed to go out of scope.

Agreed completely. I still think that "need to think about object lifetimes" is such a foreign concept in Python that it's at least worth a mention in the documentation. That would have saved me a few hours of debugging. If you let me know where to file it, I would be happy to do so. Otherwise, I guess this forum post serves as the documentation :-)

Post Reply