Irregular processing power

[pfalcon]

instead of 'data' being automatically discarded after going out of scope?

It's discarded - it becomes garbage. It's just not reclaimed immediately, but requires special "garbage collection".

It's clear, that the line data = ''.join(list(map(byte2bits, buf))) wastefully uses memory,

If it's clear, why do you use it, and not rewrite that program in non-wasteful way?

[Markus Gritsch]

If it's clear, why do you use it, and not rewrite that program in non-wasteful way?

The wasteful thing is, that one has to create the pattern for the LEDs all as one large chunk so that it can be handled as one SPI transfer, to get the timing right.

[Markus Gritsch]

I'm not familiar with the actual gc algorithm, but its possible that its not linear, which means that it may be taking much more time than the sum of the smaller garbage collections being done.

OK, might be the case. If so, this behaviour feels sub-optimal. I took the example code from Espruino, where it runs smoothly without manually taking care of such things.

[dhylands]

I'm not familiar with the actual gc algorithm, but its possible that its not linear, which means that it may be taking much more time than the sum of the smaller garbage collections being done.

OK, might be the case. If so, this behaviour feels sub-optimal. I took the example code from Espruino, where it runs smoothly without manually taking care of such things.

IIRC the espruino uses reference counting, whereas MicroPython uses garbage collection. These are 2 different approaches to managing memory, and have different sets of pros and cons.

[pfalcon]

If it's clear, why do you use it, and not rewrite that program in non-wasteful way?

The wasteful thing is, that one has to create the pattern for the LEDs all as one large chunk so that it can be handled as one SPI transfer, to get the timing right.

Ok, but that's not what I'm trying to hint. Let's look at your code - you start with efficient, inplace data structure bytearray, but from it, you go to much less efficient list, and don't even use inplace modification for it, instead creating new one each time. And what you do exactly is: pass values one by one thru external function (slow), collect such values one by one into list (slow), join them back (slow and wasteful - you split stuff just to join it back as the next step).

Surely, that piece of code is expressive, but the only way to get it efficient is to pass thru giant set of optimizations (as a bit of trivia, latest versions of Clang/LLVM easily compile to binaries of ~1Gb size - that's how much code needed to optimize (other) code to be small and fast). Surely, MicroPython can't do that (for comparison, uPy already does more optimization on bytecode than CPython does). So, if you want to write expressive code, write expressive code. And if you want efficient code, write efficient code.

I hope it's obvious what needs to be done with your code - 1) pre-allocate bytearrays for everything you need outside the loop; 2) byte2bits() should take input and output arrays, and convert one to another inplace, in one big boring loop, just like you'd do it in C. And I hope you don't challenge C's way - it does it that way, because it's the most efficient.

[fma]

I changed Markus's code to use 2 bytearrays, and I can drive 121 leds without memory error. It also works much faster (but I can't reach 60fps with 121 leds!).

[pfalcon]

Good results. I also (yeah, now ) wanted to mention that while what I posted above is based on knowledge of uPy internals and some benchmarking experience, it is still largely educated guess. I rely on you guys to prove or disprove it. That said, we have project to put quantitative results behind the guesses: http://forum.micropython.org/viewtopic.php?f=3&t=77 , and I actually got curious enough to add testcases based on this thread. Here's one (see instructions in the topic above to see rest of results!):

Code: Select all

$ ./run-bench-tests bench/bytebuf-*
bench/bytebuf:
    0.888s (+00.00%) bench/bytebuf-1-inplace.py
    25.266s (+2746.13%) bench/bytebuf-2-join_map_bytes.py
    4.827s (+443.70%) bench/bytebuf-3-bytarray_map.py
1 tests performed (3 individual testcases)

So, first one is inplace bytarray modification - zero memory allocation, and the fastest, as you see. 2nd one uses join() + map() (maybe not exactly like the code posted here, but you got the idea) - it is 30 times slower! Last one uses just map() - it's still 5 times slower.

[Markus Gritsch]

I changed Markus's code to use 2 bytearrays, and I can drive 121 leds without memory error. It also works much faster (but I can't reach 60fps with 121 leds!).

Please attach the code.

[Markus Gritsch]

@fma: Are you going to share your modified code or not?

[fma]

Sure! Here it is:

Code: Select all

import math

import pyb
import gc

NB_LEDS = 121


def tsv2rgb(hue, saturation, value):
    S = saturation / 100.
    V = value / 100.

    Ti = int((hue / 60)) % 6
    f = hue / 60. - Ti
    L = V * (1 - S)
    m = V  * (1 - f * S)
    n = V * (1 - (1 -f) * S)

    if Ti == 0:
            red = V
            green = n
            blue = L
    elif Ti == 1:
            red = m
            green = V
            blue = L
    elif Ti == 2:
            red = L
            green = V
            blue = n
    elif Ti == 3:
            red = L
            green = m
            blue = V
    elif Ti == 4:
            red = n
            green = L
            blue = V
    elif Ti == 5:
            red = V
            green = L
            blue = m
    else:
        red = green = blue = 0

    # Scale to 8 bits
    red *= 255
    green *= 255
    blue *= 255

    return int(red), int(green), int(blue)


def byte2bits(byteIn, byteOut):
    b0 = 0x03
    b1 = 0x0F
    i = 0
    for byte in byteIn:
        mask = 0x80
        while mask != 0:
            byteOut[i] = b0 if (byte & mask) == 0 else b1
            mask >>= 1
            i += 1


def main():
    spi = pyb.SPI(1, pyb.SPI.MASTER, baudrate=6400000, polarity=0, phase=1)
    spi.send(chr(0x00))

    hue = 0
    byteIn = bytearray(NB_LEDS * 3)
    byteOut = bytearray(len(byteIn) * 8)
    while True:
        for led in range(NB_LEDS):
            hue += int(360 / NB_LEDS)
            hue %= 360
            byteIn[led*3+0], byteIn[led*3+1], byteIn[led*3+2] = tsv2rgb(hue, 100, 10)
        byte2bits(byteIn, byteOut)
        pyb.disable_irq()
        spi.send(byteOut)
        pyb.enable_irq()
        gc.collect()


if __name__ == "__main__":
    main()