Fastest way to modify a bytearray?

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Post Reply
ddw
Posts: 3
Joined: Thu May 16, 2019 6:17 pm

Fastest way to modify a bytearray?

Post by ddw » Thu May 16, 2019 6:23 pm

I'm trying to find the fastest code (without using native code generation) to create one bytearray from another by applying a function to each element. So far the best I've found is

def bytearray_mod(b1, b2, mod):
b2[:] = bytearray(map(mod, b1))

Any suggestions will be greatly appreciated!

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Fastest way to modify a bytearray?

Post by dhylands » Thu May 16, 2019 6:54 pm

bytearrays are modifiable in place, so you can also just modify the elements in place. Not sure if this is faster than what you've suggested, but it uses less memory.

ddw
Posts: 3
Joined: Thu May 16, 2019 6:17 pm

Re: Fastest way to modify a bytearray?

Post by ddw » Thu May 16, 2019 7:06 pm

Thanks! What I've posted does already modify in place, right?

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Fastest way to modify a bytearray?

Post by dhylands » Thu May 16, 2019 10:07 pm

By calling bytearray it winds up allocating a new bytearray.

What I meant is that you can do something like this:

Code: Select all

>>> def bytearray_mod(b1, b2):
...     for idx, item in enumerate(b1):
...         b2[idx] = item + 100
... 
>>> b1 = bytearray([1, 2, 3, 4])
>>> b2 = bytearray(4)
>>> bytearray_mod(b1, b2)
>>> b1
bytearray(b'\x01\x02\x03\x04')
>>> b2
bytearray(b'efgh')

ddw
Posts: 3
Joined: Thu May 16, 2019 6:17 pm

Re: Fastest way to modify a bytearray?

Post by ddw » Thu May 16, 2019 10:46 pm

Right, the bytearray() in the map approach causes a temporary new allocation. But it seems to be somewhat faster:

ba_bench.py:

Code: Select all

import time
import random
import array
    
def timed_function(f, *args, **kwargs):
    myname = str(f).split(' ')[1]
    def new_func(*args, **kwargs):
        t = time.monotonic_ns()
        result = f(*args, **kwargs)
        delta = time.monotonic_ns() - t 
        print('function {}: Time = {:6.3f}ms'.format(myname, delta/1000000))
        return result
    return new_func

def show(a):
    for i in range(0,10):
        print('%4d' % int(a[i]), end='')
    print()
    
n = 10000

def f(a):
    return a+100
    
ba1 = bytearray(random.randint(0,255) for i in range(0,n))
ba2 = bytearray(random.randint(0,255) for i in range(0,n))
ba3 = bytearray(random.randint(0,255) for i in range(0,n))
    
@timed_function
def bytearray_for(b1, b2):
    for idx, item in enumerate(b1):
        b2[idx] = item + 100

@timed_function
def bytearray_map(b1, b2, f):
    b2[:] = bytearray(map(f, b1))


bytearray_for(ba1, ba2)
show(ba1)
show(ba2)
print()

bytearray_map(ba1, ba3, f)
show(ba1)
show(ba3)
print()


Running:

Code: Select all

>>> import ba_bench
function bytearray_for: Time = 530.000ms
  24 183  65 140 187 173 160 255 100 215
 124  27 165 240  31  17   4  99 200  59

function bytearray_map: Time = 384.000ms
  24 183  65 140 187 173 160 255 100 215
 124  27 165 240  31  17   4  99 200  59
>>>

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Fastest way to modify a bytearray?

Post by dhylands » Fri May 17, 2019 12:01 am

Yep - I'm not surprised, and I really just wanted to point out that there is a performance/memory usage tradeoff to make here. Pick whichever one is more important for your application.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: Fastest way to modify a bytearray?

Post by jimmo » Fri May 17, 2019 3:23 am

Also remember that you'll pay a hidden performance cost that the benchmarks don't show for the increased load on the garbage collector.

(I had this exact tradeoff for an application that involved modifying a bytearray to generate animation frames for an LED array... avoiding GC pauses turned out to have a far greater benefit on the overall visual effect than making each frame faster).

(And using things like @micropython.native / @micropython.viper also swung the balance a bit too)

Post Reply