Memoryview and readinto()

[Roberthh]

Hello all, following the hint about the optimization of code by using memoryview, I wrote that little function, which wraps teh SD-card library error:

Code: Select all

# split read, due to the bug in the SD card library, avoid reading
# more than 511 bytes at once, at a performance penalty
# required if the actual file position is not a multiple of 4
def odd_read(f, buf, n):
    BLOCKSIZE = const(512) ## a sector, but may be 4 too
    part = BLOCKSIZE - (f.tell() % BLOCKSIZE)
    if part >= n or part == BLOCKSIZE:
        return f.readinto(memoryview(buf[0:n]))
    else:
        return f.readinto(memoryview(buf[0:part])) + f.readinto(memoryview(buf[part:n]))

buf is a preallocated bytearray. That function returns the expectedt number, but does not put any data into buf. The previous version of the function, which uses read, works fine, but is slow:

Code: Select all

def odd_read(f, n):
    BLOCKSIZE = const(512) ## a sector, but may be 4 too
    part = BLOCKSIZE - (f.tell() % BLOCKSIZE)
    if part >= n or part == BLOCKSIZE:
        return f.read(n)
    else:
        return f.read(part) + f.read(n - part)

Does anyone have a clue why the memoryview version is not working? I hope it's just a simple fault of me.
Tested with: MicroPython v1.7-72-g050e645 on 2016-04-16; PYBv1.0 with STM32F405RG

Regards

[dhylands]

I got burned by exactly the same thing when I started using memoryview, so I recognized it right away.

When you do buf[0:part] you wind up allocating a new buffer which is a copy of the slice of the original buffer, and you're reading into the memoryview of the copied buffer and not the original.

The correct way to do this is to make the memoryview be of the entire buffer and take a slice of the memoryview (which is a sub-memoryview into the original memoryview).

[Roberthh]

Thanks @dhylands, that did the trick. The function now looks like:

Code: Select all

# split read, due to the bug in the SD card library, avoid reading
# more than 512 bytes at once, at a performance penalty
# required if the actual file position is not a multiple of 4
def odd_read(f, buf, n):
    BLOCKSIZE = const(512) ## a sector
    mv = memoryview(buf)
    bytes_read = 0
    for i in range(0, n - BLOCKSIZE, BLOCKSIZE):
        bytes_read += f.readinto(mv[i:i + BLOCKSIZE])
    if bytes_read < n:
        bytes_read += f.readinto(mv[bytes_read:n])
    return bytes_read

Interestingly, the method used with read, splitting the read into two chunks, where the first brings you to a 4 byte boundary, does not work with readinto(). Instead, you get os.error(5) again. I had to split the read into block of max. 512 bytes.
Best Regards, Robert