Any zlib.DecompIO examples?

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
Post Reply
martincho
Posts: 96
Joined: Mon May 16, 2022 9:59 pm

Any zlib.DecompIO examples?

Post by martincho » Mon Jun 27, 2022 5:18 pm

Trying to understand various ways to use this class. Anyone know of any nice examples out there? Google has failed me.

One of the things I am trying to figure out is how to decompress a large compressed chunk of data without using very large buffers.

I can seek to the compressed data in the file...and then I am faced with, say, 50k that expands to, say, 100k. I need to be able to do this in 1k to 4k chunks and store them in an uncompressed file. Not sure zlib.DecompIO() can do this. You can't just slice compressed data randomly and unpack it.

One option I see is to pre-process the original file to break it up into a bunch of smaller files.

In other words, take a 100K file and shard it into 100 files, each being 1K. The file names would contain a sequence number of some scheme like that. You then zip that up.

At the MicroPython level, you can then use zlib.decompress() to expand each sub-file into a reasonably sized buffer. From there you simply build-up the final uncompressed file through multiple (100 in this case) append writes.

Crazy? I am hoping there's a slick way to do this that might not require any such pre-processing.

Playing with code right now, just wondering if anyone might know of useful usage examples.

martincho
Posts: 96
Joined: Mon May 16, 2022 9:59 pm

Re: Any zlib.DecompIO examples?

Post by martincho » Mon Jun 27, 2022 7:14 pm

Well, it was easier than I thought. Here is it for the benefit of others who might need to figure this out:

Code: Select all

import zlib

compressed_text = b'\xf3LS\xa8\xcc/U\xc8*-.Q(O\xcc+Q(\xc9W(JMLQ\xc8/R(/\xca,IUHTH\xcb\xccI\x05\x00'
uncompressed_text = b'If you just want to read or write a file'

# Create file with compressed data
fp = open("test.text", "w")
fp.write(compressed_text)
fp.close()

# Uncompress it in small chunks
chunk_size = 16

fp = open("test.txt", "rb")
fp.seek(0, 2)  # Go to end of file
file_size = fp.tell()  # Store length
fp.seek(0)  # Back to the start of the file

decompressor = zlib.DecompIO(fp, -1)
result = bytearray()

while True:
    uncompressed_chunk = decompressor.read(chunk_size)
    if uncompressed_chunk:
        result += uncompressed_chunk
    else:
        break

print(f"original: {uncompressed_text}")
print(f"result:   {bytes(result)}")
Of course, this doesn't do appropriate bounds checking, etc. Just a quick test. Let me know if there's a better approach. I think this fulfills my need to be able to decompress a large compressed file in smaller chunks in order to not have memory allocation issues. I'll test with a large file once I get a chance to write the real thing.

Post Reply