Trying to understand various ways to use this class. Anyone know of any nice examples out there? Google has failed me.
One of the things I am trying to figure out is how to decompress a large compressed chunk of data without using very large buffers.
I can seek to the compressed data in the file...and then I am faced with, say, 50k that expands to, say, 100k. I need to be able to do this in 1k to 4k chunks and store them in an uncompressed file. Not sure zlib.DecompIO() can do this. You can't just slice compressed data randomly and unpack it.
One option I see is to pre-process the original file to break it up into a bunch of smaller files.
In other words, take a 100K file and shard it into 100 files, each being 1K. The file names would contain a sequence number of some scheme like that. You then zip that up.
At the MicroPython level, you can then use zlib.decompress() to expand each sub-file into a reasonably sized buffer. From there you simply build-up the final uncompressed file through multiple (100 in this case) append writes.
Crazy? I am hoping there's a slick way to do this that might not require any such pre-processing.
Playing with code right now, just wondering if anyone might know of useful usage examples.
Any zlib.DecompIO examples?
Re: Any zlib.DecompIO examples?
Well, it was easier than I thought. Here is it for the benefit of others who might need to figure this out:
Of course, this doesn't do appropriate bounds checking, etc. Just a quick test. Let me know if there's a better approach. I think this fulfills my need to be able to decompress a large compressed file in smaller chunks in order to not have memory allocation issues. I'll test with a large file once I get a chance to write the real thing.
Code: Select all
import zlib
compressed_text = b'\xf3LS\xa8\xcc/U\xc8*-.Q(O\xcc+Q(\xc9W(JMLQ\xc8/R(/\xca,IUHTH\xcb\xccI\x05\x00'
uncompressed_text = b'If you just want to read or write a file'
# Create file with compressed data
fp = open("test.text", "w")
fp.write(compressed_text)
fp.close()
# Uncompress it in small chunks
chunk_size = 16
fp = open("test.txt", "rb")
fp.seek(0, 2) # Go to end of file
file_size = fp.tell() # Store length
fp.seek(0) # Back to the start of the file
decompressor = zlib.DecompIO(fp, -1)
result = bytearray()
while True:
uncompressed_chunk = decompressor.read(chunk_size)
if uncompressed_chunk:
result += uncompressed_chunk
else:
break
print(f"original: {uncompressed_text}")
print(f"result: {bytes(result)}")