decompressing a gzip file using uzlib

[spierepf]

I am trying to decompress a gzip byte array using uzlib.decompress.

I started with

Code: Select all

$ cat /dev/null | gzip -f | hexdump -C
00000000  1f 8b 08 00 da e8 81 5e  00 03 03 00 00 00 00 00  |.......^........|
00000010  00 00 00 00                                       |....|
00000014

and attempted the following in micropython

Code: Select all

$ micropython
MicroPython v1.11 on 2020-03-29; linux version
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> import uzlib
>>> print(uzlib.decompress(b'\x1f\x8b\x08\x00\xda\xe8\x81\x5e\x00\x03\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: -3
>>> 

What am I missing?

[Christian Walther]

My guess is that uzlib.decompress doesn’t expect a gzip header. Docs say uzlib.DecompIO does, so maybe try that, or parse the header yourself, or if you have control over the input use a zlib header or a raw deflate stream. See RFC 1950, 1951, 1952 for specs.

[spierepf]

I tried that:

Code: Select all

$ micropython
MicroPython v1.11 on 2020-03-29; linux version
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> import io
>>> import uzlib
>>> file = io.BytesIO(b'\x1f\x8b\x08\x00\xda\xe8\x81\x5e\x00\x03\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00')
>>> decomp_file = uzlib.DecompIO(file)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: compression header
>>> 

Unfortunately, I don't have much control over the input. My options are .tar.gz or .zip.

[Christian Walther]

A wbits argument of 24..31 is required to indicate a gzip header:

Code: Select all

MicroPython v1.12 on 2019-12-20; TinyPICO with ESP32-PICO-D4
Type "help()" for more information.
>>> import io
>>> import uzlib
>>> file = io.BytesIO(b'\x1f\x8b\x08\x00\xda\xe8\x81\x5e\x00\x03\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00')
>>> decomp_file = uzlib.DecompIO(file, 31)
>>> decomp_file.read()
b''

See also what upip does to decompress .tar.gz.