urequests - any way to download files by chunks

[0_djek]

Hello,
Is there any way to download files from web by chunks? In normal python, you can use

Code: Select all

requests.get(url, stream=True)

and then you iterate over that response chunk by chunk. Is there any way to do the same in micropython, or you can only downlaod whole file at once? urequests library seems to accept stream as an argument, but it doesn't do anything with it.

[SpotlightKid]

The sad truth is, that urequests from micropython-lib (which is also included in some older official firmware builds) is unmaintained, outdated, incomplete and buggy.

I've written an evolved and improved version called mrequests, which can do what you ask:

Code: Select all

>>> import mrequests
>>> r = mrequests.get('http://httpbin.org/image', headers={'accept': 'image/png'})
>>> r.save("image.png")

You can also pass a chunk size to the save method as the second parameter, which specifies how many bytes are written from the socket and written to the file at once. The default is 1024.

You can also read and process the response in chunks manually instead:

Code: Select all

>>>while True:
>>>    chunk = r.read(chunksize)
>>>    if not chunk:
>>>        break
>>>    process(chunk)

mrequests is part of my micropython-stm-lib (but works - to different degrees - on all major MicroPython ports with network support):

https://github.com/SpotlightKid/mrequests/

[furia]

I am trying to download a binary file smaller than 1MB through urequests with basic authentication. The authentication part works and I get the expected response.

but the truth is I feel lost since I can't download the file I need and I have to do it through urequests.


your code works fine, but only for the image example.

Code: Select all

>>> import mrequests
>>> r = mrequests.get('http://httpbin.org/image', headers={'accept': 'image/png'})
>>> r.save("image.png")

I have not been able to use it to download a 500kb file

What can I do to be able to do something like this?

Code: Select all

>>> while True:
>>> chunk = r.read (chunksize)
>>> if not chunk:
>>> break
>>> process (chunk)

it would help me get a lot going since I'm stuck at this point.

[SpotlightKid]

Please show the actual code that doesn't work (please link to code, or copy and paste it exactly here, no screenshots or incomplete code) and also copy & paste the exact error message, if any.

[furia]

Please show the actual code that doesn't work (please link to code, or copy and paste it exactly here, no screenshots or incomplete code) and also copy & paste the exact error message, if any.

Currently I have been able to accomplish my purpose as follows:

Code: Select all

    import mrequests
    username = 'test'
    password = 'test'
    auth_str = '%s:%s' % (username, password)
    b64_auth_str = b2a_base64(auth_str)
    headers = {'Authorization': 'Basic %s' % b64_auth_str.decode('utf-8'),'accept': 'multipart/form-data'}
    url = "https://domainexample.com/file.bin"
    r = mrequests.get(url, headers=headers)
    r.save("file.csv",1024)
    r.close()

my stupid problem is that I had insisted on downloading a dynamic php file that did not return anything.

Has my methodology been correct? or does it only work because memory has not overflowed?

[SpotlightKid]

Yes, looks good. But instead of setting the Authorization header yourself, you can just pass auth=(username, password) as a keyword parameter to mrequests.get().

Using response.save(filename, chunk_size) should ensure that only as much memory as is needed for chunk_size bytes is allocated.

[mattyt]

Just a quick note that SpotlightKid moved his excellent mrequests library a little while ago so the link in this thread is broken; it's now at mrequests.

Perhaps we should consider consolidating urequest and mrequest? I would be happy to see mrequests replace urequest in micropython-lib or urequests simply removed from micropython-lib - it just seems like it will be painful to have them both exist...

[SpotlightKid]

I have updated the link to the mrequests repo above.