Page 1 of 1
urequests - any way to download files by chunks
Posted: Wed May 19, 2021 12:25 pm
by 0_djek
Hello,
Is there any way to download files from web by chunks? In normal python, you can use
and then you iterate over that response chunk by chunk. Is there any way to do the same in micropython, or you can only downlaod whole file at once? urequests library seems to accept stream as an argument, but it doesn't do anything with it.
Re: urequests - any way to download files by chunks
Posted: Wed May 19, 2021 7:40 pm
by SpotlightKid
The sad truth is, that
urequests from
micropython-lib (which is also included in
some older official firmware builds) is unmaintained, outdated, incomplete and buggy.
I've written an evolved and improved version called
mrequests, which can do what you ask:
Code: Select all
>>> import mrequests
>>> r = mrequests.get('http://httpbin.org/image', headers={'accept': 'image/png'})
>>> r.save("image.png")
You can also pass a chunk size to the save method as the second parameter, which specifies how many bytes are written from the socket and written to the file at once. The default is 1024.
You can also read and process the response in chunks manually instead:
Code: Select all
>>>while True:
>>> chunk = r.read(chunksize)
>>> if not chunk:
>>> break
>>> process(chunk)
is part of my
micropython-stm-lib (but works - to different degrees - on all major MicroPython ports with network support):
https://github.com/SpotlightKid/mrequests/
Re: urequests - any way to download files by chunks
Posted: Wed Aug 11, 2021 4:44 pm
by furia
I am trying to download a binary file smaller than 1MB through urequests with basic authentication. The authentication part works and I get the expected response.
but the truth is I feel lost since I can't download the file I need and I have to do it through urequests.
your code works fine, but only for the image example.
Code: Select all
>>> import mrequests
>>> r = mrequests.get('http://httpbin.org/image', headers={'accept': 'image/png'})
>>> r.save("image.png")
I have not been able to use it to download a 500kb file
What can I do to be able to do something like this?
Code: Select all
>>> while True:
>>> chunk = r.read (chunksize)
>>> if not chunk:
>>> break
>>> process (chunk)
it would help me get a lot going since I'm stuck at this point.
Re: urequests - any way to download files by chunks
Posted: Wed Aug 11, 2021 6:12 pm
by SpotlightKid
Please show the actual code that doesn't work (please link to code, or copy and paste it exactly here, no screenshots or incomplete code) and also copy & paste the exact error message, if any.
Re: urequests - any way to download files by chunks
Posted: Wed Aug 11, 2021 6:52 pm
by furia
SpotlightKid wrote: ↑Wed Aug 11, 2021 6:12 pm
Please show the actual code that doesn't work (please link to code, or copy and paste it exactly here, no screenshots or incomplete code) and also copy & paste the exact error message, if any.
Currently I have been able to accomplish my purpose as follows:
Code: Select all
import mrequests
username = 'test'
password = 'test'
auth_str = '%s:%s' % (username, password)
b64_auth_str = b2a_base64(auth_str)
headers = {'Authorization': 'Basic %s' % b64_auth_str.decode('utf-8'),'accept': 'multipart/form-data'}
url = "https://domainexample.com/file.bin"
r = mrequests.get(url, headers=headers)
r.save("file.csv",1024)
r.close()
my stupid problem is that I had insisted on downloading a dynamic php file that did not return anything.
Has my methodology been correct? or does it only work because memory has not overflowed?
Re: urequests - any way to download files by chunks
Posted: Wed Aug 11, 2021 7:39 pm
by SpotlightKid
Yes, looks good. But instead of setting the Authorization header yourself, you can just pass auth=(username, password) as a keyword parameter to mrequests.get().
Using response.save(filename, chunk_size) should ensure that only as much memory as is needed for chunk_size bytes is allocated.
Re: urequests - any way to download files by chunks
Posted: Thu Aug 12, 2021 2:08 am
by mattyt
Just a quick note that SpotlightKid moved his excellent mrequests library a little while ago so the link in this thread is broken; it's now at
mrequests.
Perhaps we should consider consolidating urequest and mrequest? I would be happy to see mrequests replace urequest in micropython-lib or urequests simply removed from micropython-lib - it just seems like it will be painful to have them both exist...
Re: urequests - any way to download files by chunks
Posted: Thu Aug 12, 2021 5:43 am
by SpotlightKid
I have updated the link to the mrequests repo above.