Page 1 of 4
urllib / urequests for Micropython
Posted: Thu Oct 29, 2015 1:23 pm
by lucien2k
UPDATE for moderators: urllib.urequest and urequests modules are now available in the MicroPython's standard library, micropython-lib:
=========================================================
I couldn't find a urllib library for Micropython, so I have written one.
Available at
https://github.com/lucien2k/wipy-urllib
Currently it supports:
1. HTTP GET and POST
2. 301 and 302 redirects
It works in a similar way to urllib:
Code: Select all
import urllib
f=urllib.urlopen('http://www.microsoft.com')
f.code
f.read()
More work required, but this is a starting point. I am mainly aiming to use it for APIs (pushing data etc).
Update 30th October:
- Fixed a few of the issues that were causing memory problems. It is important to note that memory is limited on the wipy though, so you may run into problems if you try to read bigger sites.
- Thanks to Damien and Daniel for helping with the problems I ran into.
Update 8th November:
- Added a new version called urequests that works in a similar way to the python-requests library.
- Available at
https://github.com/lucien2k/wipy-urllib ... equests.py /
https://pypi.python.org/pypi/urequests
Re: urllib for Micropython
Posted: Thu Oct 29, 2015 1:28 pm
by danicampora
Hi lucien2k,
Good job, I'll check it out!
A few comments:
SSL support, this works fine in normal python but ussl appears to not be functional.
The socket needs to be created slightly different than in CPython. There are notes about this in the docs and can also be seen in the smtplib than I posted here in the forum last week.
Unicode is not supported by the wipy
It is supported, so I'll investigate.
Re: urllib for Micropython
Posted: Thu Oct 29, 2015 1:33 pm
by danicampora
In order to make ssl work, change line 27:
Code: Select all
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
to (as noted in:
http://micropython.org/resources/docs/e ... #functions):
Code: Select all
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, socket.IPPROTO_SEC)
I'll make more explicit that this is an important difference between the WiPy and CPython.
Re: urllib for Micropython
Posted: Thu Oct 29, 2015 1:44 pm
by danicampora
I just had a quick look and looks very neat!
I'll try to finish the work on the smtplib so that we can start making the libraries folder grow!
Cheers,
Daniel
Re: urllib for Micropython
Posted: Thu Oct 29, 2015 2:07 pm
by lucien2k
Confirmed on the SSL, new version committed with that fix.
I'll see if I can put together some test cases for the weird memory errors I got while testing. I think it might be unicode related too.
Re: unicode support, it is mostly related to the "unicode" type (doesn't seem to exist).
Re: urllib for Micropython
Posted: Thu Oct 29, 2015 2:24 pm
by danicampora
In Python 3 there's no unicode type, every string is "unicode" by default. I'd suggest checking the Python 3 version of the standard urlib in order to see how it's handled there.
Re: urllib for Micropython
Posted: Thu Oct 29, 2015 3:04 pm
by lucien2k
Ah of course, I don't work in python3 much so I completely forgot about that.
Re: urllib for Micropython
Posted: Thu Oct 29, 2015 3:26 pm
by kfricke
I would suggest moving this topic to "Programs, Libraries and Tools"
Re: urllib for Micropython
Posted: Thu Oct 29, 2015 3:50 pm
by mbirth
I also wanted to start a HTTP module (need GET and POST to communicate via Open Spherical Camera API) and just finished the URL parser which is somewhat oriented on the standard urllib.parse module. Maybe you can make use of it:
https://github.com/mbirth/wipy-theta/bl ... er/http.py
Re: urllib for Micropython
Posted: Fri Oct 30, 2015 12:02 am
by lucien2k
Just to update, I PM'd my example to Daniel.
There is definitely something a bit weird and it manifests as memory issues occasionally. This library does encounter problems with some sites, but works great for simple stuff. I don't get those problems with CPython and obviously the wipy has limited memory but it seems to run into those problems before I would expect (we're talking 3-4kb pages rather than 100kb+).
That said it works fine with APIs, which I suspect are the major use case for this.