urllib / urequests for Micropython

Discussion about programs, libraries and tools that work with MicroPython. Mostly these are provided by a third party.
Target audience: All users and developers of MicroPython.
lucien2k
Posts: 24
Joined: Sun Oct 11, 2015 5:34 pm

urllib / urequests for Micropython

Post by lucien2k » Thu Oct 29, 2015 1:23 pm

UPDATE for moderators: urllib.urequest and urequests modules are now available in the MicroPython's standard library, micropython-lib:
=========================================================

I couldn't find a urllib library for Micropython, so I have written one.

Available at https://github.com/lucien2k/wipy-urllib

Currently it supports:
1. HTTP GET and POST
2. 301 and 302 redirects

It works in a similar way to urllib:

Code: Select all

import urllib
f=urllib.urlopen('http://www.microsoft.com')
f.code
f.read()
More work required, but this is a starting point. I am mainly aiming to use it for APIs (pushing data etc).

Update 30th October:
- Fixed a few of the issues that were causing memory problems. It is important to note that memory is limited on the wipy though, so you may run into problems if you try to read bigger sites.
- Thanks to Damien and Daniel for helping with the problems I ran into.

Update 8th November:
- Added a new version called urequests that works in a similar way to the python-requests library.
- Available at https://github.com/lucien2k/wipy-urllib ... equests.py / https://pypi.python.org/pypi/urequests
Last edited by lucien2k on Sun Nov 08, 2015 6:00 pm, edited 6 times in total.

User avatar
danicampora
Posts: 342
Joined: Tue Sep 30, 2014 7:20 am
Contact:

Re: urllib for Micropython

Post by danicampora » Thu Oct 29, 2015 1:28 pm

Hi lucien2k,

Good job, I'll check it out!

A few comments:
SSL support, this works fine in normal python but ussl appears to not be functional.
The socket needs to be created slightly different than in CPython. There are notes about this in the docs and can also be seen in the smtplib than I posted here in the forum last week.
Unicode is not supported by the wipy
It is supported, so I'll investigate.

User avatar
danicampora
Posts: 342
Joined: Tue Sep 30, 2014 7:20 am
Contact:

Re: urllib for Micropython

Post by danicampora » Thu Oct 29, 2015 1:33 pm

In order to make ssl work, change line 27:

Code: Select all

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
to (as noted in: http://micropython.org/resources/docs/e ... #functions):

Code: Select all

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, socket.IPPROTO_SEC)
I'll make more explicit that this is an important difference between the WiPy and CPython.

User avatar
danicampora
Posts: 342
Joined: Tue Sep 30, 2014 7:20 am
Contact:

Re: urllib for Micropython

Post by danicampora » Thu Oct 29, 2015 1:44 pm

I just had a quick look and looks very neat! :-)

I'll try to finish the work on the smtplib so that we can start making the libraries folder grow! :-)

Cheers,
Daniel

lucien2k
Posts: 24
Joined: Sun Oct 11, 2015 5:34 pm

Re: urllib for Micropython

Post by lucien2k » Thu Oct 29, 2015 2:07 pm

Confirmed on the SSL, new version committed with that fix.

I'll see if I can put together some test cases for the weird memory errors I got while testing. I think it might be unicode related too.

Re: unicode support, it is mostly related to the "unicode" type (doesn't seem to exist).

User avatar
danicampora
Posts: 342
Joined: Tue Sep 30, 2014 7:20 am
Contact:

Re: urllib for Micropython

Post by danicampora » Thu Oct 29, 2015 2:24 pm

In Python 3 there's no unicode type, every string is "unicode" by default. I'd suggest checking the Python 3 version of the standard urlib in order to see how it's handled there.

lucien2k
Posts: 24
Joined: Sun Oct 11, 2015 5:34 pm

Re: urllib for Micropython

Post by lucien2k » Thu Oct 29, 2015 3:04 pm

Ah of course, I don't work in python3 much so I completely forgot about that.

User avatar
kfricke
Posts: 342
Joined: Mon May 05, 2014 9:13 am
Location: Germany

Re: urllib for Micropython

Post by kfricke » Thu Oct 29, 2015 3:26 pm

I would suggest moving this topic to "Programs, Libraries and Tools"

User avatar
mbirth
Posts: 25
Joined: Mon Oct 20, 2014 1:00 pm
Location: Berlin, Germany
Contact:

Re: urllib for Micropython

Post by mbirth » Thu Oct 29, 2015 3:50 pm

I also wanted to start a HTTP module (need GET and POST to communicate via Open Spherical Camera API) and just finished the URL parser which is somewhat oriented on the standard urllib.parse module. Maybe you can make use of it: https://github.com/mbirth/wipy-theta/bl ... er/http.py
pyBoard v1.0 + LCD32MKv1.0 | WiPy + Expansion Board | GitHub

lucien2k
Posts: 24
Joined: Sun Oct 11, 2015 5:34 pm

Re: urllib for Micropython

Post by lucien2k » Fri Oct 30, 2015 12:02 am

Just to update, I PM'd my example to Daniel.

There is definitely something a bit weird and it manifests as memory issues occasionally. This library does encounter problems with some sites, but works great for simple stuff. I don't get those problems with CPython and obviously the wipy has limited memory but it seems to run into those problems before I would expect (we're talking 3-4kb pages rather than 100kb+).

That said it works fine with APIs, which I suspect are the major use case for this.

Post Reply