urllib / urequests for Micropython

[lucien2k]

With some help from Daniel and Damien, the issues are solved and a new version has been pushed.

I didn't realised we are so limited on memory, the new updates will help but there will be problems loading a page if it is bigger than about 10-20kb.

[danicampora]

Thanks for the update Alex

Well, memory is always limited in embedded systems. Anyway, like you said, the main use of this lib is for APIs, and the memory requirements for that are much lower.

Cheers,
Daniel

[Damien]

(This topic is now moved to "Programs, Libraries and Tools".)

I'd suggest adding this urllib implementation to the micropython-lib repo: https://github.com/micropython/micropython-lib Then it can be used by all platforms.

[pfalcon]

Code: Select all

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, socket.IPPROTO_SEC)

danicampora, Can you please do something about this? There should be a way. For example, you could defer creation of the underlying socket until it will be know if user intends to use it with SSL or not. Unfortunately, that's the price to pay for using proprietary vendor libraries - they should be worked around, and if not in C code, then by actual users, and that's much worse choice.

[pfalcon]

As for the lib itself, it's neat, and gives good answer to questions raised in this thread: http://forum.micropython.org/viewtopic.php?f=15&t=1039: "There's no HTTP lib for micropython" - "yes, but you can write one in half an hour". Indeed, that's easy. What's hard is to write a lib which works for everyone, with everyone having varying conditions, as discussion on this thread already shows. That's why I didn't try to write one for micropython-lib so far.

One thing I can ask against right away is calling it "urllib" - it is very far away from CPython's urllib, so such name is misleading and will conflict as soon as more "real" urllib will be provided. YMMV.

As for adding something similar to micropython-lib (called e.g. uurllib), I'm all for it, but code writing will be 5% of the effort required for this, and 95% will be discussion of what should go into it. I'm already driving bunch of such discussions, and can't drive this one. If someone is interested, please do (suggestion is to start with *absolutely* minimal API, as memory constraints of different systems is indeed the biggest problem here).

[danicampora]

danicampora, Can you please do something about this? There should be a way. For example, you could defer creation of the underlying socket until it will be know if user intends to use it with SSL or not. Unfortunately, that's the price to pay for using proprietary vendor libraries - they should be worked around, and if not in C code, then by actual users, and that's much worse choice

I saw this one coming... the problem with deferring socket creation is that people should be able to set socket options and do all kind of stuff before converting the socket into a secure one, then what to do? create a queue with all this deferred operations? I rather not go this way...

I know the current way is not ideal, but TI is already working on letting the CC3200 turn a normal socket into a secure one at any point, just like with most IP stacks, but for the moment this is what we have. Could we as a workaround do:

Code: Select all

import os
if 'CC3200' in os.uname().machine:
    # create 'special' secure socket for the WiPy
else:
    ...

??

[pfalcon]

I saw this one coming... the problem with deferring socket creation is that people should be able to set socket options and do all kind of stuff before converting the socket into a secure one, then what to do? create a queue with all this deferred operations? I rather not go this way...

That, or perhaps just cache those options in python socket structure. Not to pleasant, but if TI wanted you to go that way, why wonder that's the way. How many sockets you expect to be open at the same time? I remember that number of SSL sockets is artificially limited to some small number, like 3-4, and for normal sockets, buffers is limiting factor anyway. So, even if socket structure grows by 20 bytes, it's not going to change much.

I know the current way is not ideal, but TI is already working on letting the CC3200 turn a normal socket into a secure one at any point, just like with most IP stacks, but for the moment this is what we have.

Good, then you will be able to release wipy 1.5 or 2.0, and write in changelog: "Sockets require less memory!". But at all times wipy will be Python- and MicroPython-compatible.


Could we as a workaround do:

Code: Select all

import os
if 'CC3200' in os.uname().machine:
    # create 'special' secure socket for the WiPy
else:
    ...

??

I'd prefer not to add such stuff to micropython-lib. Of course, for single module, it can be done in the name of cross-port compatibility. But the point is that it needs to be done *every* time when working with SSL sockets. And wipy folks of course won't be doing that, producing code which is not Python compatible. And other folks won't be doing that, and produce code which is not wipy compatible.

[pfalcon]

As for adding something similar to micropython-lib (called e.g. uurllib), I'm all for it, but code writing will be 5% of the effort required for this, and 95% will be discussion of what should go into it. I'm already driving bunch of such discussions, and can't drive this one. If someone is interested, please do (suggestion is to start with *absolutely* minimal API, as memory constraints of different systems is indeed the biggest problem here).

Ok, I actually gave that some thought. The first big issue is how to name it. In Python3, "urllib" is a package. What corresponds to Python'2 urllib/urllib2 in Python3 is urllib.request. And micropython-lib supports separate modules of same package (effectively, namespace packages). Then, urllib.urequest sounds like a good name for discussed minimal module.

Regarding API, it's fair to say that the minimal module should just provide urlopen() function, nothing else - no URLOpen, not Request, no nothing. The problem is that standard urlopen() doesn't allow to specify extra headers. That's btw another reason I didn't haste to implement urllib - I thought about implementing "requests" lib work-alike instead (especially as there're talks that requests may be merged into stdlib).

[pfalcon]

Ah, and let's not forget that urllib isn't only about HTTP - it supports FTP, etc. So, another reason I avoided it and started with http.client instead (which I kinda even made work (CPython version), though it's useless of course - hundreds of kilobytes of bloat).

[Damien]

the problem with deferring socket creation is that people should be able to set socket options and do all kind of stuff before converting the socket into a secure one, then what to do? create a queue with all this deferred operations? I rather not go this way...

I agree it secure socket creation needs to be fixed so that the same code runs under CPython. That really is the aim of all uPy code, to be able to write scripts that run under CPython and all uPy ports and have the same behaviour. You may have restrictions on what you can write, but there should never be incompatibilities.

So how about the following: on WiPy you force the user to create an ssl socket straight after creating the normal socket (or at least, there can be no method calls to the socket between creation and wrapping). Ie:

Code: Select all

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock = ssl.wrap_socket(sock, ...)
# use sock as usual, set timeout, connect, bind, etc

And then you educate people that this is the way to create secure sockets. It works correctly in CPython, and can be made to work easily with WiPy without much (any?) extra RAM usage.

When WiPy 1.5 or 2.0 comes out, then the same code will work, but there won't be the restriction of having to wrap straight away.