usocket UDP and sockaddr questions

Discussion about programs, libraries and tools that work with MicroPython. Mostly these are provided by a third party.
Target audience: All users and developers of MicroPython.
SpotlightKid
Posts: 416
Joined: Wed Apr 08, 2015 5:19 am

usocket UDP and sockaddr questions

Post by SpotlightKid » Fri Nov 20, 2015 11:19 am

I'm implementing a UDP socket server (and client) with micropython for the OSC (http://opensoundcontrol.org) protocol using the usocket module.

I've come across a few limitations of the usocket module regarding UDP and I'm not sure how to handle them.

1. Is there a convenient way to convert the soackaddr struct returned by recvfrom() into a (ip, port) tuple (e.g. for logging)?

Or do I have to use ustruct to parse the struct? How do I convert the in_addr member to the dotted IP address representation then since inet_ntop resp. inet_ntoa are not implemented? Does anyone have or know a pure Python implementation of one of these functions?

(I know I can just pass the binary sockaddr struct to sendto() for replying.)

2. Is there a special reason recvfrom_into() is not implemented?

3. When I was looking at the usocket documention and trying to figure out why

Code: Select all

sock.bind(('127.0.0.1', 9000))
was giving me a "TypeError: object with buffer protocol required", I was confused, because it says: "The format of address is: (ipv4 address, port)"

Only when you look at the getaddrinfo() documentation, you realize that usocket, unlike CPython's socket object, does not take a (host, port) tuple for bind() and connect() but a binary sockaddr structure. I think this should be explicitly mentioned and the misleading sentence quoted above amended. I see that micropython-socket has a convenience function create_connection(), which handles the address conversion automatically. Maybe it would helpful to add this to the connect and bind methods of the socket wrapper in this module as well (for AF_INET sockets)? [Update: ups, sorry, it already does!]

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: usocket UDP and sockaddr questions

Post by pfalcon » Fri Nov 20, 2015 1:31 pm

Well, you wrote quite a detailed message, but didn't tell which MicroPython port you use.
Only when you look at the getaddrinfo() documentation, you realize that usocket, unlike CPython's socket object
Each port has own documentation, are you sure you looked at the documentation for the right port?

And even then, documentation may indeed be not complete/not emphasize important points, and generally need some love. We need help with that, because developers, giving a chance to write code or documentation almost always choose to write code. So, users who actually start to use that code and face missing/unclear documentation are in the best position to lead on fixing it (for developers, it all looks pretty obvious).
Maybe it would helpful to add this to the connect and bind methods of the socket wrapper in this module as well (for AF_INET sockets)? [Update: ups, sorry, it already does!]
Indeed, it does. The general idea is that "usocket" is MicroPython-specific module, and there're special rules how to use it (all forward-compatible with CPython though, so code written for usocket should work the same with CPython's socket). Docs on that needs to be more clear, patches welcome.

But if you want to use something compatible with CPython right away, micropython-lib offers proper "socket" module, which is targeted to be as much CPython compatible as makes sense (CPython's socket module is *very* bloated). It's not yet there, but as you can see, it already does more than you think it might.
1. Is there a convenient way to convert the soackaddr struct returned by recvfrom() into a (ip, port) tuple (e.g. for logging)?
Cast your vote here: https://github.com/micropython/micropython/issues/1375

Generally, development of MicroPython is driven by actual usecases and actual people driving implementation of them. So far, only myself was interested in that stuff, so despite pretty big (as for one man) work done, there're lot of gaps here and there. I'm glad that now more people interested in that, just mind that I can't drive it all, other people should contribute (and contribute not just "any" solution, but the "best", as that's the motto of MicroPython).
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

SpotlightKid
Posts: 416
Joined: Wed Apr 08, 2015 5:19 am

Re: usocket UDP and sockaddr questions

Post by SpotlightKid » Fri Nov 20, 2015 2:23 pm

I'm currently implementing this on the unix port, but the ultimate goal would be to have a module, that works with the usocket implementation of the stmhal port (with low-level IP functionality provided by some other module, e.g. slip). Possibly enhanced by micropython-lib's socket module.
Each port has own documentation, are you sure you looked at the documentation for the right port?
Not really. The docs I looked at were http://docs.micropython.org/en/latest/l ... -libraries resp. http://docs.micropython.org/en/latest/l ... ocket.html . There's no mention that this is specific to a certain port. Are you saying, that these docs are only valid for the pyboard/stmhal port? Where do I find the docs for the unix port then? I don't see it linked anywhere.

I hope that I don't give the impression that I'm only complaining about the documentation here. I'm just reporting the difficulties I'm experiencing in finding the information I need to implement my stuff. Maybe I'm just too dumb ;)


Regarding inet_ntoa / inet_ntop, I'm trying to use ffi (though that only helps on the unix port):

Code: Select all

# wrappers from mciropython-lib
import socket
import struct

inet_ntoa = getattr(socket, 'inet_ntoa', None)

if not inet_ntoa:
    import ffilib
    inet_ntoa = ffilib.libc().func("s", "inet_ntoa", "p")

def get_hostport(sockaddr):
    af, port, addr = struct.unpack('>hHL', sockaddr[:8])
    return inet_ntoa(addr), port
But I get the dotted IP address backwards (e.g. ('1.0.0.127', 48301)). I'm not sure I interpret the struct correctly. I used this as a reference: http://www.retran.com/beej/sockaddr_inman.html

And I can't figure out the correct way to set up/call inet_ntop. Again the main hurdle is lacking documentation. I tried to find more info on the ffi module, but could only find the few tests and examples in the repo, which do not cover pointers and buffers much.

SpotlightKid
Posts: 416
Joined: Wed Apr 08, 2015 5:19 am

Re: usocket UDP and sockaddr questions

Post by SpotlightKid » Fri Nov 20, 2015 2:39 pm

SpotlightKid wrote:Regarding inet_ntoa / inet_ntop, I'm trying to use ffi (though that only helps on the unix port):
Ok, actually, this is easier (assuming AF_INET):

Code: Select all

def get_hostport(sockaddr):
    port = struct.unpack('>H', sockaddr[2:4])[0]
    ip = '%i.%i.%i.%i' % tuple(sockaddr[4:8])
    return ip, port

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: usocket UDP and sockaddr questions

Post by pfalcon » Fri Nov 20, 2015 2:58 pm

Not really. The docs I looked at were http://docs.micropython.org/en/latest/l ... -libraries resp. http://docs.micropython.org/en/latest/l ... ocket.html . There's no mention that this is specific to a certain port.
If you click on "Docs" breadcrumbs on pages you ink above, you'll notice it links too
http://docs.micropython.org/en/latest/p ... index.html . And if you start http://docs.micropython.org/en/latest/ , like you likely did, it says explicitly:

You are currently viewing the documentation for the pyboard.


Yes, all the above isn't good enough. But then it's not the case that it doesn't say that it's PyBoard's documentation either.
Are you saying, that these docs are only valid for the pyboard/stmhal port? Where do I find the docs for the unix port then? I don't see it linked anywhere.
Well, there was a link somewhere which you need to know beforehand. Surely, that doesn't work, I for example don't remember it now, and can't tell you.
I hope that I don't give the impression that I'm only complaining about the documentation here. I'm just reporting the difficulties I'm experiencing in finding the information I need to implement my stuff. Maybe I'm just too dumb
No, you raise very valid issues, and I'm not surprised about them at all, because I raised them previously to Damien myself. But then it was ~ only me who looked at that. So, if you face it now, please go to uPy bugtracker and submit bug titled "Cannot easily find unix port documentation", to let developers know that it's now real, not speculative, issue.
Regarding inet_ntoa / inet_ntop,
Again, the first best thing you can do about that is to add comment to the ticket I linked above "I face this issue too!" We don't want everyone to dig it on their own at different times, we want to resolve it once and for all (even if not right away).
I tried to find more info on the ffi module, but could only find the few tests and examples in the repo, which do not cover pointers and buffers much.
The first place to look at should be a source of a module. And ffi module specifically is used a lot in micropython-lib, you can look there for various usage examples.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

SpotlightKid
Posts: 416
Joined: Wed Apr 08, 2015 5:19 am

Re: usocket UDP and sockaddr questions

Post by SpotlightKid » Fri Nov 20, 2015 3:06 pm

pfalcon wrote:The first place to look at should be a source of a module.
I tried, but it's a bit past my C expertise.
pfalcon wrote:And ffi module specifically is used a lot in micropython-lib, you can look there for various usage examples.
Allright, it didn't occur to me to look there :(

So, I always was under the impression, that the modules in micropython-lib should be potentially usable on the pyboard as well (within memory constraints). But if ffi is used library, that can't be the case, can it?

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: usocket UDP and sockaddr questions

Post by pfalcon » Fri Nov 20, 2015 3:19 pm

I tried, but it's a bit past my C expertise.
Your C expertise doesn't let you read comments in English: https://github.com/micropython/micropyt ... dffi.c#L39 ?
So, I always was under the impression, that the modules in micropython-lib should be potentially usable on the pyboard as well (within memory constraints). But if ffi is used library, that can't be the case, can it?
Each module is a world of its own. Some can run on pyboard, some can't, for some it wouldn't even make sense. Generally, micropython-lib is primarily intended for unix port. That doesn't mean it can't work on pyboard - it means someone yet should care to try it, maintain that, and improve that. (I from my side, as a maintainer, care about the most important part on a way to that - to keep modules small, but that of course conflicts with CPython compatibility).
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

SpotlightKid
Posts: 416
Joined: Wed Apr 08, 2015 5:19 am

Re: usocket UDP and sockaddr questions

Post by SpotlightKid » Fri Nov 20, 2015 3:46 pm

pfalcon wrote:Your C expertise doesn't let you read comments in English: https://github.com/micropython/micropyt ... dffi.c#L39 ?
It actually is sufficient to infer most of the information in this comment* from https://github.com/micropython/micropyt ... ffi.c#L101, but not to apply it to the info found in inet_ntop(3):

Code: Select all

       #include <arpa/inet.h>

       const char *inet_ntop(int af, const void *src, char *dst, socklen_t size);
I also wasn't sure whether I should pass a bytes or a bytearray instance where a buffer is required, but I see now, that the pcre module uses bytes.

* BTW, the info in that comment would make a great start for a README in micropython-ffilib! :)

pfalcon
Posts: 1155
Joined: Fri Feb 28, 2014 2:05 pm

Re: usocket UDP and sockaddr questions

Post by pfalcon » Fri Nov 20, 2015 3:55 pm

I also wasn't sure whether I should pass a bytes or a bytearray instance where a buffer is required, but I see now, that the pcre module uses bytes.
If you know Python, it should be pretty clear, that you use "bytes" for input (read only) parameter, and bytearray for output (read/write) parameter.
* BTW, the info in that comment would make a great start for a README in micropython-ffilib!
No, for docs/library/ffi.rst file. Patches welcome.
Awesome MicroPython list
Pycopy - A better MicroPython https://github.com/pfalcon/micropython
MicroPython standard library for all ports and forks - https://github.com/pfalcon/micropython-lib
More up to date docs - http://pycopy.readthedocs.io/

SpotlightKid
Posts: 416
Joined: Wed Apr 08, 2015 5:19 am

Re: usocket UDP and sockaddr questions

Post by SpotlightKid » Fri Nov 20, 2015 4:01 pm

pfalcon wrote:If you know Python, it should be pretty clear, that you use "bytes" for input (read only) parameter, and bytearray for output (read/write) parameter.
Sorry, I program Python for almost 20 years, but I don't see how that's "pretty clear". To a C programmer, maybe.

Post Reply