New uasyncio breaks code (websocketserver)

All ESP8266 boards running MicroPython.
Official boards are the Adafruit Huzzah and Feather boards.
Target audience: MicroPython users with an ESP8266 board.
Post Reply
jomas
Posts: 59
Joined: Mon Dec 25, 2017 1:48 pm
Location: Netherlands

New uasyncio breaks code (websocketserver)

Post by jomas » Sun Apr 05, 2020 4:54 pm

The new uasyncio breaks the code of the websocket server example from micropython-lib

Here the code that shows the problem. I adapted the function 'echo' a bit to show the problem:

Code: Select all

import uasyncio
import uhashlib, ubinascii
import websocket

def make_respkey(webkey):
    d = uhashlib.sha1(webkey)
    d.update(b"258EAFA5-E914-47DA-95CA-C5AB0DC85B11")
    respkey = d.digest()
    respkey = ubinascii.b2a_base64(respkey) #[:-1]
    return respkey


class WSWriter:

    def __init__(self, reader, writer):
        # Reader is passed for symmetry with WSReader() and ignored.
        self.s = writer

    async def awrite(self, data):
        assert len(data) < 126
        await self.s.awrite(b"\x81")
        await self.s.awrite(bytes([len(data)]))
        await self.s.awrite(data)


def WSReader(reader, writer):
        webkey = None
        while 1:
            l = yield from reader.readline()
            print(l)
            if not l:
                raise ValueError()
            if l == b"\r\n":
                break
            if l.startswith(b'Sec-WebSocket-Key'):
                webkey = l.split(b":", 1)[1]
                webkey = webkey.strip()

        if not webkey:
            raise ValueError("Not a websocker request")

        respkey = make_respkey(webkey)

        await writer.awrite(b"""\
HTTP/1.1 101 Switching Protocols\r
Upgrade: websocket\r
Connection: Upgrade\r
Sec-WebSocket-Accept: """)
        await writer.awrite(respkey)
        await writer.awrite("\r\n")
        print("Finished webrepl handshake")
        if '__version__' in dir(uasyncio):
            print("New uasyncio")
            ws = websocket.websocket(reader.s)
            rws = uasyncio.StreamReader(reader.s, ws)
        else:
            print("Old uasyncio")
            ws = websocket.websocket(reader.ios)
            rws = uasyncio.StreamReader(reader.ios, ws)
        return rws

def echo(reader, writer):
    yield from reader.readline() # Consume GET line
    reader = yield from WSReader(reader, writer)
    writer = WSWriter(reader, writer)
    while 1:
        l = yield from reader.read(256)
        print("received: ", l)
        if l == b"\r":
            await writer.awrite(b"\r\n")
        else:
            if l[0] == 0x81:
                await writer.awrite("Received %d junk-bytes" % len(l))
            else:
                await writer.awrite(l)

loop = uasyncio.get_event_loop()
loop.create_task(uasyncio.start_server(echo, "0.0.0.0", 80))
loop.run_forever()
loop.close()

To test this echo-server I used http://www.websocket.org/echo.html (Don't use https://www.websocket.org/echo.html)
With the old uasyncio I can receive (and then send back) data as expected.

With the new uasyncio, connection is established, but then when sending data from the client, I receive what looks like a raw websocket-packet in l.
It always starts with 0x81, 0x83 (length, when sending 3 bytes) and then some random bytes.
These random bytes differ when sending the same data from the client. (only the first 2 bytes are the same.)

Am I doing something wrong? can someone help?

User avatar
Osipov Andrey
Posts: 9
Joined: Mon Apr 06, 2020 12:16 pm
Location: Saint-Petersburg, Russia

Re: New uasyncio breaks code (websocketserver)

Post by Osipov Andrey » Wed Apr 08, 2020 10:54 am

And following question:

How i can wrap this websocket server to ssl and use WebSocketSecure? :ugeek:

User avatar
tve
Posts: 216
Joined: Wed Jan 01, 2020 10:12 pm
Location: Santa Barbara, CA
Contact:

Re: New uasyncio breaks code (websocketserver)

Post by tve » Wed Apr 08, 2020 7:41 pm


jomas
Posts: 59
Joined: Mon Dec 25, 2017 1:48 pm
Location: Netherlands

Re: New uasyncio breaks code (websocketserver)

Post by jomas » Thu Apr 09, 2020 3:00 pm

@Osipov @tve, please open your own topic about ssl websockets.
I really am not interested in ssl websockets.

This topic is about the new uasyncio that is incompatible with the old uasyncio. And therefore does not work with websockets.

We have now:
1. New uasyncio which is incompatible with the old one, is slow (what I have heard), and has missing components.
2. Old uasyncio which has not been updated for about 2 years.
3. uasync of pythoncoder which is based on old or new uasyncio and therefore also breaks code.
4. uasyncio of pfalcon which is maintained by him, but can only be used with his own version of Micropython (PyCopy) and pycopy-lib.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

New uasyncio breaks code - the reasons

Post by pythoncoder » Fri Apr 10, 2020 4:19 pm

The new uasyncio V3 is generally fast, now that it is partially implemented in C. There is one aspect which I'm hoping will be made faster: I/O scheduling. But even in this respect it's twice as fast as uasyncio V2.

My fast_io version is a drop-in replacement for V2. I'm not aware of it breaking any code which runs under V2. I hope it will be obsoleted by the new version: it exists for two reasons. Firstly to provide workrounds for bugs in V2 which were never addressed despite PR's. Secondly to offer an option for fast I/O scheduling. The bugs are fixed in V3. If V3 gains an option to prioritise I/O I will deprecate fast_io.

V3 aims to be compatible with Python 3.8 syntax. V2 is not, so there are differences mainly confined to more advanced features such as task cancellation. Some event loop methods deprecated in CPython are no longer supported such as call_soon. Using yield in coroutines is now an error as is the case in CPython.

I am in the process of porting applications to V3 and it's not difficult.

Paul's version requires his unofficial firmware. I have little knowledge of its capabilities.

In my opinion the new uasyncio is a much better design.
Peter Hinch
Index to my micropython libraries.

User avatar
jimmo
Posts: 2754
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia
Contact:

Re: New uasyncio breaks code (websocketserver)

Post by jimmo » Tue Apr 14, 2020 2:31 am

@jomas

In addition to everything that Peter said (i.e. several people put a lot of work into trying to finally make a really good, compatible, asyncio implementation for MicroPython), I'd say the real issue here is just that micropython-lib is a bit "left behind", and it's unfortunate that the websocketserver library hasn't been updated (yet). PR's welcome. (see also https://github.com/micropython/micropython-lib/pull/376 which might help explain my perspective on how we should proceed with micropython-lib).

Post Reply