ws: tiny asynchronous webserver framework without uasyncio

[jacob019]

http://jacobstoner.com/ws.py

Usage:

Code: Select all

import ws
ws.routefile('/webrepl.html','webrepl.html.gz')
@ws.route('/test')
def test(query=None):
	print(query)
	return 'test'
ws.serve()
#---or---
while True:
	ws.serve(1000) #milliseconds

This is a work in progress. It's a tiny asynchronous web server / framework that I've been working on for use with the ESP8266. It uses uselect and non-blocking sockets, and seems to perform reasonably well so far. ws.serve can be called from anywhere within the main loop for as short as 1ms if there's nothing to do, or it can loop forever.

It took a lot of trial and error to get reasonable performance with uselect. For reasons I don't understand, uselect.poll hangs for a long time if it isn't in a tight loop with native code.

The purpose is to have a reliable web server without using most of the ram for uasyncio. I'm looking for some constructive criticism. Is this a reasonable route to go? Are there problems with this approach that I haven't thought of?

[pfalcon]

Likewise, see https://github.com/pfalcon/picoweb/issues/20 .

Are there problems with this approach that I haven't thought of?

Yes. But you'll need to discover them as you go along. (To start with, you don't follow PEP8.)

[jacob019]

Thanks for looking. Re PEP8, I know, you're right... at this point it's just a small working prototype. I thought I would throw it out there to see if I could get any feedback or reasons not to continue down this path, before I spend more time on it. If I continue with the project I will restyle in accordance with PEP8. I figured other people might have considered the same thing and moved on. I see no reason not to proceed, but without a deeper understanding of the uselect implementation, it's hard for me to know if this is a foolish path. Certainly there will be problems that need solving throughout development, that's normal.

I see your point about picoweb and fragmentation. I'm not trying to replace it, but it doesn't fit my needs as I cannot spare all the memory for asyncio.

[pfalcon]

fragmentation

So, if you understand that, it saves me from repeating that if everyone just digs their own hole, there's little progress in the community. Fortunately, we're past that stage, many people are doing pretty exciting things with MicroPython, and everyone who wants definitely can and should.

I cannot spare all the memory for asyncio.

Which only shows that you aren't familiar with the frozen bytecode. Unfortunately, with ESP8266, you won't be able to do much beyond "trivial" and "simple" without it. That would be my suggestion - after just scratching the surface of MicroPython and see that it's cool and easy, don't jump into writing your own frameworks, but study what further functionality and infrastructure it offers.

[jacob019]

I suppose that's the kind of encouragement I was really looking for. It was my understanding that even with the frozen bytecode uasyncio uses half the memory on the ESP8266, and I have a largish project planned with several components. I would have used frozen bytecode for my code, but it is not possible for native code without which the performance is unacceptable. I think that using uselect can be used directly for a performant asynchronous server with less memory overhead than asyncio, but I suppose it will be better to cross that bridge only if I come to it. I will take your advise, for now. Thanks for it and for your work in the community.

[pythoncoder]

Can I suggest actual measurement as a substitute for handwaving?
tl;dr as frozen bytecode it uses 3072 bytes, ~10% of the free RAM after a reboot.

Code: Select all

MicroPython v1.9.2-152-gc15be98 on 2017-10-08; ESP module with ESP8266
Type "help()" for more information.
>>> import micropython
>>> micropython.mem_info()
stack: 2112 out of 8192
GC: total: 35968, used: 5168, free: 30800
 No. of 1-blocks: 25, 2-blocks: 8, max blk sz: 264, max free sz: 1916
>>> import uasyncio
>>> micropython.mem_info()
stack: 2112 out of 8192
GC: total: 35968, used: 8240, free: 27728
 No. of 1-blocks: 90, 2-blocks: 20, max blk sz: 264, max free sz: 1672
>>> 

My advice would be to use uasyncio.

[jacob019]

Thanks Peter, that's where I should have started. I appreciate you guys taking the time to set me on the right path. I'm loving MicroPython and the community.

[jacob019]

Did some testing with frozen asyncio and picoweb. It seems to suffer from the same performance issues that I found when using uselect without native code.

Using this 61116 byte sample file:
http://jacobstoner.com/webrepl.html
(that's a minified webrepl with the JS included)

It takes a full 15 seconds to serve it with picoweb:
import picoweb
ROUTES = [("/", lambda req, resp: (yield from app.sendfile(resp, "webrepl.html")))]
app = picoweb.WebApp(__name__, ROUTES)
app.run(host="0.0.0.0", port=80)

It only takes two seconds to serve it with the script that I posted above. Am I doing something wrong?

[bitninja]

Am I doing something wrong?

I tested your code with the non-gzipped version of webrepl.html and had about the same load time. Could that be the difference? The gzipped version seems to be about 28% of the original so maybe it makes the difference. Cute trick though stuffing all the JS in there.

Anyway, I wanted to encourage you to continue your work regardless of the path you take. I use PicoWeb for various projects... but not every project... so there are reasons not to use it... usually when I'm not looking for a "framework" as such.

[jacob019]

Thanks for trying it and for the encouragement. No, I tested both without gzip. With gzip my script serves it in about 200ms. I didn't try the gzipped version with picoweb. Using the uncompressed test file that I posted, I am consistently getting about two seconds load time with ws and about 15 seconds with picoweb. It seems to run a faster with the ESP8266 in STA mode rather than AP mode. If I disable native code in my script--or use more abstraction, then the load time is comparable with picoweb. I'm glad you tried it and surprised that you didn't get the same results. Are you getting shorter load times with picoweb, or longer times with ws?

Edit, more testing...
I have an ESP8266 with an RGB LED on GPIO 5,12, and 13. I have prepared tests that respond to GET requests and set RGB values according to the GET query:
ws:

Code: Select all

import machine
import ws
r = machine.PWM(machine.Pin(13, machine.Pin.OUT), freq=500, duty=1023)
g = machine.PWM(machine.Pin(12, machine.Pin.OUT), freq=500, duty=1023)
b = machine.PWM(machine.Pin(5, machine.Pin.OUT), freq=500, duty=1023)
@ws.route('/rgb')
def rgb(query):
    query = query.split(b',')
    r.duty(int(query[0]))
    g.duty(int(query[1]))
    b.duty(int(query[2]))
ws.serve()

picoweb:

Code: Select all

import machine
import picoweb
r = machine.PWM(machine.Pin(13, machine.Pin.OUT), freq=500, duty=1023)
g = machine.PWM(machine.Pin(12, machine.Pin.OUT), freq=500, duty=1023)
b = machine.PWM(machine.Pin(5, machine.Pin.OUT), freq=500, duty=1023)
app = picoweb.WebApp(__name__)
@app.route("/rgb")
def index(req, resp):
    query = req.qs.split(',')
    r.duty(int(query[0]))
    g.duty(int(query[1]))
    b.duty(int(query[2]))
    yield from resp.awrite("HTTP/1.0 200 OK\r\n\r\n")
app.run(host="0.0.0.0",port=80,debug=False)

Here's a shell command that cycles RGB every 200ms, and times each request:

Code: Select all

for ((n=0;n<10;n++)); do sh -c '
    time -f "%e" wget -O/dev/zero -q http://10.0.0.191/rgb?1023,0,0; 
    sleep 0.2;
    time -f "%e" wget -O/dev/zero -q http://10.0.0.191/rgb?0,1023,0;
    sleep 0.2; 
    time -f "%e" wget -O/dev/zero -q http://10.0.0.191/rgb?0,0,1023;    
    sleep 0.2
'; done

ws
total reqest time for 30 requests: 0.69s
free memory after serving requests and collecting garbage: 24624

picoweb
total request time for 30 requests: 2.49s
free memory after serving requests and collecting garbage: 20448

ws is a barebones prototype and picoweb is a proper framework, so it's not a fair comparison. Adding a few features and more error handling is likely to slow ws down a bit, but this is not a small difference. Even without frozen bytecode, ws is running faster and with less memory than picoweb. I'm going to continue working on it, and I am going to prove that it is possible to use MicroPython without uasyncio for non-trivial asynchronous applications.