Page 1 of 3

Running webserver and other code simultaneously

Posted: Fri May 25, 2018 4:12 pm
by simonmcc
Hi, I’m using the ws.py (http://jacobstoner.com/ws.py) minimalist webserver and it works well in a forever loop with ws.serve(1000). I’m using the web service for configuration values.

However, for my application I also want to have a tcp client running too, and I want it to be reactive, so I call select() there, and normally that is a blocking call. Is my only option to change that to a non blocking call, and call the ws.serve() when the other socket has nothing incoming?

I hope I explained that reasonably well, basically I’ve got two normally blocking things that I want to run in one thread - any pointers welcome!

Re: Running webserver and other code simultaneously

Posted: Fri May 25, 2018 5:40 pm
by pythoncoder
This is something of a FAQ. I suggest you read this thread and look at this example code.

The consensus view is to use uasyncio. A tutorial on this MicroPython asyncio subset may be found here.

Re: Running webserver and other code simultaneously

Posted: Sat May 26, 2018 6:49 am
by simonmcc
Peter, my apologies for asking an FAQ, and thanks for your pointer, I'll take a look. I had looked at the asyncio stuff briefly but it didn't work for me on my D1 mini - maybe the same issue with the outdated firmware I started off with.

I'll look a bit further into it. It seems the ws.py isnt the silver bullet either, I left it running overnight and it seems to have disconnected - program looks like it's running, but it isn't accepting connections on port 80 anymore (until a reboot)

Thanks again.

Re: Running webserver and other code simultaneously

Posted: Sat May 26, 2018 7:01 pm
by bitninja
Here is a simple web server based on the uasyncio example, that I know works on the ESP8266 and the D1 Mini. I have yet to implement another coro but perhaps Peter's tutorial will help you with that.

Code: Select all

#
# Simple HTTP server based on the uasyncio example script
# by J.G. Wezensky (joewez@gmail.com)
#

import uasyncio as asyncio
import uos
import pkg_resources

webroot = 'wwwroot'
default = 'index.html'

# Breaks an HTTP request into its parts and boils it down to a physical file (if possible)
def decode_path(req):
    cmd, headers = req.decode("utf-8").split('\r\n', 1)
    parts = cmd.split(' ')
    method, path = parts[0], parts[1]
    # remove any query string
    query = ''
    r = path.find('?')
    if r > 0:
        query = path[r:]
        path = path[:r]
    # check for use of default document
    if path == '/':
        path = default
    else:
        path = path[1:]
    # return the physical path of the response file
    return webroot + '/' + path

# Looks up the content-type based on the file extension
def get_mime_type(file):
    if file.endswith(".html"):
        return "text/html", False
    if file.endswith(".css"):
        return "text/css", True
    if file.endswith(".js"):
        return "text/javascript", True
    if file.endswith(".png"):
        return "image/png", True
    if file.endswith(".gif"):
        return "image/gif", True
    if file.endswith(".jpeg") or file.endswith(".jpg"):
        return "image/jpeg", True
    return "text/plain", False

# Quick check if a file exists
def exists(file):
    try:
        s = uos.stat(file)
        return True
    except:
        return False    

@asyncio.coroutine
def serve(reader, writer):
    try:
        file = decode_path((yield from reader.read()))
        if exists(file):
            mime_type, cacheable = get_mime_type(file)
            yield from writer.awrite("HTTP/1.0 200 OK\r\n")
            yield from writer.awrite("Content-Type: {}\r\n".format(mime_type))
            if cacheable:
                yield from writer.awrite("Cache-Control: max-age=86400\r\n")
            yield from writer.awrite("\r\n")

            f = open(file, "rb")
            buffer = f.read(512)
            while buffer != b'':
                yield from writer.awrite(buffer)
                buffer = f.read(512)
            f.close()
        else:
            yield from writer.awrite("HTTP/1.0 404 NA\r\n\r\n")
    except:
        raise
    finally:
        yield from writer.aclose()

def start():
    import logging
    logging.basicConfig(level=logging.ERROR)

    loop = asyncio.get_event_loop()
    loop.call_soon(asyncio.start_server(serve, "0.0.0.0", 80, 20))
    loop.run_forever()
    loop.close()
Good luck!

Re: Running webserver and other code simultaneously

Posted: Sat May 26, 2018 7:07 pm
by simonmcc
bitninja wrote:
Sat May 26, 2018 7:01 pm
Here is a simple web server based on the uasyncio example, that I know works on the ESP8266 and the D1 Mini. I have yet to implement another coro but perhaps Peter's tutorial will help you with that.

Code: Select all

#
# Simple HTTP server based on the uasyncio example script
# by J.G. Wezensky (joewez@gmail.com)
#

import uasyncio as asyncio
import uos
import pkg_resources

webroot = 'wwwroot'
default = 'index.html'

# Breaks an HTTP request into its parts and boils it down to a physical file (if possible)
def decode_path(req):
    cmd, headers = req.decode("utf-8").split('\r\n', 1)
    parts = cmd.split(' ')
    method, path = parts[0], parts[1]
    # remove any query string
    query = ''
    r = path.find('?')
    if r > 0:
        query = path[r:]
        path = path[:r]
    # check for use of default document
    if path == '/':
        path = default
    else:
        path = path[1:]
    # return the physical path of the response file
    return webroot + '/' + path

# Looks up the content-type based on the file extension
def get_mime_type(file):
    if file.endswith(".html"):
        return "text/html", False
    if file.endswith(".css"):
        return "text/css", True
    if file.endswith(".js"):
        return "text/javascript", True
    if file.endswith(".png"):
        return "image/png", True
    if file.endswith(".gif"):
        return "image/gif", True
    if file.endswith(".jpeg") or file.endswith(".jpg"):
        return "image/jpeg", True
    return "text/plain", False

# Quick check if a file exists
def exists(file):
    try:
        s = uos.stat(file)
        return True
    except:
        return False    

@asyncio.coroutine
def serve(reader, writer):
    try:
        file = decode_path((yield from reader.read()))
        if exists(file):
            mime_type, cacheable = get_mime_type(file)
            yield from writer.awrite("HTTP/1.0 200 OK\r\n")
            yield from writer.awrite("Content-Type: {}\r\n".format(mime_type))
            if cacheable:
                yield from writer.awrite("Cache-Control: max-age=86400\r\n")
            yield from writer.awrite("\r\n")

            f = open(file, "rb")
            buffer = f.read(512)
            while buffer != b'':
                yield from writer.awrite(buffer)
                buffer = f.read(512)
            f.close()
        else:
            yield from writer.awrite("HTTP/1.0 404 NA\r\n\r\n")
    except:
        raise
    finally:
        yield from writer.aclose()

def start():
    import logging
    logging.basicConfig(level=logging.ERROR)

    loop = asyncio.get_event_loop()
    loop.call_soon(asyncio.start_server(serve, "0.0.0.0", 80, 20))
    loop.run_forever()
    loop.close()
Good luck!

Thanks!!

Re: Running webserver and other code simultaneously

Posted: Sun May 27, 2018 5:19 am
by pythoncoder
@simonmcc I can confirm that uasyncio does work on ESP8266. This repo uses it extensively.

Re: Running webserver and other code simultaneously

Posted: Fri Jun 01, 2018 5:00 pm
by simonmcc
so I got uasyncio loaded and running, and I got one of my existing modules converted to it, but I keep running into issues with memory. My program requires quite a few modules - TCP connection to server, ssd1306 for display, 1-wire, etc, etc.

After reading another thread, I tried to precompile some modules, but when I put them onto my Wemos D1 mini, it just tells me it cant find them. I also tried changing the order of my imports, that helped a bit. I reset the board between each run.

when I call the micropython.mem_info() it gives:
stack: 4112 out of 8192
GC: total: 35968, used: 34048, free: 1920
No. of 1-blocks: 349, 2-blocks: 75, max blk sz: 264, max free sz: 33

and FYI now running:
MicroPython v1.9.4-8-ga9a3caad0 on 2018-05-11; ESP module with ESP8266

any pointers welcome

Re: Running webserver and other code simultaneously

Posted: Fri Jun 01, 2018 11:56 pm
by bitninja
You can use frozen modules for the code that is stable and definitely required. Pre-compiled modules may be accomplishing the same thing... I have not used those yet. I just got comfortable building my own firmware and then added the modules I needed into the build.

Second... garbage collect often... you already have an uphill battle with the ESP8266's limited RAM. Try to only use the memory when you need to and give the gc time to do its work.

It's tough, but I have seen some surprisingly large apps work on the ESP8266.

Good luck.

Re: Running webserver and other code simultaneously

Posted: Sat Jun 02, 2018 6:16 am
by pythoncoder
simonmcc wrote:
Fri Jun 01, 2018 5:00 pm
...I tried to precompile some modules, but when I put them onto my Wemos D1 mini, it just tells me it cant find them...
As @bitninja said, using frozen bytecode is vital for larger applications. In the esp8266 directory issue make clean. Then, in your source tree, copy the modules into ports/esp8266/modules. Compile and flash. Then at the REPL issue

Code: Select all

>>> help('modules')
You should see your frozen modules listed. If not I would respectfully suggest that something silly is going on: make sure that a new build has been created and that the firmware you've flashed is the build you've just compiled...

Most of us have been there at some point. ;)

Re: Running webserver and other code simultaneously

Posted: Mon Jun 04, 2018 10:22 pm
by simonmcc
Peter, as you've probably detected I'm at the 'silly' stage alright. This is all new to me, and I'm loving it, although it's (clearly) taking a bit of time to get used to! I appreciate your patience

I definitely will try building my own firmware soon (if only there were more hours in the day) but what I was referring to above is using mpy-cross, to create 'mpy' files - I thought I could just dump these into the directory, and load them up like regular .py files?