Running webserver and other code simultaneously

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
simonmcc
Posts: 12
Joined: Fri May 11, 2018 5:21 pm

Running webserver and other code simultaneously

Post by simonmcc » Fri May 25, 2018 4:12 pm

Hi, I’m using the ws.py (http://jacobstoner.com/ws.py) minimalist webserver and it works well in a forever loop with ws.serve(1000). I’m using the web service for configuration values.

However, for my application I also want to have a tcp client running too, and I want it to be reactive, so I call select() there, and normally that is a blocking call. Is my only option to change that to a non blocking call, and call the ws.serve() when the other socket has nothing incoming?

I hope I explained that reasonably well, basically I’ve got two normally blocking things that I want to run in one thread - any pointers welcome!

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Running webserver and other code simultaneously

Post by pythoncoder » Fri May 25, 2018 5:40 pm

This is something of a FAQ. I suggest you read this thread and look at this example code.

The consensus view is to use uasyncio. A tutorial on this MicroPython asyncio subset may be found here.
Peter Hinch
Index to my micropython libraries.

simonmcc
Posts: 12
Joined: Fri May 11, 2018 5:21 pm

Re: Running webserver and other code simultaneously

Post by simonmcc » Sat May 26, 2018 6:49 am

Peter, my apologies for asking an FAQ, and thanks for your pointer, I'll take a look. I had looked at the asyncio stuff briefly but it didn't work for me on my D1 mini - maybe the same issue with the outdated firmware I started off with.

I'll look a bit further into it. It seems the ws.py isnt the silver bullet either, I left it running overnight and it seems to have disconnected - program looks like it's running, but it isn't accepting connections on port 80 anymore (until a reboot)

Thanks again.

bitninja
Posts: 165
Joined: Thu Sep 15, 2016 4:09 pm
Location: Spring, Texas

Re: Running webserver and other code simultaneously

Post by bitninja » Sat May 26, 2018 7:01 pm

Here is a simple web server based on the uasyncio example, that I know works on the ESP8266 and the D1 Mini. I have yet to implement another coro but perhaps Peter's tutorial will help you with that.

Code: Select all

#
# Simple HTTP server based on the uasyncio example script
# by J.G. Wezensky (joewez@gmail.com)
#

import uasyncio as asyncio
import uos
import pkg_resources

webroot = 'wwwroot'
default = 'index.html'

# Breaks an HTTP request into its parts and boils it down to a physical file (if possible)
def decode_path(req):
    cmd, headers = req.decode("utf-8").split('\r\n', 1)
    parts = cmd.split(' ')
    method, path = parts[0], parts[1]
    # remove any query string
    query = ''
    r = path.find('?')
    if r > 0:
        query = path[r:]
        path = path[:r]
    # check for use of default document
    if path == '/':
        path = default
    else:
        path = path[1:]
    # return the physical path of the response file
    return webroot + '/' + path

# Looks up the content-type based on the file extension
def get_mime_type(file):
    if file.endswith(".html"):
        return "text/html", False
    if file.endswith(".css"):
        return "text/css", True
    if file.endswith(".js"):
        return "text/javascript", True
    if file.endswith(".png"):
        return "image/png", True
    if file.endswith(".gif"):
        return "image/gif", True
    if file.endswith(".jpeg") or file.endswith(".jpg"):
        return "image/jpeg", True
    return "text/plain", False

# Quick check if a file exists
def exists(file):
    try:
        s = uos.stat(file)
        return True
    except:
        return False    

@asyncio.coroutine
def serve(reader, writer):
    try:
        file = decode_path((yield from reader.read()))
        if exists(file):
            mime_type, cacheable = get_mime_type(file)
            yield from writer.awrite("HTTP/1.0 200 OK\r\n")
            yield from writer.awrite("Content-Type: {}\r\n".format(mime_type))
            if cacheable:
                yield from writer.awrite("Cache-Control: max-age=86400\r\n")
            yield from writer.awrite("\r\n")

            f = open(file, "rb")
            buffer = f.read(512)
            while buffer != b'':
                yield from writer.awrite(buffer)
                buffer = f.read(512)
            f.close()
        else:
            yield from writer.awrite("HTTP/1.0 404 NA\r\n\r\n")
    except:
        raise
    finally:
        yield from writer.aclose()

def start():
    import logging
    logging.basicConfig(level=logging.ERROR)

    loop = asyncio.get_event_loop()
    loop.call_soon(asyncio.start_server(serve, "0.0.0.0", 80, 20))
    loop.run_forever()
    loop.close()
Good luck!

simonmcc
Posts: 12
Joined: Fri May 11, 2018 5:21 pm

Re: Running webserver and other code simultaneously

Post by simonmcc » Sat May 26, 2018 7:07 pm

bitninja wrote:
Sat May 26, 2018 7:01 pm
Here is a simple web server based on the uasyncio example, that I know works on the ESP8266 and the D1 Mini. I have yet to implement another coro but perhaps Peter's tutorial will help you with that.

Code: Select all

#
# Simple HTTP server based on the uasyncio example script
# by J.G. Wezensky (joewez@gmail.com)
#

import uasyncio as asyncio
import uos
import pkg_resources

webroot = 'wwwroot'
default = 'index.html'

# Breaks an HTTP request into its parts and boils it down to a physical file (if possible)
def decode_path(req):
    cmd, headers = req.decode("utf-8").split('\r\n', 1)
    parts = cmd.split(' ')
    method, path = parts[0], parts[1]
    # remove any query string
    query = ''
    r = path.find('?')
    if r > 0:
        query = path[r:]
        path = path[:r]
    # check for use of default document
    if path == '/':
        path = default
    else:
        path = path[1:]
    # return the physical path of the response file
    return webroot + '/' + path

# Looks up the content-type based on the file extension
def get_mime_type(file):
    if file.endswith(".html"):
        return "text/html", False
    if file.endswith(".css"):
        return "text/css", True
    if file.endswith(".js"):
        return "text/javascript", True
    if file.endswith(".png"):
        return "image/png", True
    if file.endswith(".gif"):
        return "image/gif", True
    if file.endswith(".jpeg") or file.endswith(".jpg"):
        return "image/jpeg", True
    return "text/plain", False

# Quick check if a file exists
def exists(file):
    try:
        s = uos.stat(file)
        return True
    except:
        return False    

@asyncio.coroutine
def serve(reader, writer):
    try:
        file = decode_path((yield from reader.read()))
        if exists(file):
            mime_type, cacheable = get_mime_type(file)
            yield from writer.awrite("HTTP/1.0 200 OK\r\n")
            yield from writer.awrite("Content-Type: {}\r\n".format(mime_type))
            if cacheable:
                yield from writer.awrite("Cache-Control: max-age=86400\r\n")
            yield from writer.awrite("\r\n")

            f = open(file, "rb")
            buffer = f.read(512)
            while buffer != b'':
                yield from writer.awrite(buffer)
                buffer = f.read(512)
            f.close()
        else:
            yield from writer.awrite("HTTP/1.0 404 NA\r\n\r\n")
    except:
        raise
    finally:
        yield from writer.aclose()

def start():
    import logging
    logging.basicConfig(level=logging.ERROR)

    loop = asyncio.get_event_loop()
    loop.call_soon(asyncio.start_server(serve, "0.0.0.0", 80, 20))
    loop.run_forever()
    loop.close()
Good luck!

Thanks!!

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Running webserver and other code simultaneously

Post by pythoncoder » Sun May 27, 2018 5:19 am

@simonmcc I can confirm that uasyncio does work on ESP8266. This repo uses it extensively.
Peter Hinch
Index to my micropython libraries.

simonmcc
Posts: 12
Joined: Fri May 11, 2018 5:21 pm

Re: Running webserver and other code simultaneously

Post by simonmcc » Fri Jun 01, 2018 5:00 pm

so I got uasyncio loaded and running, and I got one of my existing modules converted to it, but I keep running into issues with memory. My program requires quite a few modules - TCP connection to server, ssd1306 for display, 1-wire, etc, etc.

After reading another thread, I tried to precompile some modules, but when I put them onto my Wemos D1 mini, it just tells me it cant find them. I also tried changing the order of my imports, that helped a bit. I reset the board between each run.

when I call the micropython.mem_info() it gives:
stack: 4112 out of 8192
GC: total: 35968, used: 34048, free: 1920
No. of 1-blocks: 349, 2-blocks: 75, max blk sz: 264, max free sz: 33

and FYI now running:
MicroPython v1.9.4-8-ga9a3caad0 on 2018-05-11; ESP module with ESP8266

any pointers welcome

bitninja
Posts: 165
Joined: Thu Sep 15, 2016 4:09 pm
Location: Spring, Texas

Re: Running webserver and other code simultaneously

Post by bitninja » Fri Jun 01, 2018 11:56 pm

You can use frozen modules for the code that is stable and definitely required. Pre-compiled modules may be accomplishing the same thing... I have not used those yet. I just got comfortable building my own firmware and then added the modules I needed into the build.

Second... garbage collect often... you already have an uphill battle with the ESP8266's limited RAM. Try to only use the memory when you need to and give the gc time to do its work.

It's tough, but I have seen some surprisingly large apps work on the ESP8266.

Good luck.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Running webserver and other code simultaneously

Post by pythoncoder » Sat Jun 02, 2018 6:16 am

simonmcc wrote:
Fri Jun 01, 2018 5:00 pm
...I tried to precompile some modules, but when I put them onto my Wemos D1 mini, it just tells me it cant find them...
As @bitninja said, using frozen bytecode is vital for larger applications. In the esp8266 directory issue make clean. Then, in your source tree, copy the modules into ports/esp8266/modules. Compile and flash. Then at the REPL issue

Code: Select all

>>> help('modules')
You should see your frozen modules listed. If not I would respectfully suggest that something silly is going on: make sure that a new build has been created and that the firmware you've flashed is the build you've just compiled...

Most of us have been there at some point. ;)
Peter Hinch
Index to my micropython libraries.

simonmcc
Posts: 12
Joined: Fri May 11, 2018 5:21 pm

Re: Running webserver and other code simultaneously

Post by simonmcc » Mon Jun 04, 2018 10:22 pm

Peter, as you've probably detected I'm at the 'silly' stage alright. This is all new to me, and I'm loving it, although it's (clearly) taking a bit of time to get used to! I appreciate your patience

I definitely will try building my own firmware soon (if only there were more hours in the day) but what I was referring to above is using mpy-cross, to create 'mpy' files - I thought I could just dump these into the directory, and load them up like regular .py files?

Post Reply