Simple HTTP framework

[ernitron]

Hi @fdushin,

I had a look at your http server framework and it's amazing. I am impressed, really.

I developed my own and it's ugly (I know) but it makes its job and it's extremely compact. But the temptation to use an elegant, robust and versatile httpd framework is huge.

For my purposes this is something I am looking for in a microhttpd framework (let's call these requirements
- Handlers for dynamic functions. Let's say I want to call a function that gets temperature/humidity and format them in nice html using template.
- HTML templating - I wonder if (the remarkable) utemplate by @pfalcon https://github.com/pfalcon/utemplate could be integrated or just another templating mechanism that is small enough to be competitive with custom made dynamic functions that adds html strings to the code.

Of course frozen byte code instead of cross-compiled should be better. But this is up to the developers

Thank you

[fdushin]

@ernitron I appreciate the comments, but really the credit goes to the makers of the ESP8266, the ESP SDK, the developers of micropython, and the broader micropython community, which makes all of this possible.

As far as adding handlers for templates, the framework certainly supports doing something like that. I personally prefer moving as much of the rendering and business logic of the web application to the client-side, since the resources are pretty constrained in an ESP8266. And which the plethora of rich client-side MVC frameworks you can chose from these days, one can theoretically write very rich user experiences without impacting the ESP8266 very much -- it just becomes a file server and a REST endpoint, serving up JSON. Let the browser do all the work!

I also think that for many applications (though maybe not all), web UIs are useful for two modes of operation: 1) initial setup, and 2) troubleshooting. For example, a new device may need to be configured to set up the network and other parameters, and for that, a web UI is suitable for a user who is disinclined to use UART or webrepl. From that point forward (for many applications), the web UI should not be needed until something goes wrong that requires human intervention. At least that is how I have been looking at it, in designing this framework.

Finally, I'd be happy to contribute this framework where it could be of more use to the community (micropython-lib, for example), if that would make it more useful and maintainable down the road, but I am not engaged enough yet with this community to know where that would be.

[toddeye]

Has anyone got uhttpd to run under micropython unix?

Thanks,
Todd

[fdushin]

I have not. I did a quick test to see if it would work, and while I could get some stuff to work by importing uos instead of os and usocket instead of socket, I did hit a snag with uos.listdir not being defined in unix micro python, which is strange, because I see it in the docs. We could work on fixing these issues, if there is sufficient interest. I have started a branch unix-port, in case anyone is interested.

[deshipu]

I think you need to use the os module from here in the linux version: https://github.com/micropython/micropyt ... /master/os

[fdushin]

Using the micropython-lib versions of os and socket gets me further, without using the unix-port branch, though os pull in a lot of additional micropython-lib modules, but that's okay.

The problem is in our use of socket.setsockopt, which the uhttpd.TCPServer class is using (lifted shamelessly from the telnet code). In order to do asynchronous accepts, the uhttpd.TCPServer passes in a function here. The unix implementation of usocket, which the micropython-lib socket module inherits, does not support passing in functions in this way. I suspect there is support for select in the unix port, which is not (yet?) in the esp8266 implementation, but I don't know what the roadmap is for select on esp8266.

So, short answer, no, uhttpd does not run on the unix version of micropython, but it would be nice if it could!

[pythoncoder]

I assume you're aware of the uselect module which is available on the ESP8266 ( source in extmod/moduselect.c)? I think this is an area under active development in relation to performing asynchronous I/O with uasyncio.

[Adam5Wu]

Fantastic work, fdushin!
You project saves me considerable time and efforts in building a small Web application on my ESP8266.

I do have one small suggestion:
Since current implementation is blocking and handles one request at a time, downgrading to HTTP/1.0 will yield a better user experience.

Some browsers, like Google Chrome likes to fire multiple parallel connections to HTTP/1.1 servers to speed up loading.
But for current implementation, it is doing the reverse -- parallel connections are not acknowledged, and eventually have to timeout, before the next request can proceed, meanwhile, the browser simple "hang" while spinning the loading animation.

But if the server report HTTP/1.0, then Google Chrome only use one connection at a time, which makes consecutive request load much faster.

-------
Update: Sorry I was wrong. Changing to HTTP/1.0 does not solve the problem.
Somehow I was a bit lucky during a brief test, but eventually the problem comes back.

It seems non-blocking HTTP request handling is necessary to make Google Chrome work nicely.

[fdushin]

Interesting. You might try increasing the backlog on TCP connections. I've pushed a change to master which allows you to specify the backlog via config, e.g.,

Code: Select all

server = uhttpd.Server([
        ('/api', api_handler),
        ('/test', file_handler)
    ], {
        'require_auth': True,
        'backlog': 5
    })

The backlog is now defaulted to 5 on master (it was hard coded to 0, before).

However, I am definitely getting spurious errors testing this with a simple concurrent test here

E.g.,

Code: Select all

Traceback (most recent call last):
  File "ulog.py", line 99, in do_log
  File "console_sink.py", line 37, in log
RuntimeError: maximum recursion depth exceeded
Traceback (most recent call last):
  File "uhttpd.py", line 157, in handle_request
  File "http_file_handler.py", line 106, in handle_request
  File "http_file_handler.py", line 113, in create_file_response
  File "http_file_handler.py", line 124, in generate_file
  File "http_file_handler.py", line 221, in file_size
RuntimeError: maximum recursion depth exceeded

which doesn't make any sense to me, as I have not defined any recursive functions (to my knowledge). I wonder if I am just running out of stack space.

[Roberthh]

I could result from the fact, that the code is not using threading, but handles incoming requests in a callback mode. You use the callback mechanism, introduced by @pfalcon a a kind of hack for webrepl, and then was re-used by my ftp-server and @chrisgp's telnet-server. Being a callback, it cannot really serve nested requests, especially if these call the underlying library functions of micropython.
In my ftp-server, I therefore reject requests immediately if already one request is in process. That works reasonably well. Most clients can deal with that, except OS X's finder.
@pythoncoder made a comment about using asyncio and select, which might be the technically better solution for a server being the main task on the device.