Simple HTTP framework

All ESP8266 boards running MicroPython.
Official boards are the Adafruit Huzzah and Feather boards.
Target audience: MicroPython users with an ESP8266 board.
User avatar
fdushin
Posts: 32
Joined: Thu Jul 21, 2016 5:38 pm

Re: Simple HTTP framework

Post by fdushin » Fri Dec 30, 2016 2:56 pm

@Roberthh that makes a lot of sense. I will start an investigation of porting the existing codebase to use asyncio, likely on a branch off master. In the meantime, I will add a limitation to the current README, to let people know. If anyone is willing to collaborate, please let me know.

Does anyone know the roadmap for asyncio and uselect? Will that code get rolled into a micro python image in the future?

Coincidentally, I started an investigation of using backbone.js and hit the same issues @Adam5Wu reported. I was able to work around some of the issues by loading libraries from their respective web sites and minimizing the number of scripts, but that's obviously not a long-term solution (e.g., for bootstrapping off 192.168.4.1).

Thanks for the heads up. Very helpful!

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Simple HTTP framework

Post by pythoncoder » Sat Dec 31, 2016 7:31 am

Asyncio and uselect are under active development. e.g. https://github.com/micropython/micropython/issues/2735.
Peter Hinch
Index to my micropython libraries.

Adam5Wu
Posts: 6
Joined: Wed Dec 28, 2016 3:30 am

Re: Simple HTTP framework

Post by Adam5Wu » Sun Jan 01, 2017 3:02 am

fdushin wrote:Interesting. You might try increasing the backlog on TCP connections.
Increase backlog does not help, and may make it worse, here is why:

Chrome schedules requests across live (established or pending) connections. If the new request happen to schedule to the first connection, all is well. However, if somehow the new request got scheduled to the second connection, then a "deadlock" occurs: esp is stuck on readline of the first connection, but it is idle, while chrome is waiting for the request to get through on the second connection, which is pending to be established.

This deadlock is resolved only after the time out of the socket io on the readline of the first connection.

This is what happens when backlog is zero - one established connection, and one on hold, all other connections are rejected.
If the backlog is 4, then there is a probability chrome schedules the new request on the 5th connection, then timeout have to happen 4 times before the request is served. (BTW, it seems Chrome tries to make 6 connections for each site - after 1st connection succeeds, it immediately make 5 more connections)

Since esp has such limited resource, the most feasible solution I imagine is, kind of like what Roberthh did:
1. serve one connection at a time, other connections are accepted and put into queue.
2. allow only one connection per source, all concurrent connections from the same source is immediately closed.

This do require some non-blocking or async operations, but only at very limited scope, mostly on connection management. Request processing can still be kept as is in current implementation.

Adam5Wu
Posts: 6
Joined: Wed Dec 28, 2016 3:30 am

Re: Simple HTTP framework

Post by Adam5Wu » Thu Jan 05, 2017 6:00 am

I have implemented similar logic as described in the above post, and found it works pretty well! :)
The code is a bit messy, I will cleanup and share later.

One differences is that, I found closing those extra connection right after establish makes Chrome not happy, so instead of killing the extra connections, I simply queue them up and serve one by one. I don't acknowledge keep-alive, but seems Chrome does not complain about that. For double safety, I send back HTTP/1.0, so no chance it can expect keep alive, or host header.

Along with my tests, I found out another "bug", or, port imperfection:
The backlog control on the listen function actually has no effect on esp8266.

This is because esp SDK provide a customize lwip, which has the following definition:
#define TCP_LISTEN_BACKLOG 0

More over, there is another caveat, the maximum allowed concurrent connections.
This one is also customized in ESP SDK
#define MEMP_NUM_TCP_PCB (*(volatile uint32*)0x600011FC)
Which points to reserved system RTC memory, and there is an esp SDK API for it:
espconn_tcp_set_max_con()

So only after adjusting both places, it is possible to enable HTTP server work nicely with Google Chrome.

Adam5Wu
Posts: 6
Joined: Wed Dec 28, 2016 3:30 am

Re: Simple HTTP framework

Post by Adam5Wu » Tue Jan 10, 2017 3:09 am

I have cleaned up my hacks to fdushin's code and posted it here:
https://github.com/Adam5Wu/esp8266/blob ... /uhttpd.py

It uses a slightly improved logic, which allows request to come from any established connections, although it can only handle one request at a time.

The code works to some extent. But I seem to ran into some MicroPython lower level stability issue, which I have no skill to dig deeper.

When the load goes up (multiple request of large (~200KB) files), MP seems to be suffer some kind of heap/stack corruptions.

The first sign is, randomly, error messages like the following pop up:
"TypeError 'float' object is not callable"
"TypeError 'int' object is not callable"
I don't know where they came from, but surely they did not come from any code inside uhttpd

Then, if the load remains high for a few request, the processor resets for all kinds of reasons: null pointer reference, wdt reset, etc...

I am unable to derive any useful information so couldn't even file a bug report...

But I am fairly certain it is an MP problem, tested on two ESPs with different make (D1 mini, NodeMCU), purchase date and seller, same sympton; the ESPAsyncWebServer however, is able to complete my test (not to compare apple with oranges, but just to rule out hardware or power supply problems).

Greatly appreciate anyone who have the interest and skill to help figuring out the problem... :D

User avatar
fdushin
Posts: 32
Joined: Thu Jul 21, 2016 5:38 pm

Re: Simple HTTP framework

Post by fdushin » Sun Feb 19, 2017 4:06 pm

I have started a port of the uhttpd server to @pythoncoder's new uasyncio framework. The work is on a branch, but it can be found at:

https://github.com/fadushin/esp8266/tre ... hon/uhttpd

I am still getting some failures with my concurrency unit tests -- I still get intermittent [Errno 54] Connection reset by peer errors, though most of the concurrent clients still succeed. So I don't know if I am still doing something wrong. Iterators, Generators, and yields are new to me (being a classically trained functional programmer), so I could be missing something.

The motivating use case for this is a proof of concept web application, which you can find at

https://github.com/fadushin/esp8266/tre ... eb-console

The basic idea is that it is a web application using backbone.js, and a simple REST API for GET'ing and PUT'ing data on the device; most of the heavy lifting is done in the web browser (display, transitions, rendering, etc). I am still having issues with it on Chrome, but it is better behaved on Safari. Unfortunately, I still can't get the large-ish backbone.js, jquery, and other Javascript and CSS dependencies to load directly from the device, which would be required for bootstrapping the device off its Access Point. It may simply be that the device is too underpowered, and my uhttpd code too bloated, to run this application reliably off a device this size. But it's been fun trying!

Post Reply