Standalone webserver

[jameshewitt95]

Background:

As part of my internship, we are using ESP8266-01 devices connected to PIC controllers with the ability to send/receive data each way. The company already had a well established Lua server program going, but now I've been tasked with looking at potentially swapping to Micropython.

Problem:

After playing around for a day or so, I've managed to get the web server working, and displaying a very basic HTML page (from the tutorial on the docs(dot)micropython(dot)org document repository). However, it requires constant connection to the terminal window (PuTTY), which is fine for testing, but does not cut it for the potential applications.

How would I go about creating a web server on the ESP that can be run without PuTTY and then also be used via UART communication to a PIC chip?

Notes:

I'm running Windows 10, using a combination or PuTTY and Git Bash to program/control my ESP8266-01 1MB chip with a Silicon Labs CP210 USB-Serial adapter.

[fdushin]

You might consider looking at

https://github.com/fadushin/esp8266/tre ... on/daemons

and see if it fits your needs. It has support for file service as well as REST-based JSON APIs.

[jameshewitt95]

[quote="fdushin"]https://github.com/fadushin/esp8266/tre ... on/daemons[/quote]

You mention "Some of the modules in this package require significant amounts of RAM to load and run", do you mean the ESP or another controller that it is connected to? Because the ESP I have available has very little RAM I believe.

What chip did you write it on? Because it does look close to what I want to do, but if the chip can't handle it, then I'll need to find something else.

[fdushin]

I think it depends a lot on what you are doing. Uploading all the python files to the micro python file system and having the runtime compile all of them, while running webrepl and your application code will quickly run you out of RAM. On the other hand, if you freeze the byte code on an image, the results are far better.

For example, here is a memory dump on a raw image (no webrepl or uhttpd loaded or running) right after restart:

Code: Select all

MicroPython v1.8.5-128-g5630778 on 2016-12-04; ESP module with ESP8266
Type "help()" for more information.

>>> gc.mem_alloc()
7760
>>> gc.mem_free()
28416
>>> import micropython
>>> micropython.mem_info(1)
stack: 2144 out of 8192
GC: total: 36288, used: 8064, free: 28224
 No. of 1-blocks: 31, 2-blocks: 9, max blk sz: 263, max free sz: 1755
GC memory layout; from 3ffef2b0:
00000: h=AhhhhMAhDBBBDhB=BBhBh===h====hhh==h===========================
00400: ================================================================
00800: ================================================================
00c00: ================================================================
01000: ===========================================h==h=hBh=======h=====
01400: ================================================================
01800: ================================================================
01c00: ================================Dhhhh==Bh=Bh=h=hB..h=h...h=....h
02000: =...............................................................
       (26 lines all free)
08c00: ............................

And here a dump after loading the file handler and stats api handler, when the byte code is frozen (as you would in a "production" application):

Code: Select all

>>> import uhttpd
loaded sink console
>>> import http_file_handler
>>> import http_api_handler
>>> import stats_api
>>> 
>>> api_handler = http_api_handler.Handler(
...     [(['stats'], stats_api.Handler())]
...     )
>>> file_handler = http_file_handler.Handler()
>>> server = uhttpd.Server([
...     ('/api', api_handler),
...         ('/', file_handler)
...         ], use_ssl=False)
>>> server.start()
2000-01-01T00:54:54.005 [info] esp8266: TCP server started on 192.168.4.1:80
2000-01-01T00:54:54.005 [info] esp8266: TCP server started on 192.168.1.174:80
>>> 
>>> 
>>> 
>>> 
>>> gc.mem_alloc()
12512
>>> gc.mem_free()
23696
>>> micropython.mem_info(1)
stack: 2144 out of 8192
GC: total: 36288, used: 12688, free: 23600
 No. of 1-blocks: 138, 2-blocks: 30, max blk sz: 263, max free sz: 1371
GC memory layout; from 3ffef2b0:
00000: MDhhhhMMDhDBhBDDB=BBhBh===h====Lhh==h===========================
00400: ================================================================
00800: ================================================================
00c00: ================================================================
01000: ===========================================hBhhhBBh=======h=====
01400: ================================================================
01800: ================================================================
01c00: ================================DhTh=hhLhhh===Bh.h=hB=h=.B=h=.B=
02000: ..h=MDB=Mh=DB=...Dh...B..BDBB=.B=BB=B=B=BBBB=B=BBh===h========BT
02400: BBTDhhDT..h=Lhh=LMDDBh=====hB=h=MDhh===h===h====h===BBBBBBBBBBBB
02800: BBh===h===..h=h=Dhh===========DBh=h=h==hB=BBBBBBBBBBBBh===....hh
02c00: ========.h===MDDh=BBBBBh==h===h===h======Dh=BB..BBBBBBBBBBBBBBBB
03000: h===.......h==h===========h===.........h=======...........h=====
03400: ===...Ah=======.................................................
03800: h...............................................................
       (20 lines all free)
08c00: ............................

So I read this as going from 28416 to 23696 bytes of free space, or 4720 bytes of RAM, which isn't too bad for a running HTTP server.

And here is a dump after running a call on the stats API handler (which can allocate a lot of structures):

Code: Select all

>>> gc.mem_alloc()
18000
>>> gc.mem_free()
18208
>>> micropython.mem_info(1)
stack: 2144 out of 8192
GC: total: 36288, used: 18176, free: 18112
 No. of 1-blocks: 329, 2-blocks: 51, max blk sz: 263, max free sz: 1122
GC memory layout; from 3ffef2b0:
00000: MDShhhMMDhDShBDDB=BBhBh===h====Lhh==h===========================
00400: ================================================================
00800: ================================================================
00c00: ================================================================
01000: ===========================================LhhShBBh=======h=====
01400: ================================================================
01800: ================================================================
01c00: ================================DhTh=hhLhhh===hhSh=hB=h=SB=hShB=
02000: Th=hMDB=MDhDB=h==DhhShBhhBDBB=hB=BB=B=B=BBBB=B=BBh===h========BT
02400: BBTDhhDTDSShLhSLLMDDBh=====hB=h=MDhh===h===h====h===BBBBBBBBBBBB
02800: BBh===h===hSh=h=Dhh===========DBh=h=h==hB=BBBBBBBBBBBBh===h==hhh
02c00: ========Sh===MDDh=BBBBBh==h===h===h======Dh=BBh=BBBBBBBBBBBBBBBB
03000: h===h==h=Shh==h===========h===hBhhhBhThh=======Sh==Shh===hh=====
03400: ===h=SAh=======SLhShShShThSSLhShShShTh=ShSh=hhhShLhShhhLhhLhTh=h
03800: hhSSDh==Dhh===DhLThhTTTTTh=TTShhSShShhSSDhDh=DhDh=ShShShShT=DhDh
03c00: =ShShShShT=Dhh=DhSSh=Dh==ShhSShhSSSh=h=h========================
04000: ====================================SSBhDh=DTh=B.SShhTShSh=h=STS
04400: hShSh=h==STShSSSh=h==h====ShSSS.h====h======h=....h=....h=......
       (17 lines all free)
08c00: ............................

That allocated 5488 bytes! But running the garbage collector reclaimed all but 6 bytes, albeit with some heap fragmentation:

Code: Select all

>>> gc.collect()
>>> gc.mem_alloc()
12528
>>> gc.mem_free()
23680
>>> micropython.mem_info(1)
stack: 2144 out of 8192
GC: total: 36288, used: 12704, free: 23584
 No. of 1-blocks: 145, 2-blocks: 27, max blk sz: 263, max free sz: 1195
GC memory layout; from 3ffef2b0:
00000: MDhhhhMMDhDBhBDDB=BBhBh===h====Lhh==h===========================
00400: ================================================================
00800: ================================================================
00c00: ================================================================
01000: ===========================================hhhBhBBh=======h=====
01400: ================================================================
01800: ================================================================
01c00: ================================DhTh=hhLhhh===hhhh=hB=h=BB=h=hB=
02000: B.h=MDB=Mh=DB=h=.Dh...Bh=BDBB=.B=BB=B=B=BBBB=B=BBh===h========BT
02400: BBTDhhDT....Lh..LMDDBh=====hB=..MDhh===h===h====h===BBBBBBBBBBBB
02800: BBh===h===......Dhh===========DBh=..h==hB=BBBBBBBBBBBBh===....hh
02c00: ========.h===MDDh=BBBBBh==h===h===h======Dh=BB..BBBBBBBBBBBBBBBB
03000: h===.......h==h===========h===h.h......h=======...........h=====
03400: ===...Ah=======.................................................
03800: h...............................................................
03c00: ................................................................
04000: .............................................h=.h...............
       (18 lines all free)
08c00: ............................

I have not done a lot of analysis of the uhttpd code to determine what space optimizations can be made. I suspect there is opportunity for improvement.

So I don't know how well you'd do elsewhere, or even if you wrote the code from scratch. Yes, the uhttpd modules do contain several levels of abstraction (in order to encapsulate protocols at their appropriate levels), but I don't know how much memory improvement you'd get by effectively writing the same code in one module. I've never measured the RAM impact of factoring code into lots of smaller modules, vs one big module. Maybe someone here has?

Also, please beware the the uhttpd modules are pretty new and are not well tested. In time they will be, but this is really a hobby project that has not seen a lot of exposure. And of course there is no professional support. On the other hand, the code is small, open source with a BSD-style license, and as I mentioned before, PRs are always welcome

I am also looking forward to some improvements to make, including:

* Support for POST/PUT through the REST API (will likely require some interface changes)
* Support for HTTP basic auth
* SSL support, if at all possible. (There is some code in there now to support SSL, but there is no auth or certificate management, so its value is dubious from a security perspective)

These changes may image the stability of interfaces and APIs, so anyone using this software should be prepared for the possibility of API-incompatible changes.

-Fred