uPython vs NodeMCU HTTP server

[ernitron]

I have ported my HTTP server to uPython from LUA/NodeMCU implementation. You can find both here:
https://github.com/ernitron/uPython-esp8266-httpserver
https://github.com/ernitron/esp8266-temp-server

For me it is a sort of benchmark. Main use is read from DS18b20 temperature sensor. But it can also serve as a general purpose http server and is very easily customizable.
Temperature server works with a fancy DIY LESS than 5$ IOT-temperature-sensor I use as proof of concept in IOT and some small deployment. See pix here https://www.dropbox.com/s/g9nx97rabusji ... r.jpg?dl=0

I use WeMos ESP8266 and developing on Ubuntu 14.06.

Now the considerations:

NodeMCU/LUA is fully working and also sends an http request to a server to record temperature (my IoT server). It can also be configured with another server

uPython: is working 50% and soon it hangs because of no memory or I dont know why as it simply doesn't reply on UART. Especially when it loads the Temperature Sensor class. I have manage to read temperature so I know at least that is working.
I have also observed that the EPS8266 is hotter with uPython than in NodeMCU. Measured with my fingers

So far it seems to me that uPython is less efficient that NodeMCU lua. But I can also have made some error in coding. Maybe you will spot some bugs or errors. If you want to contribute to development just let me know.

Any comments and suggestions are appreciated.

Thanks

[deshipu]

I just had a quick look, but one thing that jumped out on me, you really don't want to do

Code: Select all

"foo" +  "bar"

do this instead

Code: Select all

"foo" "bar"

when constant strings are involved.

[ernitron]

Thanks deshipu.

I have improved a lilltle bit the server and now it is working for a few requests but eventually it hits the memory allocation problem. I have disseminated the code with gc.collect() hoping to gather some memory before calling the most memory consuming functions. I suspect memory allocation is the main problem but I have no clue on how to manage it.

gc.enable and gc.collect conflicts?

But there should be some leak somewhere...


EDIT ADDITION: is amazingly fast compared to Lua/NodeMCU implementation

[ernitron]

Some more information for those interested.

The trick I used to make it working was to import ds18b20 in the main module (where it is NOT called) rather tnah in content.py where it is actually used. In this way at least the server starts and the first temperature measurement is done. After few calls to temperature the server crashes with no memory despite I have put the calls into try-except.

Calling repetively just the json is pretty safe, but calling 2 or 3 times consecutively the index or temperature pages make it crash.

It is therefore very critical where the import is done. Probably having the compiler in a small module is better than to call in a larger module.

Moreover I check the memory before calling routines and call gc.collect() several times to be sure there is enough memory (look cb_temperature_init() in content.py

The problem seems that memory is fragmented and sometimes there is no contiguous space for allocating some long variable. Could it be?

The randomness is another piece of the problem... that's why I think there should be a way to know what's going on with the garbage collector.

Thanks

[deshipu]

If you call micropython.mem_info(1), it should show you a memory map.

[ernitron]

I think I am getting close to the problem.

The result after just 1 or 2 call to /index is as follows:

Code: Select all

HTTP server  0.0.0.0 : 8805
BEFORE LINE 136
stack: 2704 out of 8192
GC: total: 28160, used: 24544, free: 3616
 No. of 1-blocks: 281, 2-blocks: 32, max blk sz: 263, max free sz: 57
BEFORE LINE 136
stack: 2704 out of 8192
GC: total: 28160, used: 23952, free: 4208
 No. of 1-blocks: 263, 2-blocks: 31, max blk sz: 263, max free sz: 48
BEFORE LINE 136
stack: 2704 out of 8192
GC: total: 28160, used: 24784, free: 3376
 No. of 1-blocks: 270, 2-blocks: 31, max blk sz: 263, max free sz: 37
Traceback (most recent call last):
  File "main.py", line 74, in <module>
  File "httpserver.py", line 42, in activate_server
  File "httpserver.py", line 131, in wait_for_connections
  File "content.py", line 138, in httpheader
MemoryError: memory allocation failed, allocating 700 bytes

MicroPython v1.8.3-13-g0be4a77-dirty on 2016-08-15; ESP module with ESP8266
Type "help()" for more information.

1. Server runs for few requests
2. Then it fails with a MemoryError
3. Just before the memory failure the mem_info shows enough space in stack and heap (i think)
4. The code that leads to this is:

Code: Select all

   header = "HTTP/1.1 " + HTTPStatusString + "\r\nServer: tempserver\r\nContent-Type: " + MimeType + "\r\nCache-Control: private, no-store\r\n" + "Connection: close\r\n\r\n"
   if extension == 'html' :
       header += '<!DOCTYPE html>\n' \
                 '<html lang="en">\n<head>\n<title>Temp ' + title + '</title>\n' + refresh
       print ('BEFORE LINE 136')
       micropython.mem_info()
       header += '<meta name="generator" content="esp8266-server">\n<meta charset="UTF-8">\n' \
                 '<meta name="viewport" content="width=device-width, initial-scale=1">\n' \

5. "header" is a local variable and I assume it is in the stack. The constant strings are allocated in the heap (I guess) with malloc() at runtime. Heap has plenty of space for allocating but it fails. I guess that heap is fragmented and 700 bytes cannot be found by malloc() or there is some other problem.

6. The garbage collector, as far as I understand, runs asynchronously but it seems not to compact free spaces so after few runs the heap is so badly fragmented that even few bytes cannot be found.

7. If this is right, there is a way to compact memory when gc_collect run to avoid fragmentation?

[deshipu]

As I said, don't use the + operator (or +=) with constant strings. The way this operator works is allocating enough memory to fit both strings, then copying them there, and returning the new string. This is very bad for memory. Just use one big string, there is no reason to use += there.

[ernitron]

I will refactor.

I have some questions:

1. Where local constant strings are kept? (I thought in the stack)
2. Where global constant strings are kept? (I guess in the heap)

[pfalcon]

Sorry for late reply. I hoped to sit and write detailed reply, but even weekend doesn't help with that, so please excuse my brevity. There's another excuse for brevity though: all, and I mean, all topics of GC in MicroPython were already discussed here on the forum (well, partly maybe on github). If you're interested, search is your friend. I undertstand that end users may not be interested in that, but: 1) we, developers, work to make it work in "background" as good as it can; 2) there're *too few* resources, and it's very to use VHLL to exhaust them, especially if a VHLL is used in a typical manner it's used on "desktop"; so, to write working apps for MCU-style systems, you do need to be concerned with memory (and other resources) matters.

uPython vs NodeMCU HTTP server

Thanks for making that comparison. Well, you can't reasonably compare apples to oranges. Even cursory look at 2 implementations shows they're written in 2 different manners. Again, apples and oranges: MicroPython is a portable language running on many boards, with standard API, while NodeMCU is ESP8266-specific project with an adhoc API, and as far as I understand, it's not even a Lua any longer, as some hacks were applied to the language itself. If you say that NodeMCU runs better on ESP8266, well, could be. The point of MicroPython/ESP8266 project is too make MicroPython run not worse. Actually, the aim is to run as best as possible. We already offer things not available in other languages for ESP8266 - normal file system (not something flat from 1980ies) and standard networking API. In other areas, we may be not yet there, and ready to do our homework. But a lot depends on how you write apps for your hardware systems, and that's what worth concentrating on, i.e. how to optimize your MicroPython app. I appreciate that this thread went this, and not some other direction.

(I said I'll be brief, and it's already long, and I'm not yet finished.)

[pfalcon]

Fragmentation

Any system which does runtime memory allocation is subject to fragmentation. Python, Lua, JavaScript, Java, C#, C programs are subject to memory fragmentation. Yes, you can write an app in C and it will fragment memory. The reason it's not that big problem on desktops is because of "infinite memory". RAM amount if very big, so it takes time to exhaust it, and then virtual memory kicks in, so you app may thrash swap on you harddisk for hours, days, weeks, and months, before it fails from lack of memory (e.g. due to fragmentation). On servers, processes die from lack of memory (e.g. due to fragmentation) regularly, loudly and spectacularly, leaving thousands of workers/customers without service for some time - everyone who supported large Java/C# projects knows that.

With MicroPython-sized systems, you can trigger fragmentation very easily.

The only way to avoid fragmentation is not to allocate memory (well, there's another way - to always allocate memory of the same size). That's of course not practical, and yet minimizing allocation is the best practical solution, including (and especially) for MicroPython, as Python language does offer good inventory to work with pre-allocated containers (e.g. buffers).