Server connection reset issue

laukejas · Post by **laukejas** » Tue Mar 31, 2020 2:36 am

Hi,

I am trying to host a webpage from my NodeMCU that displays a table of random integers (0 or 1). NodeMCU is in station mode. My code that creates the table and hosts the page looks like this:

Code: Select all

import socket
from utilities.mathUtils import randint

N = 19  
table = [[randint(0,1) for i in range(N)] for j in range(N)]


html = """<!DOCTYPE html>
<html>
    <head> <title>Page</title> </head>
    <body> <h1>Random numbers</h1>
        <font face='Courier New'>
          %s
        </font>
    </body>
</html>
"""

def generateHTMLrows(table):
  stringRows = ''
  
  for c in range(len(table[0])):    
    stringRow = ''    
    for r in range(len(table)):
        stringRow += str(table[c][r]) + ' '    
    stringRows += stringRow + '<br>'
    
  return stringRows

def connect():
  addr = socket.getaddrinfo('0.0.0.0', 80)[0][-1]

  s = socket.socket()
  s.bind(addr)
  s.listen(1)

  print('listening on', addr)
  
  while True:
      cl, addr = s.accept()
      print('client connected from', addr)
      cl_file = cl.makefile('rwb', 0)
      while True:
          line = cl_file.readline()
          if not line or line == b'\r\n':
              break
      rows = generateHTMLrows(table)
      response = html % ''.join(rows)
      cl.send('HTTP/1.0 200 OK\r\nContent-type: text/html\r\n\r\n')
      cl.send(response)
      cl.close()
      print(gc.mem_free())
connect()

As you can see at the top of my code, I define how many rows and columns my random number table will have. If this number is anywhere below 19, all is good, the webpage displays correctly:

But if I make that table any larger... Set N to 20 or more - the page fails to load. Says "connection reset". NodeMCU console shows nothing. But if I increase the N to say 30, then the page fails to load too, and the console prints "memory allocation failed, allocating 8192 bytes". Which is really weird, because I run gc.mem_free(), and it shows that there is actually plenty of memory available. Besides, such a simple table is nothing, it definitely shouldn't take much memory anyway. I am not running anything else. And besides, if I set the random number generator to generate numbers say, between 0 and 10, the page fails to load even with with lower N value.

Why am I seeing this? Is this really a memory error? How can that possibly be with such a small table?

laukejas · Post by **laukejas** » Wed Apr 01, 2020 10:53 pm

Testing this further, I tried creating an array of bytes through the use of

Code: Select all

array.array('B')

, since these take far less memory than traditional Python lists. The memory usage is clearly lower, and yet still I can't make that

Code: Select all

table

variable any larger without getting server connection reset when opening the hosted page. I now strongly suspect that it is not a memory issue, but something related to displaying that html, perhaps? Maybe the

Code: Select all

response

variable, which is a string, becomes too large?

One peculiar detail: before I get the server connection reset, the actual page that I need does try to load for a split second, and it is visible for a very brief time before I'm forwarded to connection reset page. I thought that perhaps this could be a timeout, so I set the timeout with

Code: Select all

socket.settimeout(10.0)

(10 seconds), but it didn't help anything.

I am shooting in the dark here, can anyone give any suggestions as to where to look for the problem? The debugger tools in Firefox don't tell me why the reason behind the error, and I am not sure how to debug it from MicroPython environment... Can anyone reproduce this issue?

jimmo · Post by **jimmo** » Thu Apr 02, 2020 12:48 am

I suspect the issue here is that as the string grows it becomes harder and harder to find a way to allocate it. Basically for each allocation it needs to find room for the new allocation, while still holding onto the old data until it can copy to the new location.

Instead of pre-generating a string, it would be much more efficient to stream out the table as you go.

Something like:

Code: Select all

N = 19
table = [[randint(0,1) for i in range(N)] for j in range(N)]

pre_html = """<!DOCTYPE html>
<html>
    <head> <title>Page</title> </head>
    <body> <h1>Random numbers</h1>
        <font face='Courier New'>
"""

post_hot = """
        </font>
    </body>
</html>
"""

      cl.send('HTTP/1.0 200 OK\r\nContent-type: text/html\r\n\r\n')
      cl.send(pre_html)
      for r in range(N):
        for c in range(N):
          cl.send(str(table[r][c]))
          cl.send(' ')
        cl.send('<br>')
      cl.send(post_html)
      cl.close()

You're right that you could optimise the `table` variable too.

Code: Select all

table = bytes(random.randint(0,1) for _ in range(N*N))

# Then access using table[r*N + c]

laukejas · Post by **laukejas** » Thu Apr 02, 2020 6:17 pm

Thank you for your reply, Jimmo, I had almost lost hope by now! You were right, it must be something related to generating that string. I am not entirely sure what, but in any case, I used the suggestions you provided, and now it's much better. There were only a few slight changes regarding the creation of table with Byte type for storage, and then converting it to String for streaming:

Code: Select all

table = [[bytes([randint(0,1)]) for _ in range(N)] for _ in range(N)]

Code: Select all

cl.send(str(int.from_bytes(table[r][c], 'big')))

Full working code:

Code: Select all

import socket
import array
from utilities.mathUtils import randint

N = 20
table = [[bytes([randint(0,1)]) for _ in range(N)] for _ in range(N)]

def build_table(N):
    rng = range(N)
    result = []
    for i in rng:
      arr = array.array('B') #unsigned byte
      for j in rng:
          arr.append(randint(0,1))
      result.append(arr)
    return result  
  
pre_html = """<!DOCTYPE html>
<html>
    <head> <title>Page</title> </head>
    <body> <h1>Random numbers</h1>
        <font face='Courier New'>
"""

post_html = """
        </font>
    </body>
</html>
"""


def connect():
  gc.collect()
  addr = socket.getaddrinfo('0.0.0.0', 80)[0][-1]

  s = socket.socket()
  s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
  s.bind(addr)  
  s.listen(1)

  print('listening on', addr) 
  
  while True:
      cl, addr = s.accept()
      print('client connected from', addr)
      cl_file = cl.makefile('rwb', 0)
      print(cl_file)
      while True:
          line = cl_file.readline()
          if not line or line == b'\r\n':
              break
      cl.send('HTTP/1.0 200 OK\r\nContent-type: text/html\r\n\r\n')
      cl.send(pre_html)
      for r in range(N):
        for c in range(N):
          cl.send(str(int.from_bytes(table[r][c], 'big')))
          cl.send(' ')
        cl.send('<br>')
      cl.send(post_html)
      cl.close()
      print(gc.mem_free())
      
      
connect()

Again, thank you very much!

MicroPython Forum (Archive)

Server connection reset issue

Server connection reset issue

Re: Server connection reset issue

Re: Server connection reset issue

Re: Server connection reset issue