Socket problems on Pyboard 'D'

The official PYBD running MicroPython, and its accessories.
Target audience: Users with a PYBD
Post Reply
nherriot
Posts: 22
Joined: Wed Oct 19, 2016 1:02 pm

Socket problems on Pyboard 'D'

Post by nherriot » Tue Sep 17, 2019 10:37 am

Hi MicroPython,

I'm working with a tiny micro-webserver running on my pyboard 'D'. The link explains how to get the code up and running on the pyboard 'D'.
I've noticed that for large files (for me an png image of 71k) that sometimes the whole image will not be returned. The board seems to hang slightly, then the connection is broken. My image file that I used for testing is here

To investigate further I hit the web server with curl requests. Waiting first till the file was served before hitting the device with another curl request. My curl requests look like:

Code: Select all

nherriot@Zenbook-UX32A ~ $ curl -X GET http://192.168.110.143:8000/thumb.png > thumb.png
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 71268  100 71268    0     0   205k      0 --:--:-- --:--:-- --:--:--  205k
nherriot@Zenbook-UX32A ~ $ curl -X GET http://192.168.110.143:8000/thumb.png > thumb.png
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 71268  100 71268    0     0   220k      0 --:--:-- --:--:-- --:--:--  220k
nherriot@Zenbook-UX32A ~ $ curl -X GET http://192.168.110.143:8000/thumb.png > thumb.png
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 10 71268   10  7403    0     0   2199      0  0:00:32  0:00:03  0:00:29  2199
[color=#BF0000]curl: (18) transfer closed with 63865 bytes remaining to read[/color]
Here you can see the last request fails. Looking at the code I can see no reason for this behaviour and I’ve place some print statements on the section which deals with sending the file to the client - it all looks perfectly fine. So from my web server my output looks like:

Code: Select all

Connected on IP: 192.168.110.143
Accepted 'client': <socket state=3 timeout=-1 incoming=0 off=0> and 'client address': ('192.168.110.147', 42094)
Processing request HTTP Method: GET Path: /thumb.png Version: HTTP/1.1
Server writing file: www//thumb.png of size: 71268 of type: image/png to host: ('192.168.110.147', 42094)
Last: 612 octets being sent
Accepted 'client': <socket state=3 timeout=-1 incoming=0 off=0> and 'client address': ('192.168.110.147', 42152)
Processing request HTTP Method: GET Path: /thumb.png Version: HTTP/1.1
Server writing file: www//thumb.png of size: 71268 of type: image/png to host: ('192.168.110.147', 42152)
Last: 612 octets being sent
Connected on IP: 192.168.110.143
Accepted 'client': <socket state=3 timeout=-1 incoming=0 off=0> and 'client address': ('192.168.110.147', 42178)
Processing request HTTP Method: GET Path: /thumb.png Version: HTTP/1.1
Server writing file: www//thumb.png of size: 71268 of type: image/png to host: ('192.168.110.147', 42178)
You can see for the first 2 requests to get the thumb.png the server process is fine. There is a print statement to say when the last 'chunk' of the file is sent. In this case chunk size is 1024 octets. The print line is:
Last: 612 octets being sent
The code that deals with sending the file over a socket looks like:

Code: Select all

 def WriteResponseFile(self, filepath, contentType=None, headers=None):
            """
            A method to write a file to the client. It takes the path of the file, calculates it's size and copies the file in chunk
            sizes of 1024 octets via the low level socket interface. The method first builds the first line of the HTTP request and
            provides headers, content type, reason code and size of the file. It then does the sending of the file in 1024 octet chunks
            to the client.
            If there is a failure in reading the file a WriteREsponseNotFound is sent.
            If there is a faulure in sending the file to the client a WriteResponseInternalServerError is sent.
            :param filepath:    (e.g. www/style.css)
            :param contentType: (e.g. text/html)
            :param headers:     (e.g. {'Cache-Control': 'max-age=315360000', 'Last-Modified': 'Fri, 1 Jan 2018 23:42:00 GMT'})
            :return: Boolean
            """
            try :
                size = stat(filepath)[6]
                print("Server writing file: {} of size: {} of type: {} to host: {}".format(filepath, size, contentType, self._client._addr ))
                if size > 0 :
                    with open(filepath, 'rb') as file :                                 # Open file for reading in binary mode
                        self._writeBeforeContent(200, headers, contentType, None, size) # Write our HTTP header
                        try :
                            buf = bytearray(1024)
                            while size > 0 :
                                x = file.readinto(buf)
                                if x < len(buf) :
                                    buf = memoryview(buf)[:x]
                                    print("Last: {} octets being sent".format(x))
                                self._write(buf)                                        # call up low level socket write function
                                size -= x
                            return True
                        except :
                            self.WriteResponseInternalServerError()
                            return False
            except :
                pass
            self.WriteResponseNotFound()
            return False
The low level socket function looks like:

Code: Select all

def _write(self, data):
            if data :
                if type(data) == str :
                    data = data.encode()
                return self._client._socketfile.write(data)
            return 0
The code is accessible from github here and line 766 shows the print line. When the failure happens the except clauses never get called!???

Things to note. The microWebSrv is using a single thread to handle all requests that come in to the web server. What i don't understand is:
1) Why the except is not called when it fails?
2) Why it mostly works, but sometimes not?

The micropython build I'm using is:

Code: Select all

repl
Entering REPL. Use Control-X to exit.
>
MicroPython v1.11-63-gd889def-dirty on 2019-07-02; PYBD-SF2W with STM32F722IEK
Type "help()" for more information.
I compiled this myself with threading added which i needed for the web server.

If someone could let me know what other things can i do in terms of debugging? How can I see what is going on in the embedded device to somehow totally stop running this piece of code????
I've run the same code directly on my PC with python 3.6 and none of the issues exist, so it seems to be unique to the board. Furthermore nobody on the micrWebSrv github repo seem to report or notice this when testing on the ESP Boards, so maybe this only on the new pyboard 'D' that it happens?

Any help is gratefully appreciated. :-)

Kind regards, Nicholas.

User avatar
jimmo
Posts: 791
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia

Re: Socket problems on Pyboard 'D'

Post by jimmo » Fri Sep 20, 2019 6:08 am

hi Nicholas,
Sorry I've been meaning to sit down and try and investigate this fully, but haven't had a chance.
I initially wondered if it was related to https://github.com/micropython/micropython/issues/5071 but it sounds like you're using a more recent version of MicroPython maybe?
I'll try and have a look on the weekend.

User avatar
jczic
Posts: 11
Joined: Fri Sep 01, 2017 2:10 am
Location: Paris, France
Contact:

Re: Socket problems on Pyboard 'D'

Post by jczic » Fri Sep 20, 2019 10:09 pm

Hello Nicholas and thank you for your post.
I've just committed a fix in the write socket function (commit https://github.com/jczic/MicroWebSrv/co ... fbc0c44203).
Could you retry your code and tell me if the problem occurs again please :) ?
Thanks!

nherriot
Posts: 22
Joined: Wed Oct 19, 2016 1:02 pm

Re: Socket problems on Pyboard 'D'

Post by nherriot » Thu Sep 26, 2019 12:58 pm

Hi Jean-Christophe,

will try this out later in the week. Thanks for your reply and commit! :-)
Just to let you know I ended up using the logging library to find out what exceptions were happening.
They were time-out OS related. So your fix would probably not help the underlying condition.

I added logging here:
https://github.com/SamsungResearchUK-Io ... /libraries
And I added like this:

Code: Select all

from libraries.logging.logging import *

basicConfig(level=DEBUG)                 # Can be one of NOTSET, DEBUG, INFO, WARNING, ERROR, CRITICAL
log = getLogger("microWebSrv")
Once I'm satisfied that it all works well, I'll do a pull request with the logging and doc strings for your valued feedback.
Code is here:

https://github.com/SamsungResearchUK-Io ... oWebSrv.py

I'm finding the logging invaluable!!!
At the moment I have an issue with my AR demo which I've just found out requires HTTPS connections when viewed via mobile web browsers. So if you have any ideas on how to get the socket streams wrapped up in a TLS context and examples please let me know! :-/

Kind regards, Nicholas

Post Reply