Picoweb request doesn't return all form data

[manseekingknowledge]

micropython commit 5c34c2ff7f0c44ec9e1d77059162584c6bd99c92
picoweb commit 973d6cd33841a2ab2457d0f7247fc423824f46b7
uasyncio commit 5149a54f918361d56e3974c1ace8d0a2bad7a086
uasyncio.core commit 4c63ecf5a64d5db5362e940b33b1687d882c38c8
uasyncio.queues commit 614f5e61e9952602ea273ecc0a446f1232ef1d5e

I'm trying to send some form data to my ESP8266. The form contains network configuration information. About 50% of the time I get the all the data as expected, but other times I don't. There is no pattern that I can see to it working or not working. I can submit the form 5 times waiting 10 seconds between each submit and they may all work, fail, or it might be a mixed bag. I'm not a Wireshark expert, but I took a look at the traffic and a working request looked the same as a non-working request to me. Here are the important bits of the Javascript I use to submit the form:

Code: Select all

$("#network_config_form").submit(function(event) {
    event.preventDefault();
    event.stopImmediatePropagation();
    event.stopPropagation();

    $.ajax({
        url: $(this).prop("action"), // http://192.168.100.100/update_network_config
        data: $(this).serialize(),
        type: "POST",
        timeout: 5000
    });
});

And here are the important bits of the ESP8266's code. Note the print statements:

Code: Select all

@APP.route('/update_network_config', methods=['GET', 'POST'])
def update_network_config(request, response):
    result = {}

    if request.method == 'POST':
        yield from request.read_form_data()

        print('request.__dict__')
        print(request.__dict__)
        print("")

        print('request.form')
        print(request.form)
        print("")

    yield from picoweb.start_response(response)
    yield from response.awrite(result)

Here is the result of the print statements when Picoweb gives me all the request data:

Code: Select all

request.__dict__
{'path': '/update_network_config', 'headers': {b'Content-Type': b'application/x-www-form-urlencoded; charset=UTF-8', b'Accept': b'*/*', b'Referer': b'http://192.168.100.100/', b'X-Requested-With': b'XMLHttpRequest', b'Accept-Encoding': b'gzip, deflate', b'Origin': b'http://192.168.100.100', b'Accept-Language': b'en-US,en;q=0.9', b'Connection': b'keep-alive', b'User-Agent': b'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/75.0.3770.90 Chrome/75.0.3770.90 Safari/537.36', b'Content-Length': b'121', b'Host': b'192.168.100.100', b'DNT': b'1'}, 'method': 'POST', 'qs': '', 'reader': <StreamReader <socket state=2 timeout=0 incoming=0 off=0> <socket state=2 timeout=0 incoming=0 off=0>>, 'form': {'ssid': ['AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA', 'BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB'], 'port': ['8675', '8675'], 'password': ['aaa', 'bbb']}}

request.form
{'ssid': ['AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA', 'BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB'], 'port': ['8675', '8675'], 'password': ['aaa', 'bbb']}

And here is the result of the print statements when Picoweb gives me only some of the request data:

Code: Select all

request.__dict__
{'path': '/update_network_config', 'headers': {b'Content-Type': b'application/x-www-form-urlencoded; charset=UTF-8', b'Accept': b'*/*', b'Referer': b'http://192.168.100.100/', b'X-Requested-With': b'XMLHttpRequest', b'Accept-Encoding': b'gzip, deflate', b'Origin': b'http://192.168.100.100', b'Accept-Language': b'en-US,en;q=0.9', b'Connection': b'keep-alive', b'User-Agent': b'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/75.0.3770.90 Chrome/75.0.3770.90 Safari/537.36', b'Content-Length': b'121', b'Host': b'192.168.100.100', b'DNT': b'1'}, 'method': 'POST', 'qs': '', 'reader': <StreamReader <socket state=2 timeout=0 incoming=0 off=0> <socket state=2 timeout=0 incoming=0 off=0>>, 'form': {'ssid': 'AAAAAAAAAAAAAAAAAAAAAAAAAA'}}

request.form
{'ssid': 'AAAAAAAAAAAAAAAAAAAAAAAAAA'}

Spoiler alert: The print statements are the same (including the Content-Length) except for the form data. I put some print statements in Picoweb/__init__.py and found that the line data = yield from self.reader.read(size) does not populate the data variable properly in the instances Picoweb doesn't return all the request data. I thought I would post here before I dug into uasyncio.StreamReader.

Any insight as to why this is happening would be much appreciated.

[manseekingknowledge]

I didn't figure out why this is happening, but I did find a way around it. The "readexactly" function in uasyncio does not exhibit the same problem that I'm seeing with the read function. I could modify Picoweb directly, but I decided to work around it in my code. Instead of calling yield from request.read_form_data() I make a series of other calls to obtain the form data as seen below. I've made about 200 requests with this workaround and obtained the expected form data every time:

Code: Select all

@APP.route('/update_network_config', methods=['GET', 'POST'])
def update_network_config(request, response):
    result = {}

    if request.method == 'POST':
        # Comment out read_form_data()
        # yield from request.read_form_data()
        
        # Sub in these 4 lines
        size = int(request.headers[b"Content-Length"])
        qs = yield from request.reader.readexactly(size)
        request.qs = qs.decode()
        request.parse_qs()

        print('request.__dict__')
        print(request.__dict__)
        print("")

        print('request.form')
        print(request.form)
        print("")

    yield from picoweb.start_response(response)
    yield from response.awrite(result)

[pythoncoder]

I think the issue here is that Picoweb uses nonblocking sockets (code here) and these can return partial data: the socket.read() method is called by the polling mechanism as soon as any data is available. Because it is guaranteed to return "immediately" the method will return whatever is available at that moment.

As you have discovered the StreamReader.readexactly() method will pause (allowing other tasks to run) until the requested data quantity has been received.

I would suggest that it is working as designed.

[manseekingknowledge]

Why would we want an immediate return from the function if there is a chance it will not contain the expected data? Seems like the read_form_data() function might need to call readexactly() instead of read() unless there is some other way that read_form_data should be being used that I'm missing.

[pythoncoder]

That's a good question but alas rather outside of my experience. My guess is that Paul Sokolovsky wrote it that way for a reason. Perhaps some web developers might care to comment?

Failing that you could raise an issue against Picoweb.

[manseekingknowledge]

Fixed in picoweb commit 702706cc45be00c90ab5736aaab8c77d65078c17