ESP WROOM 32 urequests POST to Google Forms

[eGRan]

Dear Community,

I am new to the forum and I am posting because I am pulling my hair out trying to get urequests to POST through Forms.

So far I have been able to create code that does POST to a Forms Sheet. Each time the code is run its generates a timestamp. However, no data posts with it. Additionally, the same code works on my computer using the requests library and does post data. The boot.py imports urequests and networks as well as connects to the local WiFi. The following is the main.py

[code]
url = 'https://docs.google.com/forms/d/e/form_key/formResponse'
form_data = '{"entry.entry_id_1":"88","entry.entry_id_2"="99"}'#,draftResponse=[],pageHistory=0)
headers = {'Content-Type': 'application/json'}
r = urequests.post(url, data=form_data, headers=headers)
r.close()
[/code]

The above code submits a POST as it generates a Timestamp, but posts no data with it.
I am out of ideas. Any help would be greatly appreciated.
Thank you

[eGRan]

Still messing around with the code. No success yet, however I did want to post what I am trying.

First, I went to confirm that my desktop Anaconda distribution running 3.7 was still posting data and to also get the logging data from the requests package. It still posts data and the logs are below:

Code: Select all

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): docs.google.com:443
send: b'POST /forms/d/e/1FAIpQLSdYLScXqq-cCoxqnrYXJd4zNzjF0tmbRYJd3eNBt4B7z55y1g/formResponse HTTP/1.1\r\nHost: docs.google.com\r\nUser-Agent: python-requests/2.21.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 48\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n'
send: b'entry.1401426727=12346&entry.773634273=Hello%3A1'
DEBUG:urllib3.connectionpool:https://docs.google.com:443 "POST /forms/d/e/1FAIpQLSdYLScXqq-cCoxqnrYXJd4zNzjF0tmbRYJd3eNBt4B7z55y1g/formResponse HTTP/1.1" 200 None
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: text/html; charset=utf-8
header: X-Robots-Tag: noindex, nofollow, nosnippet
header: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
header: Pragma: no-cache
header: Expires: Mon, 01 Jan 1990 00:00:00 GMT
header: Date: Fri, 31 May 2019 18:13:01 GMT
header: Content-Encoding: gzip
header: P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
header: P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
header: X-Content-Type-Options: nosniff
header: X-XSS-Protection: 1; mode=block
header: Server: GSE
header: Set-Cookie: NID=184=cookie_key_00;Domain=.google.com;Path=/;Expires=Sat, 30-Nov-2019 18:13:01 GMT;HttpOnly
header: Set-Cookie: NID=184=cookie_key_01;Domain=.google.com;Path=/;Expires=Sat, 30-Nov-2019 18:13:01 GMT;HttpOnly
header: Set-Cookie: NID=184=cookie_key_02;Domain=.google.com;Path=/;Expires=Sat, 30-Nov-2019 18:13:01 GMT;HttpOnly
header: Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39"
header: Transfer-Encoding: chunked

The cookie_ids were replaced to shorten the post.

Next what I attempted to do was to use the usocket to send the POST directly:

Code: Select all

url = 'https://docs.google.com/forms/d/e/1FAIpQLSdYLScXqq-cCoxqnrYXJd4zNzjF0tmbRYJd3eNBt4B7z55y1g/formResponse'
s = usocket.socket(usocket.AF_INET, usocket.SOCK_STREAM, 0)
s.connect(('216.58.192.174',443))#('docs.google.com',443)
s.send(b'POST /forms/d/e/1FAIpQLSdYLScXqq-cCoxqnrYXJd4zNzjF0tmbRYJd3eNBt4B7z55y1g/formResponse HTTP/1.1\r\nHost: docs.google.com\r\nUser-Agent: python-requests/2.21.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 48\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n')
s.send(b'entry.1401426727=123&entry.773634273=Hello')
s.close()
[\code]
This one fail completely and did not connect or generate a timestamp with the google forms record.

Lastly, I tried including the additional headers from the Desktop Python into the urequests module:
[code]
url = 'https://docs.google.com/forms/d/e/1FAIpQLSdYLScXqq-cCoxqnrYXJd4zNzjF0tmbRYJd3eNBt4B7z55y1g/formResponse'
form_data = {"entry.1401426727":"88","entry.773634273":"99"}#,draftResponse=[],pageHistory=0)
print(ujson.dumps(form_data))
headers = {'User-Agent': 'python-requests/2.21.0',
           'Accept-Encoding': 'gzip, deflate',
           'Accept': '*/*',
           'Connection': 'keep-alive'}#,Content-Type: application/x-www-form-urlencoded\r\n\r\n')}
r = post(url, data=ujson.dumps(form_data), headers=headers)
r.close()

It posted a timestamp, but nothing else.

Is there anything else I should try? Is there a response handling schema required? Even more basic, is it even theoretically possible to POST data from micropython to gxxgle Forms?

Thanks again for your time.

[jimmo]

Just to confirm, you're able to do this from "big" python on your PC? (That's what you were talking about Anaconda?)

I'm pretty sure from memory when you post json with urequests, there's json= kwarg you can use to send a dict directly (rather than using data=). (I don't think that's your problem, the result should be the same, but just FYI)

[jimmo]

Code: Select all

send: b'POST /forms/d/e/1FAIpQLSdYLScXqq-cCoxqnrYXJd4zNzjF0tmbRYJd3eNBt4B7z55y1g/formResponse HTTP/1.1\r\nHost: docs.google.com\r\nUser-Agent: python-requests/2.21.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 48\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n'
send: b'entry.1401426727=12346&entry.773634273=Hello%3A1'

I just took a closer look at your post -- this isn't sending JSON, it's sending form data (i.e. x-www-form-urlencoded). i.e. it's the same as if you made a webpage using a <form method="POST"> HTML element. That would explain what you're seeing (just a timestamp but no field values).

[eGRan]

Thank you for the reply.

I can confirm that there is no difference between data and json submission that I have tried. I have tried at least all of the following combinations (one at a time) plus/minus headers:

Code: Select all

url = 'https://docs.google.com/forms/d/e/1FAIpQLSdYLScXqq-cCoxqnrYXJd4zNzjF0tmbRYJd3eNBt4B7z55y1g/formResponse'
form_data = {"entry.1401426727":"88","entry.773634273":"99"}#,draftResponse=[],pageHistory=0)
headers = {'User-Agent': 'python-requests/2.21.0',
           'Accept-Encoding': 'gzip, deflate',
           'Accept': '*/*',
           'Connection': 'keep-alive'}#,Content-Type: application/x-www-form-urlencoded\r\n\r\n')}
#Combination 1
r = urequests.post(url, data=form_data)# , headers=headers)

#Combination 2
r = urequests.post(url, data=ujson.dumps(form_data)# , headers=headers)

#Combination 3
r = urequests.post(url, json=form_data)# , headers=headers)

#Combination 4
r = urequests.post(url, json=ujson.dumps(form_data)# , headers=headers)
r.close()

The response was from the BIG python.

Code: Select all

send: b'POST /forms/d/e/1FAIpQLSdYLScXqq-cCoxqnrYXJd4zNzjF0tmbRYJd3eNBt4B7z55y1g/formResponse HTTP/1.1\r\nHost: docs.google.com\r\nUser-Agent: python-requests/2.21.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 48\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n'
send: b'entry.1401426727=12346&entry.773634273=Hello%3A1'

Sorry if that wasn't super clear. It looks like there is a urlencode package for micropython, I could try wrapping the data/json kwargs before submitting them.

[jimmo]

Sorry if that wasn't super clear. It looks like there is a urlencode package for micropython, I could try wrapping the data/json kwargs before submitting them.

Sounds like it's worth trying! Unless you have documentation that also suggests that it also works with JSON?

[eGRan]

Gave it a try. Still nothing.
One the up side, I uncommented the line in urequests.py that prints the response line-by-line. I was also able to get micropython output (by uncommenting that line) to look exactly like BIG python logging. So at least I am more confident that micro and BIG python are posting the data.

However, during that exploration, I noticed something from that may contribute:

Code: Select all

header: Transfer-Encoding: chunked

It doesn't look like urequests.py handles chunked data or redirects well. I think I am going to have to move over to uaiohttpclient.py and modify the package to pass data and headers. Can anyone point me in a direction to get started? uAiohttpclient posting chunked data or awaiting a chunked response? Not gonna lie, I am not familiar with both HTTPS and asyncio. At least it will be a learning experience.

[jimmo]

It doesn't look like urequests.py handles chunked data or redirects well

No, but this shouldn't prevent you posting the data. (i.e. the chunked result is only on the server back to you). Stretching my memory a bit, but are you still setting connection:keep-alive which you mentioned earlier. If you turn that off (urequests doesn't handle it anyway), it's possible the server might stop setting transfer-encoding: chunked. (But I might be misremembering)

I was also able to get micropython output (by uncommenting that line) to look exactly like BIG python logging.

So both the request and the response were identical? Did a row show up in the spreadsheet? (i.e. just an empty timestamp row)

[jimmo]

I just tested this out on my own Google form:

Code: Select all

>>> import urequests
>>> h = {'Content-Type': 'application/x-www-form-urlencoded' }
>>> form_url = 'https://docs.google.com/forms/d/e/<doc id>/formResponse'
>>> form_data = 'entry.714340709=micropython'
>>> r = urequests.post(form_url, data=form_data, headers=h)
>>> r.status_code
200

It seems to work (I got "micropython" in the spreadsheet)

Might be worth having a closer look at your URL encoding? Can you share your code?

[eGRan]

Jimmo, you are awesome! I was able to duplicate your success with just those simple changes! This will make the IoT logging projects I have been working on much more stream lined to deploy.