urequests VS requests Issue

[dcarson1661]

Hi All,

I'm hoping to get a bit of help. I have some working code written in Python3 which works great on a windows PC and a Raspberry Pi 3. I'd like to use micropython on an ESP8266 instead. The code is making an http POST request to an ArcGIS Rest API. For some reason the API does not like the json body when sent via urequests. Again, this same request is properly received when using Python3 requests lib. It's not an internet connection issue, I did not include it in the code below but I do connect to the internet just fine. The API does return a response, but it is not the desired response like the Python3 code does. The API should be making a data update and return json indicating a successful update, instead I get the html from the url as though the data sent was never posted to the form.

Thanks in advance for any help on this!

Code: Select all

#Working Python3 code
import requests

url = 'https://services9.arcgis.com/jyf59MjuiWfY46oy/arcgis/rest/services/IOT/FeatureServer/0/updateFeatures'

data = {
    'f': 'json',
    'features': "[{'attributes': {'objectid': '1','READ_VALUE': '13'}}]"
}

x = requests.post(url,data)
print(x.text)

Code: Select all

#micropython code (not working)
#code after connecting to internet
import urequests
  
url = 'https://services9.arcgis.com/jyf59MjuiWfY46oy/arcgis/rest/services/IOT/FeatureServer/0/updateFeatures'



data = {
    'f': 'json',
    'features': "[{'attributes': {'objectid': '1','READ_VALUE': '13'}}]"
}


x = urequests.post(url,json=data)
print (x.text)

[jimmo]

What's going on here is that in Python requests, when the "data" argument is a dictionary, then

Code: Select all

data = { ... }
requests.post(url, data)

Then it interprets data key/value pairs that are URL encoded, and any non-scalar values are then JSON encoded (and URL escaped).

Here's teh corresponding request:

Code: Select all

POST jyf59MjuiWfY46oy/arcgis/rest/services/IOT/FeatureServer/0/updateFeatures HTTP/1.1

{'User-Agent': 'python-requests/2.24.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '110', 'Content-Type': 'application/x-www-form-urlencoded'}

f=json&features=%5B%7B%27attributes%27%3A+%7B%27objectid%27%3A+%271%27%2C%27READ_VALUE%27%3A+%2713%27%7D%7D%5D

whereas in the MicroPython urequests, the URL encoding scheme is not supported. If you set "data" it's always assumed to be just bytes to send. If you set "json" like in your example it does the right thing, but it appears that the ArcGIS endpoint doesn't understand JSON.

(Supporting URL encoding is a often-requested feature for urequests...)

Here's a simplest possible MicroPython example to work around urequests not supporting that:

Code: Select all

import urequests
import ujson

url = 'https://services9.arcgis.com/jyf59MjuiWfY46oy/arcgis/rest/services/IOT/FeatureServer/0/updateFeatures'

features = [{'attributes': {'objectid': '1','READ_VALUE': '13'}}]

def url_escape(s):
    return ''.join(c if c.isalpha() or c.isdigit() else '%%%02x' % ord(c) for c in s)

x = urequests.post(url,data='f=json&features='+url_escape(ujson.dumps(features)),headers={'Content-Type': 'application/x-www-form-urlencoded'})
print (x.text)

[dcarson1661]

jimmo, I cannot express enough gratitude for your response. This is exactly the help I needed. THANK YOU!