how to make urequests asynchronous?

All ESP32 boards running MicroPython.
Target audience: MicroPython users with an ESP32 board.
kenjican
Posts: 13
Joined: Sat Sep 15, 2018 1:32 am

how to make urequests asynchronous?

Post by kenjican » Thu Jan 27, 2022 11:49 am

I used to post json by urequests module. Now I need it to be none blocking for efficiency.

I tried uaiohttpclient ,but it seems method get only, no post method. And wrap urequest.post in generator , it doesn't work either. Still blocked.

If ascyncio could make urequests to be async?

KJM
Posts: 158
Joined: Sun Nov 18, 2018 10:53 pm
Location: Sydney AU

Re: how to make urequests asynchronous?

Post by KJM » Thu Jan 27, 2022 8:29 pm

urequests normally supplies output via return, if you make it non blocking what are your expectations for how to receive it's output? I've been able to time out urequests (so my main script doesn't hang waiting for it) using either _thread or uasyncio but both required some mods to the urequests lib to replace the return with an alternative.

kenjican
Posts: 13
Joined: Sat Sep 15, 2018 1:32 am

Re: how to make urequests asynchronous?

Post by kenjican » Fri Jan 28, 2022 5:18 am

My expectation is fire and forget. Response is not necessary. Post method of urequests module doesn't have timeout option. Async is also a good alternative.

Could you please show me how to modify urequests.py ?

KJM
Posts: 158
Joined: Sun Nov 18, 2018 10:53 pm
Location: Sydney AU

Re: how to make urequests asynchronous?

Post by KJM » Fri Jan 28, 2022 5:51 am

If you don't need the response from request there is no need to mod it I guess. If you're doing posts only, we just need to mod the calling function underneath it in the lib.

Code: Select all

def post(url, **kw): import _thread; _thread.start_new_thread(request, ("POST", url, data))
 
alternatively you could try

Code: Select all

def post(url, **kw): import uasyncio as asyncio; asyncio.run(request("POST", url, data))
then edit

Code: Select all

 def request(method, url, data=None)
to

Code: Select all

 async def request(method, url, data=None)
I run a modified urequests as a function in my main.py, I'm not sure if these suggestions will work on urequests imported as a lib. If it doesn't work let me know & we can try something different.

kenjican
Posts: 13
Joined: Sat Sep 15, 2018 1:32 am

Re: how to make urequests asynchronous?

Post by kenjican » Sat Jan 29, 2022 5:30 am

I've tried both.
1: _thread. It is blocking also. I think urequests is block in the new thread.
2: asyncio. I test it in REPL, it throws out message: "coroutine expected"
Now, I'm trying to mod urequests . My idea is to make s.read() awaitble...

KJM
Posts: 158
Joined: Sun Nov 18, 2018 10:53 pm
Location: Sydney AU

Re: how to make urequests asynchronous?

Post by KJM » Sat Jan 29, 2022 7:22 am

Blocking your main program form inside a thread is weird, are you running 1.17 upython on an ESP32?

kenjican
Posts: 13
Joined: Sat Sep 15, 2018 1:32 am

Re: how to make urequests asynchronous?

Post by kenjican » Sat Jan 29, 2022 8:46 am

I'm running 1.18 on ESP32 Wrover. As I know, threads run in turns. So,,,blocking should reasonable.

Now, I'm trying uayncio.open_connection, it would return awaitable socket. Therefor, We should modify urequests.py to make sockets awaitable.

kenjican
Posts: 13
Joined: Sat Sep 15, 2018 1:32 am

Re: how to make urequests asynchronous?

Post by kenjican » Sat Jan 29, 2022 2:29 pm

Let's make a asyncio socket awaitable, see this https://docs.python.org/3/library/asyncio-stream.html in python docs
  • First step: mock a tcp server: netcat -lk -p 8888 , we could see if ESP32 send message
  • Second step: chekc if socket established: watch 'ss -tp| grep 100.235' 192.168.100.235 is IP of ESP32
  • Third step: run asyncio socket in ESP32
mock tcp server ip is : 192.168.100.230

Code: Select all

import uasyncio as asyncio
async def test(ip):
    sReader,sWriter = await asyncio.open_connection(ip,8888)
    sWriter.write("test async socket write data")

asyncio.run(test("192.168.100.230"))
In this command : watch 'ss -tp| grep 100.235' , we could see the socket established
But in netcat , there is no message comes in.
I'm not sure whether the code is wrong or it's a bug of uasyncio

User avatar
curt
Posts: 25
Joined: Thu Jul 29, 2021 3:52 am
Location: Big Lake, Alaska

Re: how to make urequests asynchronous?

Post by curt » Sat Jan 29, 2022 5:32 pm

Have you considered UDP? It is non-blocking and much faster than TCP.

Curt

marcidy
Posts: 133
Joined: Sat Dec 12, 2020 11:07 pm

Re: how to make urequests asynchronous?

Post by marcidy » Sat Jan 29, 2022 7:37 pm

kenjican wrote:
Sat Jan 29, 2022 2:29 pm
Let's make a asyncio socket awaitable, see this https://docs.python.org/3/library/asyncio-stream.html in python docs
  • First step: mock a tcp server: netcat -lk -p 8888 , we could see if ESP32 send message
  • Second step: chekc if socket established: watch 'ss -tp| grep 100.235' 192.168.100.235 is IP of ESP32
  • Third step: run asyncio socket in ESP32
mock tcp server ip is : 192.168.100.230

Code: Select all

import uasyncio as asyncio
async def test(ip):
    sReader,sWriter = await asyncio.open_connection(ip,8888)
    sWriter.write("test async socket write data")

asyncio.run(test("192.168.100.230"))
In this command : watch 'ss -tp| grep 100.235' , we could see the socket established
But in netcat , there is no message comes in.
I'm not sure whether the code is wrong or it's a bug of uasyncio
You need to drain the stream after writing to actually send. write adds to the internal buffer, stream sends the buffer over the socket.

Code: Select all

await sWriter.drain()
I have a few examples here which may be helpful: https://github.com/marcidy/micropython- ... webexample

The http examples are a server but the code may help you along.

Post Reply