ESP32 Network Behavior

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
groger57
Posts: 13
Joined: Sun Sep 22, 2019 8:43 pm

ESP32 Network Behavior

Post by groger57 » Sun Sep 22, 2019 8:56 pm

Hi Folks:

I am running the ESP32 as an access point. It's set up to wait for a connection, and when it gets a connection, it sends data continuously from an ADC (while True loop) with a 1ms sleep to throttle it. Overall, it runs quite well.
On the connection client side, I have an app written in VS/C# that connects to it, and when it gets connection, starts to receive the data being sent. It seems to be pretty robust, but...the esp32/micropython is throwing a weird message that I cannot find any reference to.
This is what is in the Thonny debug window:

dhcps: send_nak>>udp_sendto result 0

When this happens, it breaks the connection at the client side, which I manage with a try/catch statement. In the real application I cannot have this occurring - I really need the application to push data so long as it's powered on.

What does this mean, what is the cause of it? If I knew, I could handle it in the code with perhaps a try/except.

Thanks!

groger57
Posts: 13
Joined: Sun Sep 22, 2019 8:43 pm

Re: ESP32 Network Behavior

Post by groger57 » Mon Sep 23, 2019 3:57 pm

Hi,

If no one understands the message, how can I find out, where can I look?

Thanks.

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: ESP32 Network Behavior

Post by pythoncoder » Mon Sep 30, 2019 7:00 am

I would investigate this by seeing if it occurs when you increase the transmission interval to 10ms or more. It's possible that some internal buffer in the ESP32 is overflowing. If this is the case, you may need to buffer a number of readings to send data at a lower rate (albeit with longer messages).
Peter Hinch
Index to my micropython libraries.

groger57
Posts: 13
Joined: Sun Sep 22, 2019 8:43 pm

Re: ESP32 Network Behavior

Post by groger57 » Tue Oct 01, 2019 3:14 am

Hi pythoncoder,

Getting an answer back from one of the most knowledgeable people on micropython is great. thanks for looking at my post.

That bizarre message is an unusual issue, but not the real issue, but maybe related!
I can try that suggestion - but already the link is so much slower than I could imagine. Please let me explain - any insight is appreciated.
I'm connecting to the station with a laptop, using a basic server application written in either a console app, or the c# forms app I am now using. They both run about the same. The client laptop has a very good USB wifi modem, and using it, for example, on a regular connection I can pretty easily get 200mbps, i.e., www.fast.com - so I know the wifi is solid.

In the python app on the esp32, I have it set up to connect reliably and hold the connection, so no issues there. Also, I set the esp32 speed at 240Mhz. Once the "client" connects, I begin sending data in a "while True" loop, with a 500us sleep - I want it to arrive as fast as possible. Ideally, 1ksample/sec would be ideal! I am reading 3 adc channels, and sending the raw adc values ( 0 to 4095) as a list with a newline to delineate the blocks of 3:

send.conn( str(adc1,adc2,adc3) + '\')

The very best rate I can attain is about 340 samples/second.( over 3 channels) Cannot call it bytes because there's 3 channels plus the overhead bytes. Even worst case, let's say with a 2:1 overhead to data, that's: 9 x 340 x 8bits/byte = 25Kbps. That's not right at all ! I can say with a huge degree of certainty that the windows client is not the issue either.

Maybe the question should be - what I am doing is not right and needs to be changed...then what is the correct way to set up and transmit data from a station to a client? Is it not pack to a list, then "conn.send(myList) ? There are zero examples out there...I cannot find any. Is there an optimized size or format of data structure to ensure a faster transmit speed or efficiency? It's got to be something very obvious that I have overlooked. The expressif documentation says even with UDP a 1mbps rate is pretty achievable. If it is, what do I need to do to get better than 25, 30, or even 50kbps?

Would really appreciate a push in the right direction, and thanks again!

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: ESP32 Network Behavior

Post by kevinkk525 » Tue Oct 01, 2019 5:51 am

Can you post your code?

Also I would try to avoid any changes to the data that would result in processor time like str(adc1,adc2,adc3)+"\"
You could just send each value to the socket directly like:

Code: Select all

send.conn(adc1)
send.conn(b",")
send.conn(adc2)
send.conn(b",")
send.conn(adc3)
send.conn(b"\")
That could increase the speed.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: ESP32 Network Behavior

Post by pythoncoder » Tue Oct 01, 2019 8:16 am

I have no experience of running sockets at high data rates. Judging by the lack of responses to your query I suspect you may be the first in here to attempt this. So my comments are guesses based on general experience of real time systems.

The ESP32 runs an RTOS (realtime OS). I have done measurements of interrupt latency which suggests that the RTOS grabs time slices from MicroPython: the ESP32 interrupt latency is poor compared to Pyboards. This general latency may affect socket performance. There are two possible issues: the mean throughput in bytes/sec and the message rate (messages/sec). You're going to have to do your own tests to determine the limits of the platform. If you do, please report your results.

If I wanted high realtime performance over WiFi I'd use a Pyboard D. This has a much better architecture with no RTOS and with WiFi operation devolved to a separate chip.
Peter Hinch
Index to my micropython libraries.

OutoftheBOTS_
Posts: 847
Joined: Mon Nov 20, 2017 10:18 am

Re: ESP32 Network Behavior

Post by OutoftheBOTS_ » Tue Oct 01, 2019 9:46 am

kevinkk525 wrote:
Tue Oct 01, 2019 5:51 am
Can you post your code?

Also I would try to avoid any changes to the data that would result in processor time like str(adc1,adc2,adc3)+"\"
You could just send each value to the socket directly like:

Code: Select all

send.conn(adc1)
send.conn(b",")
send.conn(adc2)
send.conn(b",")
send.conn(adc3)
send.conn(b"\")
That could increase the speed.
I maybe out of my depth a bit here so don't have experience with sockets but do have lots with sending data over serial wire connection.

If you want fast transfer then sending it as binary data is much faster. If I understand what your doing is reading binary data from the ADC then converting it to a string then sending it via a socket then at the other end you receive the string and turn it back into a binary form.

Can you just use ustruct.pack('HHHc', adc1, adc2, adc3, 92) and pass all the data in a total of 7 bytes. I assume this will be very fast as the pack doesn't need to process the adc data into a string but rather just copies it in binary format and also the python fuction pack is only called once so ti has onyl the start up over heads once

groger57
Posts: 13
Joined: Sun Sep 22, 2019 8:43 pm

Re: ESP32 Network Behavior

Post by groger57 » Tue Oct 01, 2019 2:41 pm

pythoncoder and kevinkk525;

Thanks to both of you, excellent replies and I will try both and let you know how the results turn out.
As for switching to the pyBoard, I may end up doing that if I am unable to obtain a more consistent and better result on this setup.

groger57
Posts: 13
Joined: Sun Sep 22, 2019 8:43 pm

Re: ESP32 Network Behavior

Post by groger57 » Wed Oct 02, 2019 4:55 am

I must have fallen off the python bus, or asleep in class. Looks like sending bytes is not so simple!
I am unable to solve this seemingly simple problem - I cannot convert an integer (> 255) to bytes, tried this and to no avail:

ctr = 0

while True:
ctr += 1
myArray = bytearray()
ctr.to_bytes(2,'little')
myArray.append(ctr)

It gets to 255, and surprise, at 256 it's back at zero. Just the very thing I told it not to do with ctr.to_bytes(2,'little'). It seems to just ignore that statement. Is this a micropython thing?
In any case, a search of the internet revealed not a single example of sending a byte array of values over a socket that has integers of larger values, i.e., what we might find in a data converter for example. This must be never done, or done and no one has ever asked? Not sure which.

What have I missed? This problem is getting more complex instead of easier, just because I want to send data efficiently. It''s almost like getting taxed to death because of home improvements.

kevinkk525
Posts: 969
Joined: Sat Feb 03, 2018 7:02 pm

Re: ESP32 Network Behavior

Post by kevinkk525 » Wed Oct 02, 2019 5:29 am

There's a few things wrong:
ctr.to_bytes(2,'little') returns the bytes and does not assign it to ctr. So when you append ctr to myArray, you actually append the value before to_bytes. However appending the to_bytes result to the bytearray doesn't work:

Code: Select all

b.append(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can't convert bytes to int
Generally: If you want to improve performance, avoid allocating heap space because it is the slowest operation on microcontrollers. Appending to a bytearray allocates heap space, therefore slowing your program down.
Kevin Köck
Micropython Smarthome Firmware (with Home-Assistant integration): https://github.com/kevinkk525/pysmartnode

Post Reply