ESP32 Network Behavior

[OutoftheBOTS_]

I am unable to solve this seemingly simple problem - I cannot convert an integer (> 255) to bytes, tried this and to no avail:

if it is >255 then it can easily be packed in to a char

Code: Select all

import ustruct
positive_int_less_than_255 = 210
byte_var = ustruct.pack("B", positive_int_less_than_255)

You can pack multiple variables at once if you like

Code: Select all

import ustruct
positive_int_less_than_255_1 = 210
positive_int_less_than_255_2 = 185
positive_int_less_than_255_3 = 94
byte_var_array = ustruct.pack("BBB", positive_int_less_than_255_1, positive_int_less_than_255_2, positive_int_less_than_255_3)

I use this method to pack all sorts of data into byte arrays for sending over serial

[groger57]

so to send an integer value greater than 255 requires....what? Do I need to resort to old-fashioned bit-twiddling? there's no built in methods to do this?
If I cannot append to an array, then how are the bytes sent over the socket? I could get a book and do some reading, but with the almost non-existent info on this subject I have found on the web, I'd be worried most books will not have much either.
If I were to pack a struct, for example - how would this be converted to bytes that I could use "conn.send()" ?

sorry, lots of things new to me here - thanks for helping me find my way.

[kevinkk525]

As I said earlier, just send the object directly to the socket:

socket.send(ctr)

[groger57]

have you tried this on the esp32?
it throws an error! you can only send bytes or strings, yes?

[groger57]

ok, I seem to be finally be making progress!
#for testing, assign 44,55,66 to the vars
txData1 = 44
txData2 = 55
txData3 = 66
myStruct = ustruct.pack('iiii',txData1,txData2,txData3,ctr)
newBytes = bytearray(myStruct, 'UTF-8')
conn.sendall(newBytes)
ctr += 1

Now on the receive side, after using the streamreader to read in the byte buffer (c#) I am using this, since the values are 4 bytes
int result1 = BitConverter.ToInt32(buffer0, 0);
int result2 = BitConverter.ToInt32(buffer0, 4);
int result3 = BitConverter.ToInt32(buffer0, 8);
int result4 = BitConverter.ToInt32(buffer0, 12);

It is printing out the correct values for result1 = 44, result2 = 55, result3 = 66, and ctr starts out well enough for the first 2 values, then it suddenly jumps by 8 and then the ctr value jumps in index between 6, 7 and 8 in no particular order. So instead of going from 1 to 2, it goes 1 to 9.
44
55
66
0

44
55
66
1

44
55
66
9

44
55
66
17

44
55
66
24

So what's going on with that? I know this is not a c# forum, but the result makes little sense? The counter is indexing correctly on the transmit side. Very bizarre!

[kevinkk525]

Ah you are right, sorry. I must have forgotten that you can't pass an int to the socket. But then you can use s.send(ctr.to_bytes(2,"little")) to send your value without writing it to a bytearray. Because every time you append to a bytearray, the whole bytearray has to be allocated on the heap.

[cgglzpy]

so to send an integer value greater than 255 requires....what? Do I need to resort to old-fashioned bit-twiddling? there's no built in methods to do this?
If I cannot append to an array, then how are the bytes sent over the socket? I could get a book and do some reading, but with the almost non-existent info on this subject I have found on the web, I'd be worried most books will not have much either.
If I were to pack a struct, for example - how would this be converted to bytes that I could use "conn.send()" ?

sorry, lots of things new to me here - thanks for helping me find my way.

Hi, I've been working on this topic a while now, not too much, but enough to say that it's a non-trivial task and for "the almost non-existent info on this subject I have found on the web", I can relate. So I will share my insights into this:

- If I've understood right, what you are trying to do is sampling (from ADCs) + streaming this data (through sockets, TCP I guess)
I would say that there are three different methods:

1) Sample the ADCs, pack into bytes (using struct) and immediately send this data through the socket: This is the slowest approach, because sending the data through the socket does take time, so if you do this after each ADC sampling, it will drop the data transfer rate.

2) Sample the ADCs, store the data into a "buffer" array (see http://docs.micropython.org/en/latest/l ... array.html) with a length between 20 or 32 (it is what I've found that works the best) and when the buffer is full, send it through the socket. Then start again to fill the buffer and so on. This is the fastest approach if you want to do sampling + streaming at the same time, and the fastest I've got so far are 300 kbit/s ~ 37 KB/s ~ at 1 KHz sampling rate (but just 1 variable) for 3 variables I've got at 860 Hz sampling rate ~ 30 KB/s ~ 240 Kbits/s .

*I would recommend to see the sample code from here https://github.com/robert-hh/ads1x15, for the sampling method, then this https://github.com/Carglglz/upydev/blob ... d#ads_init, and this https://github.com/Carglglz/upydev/blob ... nit_ADS.py

3) Sample the ADCs, store the data into a file (plain text or json), and when finished read/load the file in chunks ( of 2000 bytes size or so) and send the chunk through the socket. It is not sampling + streaming at the same time, but the data transfer rate is the highest. (I've got between 80-100 KB/s) but I guess it could be faster if the chunks where larger. see https://github.com/Carglglz/upydev/blob ... on.md#sync and https://github.com/Carglglz/upydev/blob ... nc_tool.py

** All the test where done with the esp32 and my laptop connected to the same AP (home wifi). The setup of esp32 being the AP and connect the laptop to this AP, in some initial test that I've done the speed drops by 20-25 % but further test needs to be done I guess...

***[made a typo with units, 860 Hz not KHz (already edited)]

[groger57]

Hi cgglzpy,

I really appreciate you took the time to reply and post your efforts. I know it will be helpful to me and I can say that for others. it's kind of weird - the ESP32 has been around a while, yet the amount of info about certain things is very thin (like this) One thing I am perplexed on is the Expressif claim of 1Mbps for UDP mode - which is what were running here. However, if you have sent at 100Kbytes, that's nearly 1Mbps, so there is evidence to support that, which eases concerns.

I finally figured out just how to do a correct byte array stuffing on the esp32 side, and just working through on picking it apart on the VS/c# side. I do a struct.pack, specify "iiii', then convert to a bytearray, and send that 16 byte array. I need to finish coding the client side, but it does look like it will be faster! Will post the results here.
I'm curious about the transfer size of 32 bytes you mention being more efficient. This could very well be as it's a base 2 power, and like so many things in embedded, base 2 values can be a kind of magic number!

The ADC sample rate is not slowing it down at all - it samples in under 10us.

Since I will use a circular buffer scheme on the client side, your idea of pushing to a file and then sending might have merit. So long as I am reasonable close in the sampling interval, I can buffer and interpolate on the client side - it only needs to have +/- 5ms accuracy. The dev board does not have any storage, so I'm not sure how much heap I have for storage. Will need to look.

Thanks too for the links, I will take advantage of all you posted! And will repost here soon.

[kevinkk525]

If I remember correctly, @Mike Teachman did something similar a while back.

[OutoftheBOTS_]

myStruct = ustruct.pack('iiii',txData1,txData2,txData3,ctr)
newBytes = bytearray(myStruct, 'UTF-8')
conn.sendall(newBytes)

Ok I would image you can shorten this to 1 line to make it run quicker so that it doesn't have to write to ram as ustruct.pack should return a byte-array

Code: Select all

        conn.sendall(ustruct.pack('iiii',txData1,txData2,txData3,ctr))

You can also select how you want to pack if it isn't being packed correctly see 7.3.2.1. Byte Order, Size, and Alignment here https://docs.python.org/2/library/struct.html

Most of the time I find it is little-endian/big-endian problem and you need to either add ">" or "<"

If you need it to packed without any C type padding then you can also add that the same way.

Code: Select all

        conn.sendall(ustruct.pack('=iiii',txData1,txData2,txData3,ctr))

see note

Note
By default, the result of packing a given C struct includes pad bytes in order to maintain proper alignment for the C types involved; similarly, alignment is taken into account when unpacking. This behavior is chosen so that the bytes of a packed struct correspond exactly to the layout in memory of the corresponding C struct. To handle platform-independent data formats or omit implicit pad bytes, use standard size and alignment instead of native size and alignment: see Byte Order, Size, and Alignment for details.