C data structure to python dictionary

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
OutoftheBOTS_
Posts: 642
Joined: Mon Nov 20, 2017 10:18 am

Re: C data structure to python dictionary

Post by OutoftheBOTS_ » Thu Sep 27, 2018 7:37 am

Ok I am a noob but this is my understanding of it.

C has many datatype lengths for integers and floats e.g int8_t, uint8_t, uint16_t, int32_t, float and double float.

Python has data types of an any size and will just allocate how ever much RAM is needed e.g you can just keep increase the number being stored in a variable and it will never become out of range as python will just keep allocating more memory. To me it seems the very minimum amount of memory python will allocate is 32 bits.

Often the data we receive in to python can come from another source that has fixed sizes for data types like C. What I do is unpack the C types in to python types so that python can use them with normal python functions e.g if a functions needs a parameter passed to it as a python int then it will get an error if you pass a C type uint8_t so first I convert the uint8_t to a python int using ustruct.unpack()

So if I had a bytearray that came in from a peripheral that held the following C struct

Code: Select all

typedef union var_data
{
int8_t int8_val;
uint8_t uint8_val;
int16_t int16_val;
uint16_t uint16_val;
int32_t int32_val;
uint32_t uint32_val;
long long_val;
float float_val;
char *string_val;
} var_data_t;

I would unpack it into python list like this.

Code: Select all

from ustruct import unpack

C_type_data = #where ever the data came from

python_list = unpack('bBhHiIlf', C_type_data)

#not 100% sure how you pick off the string as have never had to convert C string to python string

User avatar
pythoncoder
Posts: 3544
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: C data structure to python dictionary

Post by pythoncoder » Thu Sep 27, 2018 8:00 am

To unpack a string (actually a bytes object) you need to know its length:

Code: Select all

>>> b = b'\x80\x01\x01\x00\x00the quick brown fox'
>>> unpack('<Bi19s', b)
(128, 257, b'the quick brown fox')
Peter Hinch

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: C data structure to python dictionary

Post by jickster » Thu Sep 27, 2018 3:38 pm

OutoftheBOTS_ wrote:
Thu Sep 27, 2018 7:37 am

So if I had a bytearray that came in from a peripheral that held the following C struct
You're writing Python code; he needs to write C-code using uPy C-API to expose the data into Python.

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: C data structure to python dictionary

Post by jickster » Thu Sep 27, 2018 5:12 pm

learnerforlife wrote:
Thu Sep 27, 2018 4:34 am
Hi sorry if I was not clear, I am not a very experienced programmer, let me explain in detail.
We have a device which acts like a IoT gateway. we already have all the data acquisition part done in C. The controllers that we are querying the data from are mainly communicating over Modbus protocol. So after we have queried the data, it is passed through a decoding logic which reads from the byte arrays received from Modbus controllers and interprets them and returns a linked list which contains parameter names and their values.
ex. {name:voltage, value:250}->{name:curent, value:10}->{...}...
And this list could be of variable length.

Now we want to expose this functionality for writing the actual business logic in python to a third party who may not know C programming at all. So I need this data to be available in python in an easily accessible way (by easy I mean he shouldn't have to worry about where its coming from or how to map it in python) say like a dictionary, so the person writing logic could do something like device['voltage'] and he can get the voltage value.

If it was a single struct variable I could easily map it but what I am having problem understanding is how to do it for a list, which is also going to be variable in length.
Also following is the structure I am using in C.

typedef union var_data
{
int8_t int8_val;
uint8_t uint8_val;
int16_t int16_val;
uint16_t uint16_val;
int32_t int32_val;
uint32_t uint32_val;
long long_val;
float float_val;
char *string_val;

} var_data_t;


typedef struct data_list
{

char *prop_name;
var_data_t data;
struct data_list *next;
uint8_t data_type;
} data_list_t;
Python:

Code: Select all

data = get_data()

Code: Select all

mp_obj_t get_data()
{	
	// somewhere data_list_t * first is declared
	data_list_t * curr = first;
	
	qstr prop_name;
	
	// create a list
	mp_obj_list_t * mylist = mp_type_list.make_new(NULL, 0, 0, NULL)
	
	// contain the payload data
	mp_obj_t mp_data;
	
	// simplest way to get list[x].field functionality
	mp_obj_module_t * mod;
	
	while(curr != NULL)
	{
		prop_name = qstr_from_str(curr->prop_name);
		switch(curr->data_type)
		{			
			case INT8:
				mp_data = MP_OBJ_NEW_SMALL_INT(curr->data.int8_val);
				break;
			case UINT8:
				mp_data = MP_OBJ_NEW_SMALL_INT(curr->data.uint8_val);
				break;
			case INT16:
				mp_data = MP_OBJ_NEW_SMALL_INT(curr->data.int16_val);
				break;
			case UINT16:
				mp_data = MP_OBJ_NEW_SMALL_INT(curr->data.uint16_val);
				break;				
			case INT32:
				mp_data = MP_OBJ_NEW_SMALL_INT(curr->data.int32_val);
				break;
			case UINT32:
				mp_data = MP_OBJ_NEW_SMALL_INT(curr->data.uint32_val);
				break;
			case LONG:
				mp_data = MP_OBJ_NEW_SMALL_INT(curr->data.long_val);
				break;
			case FLOAT:
				mp_data = MP_OBJ_NEW_SMALL_INT(curr->data.float_val);
				break;
			case STRING:
				mp_data = MP_OBJ_NEW_QSTR(qstr_from_str(curr->data.string_val));
				break;
		}
		
		mod = m_new_obj(mp_obj_module_t);
		mod->base.type = &mp_type_module;
		mod->globals = MP_OBJ_TO_PTR(mp_obj_new_dict(2));
		
		mp_obj_dict_store(MP_OBJ_FROM_PTR(mod->globals), MP_OBJ_NEW_QSTR(MP_QSTR_name), MP_OBJ_NEW_QSTR(prop_name));
		mp_obj_dict_store(MP_OBJ_FROM_PTR(mod->globals), MP_OBJ_NEW_QSTR(MP_QSTR_val),  mp_data);
		
		mylist->append(mod);
		
		curr = curr->next;
	}
	
	return (mp_obj_t) my_list;
}

User avatar
pythoncoder
Posts: 3544
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: C data structure to python dictionary

Post by pythoncoder » Fri Sep 28, 2018 6:30 am

My approach would be to do as much as possible in Python and minimise the C.

On that basis would it be simpler for the C code to create a Python list of raw bytes objects? You'd then perform the parsing in Python. This might provide a simpler interface between the C and MicroPython worlds. Translating your existing decoding logic to Python should be straightforward.

[EDIT]
Perhaps you could simplify the Python/C interface further and provide a UART-like interface. Have a read_into(buf) function which populates a buffer with raw data, returning the no. of bytes in the buffer (0 if no data is available). Your Python code might look something like

Code: Select all

d = {}
buf = bytearray(100)  # Max modbus data size in bytes
async def process_data():
    res = read_into(buf)
    while res == 0:
        await asyncio.sleep_ms(100)
        res = read_into(buf)
    name, value = dbytes.parse(res)
    d[name] = value
The merit is that a UART-like interface with a preallocated buffer is easy to implement.
Peter Hinch

learnerforlife
Posts: 13
Joined: Mon Sep 17, 2018 5:38 am

Re: C data structure to python dictionary

Post by learnerforlife » Fri Sep 28, 2018 6:59 am

jickster wrote:
Thu Sep 27, 2018 5:12 pm
learnerforlife wrote:
Thu Sep 27, 2018 4:34 am
Python:

Code: Select all

data = get_data()

Code: Select all

mp_obj_t get_data()
{	
	// somewhere data_list_t * first is declared
	data_list_t * curr = first;
	
	qstr prop_name;
	
	// create a list
	mp_obj_list_t * mylist = mp_type_list.make_new(NULL, 0, 0, NULL)
	
	// contain the payload data
	mp_obj_t mp_data;
	
	// simplest way to get list[x].field functionality
	mp_obj_module_t * mod;
	
	while(curr != NULL)
	{
		prop_name = qstr_from_str(curr->prop_name);
		switch(curr->data_type)
		{			
			case INT8:
				mp_data = MP_OBJ_NEW_SMALL_INT(curr->data.int8_val);
				break;
			case UINT8:
				mp_data = MP_OBJ_NEW_SMALL_INT(curr->data.uint8_val);
				break;
			case INT16:
				mp_data = MP_OBJ_NEW_SMALL_INT(curr->data.int16_val);
				break;
			case UINT16:
				mp_data = MP_OBJ_NEW_SMALL_INT(curr->data.uint16_val);
				break;				
			case INT32:
				mp_data = MP_OBJ_NEW_SMALL_INT(curr->data.int32_val);
				break;
			case UINT32:
				mp_data = MP_OBJ_NEW_SMALL_INT(curr->data.uint32_val);
				break;
			case LONG:
				mp_data = MP_OBJ_NEW_SMALL_INT(curr->data.long_val);
				break;
			case FLOAT:
				mp_data = MP_OBJ_NEW_SMALL_INT(curr->data.float_val);
				break;
			case STRING:
				mp_data = MP_OBJ_NEW_QSTR(qstr_from_str(curr->data.string_val));
				break;
		}
		
		mod = m_new_obj(mp_obj_module_t);
		mod->base.type = &mp_type_module;
		mod->globals = MP_OBJ_TO_PTR(mp_obj_new_dict(2));
		
		mp_obj_dict_store(MP_OBJ_FROM_PTR(mod->globals), MP_OBJ_NEW_QSTR(MP_QSTR_name), MP_OBJ_NEW_QSTR(prop_name));
		mp_obj_dict_store(MP_OBJ_FROM_PTR(mod->globals), MP_OBJ_NEW_QSTR(MP_QSTR_val),  mp_data);
		
		mylist->append(mod);
		
		curr = curr->next;
	}
	
	return (mp_obj_t) my_list;
}
Thanks a lot I was able to do the similar thing in micropython although I don't think it was the best solution. I did not know I could do the conversion first in c and return the dictionary. This should help a lot.

Following is what I did:

Code: Select all

import uctypes
import invmodule

desc = {
    'nameaddr': uctypes.UINT32 | 0,
    'nextaddr': uctypes.UINT32 | 8,
    'datatype': uctypes.UINT8 | 12,
    'namelength': uctypes.UINT8 | 13,
    'vallength': uctypes.UINT8 | 14,
}


def getdict(head_addr):
    device_dict = {}
    node_addr = head_addr
    node = 0
    while(node_addr != 0):
        datatype = uctypes.struct(node_addr, {'type': uctypes.UINT8 | 12,}, uctypes.LITTLE_ENDIAN)
        if datatype.type == 0:
            desc['data'] = uctypes.INT8 | 4
            node = uctypes.struct(node_addr, desc, uctypes.LITTLE_ENDIAN)
            propName = str(uctypes.bytearray_at(node.nameaddr, node.namelength), 'utf-8', 'ignore')
            device_dict[propName] = node.data
        elif datatype.type == 1:
            desc['data'] = uctypes.UINT8 | 4
            node = uctypes.struct(node_addr, desc, uctypes.LITTLE_ENDIAN)
            propName = str(uctypes.bytearray_at(node.nameaddr, node.namelength), 'utf-8', 'ignore')
            device_dict[propName] = node.data
        elif datatype.type == 2:
            desc['data'] = uctypes.INT16 | 4
            node = uctypes.struct(node_addr, desc, uctypes.LITTLE_ENDIAN)
            propName = str(uctypes.bytearray_at(node.nameaddr, node.namelength), 'utf-8', 'ignore')
            device_dict[propName] = node.data
        elif datatype.type == 3:
            desc['data'] = uctypes.UINT16 | 4
            node = uctypes.struct(node_addr, desc, uctypes.LITTLE_ENDIAN)
            propName = str(uctypes.bytearray_at(node.nameaddr, node.namelength), 'utf-8', 'ignore')
            device_dict[propName] = node.data
        elif datatype.type == 4:
            desc['data'] = uctypes.INT32 | 4
            node = uctypes.struct(node_addr, desc, uctypes.LITTLE_ENDIAN)
            propName = str(uctypes.bytearray_at(node.nameaddr, node.namelength), 'utf-8', 'ignore')
            device_dict[propName] = node.data
        elif datatype.type == 5:
            desc['data'] = uctypes.UINT32 | 4
            node = uctypes.struct(node_addr, desc, uctypes.LITTLE_ENDIAN)
            propName = str(uctypes.bytearray_at(node.nameaddr, node.namelength), 'utf-8', 'ignore')
            device_dict[propName] = node.data
        elif datatype.type == 6:
            desc['data'] = uctypes.FLOAT32 | 4
            node = uctypes.struct(node_addr, desc, uctypes.LITTLE_ENDIAN)
            propName = str(uctypes.bytearray_at(node.nameaddr, node.namelength), 'utf-8', 'ignore')
            device_dict[propName] = node.data
        elif datatype.type == 7:
            desc['data'] = uctypes.UINT32 | 4
            node = uctypes.struct(node_addr, desc, uctypes.LITTLE_ENDIAN)
            propName = str(uctypes.bytearray_at(node.nameaddr, node.namelength), 'utf-8', 'ignore')
            propVal = str(uctypes.bytearray_at(node.data, node.vallength), 'utf-8', 'ignore')
            device_dict[propName] = propVal
        node_addr = node.nextaddr
    return device_dict


Last edited by learnerforlife on Fri Sep 28, 2018 7:21 am, edited 1 time in total.

learnerforlife
Posts: 13
Joined: Mon Sep 17, 2018 5:38 am

Re: C data structure to python dictionary

Post by learnerforlife » Fri Sep 28, 2018 7:20 am

@pythoncoder yeah I was also thinking the same thing (not the UART part) but my lead wants to keep as much code in C as possible. :P

User avatar
pythoncoder
Posts: 3544
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: C data structure to python dictionary

Post by pythoncoder » Sat Sep 29, 2018 8:29 am

Oh dear. Dogma trumps simple design once again :(

What he wants to do is surely possible. Alas it requires a knowledge of MicroPython internals which exceeds my own, so I'll leave you with the experts.
Peter Hinch

jickster
Posts: 629
Joined: Thu Sep 07, 2017 8:57 pm

Re: C data structure to python dictionary

Post by jickster » Sat Sep 29, 2018 11:39 pm

learnerforlife wrote:@pythoncoder yeah I was also thinking the same thing (not the UART part) but my lead wants to keep as much code in C as possible. :P
Do you have the answer you need? Lemme know if you need more C code.


Sent from my iPhone using Tapatalk Pro

learnerforlife
Posts: 13
Joined: Mon Sep 17, 2018 5:38 am

Re: C data structure to python dictionary

Post by learnerforlife » Wed Oct 03, 2018 8:25 am

jickster wrote:
Sat Sep 29, 2018 11:39 pm
learnerforlife wrote:@pythoncoder yeah I was also thinking the same thing (not the UART part) but my lead wants to keep as much code in C as possible. :P
Do you have the answer you need? Lemme know if you need more C code.


Sent from my iPhone using Tapatalk Pro
Yes, I think I've got it. Thanks a lot. I can explore from here.

Post Reply