[Solved] Native C: ESP32 core panic accessing dict after change

[johannes.rost]

Dear community,

after posting this topic:

viewtopic.php?f=2&t=10764

I also wanted to try a different approach: using the dictionary directly, and convert it to JSON by myself, see code below. This works, but not in all cases. As soon as I append an element to the dictionary in MicroPython, I get a core panic when I try to access this newly added element. There is no problem accessing the element before it. A check for "is string" before retrieving a string from it doesn't prevent the core panic, probably because the access to this element already is the problem.
So what's the correct way to work with this dictionary then? Is it even possible to use it from a different thread?

Thanks in advance for help and suggestions!

Used hardware: WiPy v3.0 (containing an ESP32D0WDQ6 (revision 1))
Used framework: Pycom MicroPython 1.20.2.r4 [v1.20.1.r2-363-ga37510c0-dirty] on 2021-06-30;
Used ESP-IDF: v3.3.x
MicroPython Version: (name='micropython', version=(1, 11, 0))

C part

Code: Select all

static mp_obj_t ConfigDict;

cJSON* CreateJsonElement(mp_obj_t Element, char* Name, cJSON* Json)
{
    printf("creating item for type %s\n", mp_obj_get_type_str(Element));
    cJSON* item = NULL;
    if (mp_obj_is_int(Element) || mp_obj_is_float(Element))
    {
        //cJSON_AddNumberToObject(Json, Name, mp_obj_get_float(Element));
        item = cJSON_CreateNumber(mp_obj_get_float(Element));
        printf("created number with value %f\n", mp_obj_get_float(Element));
    }
    else if (mp_obj_is_str(Element))
    {
        //cJSON_AddStringToObject(Json, Name, mp_obj_str_get_str(Element));
        item = cJSON_CreateString(mp_obj_str_get_str(Element));
        printf("created string with value %s\n", mp_obj_str_get_str(Element));
    }
    else if (strcmp("dict", mp_obj_get_type_str(Element)) == 0)
    {
        //ret = cJSON_AddObjectToObject(Json, Name);
        item = cJSON_CreateObject();
        printf("created empty object with following dict for key %s\n", Name);
    }
    else if (strcmp("list", mp_obj_get_type_str(Element)) == 0)
    {
        //get list items
        mp_obj_t* listElements;
        size_t listLen = 0;
        mp_obj_list_get(Element, &listLen, &listElements);

        //cJSON* array = cJSON_AddArrayToObject(Json, Name);
        item = cJSON_CreateArray();
        printf("created array\n");

        //iterate over items, that now should only contain base types
        for (int i = 0; i < listLen; i++)
        {
            printf("adding item of type %s to array\n", mp_obj_get_type_str(listElements[i]));
            cJSON_AddItemToArray(item, CreateJsonElement(listElements[i],"",item));
        }
    }

    if (item == NULL)
    {
        printf("could not create item, because type is unknown or an error happened during creation\n");
    }

    UBaseType_t uxHighWaterMark = 0;
    uxHighWaterMark = uxTaskGetStackHighWaterMark(NULL);
    printf("JSON: high water mark stack: %"PRIu16"\n", uxHighWaterMark);

    return item;
}

static uint8_t dictLevel = 0;
void DictToJsonRecursive(mp_obj_t DictOrList, cJSON* Json)
{
    if (Json == NULL)
    {
        Json = cJSON_CreateObject();
    }

    printf("starting to convert element of type %s to JSON\n", mp_obj_get_type_str(DictOrList));

    mp_map_t* map = NULL;
    if (strcmp("dict", mp_obj_get_type_str(DictOrList)) == 0)
    {
        map = mp_obj_dict_get_map(DictOrList);
    }
    else
    {
        printf("object is no dict\n");
    }

    cJSON* item= NULL;
    if (map != NULL)
    {
        printf("elements in map: %"PRIu32"\n", map->used);

        for (int i = 0; i < map->used; i++)
        {
            printf("creating JSON item for key ");
            if (mp_obj_is_str(map->table[i].key))
            {
                printf("%s\n", mp_obj_str_get_str(map->table[i].key));
            }
            else
            {
                printf("KEY ERROR for element %d\n", i);
            }
            item = CreateJsonElement(map->table[i].value, mp_obj_str_get_str(map->table[i].key), Json);
            printf("adding new item to object\n");
            cJSON_AddItemToObject(Json, mp_obj_str_get_str(map->table[i].key), item);
            
            if (strcmp("dict", mp_obj_get_type_str(map->table[i].value)) == 0)
            {
                dictLevel++;
                printf("going into nested dict level %d\n", dictLevel);
                DictToJsonRecursive(map->table[i].value, item);
                printf("coming back from dict level %d, going to level %d\n", dictLevel, dictLevel - 1);
                dictLevel--;
            }

            printf("processing next item of current dict on level %d\n", dictLevel);
        }
    }
    else
    {
        printf("ERROR: cannot get map from dictionary\n");
    }
}

#define JSONBUFFLEN (6 * 1024)
static char jsonBuff[JSONBUFFLEN] = {0};
void TestTask(void* PvParameters)
{
    C_CallBackExecutionTaskData* data = (C_CallBackExecutionTaskData*) PvParameters;

    // init semaphores
    printf("TestTask: creating semaphores\n");
    data->processed = xSemaphoreCreateBinary(); //semaphore is locked in initial state
    data->request = xSemaphoreCreateBinary();   //semaphore is locked in initial state

    printf("TestTask: starting loop\n");
    while (true)
    {
        // 1. Wait for new processing request (semaphore released)
        printf("TestTask: waiting for work\n");
        xSemaphoreTake(data->request, portMAX_DELAY);
        uint32_t coreId = xPortGetCoreID();
        printf("TestTask: processing on core %"PRIu32"\n",coreId);
        
        // 2. do the work
        cJSON* configJson = cJSON_CreateObject();
        DictToJsonRecursive(ConfigDict, configJson);

        cJSON_PrintPreallocated(configJson, &jsonBuff[0], JSONBUFFLEN, 0);
        printf("cJSON export of dict: %s\n", &jsonBuff[0]);
        cJSON_Delete(configJson);

        printf("TestTask: work done\n");
        
        // 3. release process ready lock
        printf("TestTask: signalling processing finished\n");
        xSemaphoreGive(data->processed);
    }
}

void TestTaskExecution(void)
{
    // release task that processes the callback
    printf("TestTaskExecution: releasing test task\n");
    xSemaphoreGive(callBackExecutionTaskData.request);

    // wait for the process to be finished
    printf("TestTaskExecution: waiting for test task to finish\n");
    BaseType_t semRet = xSemaphoreTake(callBackExecutionTaskData.processed, 10000 / portTICK_PERIOD_MS);    //wait for the response 10s at max
    
    if (semRet == pdFALSE)
    {
        printf("TestTaskExecution: task timed out\n");
    }
    else
    {
        printf("TestTaskExecution: task has returned\n");
    }
}

STATIC mp_obj_t DictTestTask(void)
{
    printf("DictTestThread: requesting TestTask execution\n");
    TestTaskExecution();
    return mp_const_none;
}
STATIC MP_DEFINE_CONST_FUN_OBJ_0(DictTestTask_obj, DictTestTask);

STATIC mp_obj_t TestTaskStart(void)
{
    // pin task to run on core 1, the same as µPython
    printf("creating test task\n");
    xTaskCreatePinnedToCore(TestTask, "TestTask", 1024 * 17, &callBackExecutionTaskData, tskIDLE_PRIORITY + 2, NULL, 1);
    return mp_const_none;
}
STATIC MP_DEFINE_CONST_FUN_OBJ_0(TestTaskStart_obj, TestTaskStart);

STATIC mp_obj_t SetDict(mp_obj_t NewDict)
{
    ConfigDict = NewDict;
    return mp_const_none;
}
STATIC MP_DEFINE_CONST_FUN_OBJ_1(SetDict_obj, SetDict);

µPython Part

Code: Select all

testDict = {"key_string":"value1","key_dict":{"keyindict2_int":1,"keyindict2_array":["array1","array2"], "keyindict2_float1": 3.4}}

def PrepareDictTest():
    print("Python: saving dict to C")
    websrv.SetDict(testDict)
    print("Python: starting dict test task in C module")
    websrv.TestTaskStart()

def PerformDictTest():
    print("Python: performing dict test called from uPython, processed in thread")
    websrv.DictTestTask()

def PerformDictChangeTest():
    global testDict
    global counter
    key = "key_string"
    testDict[key] = "value{}".format(counter)
    counter = counter + 1
    print("Python: changed test dictionary to:")
    print(testDict)
    print("Python: performing dict test called from uPython, processed in thread")
    websrv.DictTestTask()

def PerformDictAppendTest():
    global testDict
    global counter
    key = "test{}".format(counter)
    testDict[key] = "value{}".format(counter)
    counter = counter + 1
    print("Python: appended test dictionary to:")
    print(testDict)
    print("Python: performing dict test called from uPython, processed in thread")
    websrv.DictTestTask()

[johannes.rost]

I did some additional tests. My original assumption was slightly wrong: The core dump seems to happen not when accessing the newly added key, but the one that got moved one in index higher. I printed out the addresses of the entries like this:

Code: Select all

for (int i = 0; i < map->used; i++)
        {
            printf("table entry %d:\nelement address %p\nkey address %p\nvalue address %p\n", i, (void*) map->table, (void*) map->table[i].key, (void*) map->table[i].value);
            //......

As you can see below, the nested dictionary element, that formerly was on index position 1, was moved to index position 2, but now key and value do no longer point anywhere, they are just 0. Printing out the dicitionay from µPython works, and the other 2 elements of the dictionary are read correctly. Also, the number of elements in the table is just 3, so even if I skip table entries with NULL pointers, I won't get to the nested dictionary. Why does this happen? Is there another way to access the dictionary? Do I have to run some refresh on the dictionary? Or am I doing something else utterly wrong?

The behaviour does not change, no matter if other things run in the background or the test task for this dictionary test is the only other task running.

First run with unchanged dictionary:

original dictionary

Code: Select all

testDict = {"key_string":"value1","key_dict":{"keyindict2_int":1,"keyindict2_array":["array1","array2"], "keyindict2_float1": 3.4}}

Code: Select all

>>> PerformDictTest()
Python: performing dict test called from uPython, processed in thread
DictTestThread: requesting TestTask execution
TestTaskExecution: releasing test task
TestTaskExecution: waiting for test task to finish
TestTask: processing on core 1
starting to convert element of type dict to JSON
elements in map: 2

table entry 0:
element address 0x3fa0f1a0
key address 0x1fd2
value address 0x1fce
creating JSON item for key with name: key_string
creating item for type str
created string with value value1
JSON: high water mark stack: 15684
adding new item to object
processing next item of current dict on level 0

table entry 1:
element address 0x3fa0f1a0
key address 0x1fde
value address 0x3fa0f1b0
creating JSON item for key with name: key_dict
creating item for type dict
created empty object with following dict for key key_dict
JSON: high water mark stack: 15684
adding new item to object
going into nested dict level 1
starting to convert element of type dict to JSON
elements in map: 3

table entry 0:
element address 0x3fa0fa10
key address 0x3f453f00
value address 0x3
creating JSON item for key with name: keyindict2_int
creating item for type int
created number with value 1.000000
JSON: high water mark stack: 15684
adding new item to object
processing next item of current dict on level 1

table entry 1:
element address 0x3fa0fa10
key address 0x3f453ef0
value address 0x3fa0f1c0
creating JSON item for key with name: keyindict2_array
creating item for type list
created array
adding item of type str to array
creating item for type str
created string with value array1
JSON: high water mark stack: 15684
adding item of type str to array
creating item for type str
created string with value array2
JSON: high water mark stack: 15684
JSON: high water mark stack: 15684
adding new item to object
processing next item of current dict on level 1

table entry 2:
element address 0x3fa0fa10
key address 0x3f453ed8
value address 0x3f453ee8
creating JSON item for key with name: keyindict2_float1
creating item for type float
created number with value 3.400000
JSON: high water mark stack: 15684
adding new item to object

processing next item of current dict on level 1
coming back from dict level 1, going to level 0
processing next item of current dict on level 0

cJSON export of dict: {"key_string":"value1","key_dict":{"keyindict2_int":1,"keyindict2_array":["array1","array2"],"keyindict2_float1":3.4000000953674316}}
TestTask: work done
TestTask: signalling processing finished
TestTaskExecution: task has returned

Second run with new inserted key/value pair

Code: Select all

>>> PerformDictAppendTest()
Python: appended test dictionary to:
{'key_string': 'value1', 'test0': 'value0', 'key_dict': {'keyindict2_int': 1, 'keyindict2_array': ['array1', 'array2'], 'keyindict2_float1': 3.4}}
Python: performing dict test called from uPython, processed in thread
DictTestThread: requesting TestTask execution
TestTaskExecution: releasing test task
TestTaskExecution: waiting for test task to finish
TestTask: processing on core 1
starting to convert element of type dict to JSON
elements in map: 3
table entry 0:
element address 0x3fa38c90
key address 0x1fd2
value address 0x1fce
creating JSON item for key with name: key_string
creating item for type str
created string with value value1
JSON: high water mark stack: 15476
adding new item to object
processing next item of current dict on level 0

table entry 1:
element address 0x3fa38c90
key address 0x3fa37cc0
value address 0x3fa37cf0
creating JSON item for key with name: test0
creating item for type str
created string with value value0
JSON: high water mark stack: 15476
adding new item to object
processing next item of current dict on level 0

table entry 2:
element address 0x3fa38c90
key address 0x0
value address 0x0
Guru Meditation Error: Core  1 panic'ed (LoadProhibited). Exception was unhandled.
Core 1 register dump:
PC      : 0x401139e8  PS      : 0x00060b30  A0      : 0x80113b2a  A1      : 0x3fffc350
A2      : 0x3fa0f190  A3      : 0x3fff0f88  A4      : 0x00000002  A5      : 0x00000010
A6      : 0x00000000  A7      : 0x3ffcafc5  A8      : 0x801139da  A9      : 0x3fffc300
A10     : 0x00000000  A11     : 0x3ffe65b8  A12     : 0x3fa38c90  A13     : 0x00000000
A14     : 0x00000000  A15     : 0x40404040  SAR     : 0x00000004  EXCCAUSE: 0x0000001c
EXCVADDR: 0x00000000  LBEG    : 0x40091cf9  LEND    : 0x40091d09  LCOUNT  : 0xffffffff

ELF file SHA256: 0000000000000000000000000000000000000000000000000000000000000000

Backtrace: 0x401139e8:0x3fffc350 0x40113b27:0x3fffc370

[johannes.rost]

Another update:

I now skipped tabel entries, where table[ i ].key/value == NULL. Now I can repeat the key/value append test multiple times. This is what I observe:

1. add new key value pair: JSON = {"key_string":"value1","test0":"value0"}. Key/Index Pointer of nested dict = NULL, index of nested dict moved from 1 to 2
2. add new key value pair: JSON = {"key_string":"value1","test0":"value0","test1":"value1","key_dict":{"keyindict2_int":1,"keyindict2_array":["array1","array2"],"keyindict2_float1":3.4000000953674316}}. Includes all added key/value pairs AND nested dict. Index of nested dict moved from 2 to 3
3. add new key value pair: JSON = {"test2":"value2","test1":"value1","test0":"value0","key_string":"value1"}, key/index pointers of nested dict = NULL, nested dict index = 0
4. add new key value pair: JSON = {"test3":"value3","test2":"value2","test1":"value1","test0":"value0","key_string":"value1","key_dict":{"keyindict2_int":1,"keyindict2_array":["array1","array2"],"keyindict2_float1":3.4000000953674316}}. Index of nested dict = 5

So it seems after every second new entry in the dictionary, everything points to correct memory locations again and the dict is not lagging one element behind. This seems a bit strange, because in µPython the dict is always up to date when I print it from within µPython, before I start the C-Function.

The behaviour does not change if I use testDict.update({"test_upate":"test"}) instead of testDict["test_update"]="test"

More interestingly, only a (or at least this particular) nested dictionary seems to be missing every second new insertion of an element. An array and all the elements added by the PerformDictAppendTest() are always there.

And even more interestingly, now that the dictionary has several elements, even the nested dict is sometimes recognized after the second inserted new element...
With now 36 elements, of which one is the nested dictionary with 1 int element, 1 array with 2 strings, and 1 float, the nested dictionary never "stops existing" when I add a new element to the level 0 dictionary. The same is true if I add a new element inside the nested dictionary. But now some other elements go missing. E.g. when I have a dictionary with 99 elements, then 29 elements have a key/value pointer of NULL. Next time I add a new element, the null counter shows 28, and this goes on until it reaches 0 (all elements have a non-NULL ke/value pointer), and then it goes up again.

So how do I deal with (small?) dictionaries, that change in respect of the number of elements and/or structure, when I use them in native C?

[jimmo]

Used hardware: WiPy v3.0 (containing an ESP32D0WDQ6 (revision 1))
Used framework: Pycom MicroPython 1.20.2.r4 [v1.20.1.r2-363-ga37510c0-dirty] on 2021-06-30;
Used ESP-IDF: v3.3.x
MicroPython Version: (name='micropython', version=(1, 11, 0))

Hi,
Can you confirm which MicroPython version you're using -- it sounds like it's the pycom fork? (Which is a long way out of date with respect to upstream MicroPython). Are you able to replicate this on upstream MicroPython?

I notice your code iterates from 0 to map->used -- unless they're ordered dicts, this won't work. You need to use mp_map_slot_is_filled etc -- see dict_iter_next for an example.

[johannes.rost]

Hi jimmo,

thanks again a million for your suggestions and help!

Yes, you are correct: I am using PyCom's MicroPython for my tests. I will try to test this again with a current version of MicroPython.

I notice your code iterates from 0 to map->used -- unless they're ordered dicts, this won't work. You need to use mp_map_slot_is_filled etc -- see dict_iter_next for an example.

Thanks for that hint. I was looking at other examples of dictionary iterations and found the several that go from 0 to map->used, that's why I tried that. Thanks to your hint, I had a look at the print function of the dictionary with the dict_iter_next while loop and used it as a blue print. Now I can work my way through the dictionary without a problem! No more elements with NULL pointers.

For now, your hint solves this problem! I will nevertheless try to find out, why the json.dumps method won't work as expected, and if the behaviour changes when I use a newer version of MicroPython.

I will of course have a detailed look into that myself, but do you have a rough idea of how much of the API changed from 1.11 to the current version?