OSError(84,)

All ESP32 boards running MicroPython.
Target audience: MicroPython users with an ESP32 board.
edf012
Posts: 14
Joined: Sun May 09, 2021 8:51 pm

OSError(84,)

Post by edf012 » Fri May 13, 2022 12:11 pm

Has anyone encountered an OSError(84,)? In my case, in micropython code, I open a 3MB binary file for reading and write 4096 byte chunks to an NVS partition, which is also 3MB. The micropython loop fails (and gives the OSError(84,)) at offset 0x10f000 every time. The C code simply calls esp_partition_write, just like the OTA FW upgrade code does. I can use esptool.py to write the binary file to flash with zero issues. I've checked the file on the filesystem and I've checked the NVS partition. Both look correct and are identical to the original file on my Linux system. Any idea what may be causing this OSError? I am using MicroPython version 1.18.2.r7.

marcidy
Posts: 133
Joined: Sat Dec 12, 2020 11:07 pm

Re: OSError(84,)

Post by marcidy » Sat May 14, 2022 1:13 am

what's the context, is anything else going on (are you connected to network, threads, etc, )? Can you post a cut down sample which reproduces the error? I'm not sure if you load the file in via flash, then when you try to access it you get a crash, or if there is some other sequence. Code that reproduces is the best example.

This may be unrelated, curious why are you using an NVS partition instead of the filesystem?

If you can reproduce, github is probably the right way to go for this, but I'm happy to look at it if the example is available.

Interestingly, 84 is not in mperrno.h

marcidy
Posts: 133
Joined: Sat Dec 12, 2020 11:07 pm

Re: OSError(84,)

Post by marcidy » Sat May 14, 2022 1:16 am

Also please include partitions.csv you are using

edf012
Posts: 14
Joined: Sun May 09, 2021 8:51 pm

Re: OSError(84,)

Post by edf012 » Sat May 14, 2022 11:56 am

Hi Marcidy, thanks for responding. The context is a device that is connected to the utility grid. There are occasional power outages and brownouts. Due to bad experience with those outages and file system corruption over the past few years, with no success trying various "fixes", we decided to move our data storage to NVS. As part of that effort, I OTA'd a new partition table, including the NVS keys and a re-sized file system. I then OTA'd the pre-built NVS partition - a 3MB binary file. Via REPL, I verified the contents of the OTA'd file matched the contents on the server. I then OTA'd a new firmware that looks for the new binary file at boot up, i.e. in our main micrpython application. If the file is there, it is opened and 4096 byte chunks are written to the the NVS partition. Strangely, after reaching offset 0x10f000, the operation fails and I get the OSError(84,). I've tried many things like different chunk sizes, starting from the end and going backwards, etc, etc. Nothing works. It always fails at offset 0x10f000. I triple-checked the data in the binary file on our device. It is correct. I already reached out to Espressif, and, of course, they point the finger at Micropython. We are using IDF v4.2. Below is the latest code I've tried (there have been a half dozen or more variants, this variant splits the flash write into three 1 MB parts and tries to ignore the exception):

The partition table as seen at boot up:
I (56) boot: Partition Table:
I (59) boot: ## Label Usage Type ST Offset Length
I (67) boot: 0 oldnvs WiFi data 01 02 0000e000 00004000
I (74) boot: 1 phy_init RF data 01 01 00012000 00001000
I (82) boot: 2 otadata OTA data 01 00 00013000 00002000
I (89) boot: 3 nvs_key NVS keys 01 04 00015000 00001000
I (97) boot: 4 factory factory app 00 00 00020000 00300000
I (104) boot: 5 ota_0 OTA app 00 10 00320000 00300000
I (112) boot: 6 ota_1 OTA app 00 11 00620000 00300000
I (119) boot: 7 vfs Unknown data 01 81 00920000 003e0000
I (127) boot: 8 nvs WiFi data 01 02 00d00000 00300000
I (134) boot: End of partition table

In our main micropython application, in init:
# U20-1159
offset = 0
chunk = ()
log.info('chunk size is {}'.format(len(chunk)))
with open('ubinvs.bin', 'rb') as f:
log.info('opened ubinvs.bin..iterating through the file')
f.seek(2097152)
while chunk := f.read(4096) and offset < 1048576:
try:
moecom.spi_write(offset, chunk, 4096)
except:
pass
offset += 4096
log.info('Processed 1MB')
offset = 0
with open('ubinvs.bin', 'rb') as f:
log.info('re-opened ubinvs.bin..iterating through the file')
f.seek(1048576)
while chunk := f.read(4096) and offset < 1048576:
try:
moecom.spi_write(offset, chunk, 4096)
except:
pass
offset += 4096 <-- OSError(84,) leaves me here at a REPL prompt - well that's what traceback tells me
log.info('Processed 2MB')
offset = 0
with open('ubinvs.bin', 'rb') as f:
log.info('re-opened ubinvs.bin..iterating through the file')
while chunk := f.read(4096) and offset < 1048576:
try:
moecom.spi_write(offset, chunk, 4096)
except:
pass
offset += 4096
log.info('finished iterating through the file')
os.rename('ubinvs.bin','ubinvs.save')
log.info('renamed ubinvs file')
log.info('Processed 3MB')

and in a C file:

STATIC mp_obj_t mod_moecom_spi_write(mp_obj_t _offset, mp_obj_t _data, mp_obj_t _size) {
uint32_t offset = mp_obj_get_int(_offset);
const char *data = mp_obj_str_get_str(_data);
uint32_t size = mp_obj_get_int(_size);
static esp_partition_t *partition = NULL;
static bool partition_erased = false;
unsigned int x = size / 16;

printf("got offset 0x%08x, got data %p, got size %d, x is %d\n", offset, data, size, x);

if (partition == NULL) {
// Find the partition map in the partition table
partition = (esp_partition_t *)esp_partition_find_first(ESP_PARTITION_TYPE_DATA, ESP_PARTITION_SUBTYPE_DATA_NVS, "nvs");
printf("Found %s, size %d\n", partition->label ,partition->size);
if (partition_erased == false) {
esp_partition_erase_range(partition, 0xd00000, 0x300000);
partition_erased = true;
}
}

for (;x>0;x--) {
if (ESP_OK == esp_partition_write(partition, offset, data, 16)) {
offset += 16;
}
else {
printf("esp_partition_write FAIL, offset: %d\n", offset);
}
}
return mp_const_none;
}
STATIC MP_DEFINE_CONST_FUN_OBJ_3(mod_moecom_spi_write_obj, mod_moecom_spi_write);

As I think I mentioned, I can use esptool.py to flash the NVS binary over the serial port and the device reboots and operates without error, that is, our application runs for weeks, 24/7, with no issues.

TIA for any help you can provide.

marcidy
Posts: 133
Joined: Sat Dec 12, 2020 11:07 pm

Re: OSError(84,)

Post by marcidy » Sat May 14, 2022 4:34 pm

Can you please edit your post to use code tags to preserve code formatting

[code]
...
[/code]

marcidy
Posts: 133
Joined: Sat Dec 12, 2020 11:07 pm

Re: OSError(84,)

Post by marcidy » Sun May 15, 2022 6:17 pm

I don't see anything clearly indicating an problem, however it seems you have a highly customized micropython? You mention a few changes in another thread, so it's not immediately clear to me if this is the smallest example which reproduces the error, and at that point remotely debugging someone elses code/hardware is difficult.

There's a number of curious points in what you did post (that is fails at the same spot, the point at which it fails should be a clue) but i suspect you noticed all that and are really asking if anyone has seeing this specific OSError. I suspect you have seen that this style of OSError is really a pass-through for a failed call that micropython doesn't handle, but isn't severe enough to crash the whole system.

You'll have to do some hardware debugging and trace the calls to see what is actually returning 84. Sounds like you can reproduce the error each time and at the same point in the writes, which should made debugging easier.

I wonder if you explored the functionality in esp32.Partition ? you can find nvs partitions fine, and there's a "writeblocks" method on the partition object.

Code: Select all

>>> esp32.Partition.find(1)
[<Partition type=1, subtype=2, address=36864, size=24576, label=nvs, encrypted=0>, <Partition type=1, subtype=1, address=61440, size=4096, label=phy_init, enc]
>>> for p in _:
...     print(p)
... 
<Partition type=1, subtype=2, address=36864, size=24576, label=nvs, encrypted=0>
<Partition type=1, subtype=1, address=61440, size=4096, label=phy_init, encrypted=0>
<Partition type=1, subtype=129, address=2097152, size=2097152, label=vfs, encrypted=0>
>>> p.writeblocks
I haven't tried to use it, but it calls esp_partition_write as well, and may be more thoroughly tested / error checked.

in ports/esp32/esp32_partition.c:

Code: Select all

STATIC mp_obj_t esp32_partition_writeblocks(size_t n_args, const mp_obj_t *args) {
    esp32_partition_obj_t *self = MP_OBJ_TO_PTR(args[0]);
    uint32_t offset = mp_obj_get_int(args[1]) * BLOCK_SIZE_BYTES;
    mp_buffer_info_t bufinfo;
    mp_get_buffer_raise(args[2], &bufinfo, MP_BUFFER_READ);
    if (n_args == 3) {
        check_esp_err(esp_partition_erase_range(self->part, offset, bufinfo.len));
    } else {
        offset += mp_obj_get_int(args[3]);
    }
    check_esp_err(esp_partition_write(self->part, offset, bufinfo.buf, bufinfo.len));
    return mp_const_none;
}   
STATIC MP_DEFINE_CONST_FUN_OBJ_VAR_BETWEEN(esp32_partition_writeblocks_obj, 3, 4, esp32_partition_writeblocks);

edf012
Posts: 14
Joined: Sun May 09, 2021 8:51 pm

Re: OSError(84,)

Post by edf012 » Mon May 16, 2022 10:59 am

Hi Marcidy, If you mean, did we add some functionality to Micropython, the answer is yes. The code I showed is the code that opens the binary file and writes it to the NVS partition. This is done very early in the main application init function, i.e. right after the watchdog thread is started and some other UART's are initialized.

I searched through Espressif's IDF code and could not find where the OSError 84 exception is raised - except in seme tinyusb functions, which we don't use in our product.

I am still trying to wrap my head around why I see this error at all. The only info I've been able to find is that this error is an illegal byte sequence - attributed to a UTF-8 issue. In my case, it's a binary file being written to an NVS partition. IMO, I should never see this error during this particular operation.

I did see the writeblocks method in the Partition module. However, I did not see its use in any of our other code, so I elected to use the method that OTA employed. I can easily re-write the code to use the esp32.Partition functions. I'll give it a try and post the results.

Thanks for your help!

edf012
Posts: 14
Joined: Sun May 09, 2021 8:51 pm

Re: OSError(84,)

Post by edf012 » Mon May 16, 2022 1:10 pm

offset = 0x0
chunk = ()
import esp32
part = esp32.Partition("nvs")
with open('ubinvs.bin', 'rb') as f:
log.info('opened ubinvs.bin..iterating through the file')
while chunk := f.read(4096):
log.info('part is {}, chunk size is {}, offset is {}'.format(part, len(chunk), hex(offset)))
try:
esp32.Partition.writeblocks(part, chunk, offset)
#moecom.spi_write(offset, chunk, 4096)
except:
pass
offset += 4096
os.rename('ubinvs.bin','ubinvs.save')
log.info('renamed ubinvs file')

44:INFO:ubicell:part is <Partition type=1, subtype=2, address=13631488, size=3145728, label=nvs, encrypted=0>, chunk size is 4096, offset is 0x10c000
44:INFO:ubicell:part is <Partition type=1, subtype=2, address=13631488, size=3145728, label=nvs, encrypted=0>, chunk size is 4096, offset is 0x10d000
45:INFO:ubicell:part is <Partition type=1, subtype=2, address=13631488, size=3145728, label=nvs, encrypted=0>, chunk size is 4096, offset is 0x10e000
45:INFO:ubicell:part is <Partition type=1, subtype=2, address=13631488, size=3145728, label=nvs, encrypted=0>, chunk size is 4096, offset is 0x10f000
Traceback (most recent call last):
File "main.py", line 9, in <module>
File "ubicell.py", line 448, in __init__
OSError: 84

same error, same place.

edf012
Posts: 14
Joined: Sun May 09, 2021 8:51 pm

Re: OSError(84,)

Post by edf012 » Mon May 16, 2022 3:06 pm

39:INFO:ubicell:block_num is 3597, chunk size is 4096
39:INFO:ubicell:block_num is 3598, chunk size is 4096
39:INFO:ubicell:block_num is 3599, chunk size is 4096
Traceback (most recent call last):
File "main.py", line 9, in <module>
File "ubicell.py", line 448, in __init__
OSError: 84
MicroPython v1.13-575-g55f51c7d5-dirty on 2022-05-16; COMLTEQTUG with ESP32
Type "help()" for more information.
>>>

I calculate block num using the NVS partition start address divided by the block size. block num is incremented by 1 after each write.

edf012
Posts: 14
Joined: Sun May 09, 2021 8:51 pm

Re: OSError(84,)

Post by edf012 » Mon May 16, 2022 3:24 pm

The other strange behavior is the writeblocks is in a try block. The except statement is pass. Why doesn't micropython ignore this "error"?

Post Reply