Remove '\x00\x00\x00\x00\x00\x00\x00\x00.\x00\x00\x00' files?

All ESP8266 boards running MicroPython.
Official boards are the Adafruit Huzzah and Feather boards.
Target audience: MicroPython users with an ESP8266 board.
Post Reply
kwiley
Posts: 67
Joined: Wed May 16, 2018 5:53 pm
Contact:

Remove '\x00\x00\x00\x00\x00\x00\x00\x00.\x00\x00\x00' files?

Post by kwiley » Sun Jun 23, 2019 7:32 pm

My flash drive seems to have accumulated numerous files named '\x00\x00\x00\x00\x00\x00\x00\x00.\x00\x00\x00'. I don't understand where they came from, and I have no idea how to get rid of them. os.remove() doesn't work.

kevinkk525
Posts: 522
Joined: Sat Feb 03, 2018 7:02 pm

Re: Remove '\x00\x00\x00\x00\x00\x00\x00\x00.\x00\x00\x00' files?

Post by kevinkk525 » Sun Jun 23, 2019 8:22 pm

I had those as part of a filesystem corruption. You could erase the flash and flash the firmware again.
Kevin Köck
Micropython Smarthome Firmware: https://github.com/kevinkk525/pysmartnode

kwiley
Posts: 67
Joined: Wed May 16, 2018 5:53 pm
Contact:

Re: Remove '\x00\x00\x00\x00\x00\x00\x00\x00.\x00\x00\x00' files?

Post by kwiley » Sun Jun 23, 2019 11:42 pm

I suspected as much. I was hoping for a less nuclear solution.

Thanks.

User avatar
pythoncoder
Posts: 3697
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Remove '\x00\x00\x00\x00\x00\x00\x00\x00.\x00\x00\x00' files?

Post by pythoncoder » Mon Jun 24, 2019 9:26 am

I think it's the only safe solution to a corrupted filesystem.
Peter Hinch

nebelgrau
Posts: 8
Joined: Mon Jun 03, 2019 4:37 pm

Re: Remove '\x00\x00\x00\x00\x00\x00\x00\x00.\x00\x00\x00' files?

Post by nebelgrau » Mon Jul 15, 2019 3:59 pm

Why does this happen, what is the mechanism behind the creation of these "files"? I ran into this a few times and I thought the reason might have been some error that occurred while my script was writing into a file: is that it? Or it "just happens sometimes"? Is it a general MP problem or specific to ESP8266?

Thanks!

User avatar
jimmo
Posts: 878
Joined: Tue Aug 08, 2017 1:57 am
Location: Sydney, Australia

Re: Remove '\x00\x00\x00\x00\x00\x00\x00\x00.\x00\x00\x00' files?

Post by jimmo » Mon Jul 15, 2019 10:56 pm

The most common cause of this sort of filesystem corruption is if writes get interrupted (due to power off or reset).

The way flash works is you can't just write a single byte, you always have to erase an entire "page", then overwrite the page with new data. The reasons for this is that writes can only set bits to zero, and a different page-erase mechanism can set the entire page to ones. (Or the other way around).

So when you write to a file in MicroPython, what actually happens is:
-Read into memory the entire page containing the area ylu want to write (this may be cached)
- Erase the page
- Modify the in-memory copy with your changes
- Overwrite the page

(This is the same on any computer using flash memory or SD cards etc)

If you lose power (or reset) after step 2, then you will end up in a corrupted state. If the page is just your file data, then you'll see blank sections in your file. If the page is a directory table (or the FAT), then you'll see much more serious corruption, potentially including what you're seeing (likely a corrupted directory table).

Other devices (e.g. your PC) use better filesystems (e.g. with journalling) that can recover from failures. MicroPython uses FAT because it works well over USB and can be implemented with a small amount of code, although on ESP8266 the former isn't relevant.

So is it possible that your program is doing lots of filesystem activity and could be reset while it's running?

User avatar
pythoncoder
Posts: 3697
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Beware of MSC mode

Post by pythoncoder » Tue Jul 16, 2019 10:20 am

The commonest cause of corruption is when the flash drive is configured for MSC (mass storage) mode. The nature of mass storage devices such as USB sticks is that the PC expects to have full control over the (presumed dumb) drive. If the device makes any changes to the filesystem all bets are off and corruption is very likely.

I advise disabling MSC mode in boot.py. I very rarely experience corruption on any MicroPython device.
Peter Hinch

kevinkk525
Posts: 522
Joined: Sat Feb 03, 2018 7:02 pm

Re: Remove '\x00\x00\x00\x00\x00\x00\x00\x00.\x00\x00\x00' files?

Post by kevinkk525 » Tue Jul 16, 2019 1:22 pm

This does however not apply to the ESP8266 unless that's a new feature I have never heard of.
Kevin Köck
Micropython Smarthome Firmware: https://github.com/kevinkk525/pysmartnode

User avatar
pythoncoder
Posts: 3697
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: Remove '\x00\x00\x00\x00\x00\x00\x00\x00.\x00\x00\x00' files?

Post by pythoncoder » Wed Jul 17, 2019 6:50 am

Ah, you're right. Been spending too much time with Pyboards ;)
Peter Hinch

Post Reply