SD card file write

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
User avatar
SureshVakati
Posts: 42
Joined: Fri Feb 24, 2017 3:52 pm

SD card file write

Post by SureshVakati » Fri Mar 03, 2017 3:48 am

Hello,
I am trying to save some run-time configurations to a text file using ujson module.

Configurations look like this.

Code: Select all

{"ARM": "0",
 "T": "Test",
 "P4": "phone4",
 "C2": "disarm": "disarmed",
 "P1": "phone1",
 "A4": "alarm IN4",
 "P3": "phone3",
 "P2": "phone2",
 "A1": "alarm IN1",
 "A2": "alarm IN2",
 "C1": "arm": "armed",
 "D": "Reset OK",
 "A3": "alarm IN3"}
I am trying to open a text file like this.

Code: Select all

with open('config.txt','w')as f:
     f.seek(0)
     f.write(ujson.dumps(data))
     f.flush()
os.sync()
As my configuration file keep on increasing during the execution, my data is getting stripped at the end while writing to a file using above code (but if I print out what I am writing, it is showing correctly) when reading the config file again, I am getting JSON syntax error due to the stripped data while writing. Any alternatives? Is this the best way to store program data?

Am I supposed to split the data into chunks and write to a file? I am also seeing NUL character is being added at the end of the data, so added f.seek(0) before writing to the file.

If I delete default boot.py and main.py and configuration files, it is creating boot.py only on hard reset. Some times even that boot.py only contains y symbols.

Thank you!

User avatar
pythoncoder
Posts: 5956
Joined: Fri Jul 18, 2014 8:01 am
Location: UK
Contact:

Re: SD card file write

Post by pythoncoder » Fri Mar 03, 2017 7:11 am

Firstly the data you presented is not a valid Python dictionary: lines 4 and 11 have syntax errors. With a valid dictionary there should be no need to seek or sync: when you open a file in 'w' mode you will overwrite its entire contents. So, with data hacked to fix the syntax errors,

Code: Select all

data = {"ARM": "0",
 "T": "Test",
 "P4": "phone4",
 "C2": "disarm",
 "C3" : "disarmed",
 "P1": "phone1",
 "A4": "alarm IN4",
 "P3": "phone3",
 "P2": "phone2",
 "A1": "alarm IN1",
 "A2": "alarm IN2",
 "C1": "arm",
 "C2": "armed",
 "D": "Reset OK",
 "A3": "alarm IN3"}
 
with open('config.txt','w')as f:
    f.write(ujson.dumps(data))
followed by

Code: Select all

with open('config.txt','r') as f:
    d = ujson.loads(f.read())
    print(d)
works for me.
Changes to the dictionary at runtime should have no effect as you're overwriting the file from scratch each time.
Peter Hinch
Index to my micropython libraries.

User avatar
SureshVakati
Posts: 42
Joined: Fri Feb 24, 2017 3:52 pm

Re: SD card file write

Post by SureshVakati » Fri Mar 03, 2017 12:13 pm

Sorry, I manually typed the dictionary in my phone, so got these errors. This is what "C1":{"arm":"armed"}, "C2":{"disarm":"disarmed"} should be. I was doing exactly what you did above. I was seeing NUL characters each time I read the file, so added f.seek(0) for both read and write.

It is working for me too when the configuration is as I posted. When the configuration doubles, its stripping out in the middle.Is it because my code so big and left with a little memory available during runtime? But I don't see memory allocation errors. I am using gc.collect() before processing big strings.

I will post the log tomorrow, as I am typing it in phone, I don't have them now. So that we can see what's going on.

Thank you.


Sent from my ONEPLUS A3000 using Tapatalk

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: SD card file write

Post by dhylands » Fri Mar 03, 2017 4:22 pm

If you're writing to the sdcard from pyboard, then you really need to make sure you disable USB Mass Storage, otherwise you will get file corruption from the PC also overwriting the data that the pyboard wrote.

User avatar
SureshVakati
Posts: 42
Joined: Fri Feb 24, 2017 3:52 pm

Re: SD card file write

Post by SureshVakati » Fri Mar 03, 2017 5:37 pm

Thanks Dave,
I am using pyb.usb_mode('VCP+HID'). If I use pyb.usb_mode('VCP') on STM32f4, how do I reset it back to 'VCP+HID'?Is there anyway to do safe boot?

sometimes,I am also seeing time.time() is returning 0, are my board configurations correct?

Code: Select all

// HSE is 16MHz
#define MICROPY_HW_CLK_PLLM (16)
#define MICROPY_HW_CLK_PLLN (336)
#define MICROPY_HW_CLK_PLLP (RCC_PLLP_DIV2)
#define MICROPY_HW_CLK_PLLQ (7)
#define MICROPY_HW_CLK_LAST_FREQ (1)

// The pyboard has a 32kHz crystal for the RTC
#define MICROPY_HW_RTC_USE_LSE      (1)
#define MICROPY_HW_RTC_USE_US       (0)
#define MICROPY_HW_RTC_USE_CALOUT   (1)

Code: Select all

import time
>>> start=time.time()
>>> time.time()
0
If I unplug the power and drain all the capacitance in the board by pressing hardware reset button for 5 or 6 seconds, time module works fine again.

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: SD card file write

Post by dhylands » Fri Mar 03, 2017 7:33 pm

To switch back you can do a safe boot:
http://docs.micropython.org/en/latest/p ... boot-modes

If you use safe boot it will come up with USB Mass Storage enabled.

You could also prepare an sdcard which when booted modifies the /flash filesystem.

User avatar
SureshVakati
Posts: 42
Joined: Fri Feb 24, 2017 3:52 pm

Re: SD card file write

Post by SureshVakati » Fri Mar 03, 2017 9:33 pm

Yeah, I read that document before. My board is custom made with STM32F405 and doesn't have user switch. Is there any other way doing it?

Something is wrong with my file read and write operation, I am seeing NUL characters at the end of my Config file.

sometimes time.time() returns 0, what could be the reason for it?
Attachments
NUL_File.PNG
NUL_File.PNG (4.04 KiB) Viewed 9248 times

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: SD card file write

Post by dhylands » Fri Mar 03, 2017 9:49 pm

If you don't use the user switch then you'll need to provide some other way of doing that if you want. i.e. you'll need to write some explicit code in your boot.py to check something - like a GPIO or whatever it is you want to use to indicate that you want to change the USB mode at boot time.

If I had to guess about the time, I'd imagine that it's probably related to your RTC crystal not oscillating properly or something, since that's what's used to control the time. I've not seen that behaviour on any of the boards I work with.

Without a user button, there is also no way to perform a filesystem reset. You should probably do a mass erase so that the filesystem gets reinitialized.

Using USB Mass Storage can cause all kinds of interesting file corruption, especially if python code is writing to the file system (USB Mass Storage assumes that the host has exclusive access to the filesystem and this is violated when python code writes to the filesystem). I've never seen the nul problem you describe. Does the same code running on a pyboard behave the same?

User avatar
SureshVakati
Posts: 42
Joined: Fri Feb 24, 2017 3:52 pm

Re: SD card file write

Post by SureshVakati » Fri Mar 03, 2017 9:58 pm

I don't have a pyboard to test my code on it. Could you check my RTC configurations I posted above? Do you see anything wrong in there? I have HSE Frequency(Hz), 16000000, PLL_M 16, I think I configured my build to use external 32K crystal.

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: SD card file write

Post by dhylands » Sat Mar 04, 2017 4:03 am

The settings look reasonable.

Is the crystal actually generating a 32 kHz signal?

Post Reply