Filesystem size with custom STM32F7 build

General discussions and questions abound development of code with MicroPython that is not hardware specific.
Target audience: MicroPython Users.
User avatar
TravisT
Posts: 54
Joined: Sun Feb 23, 2014 2:31 pm
Location: Portland, OR
Contact:

Filesystem size with custom STM32F7 build

Post by TravisT » Wed Mar 20, 2019 1:29 am

I have been working on updating the filesystem flash size based on the nucleo STM32F767 build. I get it to recognize (show is free using an os) a larger filesystem, but unlike my efforts with the STM32F405 edits I get file corruption when passing the 96k threshold (default build filesystem size).

I realize this might be something simply I do not understand or simple and I am overlooking it. The device runs with this code and only noticed it when I pass the 96k threshold and the files show a bunch of "……" (see below).

Code: Select all

  if counter>=600:
   (t0,t1)=(t1,utime.ticks_ms…………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
Below are the main edits I have done to try to grow the filesystem portion of the flash, and utilize more of the flash as a hole. The actual micropython firmware I am trying to push further down in the flash to take up the larger 256k segments and letting the filesystem to be in the smaller sectors and continues.

In flashdev.c I changed the seg1 number of blocks. I am not totally sure why the sector size is 32k and not 64k like on most of the other devices.

Code: Select all

#elif defined(STM32F746xx) || defined(STM32F765xx) || defined(STM32F767xx) || defined(STM32F769xx)

// The STM32F746 doesn't really have CCRAM, so we use the 64K DTCM for this.

#define CACHE_MEM_START_ADDR (0x20000000) // DTCM data RAM, 64k
#define FLASH_SECTOR_SIZE_MAX (0x08000) // 32k max
#define FLASH_MEM_SEG1_START_ADDR (0x08008000) // sector 1
#define FLASH_MEM_SEG1_NUM_BLOCKS (576) // sectors 1,2,3,4,5,6: 32k+32k+32k+64/128+64/256+64/256 = 288

Also in flash.c I increased the flash bank size on the last entry from 3 to 7 to utilize all the banks.

Code: Select all

#elif defined(STM32F7)

// FLASH_FLAG_PGSERR (Programming Sequence Error) was renamed to
// FLASH_FLAG_ERSERR (Erasing Sequence Error) in STM32F7
#define FLASH_FLAG_PGSERR FLASH_FLAG_ERSERR

static const flash_layout_t flash_layout[] = {
    { 0x08000000, 0x08000, 4 },
    { 0x08020000, 0x20000, 1 },
    { 0x08040000, 0x40000, 7 },
};

In the board mpconfigboard.mk I changed TEXT1_ADDR to a higher number to make room for the expanded filesystem.

Code: Select all

MCU_SERIES = f7
CMSIS_MCU = STM32F767xx
MICROPY_FLOAT_IMPL = double
AF_FILE = boards/stm32f767_af.csv
LD_FILES = boards/stm32f767.ld boards/common_ifs.ld
TEXT0_ADDR = 0x08000000
TEXT1_ADDR = 0x080C0000
I also updated the stm32f767.ld linker script to update the location and length of the filesystem and FLASH_TEXT.

Code: Select all

/* Specify the memory areas */
MEMORY
{
    FLASH (rx)      : ORIGIN = 0x08000000, LENGTH = 1024K
    FLASH_ISR (rx)  : ORIGIN = 0x08000000, LENGTH = 32K     /* sector 0, 32K */
    FLASH_APP (rx)  : ORIGIN = 0x08008000, LENGTH = 2016K   /* sectors 1-11 3x32K 1*128K 7*256K */
    FLASH_FS (r)    : ORIGIN = 0x08008000, LENGTH = 288K     /* sectors 1, 2, 3 (32K each) */
    FLASH_TEXT (rx) : ORIGIN = 0x080C0000, LENGTH = 1280K   /* sectors 4-7 1*128Kib 3*256KiB = 896K */
    DTCM (xrw)      : ORIGIN = 0x20000000, LENGTH = 128K    /* Used for storage cache */
    RAM (xrw)       : ORIGIN = 0x20020000, LENGTH = 384K    /* SRAM1 = 368K, SRAM2 = 16K */
}
The MCU has 12 flash sectors in what I believe is the default single bank mode.
4x 32
1x 128
7x 256

What might I be missing or doing wrong? The STM32F405 is working with much less modification. I did use Dfuse to erase the flash before updating to try to reduce possibilities of corruption from other methods (dfu-util with mass erase does not seem to work with the STM32F7).

Thanks for any help
_______________
Travis Travelstead

User avatar
dhylands
Posts: 3202
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Filesystem size with custom STM32F7 build

Post by dhylands » Wed Mar 20, 2019 4:44 am

In order to modify a flash block, it needs to be read into RAM in its entirety, modified and then written back out.

So if you add larger flash blocks to your filesystem you need to also ensure that you have enough RAM to support that.

User avatar
TravisT
Posts: 54
Joined: Sun Feb 23, 2014 2:31 pm
Location: Portland, OR
Contact:

Re: Filesystem size with custom STM32F7 build

Post by TravisT » Wed Mar 20, 2019 5:05 am

This device does have the 512k of RAM, I guess I cannot see the weak points clearly, or I am misunderstanding your point.

I can reduce the filesystem size to something less than half the RAM, which is still plenty for most things. I can also break up how I am copying files over if this is a point where writing to a block causes issues.

When I either freeze and/or minify the code below the 96k threshold I do not seem to have any issues with corruption. The funny thing is that it is always seen in the main.py, which could be because the automated process of copying the files over.

I cannot expect you to have any thorough review of this, but am I hitting the right build related files to accomplish what I am trying to do? This is one of those points that 90% of the references I can find is you Dave helping others with similar questions, but the processes has been evolving over the years so some references are incorrect now, or do not hit on all my points.

Thanks for the reply
_______________
Travis Travelstead

User avatar
dhylands
Posts: 3202
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Filesystem size with custom STM32F7 build

Post by dhylands » Wed Mar 20, 2019 3:22 pm

I should have been more specific. There is something called CACHE_MEM_START_ADDR that points to a block of permanently available RAM which is used for the flash file system.

It looks like this corresponds to the DTCM, which is 128K ?. You didn't mention exactly which chip you're using, so I'm not sure exactly how much DTCM it has. This means that you can only use upto 128K from a flash block and adding larger flash blocks into the filesystem won't work, unless you're willing to say add a 256K flash block, but only use 128K of it, in which case the calculations for the flash block numbers need to be adjusted.

User avatar
TravisT
Posts: 54
Joined: Sun Feb 23, 2014 2:31 pm
Location: Portland, OR
Contact:

Re: Filesystem size with custom STM32F7 build

Post by TravisT » Wed Mar 20, 2019 4:03 pm

Thanks for the reply.

Sorry, yeah I also should have been more specific. I am using the STM32F767, for capability and to leverage the existing work done on the Nucleo.

For playing around with expanding available filesystem, actually a lot of help from several post by you and others, I realize the limitation on the buffer RAM (often only 64k) which limits the usable size of the block.

What is strange to me compared to the other processors is the max size of sector being 32k, and was going to be my next step is to based the size on 32k instead of 64k and that is where the overrun happens just after I spill over the several 32k blocks. I did try to accommodate that i cannot use the entire flash block, but I did that for the 64k buffer and what might be a 32k actual limit.

#define CACHE_MEM_START_ADDR (0x20000000) // DTCM data RAM, 64k
#define FLASH_SECTOR_SIZE_MAX (0x08000) // 32k max
_______________
Travis Travelstead

User avatar
TravisT
Posts: 54
Joined: Sun Feb 23, 2014 2:31 pm
Location: Portland, OR
Contact:

Re: Filesystem size with custom STM32F7 build

Post by TravisT » Wed Mar 20, 2019 4:31 pm

Now having to explain it I thought of this differently and made as assumption to test against.

Based on my last post it shows that 64k of RAM is made available but the max block size is limited to 32k, which is not typical and small. My assumption is that because the default blocks used for the filesystem do not go over 32k, which might be why 32k max block size was used, and not the full available 64k.

With that I decided to change the 32k block limit to 64k. So far it seems to be working, but I it would make me feel a lot better to get it confirmed that this is not an issue.
_______________
Travis Travelstead

User avatar
dhylands
Posts: 3202
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: Filesystem size with custom STM32F7 build

Post by dhylands » Wed Mar 20, 2019 6:54 pm

Looking at the Reference Manual for the STM32F767 it looks like all of the variants have 128K of DTCM

How the flash blocks are arranged depends on whether dual-banked mode is enabled or not, and I think this may depend on the exact variant that you have (the letters that appear after STM32F767 in the chip part number)

I don't think that the DTCM is used for anything other than the flash cache, but I'm not 100% sure.

And yes, FLASH_SECTOR_SIZE_MAX is the amount of memory which needs to be reserved, and it determines the largest amount that can be stored in a flash block.

User avatar
TravisT
Posts: 54
Joined: Sun Feb 23, 2014 2:31 pm
Location: Portland, OR
Contact:

Re: Filesystem size with custom STM32F7 build

Post by TravisT » Wed Mar 20, 2019 7:30 pm

Okay I think this is making more sense now.

Code: Select all

#elif defined(STM32F746xx) || defined(STM32F765xx) || defined(STM32F767xx) || defined(STM32F769xx)

// The STM32F746 doesn't really have CCRAM, so we use the 64K DTCM for this.

#define CACHE_MEM_START_ADDR (0x20000000) // DTCM data RAM, 64k
#define FLASH_SECTOR_SIZE_MAX (0x08000) // 32k max
#define FLASH_MEM_SEG1_START_ADDR (0x08008000) // sector 1
#define FLASH_MEM_SEG1_NUM_BLOCKS (192) // sectors 1,2,3: 32k+32k+32=96k
Most likely out of convenience the F76X MCUs were attached to the F746, even they though there is a reasonable enough "class" difference (like separating the F429 and F405) which makes the configuration limiting out of the box. It would also explain why the numbers are unusually low for the F76X MCUs.

The F746 does only have 64k DTCM RAM, and likely out of convenience the focus was on the 32k blocks and that became the magic number. The F767 (what I am using) has 128k DTCM RAM, and I want to expand past the 32k blocks.

I will update a build and try it

Thanks for helping point this out
_______________
Travis Travelstead

User avatar
TravisT
Posts: 54
Joined: Sun Feb 23, 2014 2:31 pm
Location: Portland, OR
Contact:

Re: Filesystem size with custom STM32F7 build

Post by TravisT » Thu Mar 21, 2019 7:24 am

It would seem that even when trying to utilize the 128k DTCM RAM and a max sector size of 128k, it does not seem to utilize more than 64k. I am wondering if there is somewhere else that is limiting what is usable in each sector.

What I used for testing but it did not appear to provide more space than using 64k settings.

Code: Select all

#define CACHE_MEM_START_ADDR (0x20000000) // DTCM data RAM, 128k
#define FLASH_SECTOR_SIZE_MAX (0x20000) // 128k max
_______________
Travis Travelstead

chrismas9
Posts: 104
Joined: Wed Jun 25, 2014 10:07 am

Re: Filesystem size with custom STM32F7 build

Post by chrismas9 » Thu Mar 21, 2019 10:36 am

By default MicroPython only supports one continuous block of flash memory without holes in it. If you try to use multiple partial sectors, eg 64/128, then there will be 64k holes in the memory map (second half of the big sectors). Most ports use half of one big sector at the end. This is ok because the end of the filesystem is in the middle of the last sector and doesn't have to jump over a hole to another sector.

There is provision in flashbdev.c to add one more partial sector with its own start address and length. See FLASH_MEM_SEG_START_ADDRESS and FLASH_MEM_SEG_NUM_BLOCKS. You could duplicate these definitions for SEG3, etc and update the code to support more segments.

Without code changes you could enable SEG2, increase FLASH_SECTOR_SIZE_MAX to 64k and get:

32 + 32 32 + 64(of 128) + 64(of 256) = 224kB

You could also try increasing FLASH_SECTOR_SIZE_MAX to 128k and try

32 + 32 32 + 128 + 128(of 256) + 128(of 256)= 480kB

This uses 736kB of flash and would only work with a 2MB part. You could disable SEG2 and get 352kB.

The H746 uses FLASH_SECTOR_SIZE_MAX = 128k so big sectors should work, but I haven't tried this.

Note that setting FLASH_SECTOR_SIZE_MAK to 128k will break the F746 unless you split the definitions.

Post Reply