2MB flash on the wipy?

[danicampora]

Even without the 192KB flash space limitation, there's not enough RAM to load 2MB of Python code...

[iandouglas]

we don't have 128K of free RAM

Considering this was just FTP, Kickstarter advertised the WiPy with 256Kb of RAM, but it's still a good point. Many of the files I was trying to send were 4kb-30kb in size; I think the largest was about 100kb.

[dhylands]

we don't have 128K of free RAM

Considering this was just FTP, Kickstarter advertised the WiPy with 256Kb of RAM, but it's still a good point. Many of the files I was trying to send were 4kb-30kb in size; I think the largest was about 100kb.

The way that flash works, you can only set bits to zero. To set a bit to 1 you need to erase the entire page (which may be 128K) which sets all of the bits to 1. The file system is arranged as 512 byte blocks, so to update a sector which already has bits in it (like the directory containing the file entry) you need to copy the contents of the old flash block into RAM, modify it, erase the page in flash and write the updated RAM copy back.

So even creating a zero byte file still needs 128K bytes of RAM (if you're dealing with 128K flash pages. The pyboard has some 16K and 64K flash pages, and these are used for the filesystem (except for the very first 16K page which contains the interrupt vectors). The 112K comes from 3x16K + 1x64K and we have to reserve 64K of the 192K of RAM just to allow this to happen.

Just trying to explain some of the limitations even though it seems like there is lots of flash.

[marfis]

you need to copy the contents of the old flash block into RAM, modify it, erase the page in flash and write the updated RAM copy back

thats one way to do it but it's dangerous and needs careful considerations with the power supply hw. obviously you'll get corrupted data if power is removed during the write/erase process.

Another way is to do bank swapping, i.e. you have one spare flash bank so you can safely erase/write data into it and mark it as valid after write completion. if power is removed during the write the original flash page remains intact. This also has the advantage that you will only need to RAM buffer the amount of data to be written - not the whole flash page (you can patch the data during the copy process).

It does add complexity because you'll need to track the flash pages order for the filesystem.

[pythoncoder]

@marfis It would be slower: erasing a flash sector takes time (at least on devices I've used).
@iandouglas I suggest you look at this from a different starting point: that of your application. Rather than have a fully blown webserver on each WiPy, keep the nodes doing the sensing simple. Each node could communicate its readings with a central one running the webserver. The latter could be something running cPython under Linux such as a Raspberry Pi: this would (presumably) be capable of running falcon without difficulty. There are various ways in which it could acquire data from the WiPy units: one way is using sockets but others may be able to advise better or easier approaches.

I'd suggest that the WiPy is ideal for your purpose for everything except the central webserver. Even if the WiPy had vast resources it wouldn't reduce the task of porting falcon to MicroPython.

The WiPy also offers the possibility of your sensor nodes being battery powered. Safer and more likely legal in a bathroom

[dhylands]

I agree (about the reliability issue). It gets more complicated only having 3x16K and 1x64K pages to play with.

You'd probably have to waste 2 of the 128K pages in order to do it properly (only use 16K of one of the 128K pages, and 64K of the other, so that you could ensure you always had a page to use.

I could see some people wanting a "high reliability" option even though it wasted 256K of flash (IIRC the pyboard has 1 Mb and we're currently using less than 512K). Using an external device (like a serial EEPROM or flash) would make implementing something like that much easier.

And I'm not sure I would consider the FAT filesystem to be terribly reliable anyways.

Currently, in order to minimize the number of times we have to write to the flash, we try to buffer changes in memory and delay flushing them out on every write. This allows multiple writes into different sectors in the same flash block to be bundled together. So if you lost power in that time period you would also lose stuff.

To do the reliability thing justice, you really need to use a proper flash file system, like JFFS2 or YAFFS, but those filesystems are not targeted for the small environment we're running in.

For super reliability, you wouldn't use the filesystem at all, and put your code modules in as frozen code. Then it never gets rewritten.

[pythoncoder]

@dhylands The low power consumption of these boards make them ideal for running from a supply with a battery backup. Alternatively a capacitor might be used, with the application detecting imminent power loss, flushing the filesystem and shutting down.

[marfis]

Just to be clear, I have no problem with 192kb Flash left. I love the Wipy and congratulate Dani for this great achievement.

So the reliability issue is a bit off topic. Yet interesting to discuss.

@pythoncoder: you always have to erase a bank, the difference is where you store the data while erasing/writing. Or did you mean the time penalty of erasing 64kb vs 128kB?

probably have to waste 2 of the 128K pages in order to do it properly (only use 16K of one of the 128K pages, and 64K of the other, so that you could ensure you always had a page to use.

If you mirror both pages you mean?
I'm not sure how long the erase cycle takes (16kb /64kb vs 128kb). But in principle it should be possible to use n-1 pages, that is if you had e.g. 4x128kb pages you'd use 3 of them and use the spare to savely write. The spare bank is not a fixed one. E.g. if you'd write bank 1 you'd mirror/write it on bank 4 then invalidate bank 1 etc.

And this logic makes it more complex. The benefit is that you wouldnt need to RAM buffer all that data. Besides reliability of course.

would consider the FAT filesystem to be terribly reliable anyways

i agree

Alternatively a capacitor might be used, with the application detecting imminent power loss, flushing the filesystem and shutting down.

A capacitor is also dangerous to use as a buffer. Consider the f4 running at 168Mhz and a write cycle. A rough guess makes this a current of more than 100mA, lasting for maybe 50msec? So if you allow 1v drop you'd need 5000uF! Even for very low power devices such as a msp430 I wouldn't try writing flash while the supply is dropping fast.

Frozen bytecode is probably the best solution as @dhylands pointed out.

So if you lost power in that time period you would also lose stuff.

yes but the issue is not losing data but having a corrupted file system from where you can't recover

[Roberthh]

That's a good discussion about PyBoard and Flash. But coming back to WiPy. As far as I understand the schematics and the data sheets, the WiPy uses a serial flash with 4 k pages. I guess that the limitation comes from the fact, that WiPy reserves space for about 4 copies of Micropython for safe boot and update. That alone would account for about 1 MByte of flash.
But what about an option for a WiPY with larger serial flash. The data sheet I looked at shows devices with up to 64 MByte of flash.
But on the other hand, compared to the size of a power supply, a small Micro-SD adapter should always fit somewhere.
Regards

[pythoncoder]

@marfis

you always have to erase a bank

I'd assumed you'd have to erase two, to write the buffered data back to its original location. But I take your point that if you can simply remap the new sector then you only need to erase one. Does the FAT filesystem have that flexibility? Or are you assuming a layer below the filesystem? In either case the act of remapping implies a write to a mapping table with attendant possible erase. However filesystems aren't my forte: I'm entirely willing to accept I have the wrong end of the stick here

Regarding capacitors, in my measurements a busy Pyboard takes 50-60mA average. A drop of several volts would be manageable given sufficient voltage prior to the regulator. But I agree the cap would still have to be large, and calculating the worst-case time to flush the filesystem might be difficult. A one farad supercap would fix it