rsync timeout

[smhodge]

rsync displays a message "timed out or error in transfer to remote" message when I attempt to update files on /flash. I have about 10 or so total *.py files that potentially could be updated. It often gets 2 or 3 or more copied but then it bombs.

This symptom has only recently cropped up (as in the last month or so). Prior to that I used rsysnc successfully to update files countless times w/o any problem. Initially I was able to kick it enough times, doing board resets, etc., that it would eventually "take" and complete the update, but lately that has not worked.

Instead the only reliable way I have of fixing it is to do a firmware upgrade with the mass erase option. rsync will then work reliably for a "few" or "several" more times, but then it inevitably starts failing again.

My files have of course been growing in size as time goes on, so I'm wondering if I'm hitting a total size limit with the local filesytem on /flash. I can't find anywhere in the docs that state what that is but no doubt I'm not looking in the correct place. My files currently total about 54 KB.

Does anyone have a suspicion as to what is going on?

Thanks

[jimmo]

When you say rsync, I assume you mean via rshell?

Which Pyboard do you have (so I can check flash size etc).

Assume you're using FAT not LittleFS?

[smhodge]

Yes, the one on github by dhylands. Version 1.1 pyboard, latest firmware (1.12).

[smhodge]

And I assume FAT, whatever the default is.

[pythoncoder]

That is odd: I use rshell/rsync daily. On Pyboard 1.x I generally use an SD card, but on Pyboard D and ESPx I use flash. The only time I've had problems have been with large numbers of files on an SD card when operations like ls can fail. I assume you've checked the contents of /flash to be sure that there really are just 54K of files on there - could you be inadvertently copying hidden files like git data across?

Otherwise maybe the flash is failing - you could try an SD card.

[smhodge]

Yes, listing the files on /flash is how I got the 54K number. Also even though I can fix it by re-writing the firmware and doing a mass-erase, I recently had the issue start again after just using rshell a couple of times to tweak one or two files w/o changing their size signficantly. I'll try an SD card next time it happens but I find that less convenient when debugging. I'd still like to know if I'm close to some /flash limit with 54K. Is there a way to test if the flash is ok?

[jimmo]

I'd still like to know if I'm close to some /flash limit with 54K.

The filesystem limit on pybv1.1 is 112K.

I realise you're using rshell, but do you have USB MSC enabled? Because the way this works is by presenting the flash device almost exactly to the PC over USB, any of the Windows/Mac/Linux filesystem tools will be able to tell you about the state of the filesystem (including how much free space and capacity there is). I would be interested to see what happens if you ran a scandisk/fsck on the volume after running into the error.

I'm not sure this is the issue, but there's definitely a way that the filesystem can lose track of sectors (I forget the term that MSDOS scandisk used to call them). We actually have support for doing a rudimentary on-device scandisk/fsck for FAT in MicroPython (see https://github.com/micropython/oofatfs/ ... 1a6815ad2d) but this isn't enabled on any boards.

The other thing I'd suggest is to try moving to LittleFS instead of FAT. To do this on your Pyboard, see http://docs.micropython.org/en/latest/r ... filesystem
(My working assumption is that for everything except USB MSC, LittleFS is superior to FAT)

[smhodge]

My boot.py file contained the line:
pyb.usb_mode('VCP')
i.e., only the virtual serial port. My understanding was that was recommended(?) for rshell/rsync(?) but I can't recall for sure why I switched to that.

I just now changed it to:
pyb.usb_mode('VCP+MSC')

...and I can copy files manually using a file manager from PC to flash drive just fine. It means I have to manually select the files that need "syncing" but that's no big deal. I'm not sure whether or not rshell/rsync will now work reliably or not -- I need to do more tests, but I'll post when I know for sure.

Meanwhile, a question about LittleFS. Are you proposing that so I can use rsync/rshell? If I try that, it sounds like I would need to undo the USB MSC, i.e., go back to pyb.usb_mode('VCP'). Is that correct?

Thanks

[jimmo]

My understanding was that was recommended(?) for rshell/rsync(?) but I can't recall for sure why I switched to that.

If you're not using the MSC functionality it's best to turn it off so that there's no chance of your host PC stuffing around with the filesystem.

But to use rshell you just need VCP, it doesn't matter if you have MSC enabled or not.

My point was that MSC just provides a very convenient way to look at the block device from your host pc where you have a full set of tools to investigate it. I definitely do not recommend using MSC as your regular goto way to get files on/off your device, because it's not quite reliable enough (but it's great for quick demos and getting started).

Meanwhile, a question about LittleFS. Are you proposing that so I can use rsync/rshell? If I try that, it sounds like I would need to undo the USB MSC, i.e., go back to pyb.usb_mode('VCP'). Is that correct?

Yes, I recommend always using rshell. (Or any of the approaches based on that technique, e.g. I mostly use pyboard.py and mpreplfs, but rshell is good for your purposes).

You don't need to disable MSC to use LittleFS, it just means that your PC will complain that it doesn't understand what filesystem the device is using (although you can work around this on Linux with the LFS FUSE driver).

[dhylands]

I just postred version 0.0.28 of rshell which includes a user contribution which increases one of the timeouts and apparently improves rsync timeout problems.