ftplib.py ported to MicroPython

Discussion about programs, libraries and tools that work with MicroPython. Mostly these are provided by a third party.
Target audience: All users and developers of MicroPython.
Post Reply
SpotlightKid
Posts: 463
Joined: Wed Apr 08, 2015 5:19 am

ftplib.py ported to MicroPython

Post by SpotlightKid » Tue Jan 24, 2017 10:00 pm

Hi all,

triggered by a request in this post, todayI took ftplib.py form the CPython standard library and made it work with MicroPython.

Here's the result: https://github.com/SpotlightKid/micropython-ftplib

So far the code has been tested and confirmed to work only under the Unix port of MicroPython and against the FTP server from the pyftpdlib package. But in theory it should work on the pyboard and esp8266 too. I'll hopefully confirm this soon.

I put the code in its own repository, rather than making a pull request to micropython-lib, because at the moment the module isn't really "micro". No wonder, considering the first version of the module appeared in 1992! Maybe I'll streamline and clean this up more later and then propose inclusion into micropython-lib.

katesfb
Posts: 22
Joined: Sun Dec 18, 2016 8:09 pm

Re: ftplib.py ported to MicroPython

Post by katesfb » Wed Jan 25, 2017 1:22 am

Hi,
This is brilliant - your effort on this is much apreciated.

I suppose the next question is (given that i am very new to micropython and the pyboard and am only really a beginner programmer in python);
What is the hardware requirement/general aproach to get this to work. We have a couple of options 1) use something like the wipy/esp8266 or 2) if the application is remote logging then use an ethernet to spi adapter to communicate with a router connected to a cell modem.

Any help is much appreciated.

Cheers.

SpotlightKid
Posts: 463
Joined: Wed Apr 08, 2015 5:19 am

Re: ftplib.py ported to MicroPython

Post by SpotlightKid » Thu Jan 26, 2017 12:24 am

As I should have expected, the code for the Unix port didn't work on the esp8266 port out-of-the-box. So I created a special version of ftplib.py for the esp8266 port. It is in the esp8266 directory of the repo linked above.

I stripped this version down considerably by throwing out support for FTP methods, which aren't strictly necessary for listing, uploading and downloading files. But the module is still too big to be compiled on the esp8266 itself, I always got a MemoryError when trying to import it. So I also provide a compiled ftplib.mpy file. This version imports and works on my NodeMCU board with the latest MicroPython version (1.8.7). I was able to upload and download files to my laptop over wifi with no problems.

Share & Enjoy, Chris

User avatar
Roberthh
Posts: 3667
Joined: Sat May 09, 2015 4:13 pm
Location: Rhineland, Europe

Re: ftplib.py ported to MicroPython

Post by Roberthh » Thu Jan 26, 2017 9:19 am

That's very good, and pairs nicely with the ftp server I assembled (https://github.com/robert-hh/ESP8266-FTP-Server). I'll try this night to make a esp8266-eps8266 link, and I expect it to work.
B.T.W.: I had the same experience about the code sice, and am using the server as frozen bytecode. In combination with Filezilla, FireFTP oder some file managers it's very convenient, because you can edit file from your host PC 'in situ'.

stijn
Posts: 735
Joined: Thu Apr 24, 2014 9:13 am

Re: ftplib.py ported to MicroPython

Post by stijn » Thu Jan 26, 2017 12:04 pm

SpotlightKid wrote:I put the code in its own repository, rather than making a pull request to micropython-lib, because at the moment the module isn't really "micro". Maybe I'll streamline and clean this up more later and then propose inclusion into micropython-lib
Just some thoughts, but wouldn't that be a maintaince nightmare? I mean: ideally you now should have some way of patching which you can use to transform the original ftplib.py into something usable by uPy. This way if ftplib gets updates and/or bug fixes, you can just fetch the latest version, apply the patch again and you're done. However if you start adding a ton of changes to make it micro, keeping it in sync with the original becomes harder and harder.

SpotlightKid
Posts: 463
Joined: Wed Apr 08, 2015 5:19 am

Re: ftplib.py ported to MicroPython

Post by SpotlightKid » Thu Jan 26, 2017 12:38 pm

I don't consider the original ftplib.py module a particular good example of Pythonic programming. This was originally written in 1992 for Python 1.0 (!) or even before and over the years things have been added to it using newer features of Python, while other parts of the code have been kept as is.

Also I don't expect the FTP RFC to evolve substantially anymore. In ancient times of the internet the code had to account for various incompatible FTP server implementations and varying support of protocol features. Nowadays, servers have evolved and hopefully are pretty standard. For example, I don't think anyone uses active transfer mode much anymore, since most firewalls or routers won't allow to open arbitrary ports to the internet anyway. So I'm considering throwing out support for active transfers alltogether, which would make the code a loter simpler and shorter and also avoid having to use workrarounds for functions missing from the socket implementation of MicroPython.

The first version of the code was basically just a quick port of the original, to see whether it could be done, with just changing calls to the socket API where there are differences in argument numbers and types etc. But adapting this version to the esp8266 I quickly noticed that there are again different incompatibities to account for and trying to handle this all with the same code would quickly become a mess.

Ideally, I would write a separate implementation specifically for MicroPython from scratch. But I not sure whether the diminishing importance of FTP warrants the effort.

User avatar
dhylands
Posts: 3821
Joined: Mon Jan 06, 2014 6:08 pm
Location: Peachland, BC, Canada
Contact:

Re: ftplib.py ported to MicroPython

Post by dhylands » Thu Jan 26, 2017 4:53 pm

Well, just playing devils advocate here, but active mode is the default mode of ftp, so its probably the one that gets used/tested the most. Most firewalls know how to deal with active ftp, and the first time I've used passive ftp was with the WiPy.

It's more likely that ftp is replaced by sftp or scp.

User avatar
kfricke
Posts: 342
Joined: Mon May 05, 2014 9:13 am
Location: Germany

Re: ftplib.py ported to MicroPython

Post by kfricke » Thu Jan 26, 2017 5:22 pm

dhylands wrote:It's more likely that ftp is replaced by sftp or scp.
*signed*

I personally would prefer TFTP over FTP in embedded environments because it suites the needs of these devices better than a crippled and also insecure FTP connection. Pair it with you DHCP/BOOTP server and you use it likes the Pros do it for the last 35 years.

SpotlightKid
Posts: 463
Joined: Wed Apr 08, 2015 5:19 am

Re: ftplib.py ported to MicroPython

Post by SpotlightKid » Thu Jan 26, 2017 6:31 pm

dhylands wrote:Well, just playing devils advocate here, but active mode is the default mode of ftp, so its probably the one that gets used/tested the most. Most firewalls know how to deal with active ftp, and the first time I've used passive ftp was with the WiPy.
Active transfers do not even work when you're behind a NAT router and have no special application level firewall to extract the IP and port of the client from the PORT message and create an appropriate forwarding rule. Virtually all clients and servers nowadays switch to passive mode immediately after login or before transfers.
dhylands wrote:It's more likely that ftp is replaced by sftp or scp.
Of course, if you have the choice, there are far superior protocols than FTP for file transfer over unsecure networks (internet), e.g. SFTP or simply HTTPS, or simpler ones (e.g. the mentioned TFTP), when you have a local network that's under your control .

I see this as a way to support legacy systems, which allow only FTP upload to transfer bigger amounts of data. Running an FTP server on your board may also be useful to give access to a variety of tools for uploading, but I wouldn't design a new system needing an FTP client on an embedded system nowadays.

User avatar
kfricke
Posts: 342
Joined: Mon May 05, 2014 9:13 am
Location: Germany

Re: ftplib.py ported to MicroPython

Post by kfricke » Thu Jan 26, 2017 8:07 pm

SpotlightKid wrote:Active transfers do not even work when you're behind a NAT router and have no special application level firewall to extract the IP and port of the client from the PORT message and create an appropriate forwarding rule. Virtually all clients and servers nowadays switch to passive mode immediately after login or before transfers.
I think this is a little under-estimated. Most "better" Linux based home and SoHO routers do use the ip_conntrack kernel module to extract the needed context from the FTP sessions. Sure the router needs to support this actively and there are a few other "issues" with FTP (passive and active mode).

tbh, FTP is my most hated protocol from the old days of security by obscurity design.

Post Reply