FtpWebRequest is Broken

I’m going to take a brief break from helpful solutions to have a bit of a rant about FTP support in the .NET Framework.

When all you have is hammers..

FTP as a protocol is pretty lousy, but its support in .NET (see System.Net.FtpWebRequest) is especially terrible. If you can use an alternative, I suggest you do.

Here’s just a small list of alternatives (evaluate each based on community feedback, release/update activity, user support, and your own testing):

• Alex FTPS Client – Seems active, well-rated, and gets lots of positive community reviews on places like StackOverflow.
• Starksoft FTP and FTPS C# Client Library – Seems decent, and is written by a guy called Benton Stark which is a pretty awesome name. Check out the Google Code repository for the latest code (the CodePlex project is out of date, and I believe the binaries on the Starksoft site are also out of date as of Revision 63).
• C# FTP Library – A C# wrapper for WinINet FTP functionality. There’s a couple of other WinINet/FTP articles, including an article on WinINet FTP limitations, a C++ sample, a full WinINet-based FTP client with C++ source that originated from a well-rated The Code Project article
• There are other projects in various states (libftp, OpenNETCF FTP Library, System.Net.FtpClient).
• There are some fairly excellent-looking commercial libraries (Limiblabs Ftp.dll, Rebex FTP/SSL for .NET/.NET CF, edtFTPnet/PRO) which take every opportunity to post how they can’t help you solve your problem with the standard libraries but if you feel like forking out some cash your problem is solved great they are (Rebex being the most notorious I’ve seen for this). The benefit of these of course being you’re in good company (look at some of the clients for the more popular libraries) and have someone to blame fix any issues you have.

Why not just use FtpWebRequest?

Why not System.Net.FtpWebRequest you say?

I’ve talked before about a small framework issue when using FtpWebRequest (and have others still to post) but I’ll let that one slide in my criticism here (it’s probably System.Net.NetworkCredential‘s fault that another framework class called its internal methods).

So let’s focus on a few of my other criticisms (most of which apply equally to both FtpWebRequest and HttpWebRequest):

• The interface is not high-level enough to adequately wrap FTP functionality. This leads to a fair bit of boilerplate (as noted years ago, compare the Ruby code in that article). It also leads to inadequacy on basic tasks. For instance, there’s no real standard on directory listings, so you’re forced to try and write your own server-specific directory listing parser (I’ve previously adapted DirectoryListParser from Alex FTPS Client, itself adapted from Adarsh’s post). Likewise, server behaviour seems to vary on “FileExists” meaning the “best” solution is to wait for an exception.
• The interface does not expose adequate control of important behaviors. For instance, connection pooling is implicit and handled fairly obscurely – see Adarsh’s post, jpsander’s post, and System.Net.ServicePointManager. Don’t even try and get hold of the actual connection. FTP servers being so inconsistently implemented, the ability to manually modify even seemingly inconsequential behaviours becomes essential.
• The actual implementation is less than adequate (and largely intermittently so) and relying on MSDN documentation is likely to mislead you on best practice usage. For some example issues see David Zhang’s issue with HttpWebRequest in CF, Benjamin Nitschke’s issues with HttpWebRequest performance, and the numerous threads on (frequently intermittent and hard to reproduce when testing) “underlying connection was closed” issues.

I’ve personally dealt with these “underlying connection was closed” issues in rare circumstances where a connection is lost and therefore “breaks” protocol without properly closing the connection, which:

1. Leads to a series of connections stuck in TIME_WAIT just in case there’s more traffic to come.
2. Which leads to an exhaustion of the ServicePointManager connection pool count
3. Which means all new connections encounter the “underlying connection was closed” issue
4. Which, requires a restart of the AppDomain (i.e. your application)

Usually someone will suggest the largely hit-and-miss idea of changing KeepAlive to false (or true if you’ve already tried that). The more effective but hacky solution seems to be trying to subvert .NET connection handling and leveraging reflection hacks.

Should a mature framework really have such a leaky abstraction?

It can’t be that bad?

The sheer fact that there are so many successful commercial libraries for something as seemingly fundamental as FTP connectivity confirms to me that the standard libraries are simply inadequate.

The cherry on top of this rather unappetising cake is that Microsoft is not only well-aware of the library’s inadequacies, but has given up on its own FTP library:

• Microsoft is using Rebex FTP/SSL and SFTP in Microsoft Visual Studio 2010
• Microsoft uses the same Rebex component in place of FtpWebRequest in the BizTalk FTP Adapter (decompile the adapter if you’re curious)

Hopefully we can see Microsoft either buy (and integrate, particularly as many of the commercial libraries are fairly low-level) or build an adequate FTP library in a future version of the framework. I’m not holding my breath.