How to Change the Commands FtpWebRequest Sends

Below I discuss how I’ve worked around some limitations of the System.Net.FtpWebRequest to allow low-level customisation of the actual commands sent by the FtpWebRequest class.

This allows resolution of a couple of issues, including:

• Customising the logon sequence (allowing support for an FTP Proxy such as the Bluecoat ProxySG)
• Removing the hard-coded OPTS command for servers that do not support the command.

If you’re here for the workaround click here to see the solution, and for everyone else I’m going to be as self-indulgent as usual and detail the background of my workaround.

Background and security exploit in FtpWebRequest

My particular discovery of this workaround was unintentional – I was looking to change the USER/PASS sequence sent by FtpWebRequest (including sending an ACCT command after the USER and PASS). To achieve this I tried sending an embedded command as my PASS by setting the password of my FtpWebRequest‘s NetworkCredential.

NetworkCredential n = new NetworkCredential("SomeUser", "SomePass\nACCT SomeAcct");

Surprisingly, this actually (almost) worked! It actually sent ACCT SomeAcct as an FTP command (and this is a whole new space for a security vulnerability).

Unfortunately I was then (seemingly) getting errors on the OPTS command, which led me down the path below. What I found out later was that injecting the command caused internal miscounts on command processing, which meant the responses from the server were out of alignment with the commands sent and couldn’t be properly correlated.

How FtpWebRequest Sends Commands

It was time to decompile System.Net.FtpWebRequest and I decided to start at the GetResponse method, which seemed to be the method where actual commands were sent to the server.

Looking through the method, it seemed that the basic structure was:

1. If using a HTTP Proxy, delegate to HttpWebRequest / HttpWebResponse
2. If using asynchronous methods “do some stuff I didn’t understand at the time”
3. If using synchronous methods call the private method SubmitRequest

Checking out SubmitRequest, it seemed that that there was some stuff happening with a System.Net.FtpControlStream – an internal class which apparently was the actual type of the m_connection variable being used throughout FtpWebRequest.

 FtpControlStream ftpControlStream = this.m_Connection;

I jumped into FtpControlStream and decided to do a quick search for “OPTS”. Sure enough there were a couple of references:

The first reference was in a PipelineCallback method, which seemed to be doing some pretty hacky stuff using string evaluation after each command to work out what to do. This didn’t look good.

if (entry.Command == "OPTS utf8 on\r\n")
{
  if (response.PositiveCompletion)
    this.Encoding = Encoding.UTF8;
  else
    this.Encoding = Encoding.Default;
  return CommandStream.PipelineInstruction.Advance;
}

The second reference was in a BuildCommandsList method, which looked to be building the list of commands run as part of an FTP request. This looked much more promising, and sure enough there was a line adding the OPTS command to the queue:

  arrayList.Add((object) new CommandStream.PipelineEntry(this.FormatFtpCommand("OPTS", "utf8 on")));

This explained why Thomas Levesque’s answer on StackOverflow was that OPTS is hardcoded.

Changing the Commands FtpWebRequest Sends

I was starting to get a bit of a picture of how this worked:

1. A queue of commands (a CommandStream) is built (BuildCommandsList)
2. Each command is processed in sequence
3. A callback (PipelineCallback) is fired as each command is processed allowing actual functionality

The next question was – how could I modify this commands list before it was sent to the server?

Unfortunately Monkeypatching (rewriting existing code) isn’t currently possible in C#, but maybe I could make my own CommandStream / FtpControlStream and replace (using reflection) the m_Connection variable in FtpWebRequest with my new class?

Unfortunately Eric Lippert says this is impossible and Jon Hanna’s attempts on StackOverflow were stopped by TypeLoadExceptions. My own quick attempts ran into the same issues so this seemed a no-go.

I read through the code of CommandStream, FtpControlStream, and FtpWebRequest a bit more, and realised there were quite a few callbacks around the place – if just one of these happened to fire at the right time (after the command list was built, but before a given command was sent to the server) I might be able to hook in my own callback!

Unfortunately most of the callbacks were implemented as direct method references, and thanks to “no monkeypatching” I wouldn’t be able to change these. However, a couple of callbacks used member AsyncCallback variables, and these I could repoint to my own code!

After running a few LINQPad tests (injecting a basic “Hello World” callback of my own into these various callback options) to work out when these different callbacks fired, I’d narrowed down my target: m_WriteCallbackDelegate in System.Net.CommandStream.

m_WriteCallbackDelegate seemed to fire after each command was sent to the server – as long as I didn’t want to have a command other than USER sent first (or AUTH TLS if SSL was enabled) this was the callback I was after.

My basic strategy would be to inject a callback that would:

1. Check if the position in the command list is at the position expected to make a change (so that modifications are not applied twice, as the callback fires for every command)
2. If so, modify the built command list (for instance, remove the command if I don’t want it sent)
3. Hand back to the original callback (so everything can continue as normal)

With some reflection this part was actually surprisingly easy once I worked out what was happening.

Gotchas

Unfortunately there were two gotchas:

1. The callback was only fired when the asynchronous methods of FtpWebRequest were used (BeginGetResponse / EndGetResponse).
2. The callback variable (m_WriteCallbackDelegate) was static – this would affect every FtpWebRequest in the current AppDomain!

Resolving the first point wasn’t the end of the world – I could rewrite my code a little (and deal with the terrible asynchronous FtpWebRequest interface) to call the asynchronous methods but still work synchronously (or asynchronously if I so chose).

The second point was however a little trickier. What I was after was to only apply the callback on specific FtpWebRequest instances (those that I needed my modifications to apply to).

My first thought was to store a thread-safe list of IDs for the requests that I wanted to apply the special callback for. I could then add the ID of new requests to this list on demand, and when the callback fired it would check the current request’s instance ID against that list.

Unfortunately something like GetHashCode collides really quickly (my quick LINQPad tests showed collisions after around 10,000 objects), and there was no way for me to add a new variable to an internal class (I could add an Extension Method but these wouldn’t allow me to add a member-level property). If only there were some way to dynamically attach data to an object instance.. oh wow there is!

The ConditionalWeakTable<TKey, TValue> class enables language compilers to attach arbitrary properties to managed objects at run time.

Source: http://msdn.microsoft.com/en-us/library/dd287757.aspx

The ConditionalWeakTable meant that I could keep a thread-safe dynamic collection (with weak references removing GC issues!) of the FtpWebRequest instances that should use my new callback, and the callback could at run-time check if the current instance should apply the new rules (or simply hand back to the original callback).

The Final Workaround

So what I had worked out at this point was:

1. FtpWebRequest uses a FtpCommandStream to store a queue of commands (each represented by a PipelineEntry) to send to the FTP server.
2. This command stream generates a series of commands to execute per request.
3. This means if the command stream is modified for each request, the commands sent by FtpCommandStream can be modified.
4. Facilitating this, when using the asynchronous methods of FtpWebRequest, a callback method is fired (by FtpCommandStream‘s parent class ControlStream) after each command is sent to the server.
5. This allows us to inject/remove/change a command in the stream then hand back to the standard callback.
6. This callback is static so any replacement will need to happen once per AppDomain and ensure that changes are only applied to the desired instances of `FtpWebRequest

Wrapping this all up gave me an injected custom write callback which I chose to setup with a static initializer, and I’m hoping you can adapt to your own needs. It also required a couple of extension methods which I’ve included below for changing an array generically (without design-time declaration of its type, as we cannot declare the internal PipelineEntry type).

In my particular case I wanted to inject additional FTP commands, so my callback when fired would run through a Queue of commands to inject. Each item in this queue had a condition to check against the “last run command” (stored as Func<string, bool>) and if this condition was true, a “command” to inject as the next command. However my example below shows how to perform both a command removal and insertion.

Remember this relies on leveraging internal behaviour of the framework, which could (though I doubt it for minor releases) change without warning at any time.

If you have any questions or trouble adapting this to your particular needs feel free to leave a comment or get in touch.

// In custom class FtpCallbackInjector
// NB: Requires usage of the asynchronous FtpWebRequest methods (Begin/End)
// NB: In this sample, your request must be added to MarkedRequests

// Could strong-type "object" here - it's the associated per-request "state"
private readonly static ConditionalWeakTable<FtpWebRequest, object> MarkedRequests = new ConditionalWeakTable<FtpWebRequest, object>();

// Static Initializer
static FtpCallbackInjector()
{
    // Get access to (post)-write callback delegate
    Type commandStreamType = typeof(FtpWebRequest).Assembly.GetType("System.Net.CommandStream");
    FieldInfo commandStreamWriteCallbackField = commandStreamType.GetField("m_WriteCallbackDelegate", BindingFlags.NonPublic | BindingFlags.Static);

    // Store original delegate for chaining
    AsyncCallback originalDelegate = (AsyncCallback)commandStreamWriteCallbackField.GetValue(null); 

    // Inject our own delegate (which in turn calls the original delegate)
    commandStreamWriteCallbackField.SetValue(null, new AsyncCallback((asyncResult) =>
    {
      /* Get field/method references via reflection */
      Type commandStreamPipelineEntryType = typeof(FtpWebRequest).Assembly.GetType("System.Net.CommandStream+PipelineEntry");
      FieldInfo commandStreamCommandsField = commandStreamType.GetField("m_Commands", BindingFlags.NonPublic | BindingFlags.Instance);
      FieldInfo commandStreamIndexField = commandStreamType.GetField("m_Index", BindingFlags.NonPublic | BindingFlags.Instance);
      FieldInfo commandStreamRequestField = commandStreamType.GetField("m_Request", BindingFlags.NonPublic | BindingFlags.Instance);
      FieldInfo pipelineEntryCommandField = commandStreamPipelineEntryType.GetField("Command", BindingFlags.NonPublic | BindingFlags.Instance);
      MethodInfo formatFtpCommandMethod = typeof(FtpWebRequest).Assembly.GetType("System.Net.FtpControlStream").GetMethod("FormatFtpCommand", BindingFlags.NonPublic | BindingFlags.Instance);

      // asyncResult.AsyncState is the System.Net.CommandStream
      object commandStream = asyncResult.AsyncState; 

      // Reference to executing request
      FtpWebRequest request = (FtpWebRequest)commandStreamRequestField.GetValue(commandStream); 

      /* Check if executing request is marked */
      // Associate anything you like here with the request
      object markedState = null; 
      if (!MarkedRequests.TryGetValue(request, out markedState))
      {
        // If not marked simply chain and return
        originalDelegate(asyncResult);
        return;
      }

      // Array of PipelineEntry
      Array commands = (Array)commandStreamCommandsField.GetValue(commandStream); 
      // Current (just-executed) index in CommandStream
      int commandStreamIndex = (int)commandStreamIndexField.GetValue(commandStream); 
      // Current (just-executed) command of type System.Net.CommandStream+PipelineEntry
      object currentPipelineEntry = commands.GetValue(commandStreamIndex); 
      // Current (just-written) command string
      string currentCommand = (string)pipelineEntryCommandField.GetValue(currentPipelineEntry); 

      // If the just executed command passes some kind of filter
      // Example removing "OPTS" after "PASS"
      // (Could have also got "next command" and checked it was "OPTS")
      if (currentCommand.StartsWith("PASS")
      {
          // Remove next "OPTS" command
          commands = RemoveAt(commands, commandStreamIndex + 1);

          // Update CommandStream's command (PipelineEntry) array with the new array 
          commandStreamCommandsField.SetValue(commandStream, commands);
      }

      // Example adding "ACCT" after "PASS"
      if (currentCommand.StartsWith("PASS"))
      {
        // Create "ACCT" command
        // Get the Pipeline Entry constructor which takes a string parameter (the command)
        ConstructorInfo pipelineEntryConstructor = commandStreamPipelineEntryType.GetConstructor(BindingFlags.NonPublic | BindingFlags.Instance, null, new[] { typeof(string) }, null);

        // Call the FormatFtpCommand method, which takes a command and parameter and formats them for FTP submission
        string formattedCommand = (string)formatFtpCommandMethod.Invoke(commandStream, new[] { "ACCT", "someAcct" });

        // Create new Pipeline Entry using formatted command as parameter
        // Of type System.Net.CommandStream+PipelineEntry
        object newPipelineEntry = pipelineEntryConstructor.Invoke(new [] { formattedCommand }); 

        // Create new merged command stream array including injected command
        commands = InsertAt(commands, commandStreamIndex + 1, newPipelineEntry);

        // Update CommandStream's command (PipelineEntry) array with the new array
        commandStreamCommandsField.SetValue(commandStream, commands);
      }
    }

    // Chain back to original delegate
    originalDelegate(asyncResult);
  }));
}

public static Array RemoveAt(Array source, int index)
{
    if (source == null) 
        throw new ArgumentNullException("source"); 

    if (0 > index || index >= source.Length) 
    throw new ArgumentOutOfRangeException("index", index, "index is outside the bounds of source array"); 

    Array dest = Array.CreateInstance(source.GetType().GetElementType(), source.Length - 1); 
    Array.Copy(source, 0, dest, 0, index); 
    Array.Copy(source, index+1, dest, index, source.Length - index - 1); 

    return dest;
 }

 public static Array InsertAt(Array source, int index, object o) 
 {  
    if (source == null) 
        throw new ArgumentNullException("source"); 

    if (0 > index || index > source.Length) 
        throw new ArgumentOutOfRangeException("index", index, "index is outside the bounds of source array"); 

    Array dest = Array.CreateInstance(source.GetType().GetElementType(), source.Length + 1); 
    Array.Copy(source, 0, dest, 0, index); 
    dest.SetValue(o, index);
    Array.Copy(source, index, dest, index+1, source.Length - index); 

    return dest;
 }