Posted on by Josh Johnson

TFW Windows Interrupts Your Service

Since the release of the “Cloud Flat File” integration feature in Transaction Manager, the need for a service to sync hosted FTP folders and local file system folders has arisen. I created a simple open-source tool to accomplish this task as an option for our customers. While it isn’t a perfect fit for every scenario, it offers an easy set-and-forget model that doesn’t require setting up batch scripts and messing with Windows Task Scheduler. It isn’t perfect, though, as was recently discovered by a high-volume customer.

An Introduction to the Problem

The tool, known as “Cloud FTP Bridge”, is a desktop application built on Windows Forms with a Windows Service component. It is a Visual Studio Solution with 3 projects:

  • Tc.Psg.CloudFtpBridge: The “shared” library targeting .NET Standard. This library contains the “meat and potatoes” of the app and can be referenced and used independently of the other two projects.
  • Tc.Psg.CloudFtpBridge.UI: A Windows Forms project defining the admin/management UI for creating server connections and workflows.
  • Tc.Psg.CloudFtpBridge.Service: A Windows Service responsible for executing Workflows on a regular basis.

In today’s post, we’re focusing on that first project – the shared library. Within that library, we define a couple of important interfaces – IFolder and IFile. This allows us to implement each of these twice – once for the local files system and once for remote FTP file systems – and use the same logic to move files around, regardless of which implementation is actually doing the work. Here is what those interfaces look like:

// IFolder.cs
using System;
using System.Collections.Generic;
using System.Text;
using System.Threading.Tasks;

namespace Tc.Psg.CloudFtpBridge.IO
{
    public interface IFolder
    {
        string FullName { get; }

        Task<IFile> CreateFile(string name);
        Task<IFolder> CreateOrGetFolder(string name);
        Task<IEnumerable<IFile>> GetFiles();
        Task<IEnumerable<IFolder>> GetFolders();
    }
}

// IFile.cs
using System;
using System.IO;
using System.Threading.Tasks;

namespace Tc.Psg.CloudFtpBridge.IO
{
    public interface IFile : IDisposable
    {
        IFolder Folder { get; }
        string FullName { get; }
        string Name { get; }

        Task<bool> Exists();
        Task<Stream> GetReadStream();
        Task<Stream> GetWriteStream();
        Task<IFile> MoveTo(IFolder destinationFolder, string newFileName = null);
    }
}

A FileManager class encapsulates the necessary logic for executing a workflow, which really just consists of moving files around between folders. Here is how it works at a high level (full code linked above):

  1. Call GetFiles() on the source IFolder to get a list of IFile candidates to be transferred. These might be remote files being transferred to a local directory or vice versa. The file manager is unaware of this detail.
  2. Iterate over each IFile and call MoveTo() to move the file to a processing folder. This reduces the risk of files being touched by the source system while we are reading them (a corner case specific to a handful of business systems).
  3. Iterate over each of those “staged” IFiles and call GetReadStream() to get a stream of data to read from.
  4. Call CreateFile() on the destination IFolder to create an empty file to write to.
  5. Call GetWriteStream() on the new empty file and copy the source stream to it.
  6. Call MoveTo() again on the source IFile to move it to an archive folder.

This process worked fine in most case, but my initial implementation had a fatal flaw – it kept a list of references to the “staged” files in memory and used that list of references for steps 3 through 6. This meant that if the service crashed or was otherwise stopped unexpectedly (perhaps by an unexpected server reboot), any files left in the processing folder would be left behind, never to be seen again. This was no bueno.

Finding a Solution

So how did I fix this? The main solution was pretty simple – just scrap that in-memory list and simply grab a new list of the files in the processing folder before iterating over the staged files. This ensures we catch any other files that might be there waiting to be picked up. But I also took the opportunity to split the “staging” and “processing” phases into two separate methods in the FileManager class. This allowed me to process all staged files upon service start-up before even executing any workflows. Further, I added the workflow ID to the processing folder name so I could load the workflow metadata as well. This improved the logging story and ensured I would have access to future features that might be added to workflows that could further customize how files are moved about.

The lesson? Don’t forget to account for unexpected “crashes”. These may be actual crashes or just a user killing the process in the middle of your work. Either way, especially when writing long-running tasks, be aware of the potential consequences of interruptions.

Josh Johnson

Josh Johnson

Solutions Architect at TrueCommerce
Josh has been writing software for 7 years. He currently works on the .NET stack using .NET Core and Angular to build highly-customized solutions for the Professional Services team at TrueCommerce.
Josh Johnson

Latest posts by Josh Johnson (see all)