HTTP in Swift, Part 11: Throttling

Part 11 in a series on building a Swift HTTP framework:

I once worked on an app that used a Timer to periodically ping a server with a status update. In one build of the app, we noticed that the status server started experiencing CPU spikes, eventually culminating in it not being able to handle any more requests.

After investigation, we discovered that some simple changes to the app logic had resulted in us accidentally overwhelming the server with requests. Timer A would get set up to send a status update based on certain conditions. The timer would fire, and the update would get sent. The bug that was introduced was that the timer was not getting correctly invalidated and would stay running. When another condition would get met, we’d create a new timer (B) to send a status update. Except now we had both A and B running, so they would both try to send requests. Then when requests started timing out, the completion handlers would set up another timer to try again, and it would happen for both timers, resulting in C and D. One timer became two, which became four, which became eight, then 16, then 32… We were slamming our servers with an exponentially increasing number of requests.

The immediate fix was to update the server to immediately reject all incoming requests. Then we shipped an emergency bug fix for the app to fix the timer invalidation behavior.

But … wouldn’t it have been nice to have the app prevent this from happening in the first place? We can do this, and it’s going to be surprisingly simple.

Throttling Requests

Let’s take a look at our HTTPLoader interface to remind ourselves of our API contract:

open class HTTPLoader {

    /// Load an HTTPTask that will eventually call its completion handler
    func load(task: HTTPTask)

    /// Reset the loader to its initial configuration
    func reset(with group: DispatchGroup)
}

Remember that the load(task:) method makes no promises about when tasks will be executed. A loader may receive an HTTPTask and start executing it immediately, or it may sit on it for a few seconds, or minutes, or hours, or years. To the client of the API, there’s no promise made about execution timing.

A throttling loader can take advantage of this. When it receives an HTTPTask, it can see if it is allowed to continue loading it. If it is, it can merrily pass the task on to the next loader in the chain. If it’s not allowed to load it, it can place it on a list of tasks to execute later.

Our overall interface will look something like this:

public class Throttle: HTTPLoader {
    public var maximumNumberOfRequests = UInt.max

    private var executingRequests = [UUID: HTTPTask]()
    private var pendingRequests = [HTTPTask]()

    public override func load(task: HTTPTask) {
        if UInt(executingRequests.count) < maximumNumberOfRequests {
            startTask(task)
        } else {
            pendingRequests.append(task)
        }
    }

    private func startTask(_ task: HTTPTask) {
        let id = task.id
        executingRequests[id] = task
        task.addCompletionHandler {
            self.executingRequests[id] = nil
            self.startNextTasksIfAble()
        }
        super.load(task: task)
    }

    private func startNextTasksIfAble() {
        while UInt(executingRequests.count) < maximumNumberOfRequests && pendingRequests.count > 0 {
            // we have capacity for another request, and more requests to start
            let next = pendingRequests.removeFirst()
            startTask(next)
        }
    }
}

This (incomplete) implementation gives us the basic idea of how throttling works. When we get a request, we check to see if how many tasks we currently have executing. If it’s less than our allowed maximum, then we can start executing this request. If we’re at (or beyond) our limit, we’ll put the task into an array of “pending” requests to indicate that it needs to wait to be loaded.

Loading a task adds it to a list of currently executing tasks, much like the Autocancel loader we created last time. And when it finishes, it gets removed from the list. Additionally, when a request finishes, the loader checks to see if there are tasks waiting to load, and if it is allowed to execute them. If it is, then it pulls them off the array and starts executing them.

There are a couple of shortcomings with this simple implementation:

It is not thread-safe. Requests can be loaded from any thread, and we’re modifying a lot of state inside the loader without making sure we have exclusive access to it. We’d need a synchronization type (such as an NSLock or DispatchQueue) to make sure that we’re correctly updating state.

We have no way to react to a task being cancelled. If a task is cancelled while it’s still pending, then we should probably just pull it out of the array and call its completion handler. Fortunately, adding this would be pretty straight-forward:

 let id = task.id
 task.addCancelHandler {
     if let index = self.pendingRequests.firstIndex(where: { $0.id === id }) {
         let thisTask = self.pendingRequests.remove(at: index)

         let error = HTTPError(.cancelled, ...)
         thisTask.complete(.failure(error))
     }
 }
 pendingRequests.append(task)

We’re missing our reset() logic. Conceptually, this will be similar to our Autocancel loader from last time: When we reset, we join each pending task and executing task to the DispatchGroup. When each one completes, they respectively leave the group. We could invoke cancel() on every task, but ideally we’ve got an Autocancel loader in the chain that’s already going to do that for us.

Unthrottled Requests

While it’s nice that we have the ability to throttle requests, there may be requests that we never want to throttle. This is a great opportunity to add another request option:

public enum ThrottleOption: HTTPRequestOption {
    public static var defaultOptionValue: ThrottleOption { .always }
    
    case always
    case never
}

extension HTTPRequest {    
    public var throttle: ThrottleOption {
        get { self[option: ThrottleOption.self] }
        set { self[option: ThrottleOption.self] = newValue }
    }
}

Recall that a request option allows us to add per-request behavior. So, by default, requests are .always throttled, but we have the ability to indicate that a single request is .never throttled. All that’s left is to look for this in our loader:

    public override func load(task: HTTPTask) {
        if task.request.throttle == .never {
            super.load(task: task)
            return
        }
        ...
    }

With this, we now have a loader that we can insert into our chain to limit the number of simultaneous outgoing network requests. Also note that we’ve declared the maximumNumberOfRequests as public var, which means that we can dynamically update this value. For example, our app may download some configuration settings to indicate how fast it’s allowed to load requests.

// A networking chain that:
// - prevents you from resetting while a reset command is in progress
// - automatically cancels in-flight requests when asked to reset
// - limits the number of simultaneous requests to a maximum number
// - updates requests with missing server information with default or per-request server environment information
// - executes all of this on a URLSession

let chain = resetGuard --> autocancel --> throttle --> applyEnvironment --> ... --> urlSessionLoader

If we’d had something like this in place, we could have remotely “shut down” our mis-behaving app without having to scramble to release an app update, and we could’ve kept our server up and running.

In the next post, we’ll look at how we can automatically retry requests when they fail.

Throttling Requests

Unthrottled Requests

Related️️ Posts️