The RTO Nobody Wanted to Pay For

Four hours. Someone typed it into a spreadsheet cell during a planning workshop, and it has been quoted in every audit since. Nobody has ever watched the service come back in four hours.

Picture the workshop. A conference room, a spreadsheet projected on the wall, one row per service. The column header reads Maximum acceptable downtime. When the cursor reaches online orders, the person who runs that business thinks about abandoned carts, a call center filling up and a weekend of refunds, and says four hours. Someone types it. Nobody suggests a week; in that room, a longer answer would sound like not caring. The cursor moves to the next row.

There is no column for price. Nobody asks what the business would pay, every month, to make four hours true. The infrastructure team will get the number later, as a requirement, and nobody in the room knows that the last full rebuild of the order system from its backups, two years ago, took most of a working day. The number moves into the continuity plan, then a policy, then a slide for the board, and at each step it looks a little more like a fact.

That room is where recovery objectives are usually born: decided as a preference, recorded as a fact. The difference between four hours on paper and most of a day in practice sits in no budget and no risk register. It lives between two meetings, the one that chose the number and the one that set the budget, and the organization finds out, on the worst possible day, that it never bought the promise.

What the number actually promises

Suppose someone asks how quickly you could be back on the road after a flat tire. The honest answer depends less on your schedule than on what is already in the car. With no spare, you wait for a tow truck: hours, but you paid nothing in advance. With a spare and a jack, perhaps twenty minutes, if the spare still has air and you have changed a tire before. With roadside assistance, someone else does it, for a monthly fee. With a second car in the driveway, you are moving again almost at once, and paying for a car that mostly sits there. Every faster answer is bought before the flat, whether or not the flat ever comes.

A recovery time objective, or RTO, is an organization’s written answer to that question for one service: the longest it may stay down after a disruption before the damage becomes unacceptable. NIST’s contingency-planning guide, the usual reference, places it under a larger figure, the maximum tolerable downtime, which is the total outage the business is willing to absorb. The RTO normally has to be shorter, because people still need time to reprocess data once the system is back.

Its companion, the recovery point objective or RPO, answers a different question: how much recent work can you afford to lose? If the order database is copied every night, a failure at five in the afternoon loses the day’s orders, whatever the RTO says.

The important word is objective. It records what the business needs, not what the systems can do. Recovery time is set the way the flat-tire answer is: by what was built, copied, tested and staffed before the failure. Writing the objective down costs nothing. Making it true costs money every month.

A question asked without a price

The process that produces the number is called a business impact analysis. For each service, the people who depend on it estimate what an outage would cost them, in revenue, customers, contractual penalties, regulatory exposure or safety, and how that harm grows by the hour. As usually run, it is also a question about pain asked without a price. Ask anyone how long they could manage without something they depend on and the answer will be short, because the cost of a short answer falls on someone else.

Three groups each hold a piece of the decision, and they rarely sit in the same room. The business owns the tolerance: how long it can be without the service. Technology owns the capability: how long recovery actually takes. Finance owns the budget that decides whether the capability can change. Only the tolerance becomes a sentence in a policy, because it is the only one that is free to write. Nobody behaves badly here. Technology receives the number as a requirement with no money attached, and finance never sees a request labeled four hours.

What finance sees instead is the cost of the capability without the promise it buys: a recovery environment kept ready in a second region (a separate group of data centers, far enough away not to share the same disaster), a copy of the order system idling at low capacity, licenses for software that only matters in a disaster. On a monthly cost report, each of those looks like waste, because it is idle by design; insurance always looks unused when nothing goes wrong. So the cost review that trims idle resources, a perfectly sensible exercise, quietly lengthens the real recovery time, and nobody updates the four hours, because nobody connected the two.

What each hour costs

NIST frames the choice as a balance: the longer a disruption runs, the more it costs, and the shorter the RTO, the more expensive the recovery solution. The right target sits where the two curves meet, and it differs for every system. What the balance leaves out is that, as the target shrinks, the price changes in kind, not only in size.

From days to hours, the cost is mostly discipline: protected copies that actually restore, infrastructure described as code so an environment can be rebuilt rather than remembered, and a procedure someone walked through recently. AWS calls this backup and restore, the option with the lowest cost and complexity. Cheap to hold, slow to use: the tow truck.

From hours to minutes, you start paying for things that run while nothing is wrong. In what AWS calls a pilot light, data is copied continuously to the recovery region and the databases are on, but the application servers sit switched off, so nothing can serve a customer until someone starts them and scales them up. In a warm standby, a smaller but complete copy of the service runs all the time and can take traffic immediately, at reduced capacity. Either way, you pay every month for a second environment that, in a good month, does nothing visible. That is the spare tire and the roadside plan.

From minutes to almost nothing, the cost moves into the application itself. Running at full strength in two places at once, which the industry calls active-active, means both copies accept orders, and the software has to decide what happens when the same order changes in both within the same second. AWS calls this the most complex and costly approach, the one that can bring recovery time close to zero. It is a design problem, not an infrastructure purchase, paid for in engineering time on every later feature: the second car, kept in sync with the first.

The infrastructure bill is only part of it. Three costs never appear on it.

The first is rehearsal. A recovery path nobody has exercised recently is a hypothesis. Exercising it costs engineering days, a maintenance window and the risk that the test itself causes an outage, so it is the first thing deferred when a quarter gets busy.

The second is dependencies. A service comes back no faster than the slowest thing it needs: the login system, the DNS records (the internet’s address book, which points customers at it), the secrets and certificates it cannot start without, the payment provider on the other end. A four-hour objective for orders is a quiet claim about all of them, and many were never asked.

The third is the decision. Moving a service to its recovery site, called failover, has consequences: it may lose the last minutes of data, and returning to the primary site afterwards is often harder than leaving it. So people hesitate, and wait for someone senior enough to make the call. A failover that takes fifteen minutes to run and two hours to authorize is a recovery of two hours and fifteen minutes. Shortening that wait costs no money. It costs authority, delegated in writing before the incident, to someone who will be awake when it happens.

Four hours from when?

There is a second place the gap hides, and it is the clock itself. Four hours, measured from when? And stopped by what?

The start can be defined at least four ways: when the service stopped working, when monitoring noticed, when someone declared an incident, or when someone authorized recovery. In a quiet overnight failure, those moments can be hours apart. A team that starts its clock at declaration is measuring how fast it executes, which is useful, but not what customers lived through.

The stop is just as slippery. The database is up. The servers report healthy. The status page turns green. None of that means a customer can place an order, that the order lands in a database whose contents match the agreed recovery point, that payments clear, or that the security controls switched off during the emergency are back on. A clock that stops at “database up” can record three hours and fifty minutes for an outage customers experienced as nine. Nobody lied. They used a different clock.

The fix is a definition, written before the next test. Start the business clock when the service stopped meeting its agreed criteria, on the best evidence available, and say which proxy you used when the exact moment cannot be known. Stop it when an accountable owner confirms that the essential transactions work on data at the agreed recovery point. Underneath, track detection, decision, restoration and validation separately, because that breakdown shows where the time goes.

And decide in advance whether a degraded state counts. A temporary spare gets you moving, at reduced speed and for a limited distance. Taking orders while delaying shipment, or serving search but not checkout, may meet the real need sooner and far more cheaply than full recovery. That is a legitimate answer, as long as it is the one that was agreed.

Declared, demonstrated, accepted

Seen clearly, every RTO involves three numbers, and most organizations write down only one. The declared time is the objective in the policy. The demonstrated time is what the service actually achieved on the agreed clock, in the latest realistic test or real incident, with its date and conditions attached: whether the team knew the test was coming, whether anyone pre-warmed capacity, whether the architecture has changed since. The third is the gap, and what closing it would cost.

When the demonstrated time misses the declared one, the options are few, and each costs something. Pay to close the gap, knowing which spending buys which hours. Change the promise, with a degraded mode that meets the essential need sooner. Change the number: the business signs a longer objective with the impact in view. Or accept the difference as risk, in writing, with a name on it and a date to look again. NIST asks for something similar: when an RTO cannot be met and the tolerable downtime will not flex, document the situation and a plan to mitigate it.

What is missing from that list is the most common outcome: leaving the policy at four hours, the capability at most of a day, and calling the result compliance. That is not accepted risk. It is risk nobody accepted, presented as if somebody had.

Practical Framework

The RTO worth signing

Before an RTO goes into a policy, one page should hold:

1. The promise. The service, the disruption it covers, and the clock: what starts it, what stops it, and whether a degraded state counts.
2. The evidence. The demonstrated time on that clock, when it was measured, and under what conditions.
3. The price. What closing the gap would cost in infrastructure, rehearsal, dependencies and delegated authority, and which hours each part buys.
4. The signature. Who accepted whatever gap remains, and when they will look at it again.

A page that holds all four turns the RTO from a wish into a purchase decision. A page that holds only the first is the spreadsheet cell from the workshop.

The missing column

None of this argues for short recovery times, or against them. Some services deserve minutes and the money that buys them. Many can be down for a day with an apology and a backlog, and treating them as if they needed four hours spends money that would protect something else better. What matters is that the choice is made with the price in view, by the people who will pay it, and made again when the price or the design changes.

So go back to the workshop. Next year, add one column beside Maximum acceptable downtime: what would you pay, every month, to make that true? Put the demonstrated time next to it, and the name of whoever will own the difference. Four hours may survive that conversation. It may become twelve, with orders taken by phone, or two, with a standby environment and a budget line to match. Any of those is an objective. The number in the cell today is something else: a figure everyone agreed to and nobody bought.

Source notes

These sources support the definitions of RTO, RPO and maximum tolerable downtime, the trade-off between recovery cost and disruption cost, and the cloud recovery strategies described here. The planning workshop, the order service, the four-hour objective and every timing in the article are hypothetical illustrations, as is the flat tire. The argument that recovery targets are set without a price, the clock definitions and the one-page checklist are this article’s judgment and are not attributed to any source.

• RTO, RPO and the tolerable outage. NIST defines maximum tolerable downtime as the total outage a system owner is willing to accept, the RTO as the longest a system resource can stay unavailable before the impact becomes unacceptable, and the RPO as the point in time to which data can be recovered. It notes that the RTO must normally be shorter than the maximum tolerable downtime, and that when an RTO cannot be met, the situation and a plan to mitigate it should be documented. Source: NIST SP 800-34 Rev. 1, Contingency Planning Guide for Federal Information Systems, 2010.
• Shorter recovery costs more. The same guide treats the target as a balance: a disruption grows more costly the longer it lasts, recovery solutions grow more expensive as the RTO shortens, and the balance point differs for every organization and system. Source: NIST SP 800-34 Rev. 1, Figure 3-3, “Cost Balancing”.
• Four cloud recovery strategies. AWS describes backup and restore, pilot light, warm standby and multi-site active/active, ranging from the lowest cost and complexity to the most complex and costly. A pilot light cannot process requests until further action is taken; a warm standby can handle traffic immediately at reduced capacity. Source: AWS, “Disaster recovery options in the cloud”, current documentation.
• Objectives from business need, then tested. AWS Well-Architected says RTO and RPO should be set from business needs and that disaster-recovery implementations should be tested to validate them. Source: AWS, “Plan for Disaster Recovery (DR)”, Reliability Pillar, current documentation.