The Cloud Is Someone Else’s Failure Domain

Cloud computing has done something genuinely useful: it has spared many teams from buying servers, racking them up, nursing failed disks, and pretending a cupboard with a fan in it is a data centre. That is progress.

But it has also encouraged a comforting misunderstanding. People say, “It’s in the cloud,” with the faintly magical confidence once reserved for putting something in a filing cabinet and assuming it could never burn down.

The cloud is not the absence of infrastructure. It is infrastructure operated by somebody else, reached through networks you do not control, governed by limits you may only discover at the least convenient moment, and joined together by a surprisingly large collection of dependencies.

That does not make cloud services bad. Far from it. It means we need to describe them honestly. The cloud is someone else’s failure domain. Your job is to understand where theirs ends and yours begins, because your customers will not find that distinction especially comforting at 2am.

Availability is not a property you can purchase wholesale

A cloud provider may offer highly resilient compute, storage, databases, queues, identity services, DNS, monitoring, and a small galaxy of managed products. Each can be well engineered. Each can also fail, degrade, reject requests, or become unavailable in a particular region or through a particular API.

That last point matters. Availability is not a sticker on a product box. It is a characteristic of an entire service path.

Imagine a temperature monitoring platform. A sensor sends a reading through a gateway, the gateway resolves a hostname, establishes a secure connection, authenticates, publishes to an ingestion endpoint, writes into a queue or database, triggers alerting logic, and delivers an email, SMS, or app notification. An issue at any point can prevent a user receiving an alert.

The database may be healthy while the identity service is timing out. The application may be running while DNS resolution fails. The message queue may accept data while downstream processing has fallen behind. The public status page may even be green, which is always a charming touch when your own dashboard is having the technological equivalent of lying face down on the kitchen floor.

Cloud architecture is therefore not simply a matter of selecting services labelled “high availability”. It is the work of mapping dependencies, deciding what must happen during partial failure, and accepting that a managed service can be both excellent and unavailable.

Shared responsibility is useful, but incomplete

The shared-responsibility model is broadly sensible. A provider takes responsibility for physical facilities, underlying hardware, and much of the platform. You remain responsible for your application design, configuration, identity controls, data, and use of the services.

The trouble is that the phrase can become a legalistic umbrella under which practical questions get drenched and forgotten.

Suppose a cloud region experiences an outage. The provider may own the fault. Your organisation still owns the consequences: missed transactions, inaccessible patient records, unprocessed orders, absent alerts, and an understandably irritated support queue.

Responsibility is not the same as control, and neither is the same as impact.

This is especially important for systems that supervise physical conditions. A cold-storage alarm that relies entirely on a remote cloud path has a different risk profile from one that can alert locally when temperature exceeds a limit. If internet access disappears, the latter can still sound a local alarm, notify a local contact through an independent communication route, and retain readings for later synchronisation.

That is not an argument against cloud platforms. It is an argument against treating connectivity as a law of nature. It is merely an engineering dependency, and dependencies have the rude habit of becoming visible during storms, maintenance, routing errors, expired certificates, and someone’s enthusiastic configuration change on a Friday afternoon.

The most awkward failures sit between services

Large outages make the news, but smaller, stranger failures are often more instructive. A system can fail in ways that are neither fully up nor properly down.

A common example is retry behaviour. Your application calls a cloud API and receives a timeout. Did the provider process the request before the connection failed, or did it not receive it at all? If you retry blindly, you may create duplicate records, trigger two notifications, or charge a card twice. If you do not retry, you may lose a legitimate operation.

The usual answer is not blind optimism. It is idempotency: assigning an operation a unique identifier so repeated requests can be recognised as the same logical action. A message consumer should also assume it may receive the same message more than once. “Exactly once” is a lovely phrase, but end-to-end exactly-once processing is often an expensive promise with several footnotes attached.

Queues introduce another trade-off. They can absorb bursts and decouple components, which is valuable. They can also conceal growing backlog. If consumers are processing 100 messages per second and producers suddenly sustain 150, the queue length rises by 50 messages per second. Nothing has technically failed yet, which is precisely why it can be missed. Then alerts arrive late, dashboards show old data, and everyone discovers that elasticity did not include an infinite supply of downstream capacity.

Monitoring should therefore include lag, age of oldest message, error rates, throttling responses, failed authentication attempts, and the rate at which retry queues are growing. CPU usage alone is not a service-health strategy. It is more like checking whether the car engine is warm while ignoring the fact that the wheels have fallen off.

Design for degraded service, not heroic recovery

Resilience discussions often leap straight to multi-region architecture. There are situations where that is appropriate, but it is not a universal cure and it is certainly not free.

Running across regions introduces data replication choices, failover logic, consistency questions, additional operational complexity, and a larger bill. If two regions can both accept writes, what happens when network partitioning means each region cannot see the other? Which version of a device configuration wins? Can an alert be sent twice? Is delayed data acceptable, or dangerous?

Before building a grand cross-region fortress, ask simpler questions:

• What must continue working if the cloud service is unavailable for an hour?
• What can be delayed safely?
• What should fail closed, and what should fail open?
• What information needs to be stored locally?
• How will users know the system is degraded rather than merely quiet?

For an IoT device, local buffering is often a sensible first step. Readings can be timestamped and retained while the connection is unavailable, then uploaded when it returns. The design needs limits: storage fills up, clocks drift, and reconnecting thousands of devices simultaneously can turn a recovery into a small stampede. Backoff with jitter helps here, spreading reconnect attempts rather than having every device politely form the same queue at exactly the same second.

A local alarm path may be more valuable than a second cloud region. A clear manual procedure may be more valuable than an exquisitely drawn architecture diagram. Neither is glamorous, which is probably why they are so often the bits that save the day.

Know your dependencies before they introduce themselves

A useful exercise is to draw the journey of one important user action. Not the tidy diagram used in a sales deck, but the actual journey.

For example: a customer opens an app, signs in through a third-party identity provider, retrieves data through an API gateway, reads from a managed database, and receives a notification through an external messaging provider. Add DNS, certificate authorities, content delivery networks, software repositories, payment services, observability tools, and any feature-flag platform that can change application behaviour remotely. The picture gets busy rather quickly.

Then ask what happens when each part is slow, unavailable, misconfigured, rate-limited, or returning stale data.

This should include your own changes. Cloud outages are not always provider failures. A restrictive firewall rule, deleted route, incorrect IAM policy, exhausted quota, accidental database migration, or a bad deployment can produce the same customer experience: the service does not work. The distinction may matter later in the incident review. It does not help the person waiting for an important alert.

Test the assumptions where practical. Restore backups. Practise failing over a component. Disable a non-critical integration in a controlled environment. Verify that alerting tells you about the customer-facing problem rather than merely announcing that a virtual machine is still breathing.

Cloud services are immensely capable tools, and I would not swap them for a return to feeding tapes into a server-room cabinet. But confidence should come from deliberate design and tested recovery, not from the word “cloud” doing a bit of ceremonial hand-waving.

Your provider can run excellent infrastructure. You still need to decide what your system does when that infrastructure, or the network path to it, stops cooperating. That is not pessimism. It is a rather practical form of kindness to the people who depend on what you build, including the future version of you who would quite like to sleep through the night.