The Myth of ‘Always On’: Confronting Data Center SPOFs
True resilience depends as much on disciplined operations – training, staffing, procedures, and realistic scenario planning – as on engineering.
Texas winters can swing from mild days to freezing nights. In February 2021, that volatility turned into days of subfreezing weather and snowfall across Northern Texas, exposing fragile assumptions about grid stability, on-time fuel resupply, site access, and cross-vendor collaboration. Most data centers stayed online, but several were pushed to the brink as blackouts spread and refueling and access plans unraveled.
Data centers are engineered to withstand the kinds of conditions Texas saw in 2021, but the weather event revealed off-site dependencies that designs didn't fully account for. A typical data center carries roughly 48 hours of on-site fuel and backup generation; during the freeze, however, refueling was delayed by icy roads and interstate closures. Netrality’s chief operating officer, Josh Maes, later admitted the sustained, region-wide disruption to fuel logistics had been undermodeled.
Just a month later, a different failure mode surfaced in Strasbourg, France, when a fire traced to UPS equipment destroyed OVH’s SBG2 data center. The facility’s environmentally friendly passive cooling earned sustainability brownie points, but the same airflow helped fan the flames once the fire broke out. As responders frantically worked to cut power to the facility, the backup generators unexpectedly kicked in after the main supply was cut – an example of control logic behaving as designed in one context but dangerously in another. Fire safety is as much about airflow management and shutdown coordination as it is about suppression hardware.
Taken together, these incidents highlight the ‘unknown unknowns’ that can lurk in any data center – the single points of failure (SPOF) that may be built in from the outset, accumulate through upgrades over the facility’s life, or surface through plain human error.
Engineering Out Single Points of Failure
Of course, designers and operators anticipate potential SPOFs long before the first mouse click is made in AutoCAD.
“The key ones are power redundancy to the site,” said Matt Wilkins, Global Director of Design and Engineering at Colt Data Center Services. “We've got dual-redundant power supplies from different systems to ensure that power is uninterrupted. We've also, obviously, got backup generation on our sites to ensure that these key components can be maintained without causing disruption.”
Wilkins continued: “We have redundancy within the cooling system itself – multiple chillers – but we also have multiple redundant power supplies to them. We're also 100% carrier neutral so that our customers can use the carrier of their choice, which maximizes cost efficiencies for them [and helps ensure] redundancy and uptime.”
But many data center designs fall short of total redundancy for pragmatic cost reasons, said Ed Ansett, Global Director, Data Center Technology and Innovation at Ramboll.
“Data centers are generally designed with a high degree of fault tolerance,” Ansett explained. “That basically means that in the event that something fails, something else – a redundant component – will take over.”
“Data centers are generally designed with a high degree of fault tolerance. That basically means that in the event that something fails, something else – a redundant component – will take over.” - Ed Ansett, Global Director of Data Center Technology and Innovation, Ramboll
During data center design, teams run SPOF assessments to uncover weak links, then decide to either eliminate them or accept them, Ansett said. That decision is primarily based on likelihood, impact, and cost.
What Tiers Do – and Don't – Guarantee
This is where standards frameworks such as the Uptime Institute’s Tier Standards come in – giving prospective customers a clear idea of the assessed risks of a data center’s design. Alongside Uptime’s certification system, there are similar alternatives: the Telecommunications Industry Association’s TIA-942 and the European EN 50600 standards.
Broadly speaking, these frameworks align with the four-tier paradigm established by the Uptime Institute, with Tier IV promising complete fault tolerance.
“Tier IV says that you have a completely duplicate feed,” Ansett noted. “It’s predicated on the idea that the IT load, which is what everybody is concerned with, is dual-fed from both a power and cooling perspective. That means that if one entire system fails, there is an alternative system that will be able to pick it up seamlessly.
“But that’s quite expensive, so there’s really a big compromise in Tier III. Because of the way that the various standards work, if you don’t meet the requirements, you automatically default to the next tier down,” he said.
More importantly, certifications do not remove risks. If control settings, protection coordination, or operating procedures diverge from design assumptions, even Tier IV data centers can fail.
Minor Faults, Big Consequences
Publicly documented outages show that it's rarely a single failure that brings a data center down; more often, two or more issues act in concert. In 2014, when the Singapore Stock Exchange (SGX) experienced hours of downtime, external investigators, including Ansett’s data center consulting company i3 Solutions, detailed a chain of events. A malfunction in a component of one of the two diesel rotary uninterruptible power supplies (DRUPS) created a frequency mismatch between power sources. Downstream static transfer switches (STS – devices that transfer loads between the two sources) were not configured to handle the resulting out-of-phase transfer. The ensuing surge in current tripped breakers and cascaded across the primary data center. In other words, a sequence of failures exposed a design vulnerability that testing had not caught.
DRUPS-related issues have been implicated in several other data center outages, including ones affecting AWS’s Sydney Region in June 2016 and Sovereign House in London in November 2015. Contributing factors can include synchronization faults, voltage sags, contaminated or degraded fuel, maintenance problems, and, of course, human error.
People Problems
Indeed, it’s not just technical and design shortcomings at issue. The biggest risks often stem from the people operating the data center.
“The spectrum of human error occurring in data centers is quite broad,” said Madeleine Kudritzki, Vice President of Professional Services Program Management at Uptime Institute.
According to Kudritzki, Uptime has seen a wide range of human errors at client sites – from untrained security officers hitting the emergency power-off button instead of a door-lock release, to tools being dropped into critical equipment, to skipped inspections that missed generator leaks, and even intentional shortcuts where key procedural steps are ignored under the guise of “seniority” and “expertise.”
“There are mistakes that happen, and there are intentional lapses in judgment,” she said. “This is why mitigation of human factor risk is so complicated – it’s multi-faceted and unpredictable.”