Fault tolerance and reliability

Draft

A spacecraft is a machine that cannot be repaired, inspected or restarted by hand. Every failure must be anticipated in design, detected autonomously, and either tolerated or survived long enough for a ground team to intervene minutes or hours later.

The arithmetic

For a constant failure rate, reliability falls exponentially:

Exponential reliability

R(t) = e^{-\lambda t}, \qquad \text{MTBF} = \frac{1}{\lambda}

Inverting this for realistic mission requirements gives numbers that explain why spacecraft parts cost what they do:

Requirement	Implied failure rate	Implied MTBF
90% reliability over 15 years	0.0070 per year	142 years
95% over 15 years	0.0034 per year	292 years
99% over 5 years	0.0020 per year	497 years

A mean time between failures of 142 years for a complete spacecraft is not achievable by making every part very good. It is achievable only by combining good parts with redundancy, because reliability in series multiplies:

R_{\text{series}} = \prod_i R_i, \qquad R_{\text{parallel}} = 1 - \prod_i(1 - R_i)

A system of 100 components each at 99.9% reliability is only 90% reliable overall. That multiplication is why parts count is itself a reliability metric and why simplification is a reliability activity, not merely a cost one.

FMEA and single-point failures

Failure modes and effects analysis walks every component, enumerates how it can fail, and traces the consequence to the system level. Adding criticality gives FMECA, which ranks by severity and likelihood.

The output that matters most is the single-point failure list: items whose failure ends the mission with no workaround. Every one is then either eliminated by redundancy, reduced in likelihood by design margin, or formally accepted with rationale at a review.

Some cannot be eliminated. A deployment mechanism that must actuate once has no meaningful redundancy, since a second mechanism is a second thing that can jam. JWST carried 344 accepted single-point failures, mostly deployment events, and the programme’s answer was not redundancy but extraordinary test discipline. All 344 worked.

Redundancy and its limits

Approach	Protects against	Does not protect against
Identical block redundancy	Random hardware failure	Design faults, software bugs, common environmental causes
Cross-strapping	Failure propagating along a whole chain	The interface complexity it adds
Dissimilar redundancy	Design faults, since implementations differ	Requirement errors common to both
Functional redundancy	Loss of a capability, by achieving it another way	Loss of the underlying resource

The critical qualification is in the second column. Redundancy defends against random failure. It does nothing against a systematic one, because identical units given identical inputs fail identically at the same instant.

Ariane 5 flight 501 in June 1996 is the definitive demonstration. The inertial reference system carried over from Ariane 4 executed a conversion of a 64-bit floating-point horizontal velocity value into a 16-bit integer. Ariane 5’s trajectory produced a larger value than Ariane 4 ever had, the conversion overflowed, and the exception was unhandled. The backup unit, running identical software on identical data, had already failed the same way moments earlier. The guidance computer interpreted the resulting diagnostic pattern as flight data, commanded a full nozzle deflection, and the vehicle broke up.

Two units, complete redundancy, and no protection whatsoever. The variable in question was not even needed after liftoff; the calculation was retained from Ariane 4 for a pre-launch alignment function.

Fault detection, isolation and recovery

FDIR is the onboard logic that acts when something goes wrong and nobody is available to decide.

The design principle running through that diagram is that autonomy should be proportional to urgency. A fault that will destroy hardware in milliseconds, such as a latch-up, must be handled in hardware. A fault that merely degrades performance should be reported and left to the ground, which has more information and better judgement. Building elaborate autonomous recovery for non-urgent faults adds software that is itself a failure source, and FDIR logic misfiring on a healthy system is a well-represented category of spacecraft anomaly.

Safe mode is the terminal state and must be designed to be indefinitely survivable with no ground contact. It requires an attitude that keeps solar arrays illuminated and radiators pointed away from the Sun, minimum electrical load, and a low-gain antenna that can be heard without accurate pointing. A safe mode that depends on the failed subsystem, or that cannot hold attitude without consuming propellant, is not a safe mode.

What real failures teach

Case studies are more instructive than principles, because the failures are rarely where the analysis predicted.

Mars Climate Orbiter, 1999: Ground software supplied impulse in pound-force seconds where the navigation software expected newton-seconds. The trajectory error accumulated over months and the spacecraft entered the atmosphere. The fault was in an interface specification, not in any component.
Mars Polar Lander, 1999: Leg deployment produced a spurious touchdown signal that software latched. At 40 m altitude the software read the latched signal and shut down the descent engine. Each element behaved as designed; the interaction was never tested.
Hubble, 1990: The primary mirror was ground to the wrong figure because a null corrector was assembled with a spacing error. An independent test that would have caught it was skipped as unnecessary. The mirror was polished to exquisite precision, to the wrong shape.
Genesis, 2004: A deceleration sensor assembly was installed inverted, so entry deceleration was never detected and the parachute never deployed. The drawing error survived every review.

The common thread is that none of these was a component that broke. Every one was an interface, an interaction, an assumption or a test that was not performed. Component reliability is a solved problem in comparison; systems integration is where missions are lost.

That is the argument for the verification discipline in systems engineering, and specifically for testing the integrated system in as flight-like a configuration as possible. Mars Polar Lander’s fault would have been found by a single end-to-end test with real leg deployment; Hubble’s by running the test that was skipped.

Practices that generalise

Practice	Rationale
Test as you fly	The failure is usually in an interaction, and only an integrated test in a flight-like configuration exercises interactions
Do not skip a test because the analysis is confident	Hubble’s analysis was confident
Trend telemetry, do not just check limits	Degradation is visible in trends long before it crosses a threshold
Design safe modes first	A safe mode retrofitted around an existing design usually depends on something that can fail
Prefer fewer parts	Series reliability multiplies, so simplification is a reliability activity
Use dissimilar redundancy where design faults are credible	Identical redundancy is no defence against a systematic fault
Version-control and review ground software as flight software	Mars Climate Orbiter was lost to a ground software interface

That last row is regularly ignored. Ground systems are frequently held to a lower standard than flight software on the reasoning that they can be fixed. In the case that destroyed Mars Climate Orbiter, the ground software was as mission-critical as anything aboard, and its error was unrecoverable by the time it was visible.

References

Lions, J. L. et al. ARIANE 5 Flight 501 Failure: Report by the Inquiry Board, ESA, 1996.
Mars Climate Orbiter Mishap Investigation Board Phase I Report, NASA, 1999.
Report on the Loss of the Mars Polar Lander and Deep Space 2 Missions, JPL Special Review Board, 2000.
The Hubble Space Telescope Optical Systems Failure Report, NASA, 1990.
Rausand, M. and Høyland, A. System Reliability Theory: Models, Statistical Methods and Applications, 2nd ed., Wiley, 2004.
NASA-STD-8729.1, Reliability and Maintainability Standard for Spaceflight and Support Systems.