System Health Monitoring
System health monitoring encompasses the techniques, architectures, and methodologies that a system uses to assess its own operational status while it runs. In safety-critical applications, health monitoring is the foundation for detecting anomalies, triggering recovery mechanisms, and keeping the system within safe operating parameters throughout its service life.
Health monitoring serves two related but distinct purposes. The first is safety: a monitor must detect a dangerous fault quickly enough that the system can reach a safe state before the fault causes harm. The second is availability: trending and prognostics reveal gradual degradation early enough to schedule maintenance before an outage occurs. The two purposes impose different requirements. Safety monitoring is judged by diagnostic coverage and detection latency, and its own failure modes must be analyzed as rigorously as those of the functions it protects. Availability monitoring is judged by prediction accuracy and false-alarm rate, and it tolerates far longer time constants.
These techniques appear wherever unexpected loss of function carries serious consequences, from infusion pumps and ventilators to flight control computers, railway interlockings, industrial burner management systems, and grid protection relays. The sections below work from the low-level mechanisms that detect faults, through the quantitative requirements that determine how much detection is enough, to the architectures, data management practices, and standards that turn raw diagnostics into usable health information.
Scope and Terminology
Precise vocabulary matters in this field because the applicable standards attach specific obligations to specific terms.
Fault, error, and failure: A fault is an abnormal condition in a component, such as a shorted transistor or a corrupted memory cell. An error is the incorrect internal state that the fault produces. A failure is the loss of correct service at the system boundary. Health monitoring aims to detect faults or errors before they propagate into failures.
Detected and latent faults: A detected fault is one that a diagnostic mechanism reveals within its designed detection interval. A latent fault remains present and undetected, and it becomes dangerous when a second fault removes the redundancy that was masking it. Detecting latent faults is the specific job of periodic and start-up diagnostics, and it is measured separately from the detection of immediately dangerous faults.
Diagnostic coverage: Diagnostic coverage is the fraction of the dangerous failure rate of an element that a given safety mechanism detects. It is a rate-weighted ratio, not a count of failure modes, so a mechanism that catches many rare failure modes while missing one dominant mode yields poor coverage.
Safety mechanism versus monitoring function: A safety mechanism detects faults and either controls or avoids their harmful effects; it carries the integrity requirements of the function it protects. A monitoring function that only records data for maintenance carries no such obligation. Confusing the two leads either to unjustified safety claims or to unnecessary development cost.
Fault detection, isolation, and recovery: Often abbreviated FDIR, this sequence describes the complete response chain. Detection establishes that something is wrong, isolation identifies which element is responsible, and recovery restores service through reconfiguration, restart, or transition to a degraded or safe mode. Health monitoring supplies the detection and isolation stages; fault-tolerant design supplies the recovery stage.
Watchdog Timers
Watchdog timers are hardware or software mechanisms that detect system malfunctions by monitoring periodic activity signals. When a system fails to service the watchdog within a specified timeout, the watchdog triggers a corrective action, typically a processor reset or a transition to a safe state driven directly by hardware.
The watchdog is the diagnostic of last resort. It detects the class of failures in which the processor stops executing the intended program at all: stack corruption, unbounded loops, deadlock between tasks, clock loss, and single-event upsets that alter the program counter. Because it must remain effective when the software has failed completely, its independence from the monitored processor is the property that determines its value.
Hardware Watchdog Timers
Hardware watchdog timers operate independently of the main processor and provide protection even when software execution becomes completely unresponsive. Most microcontrollers include integrated watchdog peripherals, while external watchdog integrated circuits offer additional independence for high-reliability applications.
Independent clock sources: An internal watchdog clocked from the same phase-locked loop as the core cannot detect a clock failure, because a stopped clock also stops the watchdog counter. Microcontroller families intended for safety work therefore provide a watchdog clocked from a separate low-speed internal RC oscillator of a few tens of kilohertz, which continues to run when the main clock tree fails. External watchdogs go further by using a wholly separate oscillator on a separate die and separate supply rail, removing the shared-substrate and shared-regulator common-cause paths.
Window watchdog operation: A simple watchdog detects only late service. A windowed watchdog defines both a lower and an upper bound, so a refresh that arrives too early is treated as a fault in the same way as one that arrives too late. This catches software that has entered a tight loop containing the refresh instruction, an interrupt handler that services the watchdog on every occurrence, and timing faults caused by a clock running fast. Many microcontroller families pair an independent watchdog for gross timeout detection with a window watchdog on the peripheral bus for timing-accuracy detection, and safety manuals commonly recommend enabling both.
Question-and-answer watchdogs: The most rigorous external watchdogs require the processor to compute a correct response to a pseudo-random challenge within a bounded time. The processor must read a seed over a serial interface, transform it with a defined algorithm, and return the result. A stuck task that merely toggles an output pin cannot satisfy such a watchdog, because servicing it requires the processor to fetch, execute, and complete a nontrivial computation. Challenge-and-response watchdogs are standard in automotive system-basis chips and in safety power-management devices that also supervise supply rails.
Multiple timeout stages: Staged watchdogs generate a maskable warning interrupt before the final reset. The warning handler can capture a stack trace, record the identity of the running task, and store diagnostic context in retained memory before the reset occurs, converting an opaque reset into a traceable event. Some devices also count consecutive watchdog resets and, after a configured limit, latch the outputs into the safe state rather than continuing an endless reset loop.
Independent shutdown path: In the highest-integrity designs, the watchdog does not rely on the monitored processor to reach the safe state. Its output drives a relay coil, a gate-driver enable, or a shutdown pin directly, so that de-energizing the output is sufficient to render the equipment safe regardless of what the processor does next.
Software Watchdog Implementation
Software watchdogs extend hardware watchdog functionality by monitoring individual tasks and software components within complex systems.
Task-level monitoring: In a real-time operating system, each critical task sets a flag or increments a counter on every activation. A supervisory task or timer interrupt inspects those counters and services the hardware watchdog only when every monitored task has run the expected number of times. Without this aggregation, a single high-priority task can keep refreshing the hardware watchdog while lower-priority safety tasks have been starved for seconds.
Control flow monitoring: Sequence checkers verify that execution follows the intended control flow. Each checkpoint contributes a unique signature to an accumulator, and the accumulated value is compared against the expected value at the end of the sequence. Blocks that are skipped, executed out of order, or executed twice produce a mismatch. Standards for high-integrity software list program-sequence monitoring among the recommended techniques precisely because it detects errors that data checks cannot.
Deadline and jitter monitoring: Deadline monitors record the completion time of each periodic task against its budget and flag both overruns and excessive jitter. Because worst-case execution times are estimated during design, runtime deadline monitoring also serves as continuous evidence that the timing analysis remains valid in the field.
Alive supervision versus logical supervision: Alive supervision confirms that a task runs at the expected rate. Logical supervision confirms that it runs in the expected order relative to other tasks. Automotive watchdog stacks implement both, together with deadline supervision, as three cooperating layers, because each layer detects a class of failures that the others miss.
Watchdog Servicing Strategies
The method of servicing the watchdog determines how much it actually proves about system health.
Centralized servicing: A single task or interrupt handler services the watchdog after aggregating health conditions from multiple sources. This simplifies management and makes the health criteria explicit in one place, but every critical function must be represented in the aggregation, or its failure will go unnoticed.
Conditional servicing: The watchdog is serviced only when specific health conditions hold, such as successful completion of a diagnostic routine or receipt of plausible sensor data. This links watchdog operation directly to verified health rather than to mere execution. The design must ensure that a transient condition does not cause an unnecessary reset of an otherwise healthy system.
Distributed servicing: Independent watchdogs monitor separate subsystems, each serviced by the subsystem it supervises. This yields finer fault isolation and prevents one subsystem from masking another, at the cost of more hardware and a more complex reset architecture.
Anti-patterns to avoid: Servicing the watchdog from a timer interrupt that runs regardless of task health defeats its purpose entirely, as does servicing it inside a delay loop or disabling it during long initialization routines. Refresh code should never be placed in a general-purpose library function that any code path can call inadvertently.
Built-In Self-Test
Built-in self-test (BIST) is the capability of a system to test its own functionality using integrated test hardware and software. BIST techniques range from simple power-on diagnostics to continuous online testing that runs concurrently with normal operation.
Power-On Self-Test
Power-on self-test (POST) executes during start-up to verify hardware functionality before the system enters normal operation. Because the equipment is not yet performing its function, POST can use destructive tests that would be impossible later.
Processor tests: Core self-test routines exercise the register file, arithmetic and logic unit, shifter, branch logic, and exception handling with patterns chosen for high fault coverage in minimal time. Microcontroller vendors supply certified core self-test libraries whose coverage figures have been quantified by fault-injection campaigns, since developing and justifying such coverage independently is expensive. Larger devices include logic BIST, which loads pseudo-random patterns into internal scan chains and compresses the responses into a signature compared against a golden value.
Memory tests: RAM tests use March algorithms, which apply an ordered sequence of read and write operations in ascending and descending address order. March C− requires ten operations per cell, denoted 10N, and detects stuck-at faults, transition faults, inversion and idempotent coupling faults, and many address decoder faults. Shorter algorithms such as MATS+ run in 5N but leave coupling faults largely uncovered. Because a full March test destroys memory contents, it is normally confined to start-up, with a non-destructive variant that saves and restores each block used at runtime. ROM and program flash are verified with a cyclic redundancy check over the image, compared against a value stored at build time.
Peripheral tests: Communication interfaces are checked with internal loopback, comparing transmitted and received frames without disturbing the bus. Analog-to-digital converters are checked by measuring an internal reference and the internal ground connection through the input multiplexer and confirming that the resulting codes fall within expected tolerance, which detects reference drift, gain error, and multiplexer faults. Digital outputs are read back through separate sense inputs to confirm that the pin state matches the commanded state.
Safety interlock verification: Emergency stop circuits, protective interlocks, and shutdown paths are exercised before the system enables hazardous operations. Testing a shutdown path requires an independent means of observing that the path actually opened, which is why safety relays and gate drivers provide dedicated feedback contacts or diagnostic outputs.
Start-up budget: POST duration competes directly with availability requirements. An automotive electronic control unit may have only tens of milliseconds before it must respond on the bus, while an industrial burner controller can afford several seconds. When the full test cannot fit in the start-up budget, designers partition it, running the critical subset at every start-up and rotating the remainder across successive power cycles or into background execution.
Continuous Online Testing
Online BIST executes during normal system operation without interrupting primary functions. It provides the short detection latency that safety standards demand for dangerous faults.
Memory protection and scrubbing: Error-correcting codes provide continuous protection for memory arrays. A single-error-correcting, double-error-detecting code over a 32-bit word requires seven check bits and corrects any single-bit upset while flagging double-bit errors as uncorrectable. Because corrected errors accumulate silently, systems perform scrubbing, in which a background process reads and rewrites every location on a fixed cycle so that single-bit upsets are corrected before a second upset in the same word makes them uncorrectable. Scrubbing is a latent-fault countermeasure, and its period is chosen from the expected upset rate of the operating environment.
Processor integrity monitoring: Lockstep architectures execute the same instruction stream on duplicate cores and compare their outputs continuously. The redundant core is deliberately offset from the primary by a small fixed number of clock cycles, commonly two, so that the two cores occupy different architectural states at any instant. This temporal diversity prevents a common-source disturbance such as a clock glitch or supply transient from corrupting both cores identically and escaping the comparator. Triple-core lockstep extends the approach with majority voting, allowing the system to continue with the two agreeing cores instead of shutting down on the first mismatch. Where hardware redundancy is unavailable, software techniques such as inverse computation, redundant execution with diverse data encoding, and periodic core self-test provide partial coverage at a cost in execution time.
Analog circuit monitoring: Reference voltages, supply rails, and bias currents are sampled continuously and compared against windows derived from the design tolerances. Cross-channel comparison between redundant sensor chains detects drift that neither channel would reveal alone. Signal-chain integrity can also be checked by injecting a known small stimulus and confirming the expected response.
Communication integrity: Cyclic redundancy checks detect corrupted frames, sequence counters detect lost or duplicated messages, and timeout supervision detects a silent transmitter. Safety communication layers add an identifier for the intended receiver, protecting against misrouted messages that are individually well formed. Above the protocol layer, semantic checks confirm that received values are plausible for the current operating state.
Test access infrastructure: Boundary-scan and embedded instrument access standards allow production test structures to be reused in the field. A boundary-scan chain can verify board interconnect during maintenance, and instrument access networks let firmware start memory BIST engines and read their results without dedicated wiring.
Scheduled Diagnostic Routines
Some tests cannot execute concurrently with normal operation and must be scheduled explicitly.
Proof tests and actuator exercising: A dangerous failure that no online diagnostic can detect remains latent until a proof test reveals it. Partial-stroke testing of a shutdown valve, which moves the valve through a fraction of its travel without interrupting the process, is the classic example: it detects a seized actuator between full proof tests and extends the interval at which the plant must be shut down for a complete test. The proof-test interval is an explicit input to the probability calculations that justify a safety integrity level, so it is a design parameter rather than a maintenance convenience.
Full memory tests: Destructive March algorithms require exclusive access to the memory under test. Systems run them at start-up, during scheduled downtime, or block by block with contents saved and restored, accepting a longer effective test interval in exchange for continuous availability.
Calibration verification: Comparison against a traceable reference validates measurement accuracy. Automated sequences can correct small drift while flagging deviations beyond a defined limit for maintenance. Regulated domains require records of these checks, because the validity of the measurement is part of the safety argument.
Fault injection for diagnostic verification: The diagnostics themselves must be tested. Deliberate injection of corrupted data, forced timeouts, and simulated sensor faults confirms that each monitor detects its target condition and produces the intended reaction. Fault-injection campaigns also provide the empirical basis for claimed diagnostic coverage.
Sensor and Signal Validation
Most safety functions act on measured values, so a monitor that verifies processor and memory integrity but accepts an implausible sensor reading has left the dominant hazard uncovered. Signal validation establishes that an input represents physical reality before the system acts on it.
Range and rate checks: A reading outside the physically possible range indicates a fault rather than an extreme condition. A rate-of-change limit catches step discontinuities that no real process can produce, which is the signature of an intermittent connection or a converter fault. Both limits must be derived from the physics of the measured process, not chosen arbitrarily, or they will generate nuisance alarms during legitimate transients.
Live-zero signaling: Current-loop transmitters are the standard example of a self-revealing signal. Because the 4 to 20 milliampere range places the zero of the measurement at a nonzero current, a broken wire produces zero current, which is unambiguously distinguishable from a legitimate reading. The NAMUR NE 43 recommendation formalizes this by restricting the measurement range to approximately 3.8 to 20.5 milliamperes and reserving currents at or below 3.6 milliamperes and at or above 21 milliamperes to signal a transmitter fault. Voltage-output and resistance-based sensors achieve similar diagnosis through bias resistors that place an open or shorted circuit outside the valid signal band.
Cross-channel comparison and voting: Redundant sensors allow discrepancy detection, and three or more channels allow the faulty channel to be identified and excluded. Diverse redundancy, using sensors based on different physical principles, defends against common-cause errors such as a shared process connection plugging or a common calibration mistake.
Analytical redundancy: Where physical redundancy is impractical, a model of the process predicts one measurement from others. Comparing measured and predicted values produces a residual that should remain near zero; a persistent nonzero residual indicates a sensor fault, a process change, or a model that no longer matches the plant. Estimating the correct threshold for the residual is the difficult part, since it must accommodate legitimate modeling error without masking real faults.
Stuck-signal detection: A perfectly constant reading is suspicious in any process that has measurement noise. Monitoring the variance of a signal over a moving window detects frozen converters and cached values that are no longer being updated, a failure mode that range and rate checks cannot catch.
Actuator feedback: Commanding an output is not evidence that the output occurred. Position feedback, current sensing in a drive, and contactor auxiliary contacts close the loop between command and physical effect, and the comparison between commanded and observed state is itself a diagnostic with quantifiable coverage.
Diagnostic Coverage and Timing
Safety standards do not credit a diagnostic merely for existing. They require quantified coverage and bounded detection time, and those two quantities determine how much the monitoring architecture contributes to the safety argument.
Coverage levels: The functional safety literature conventionally groups diagnostic coverage into three bands: low at approximately 60 percent, medium at approximately 90 percent, and high at approximately 99 percent. Achieving the high band generally requires hardware redundancy such as lockstep comparison or dual-channel measurement, because software-only checks rarely detect the last percent of dangerous failure modes. Coverage is established by failure modes, effects, and diagnostic analysis, which enumerates the failure modes of each element, assigns a failure rate to each, and records which mechanism detects it.
Safe failure fraction and architectural constraints: Coverage feeds the safe failure fraction, which combines safe failures and detected dangerous failures as a proportion of total failures. The safe failure fraction, together with hardware fault tolerance, sets the maximum safety integrity level that an architecture may claim, independently of the calculated failure probability. This is why adding diagnostics can raise the achievable integrity level of an existing single-channel design without adding a second channel.
Detection timing: Coverage without timing is meaningless. The diagnostic test interval plus the time required to execute the fault reaction must be shorter than the process safety time, the interval between the onset of a dangerous fault and the occurrence of harm. Automotive practice expresses the same constraint as the fault-tolerant time interval, which must exceed the sum of the fault detection time interval and the fault reaction time interval. A diagnostic that runs once per second is useless for a hazard that develops in fifty milliseconds, regardless of its coverage.
Latent fault metrics: Automotive hardware metrics separate the two concerns explicitly. The single-point fault metric addresses faults that lead directly to a violation of the safety goal, with targets of at least 90, 97, and 99 percent for ASIL B, C, and D respectively. The latent fault metric addresses faults that are dangerous only in combination with a second fault, with targets of at least 60, 80, and 90 percent for the same levels. Start-up tests and periodic background tests are the primary contributors to the latent fault metric, which is the quantitative reason that a safety architecture cannot rely on runtime monitoring alone.
Independence of the monitor: A diagnostic that shares resources with the function it monitors cannot claim full coverage, because a single fault may disable both. Freedom from interference between the monitor and the monitored function, established through memory protection, timing partitioning, and separate hardware where necessary, is a precondition for crediting the diagnostic in the safety analysis.
Degradation Detection
Degradation detection identifies gradual deterioration in performance or component health before it becomes a failure. Where the mechanisms above answer the question "is the system working now," degradation detection answers "how much margin remains."
Parameter Trending
Tracking key parameters over time reveals patterns that no instantaneous threshold would catch.
Statistical process control: Control charts distinguish common-cause variation, which is inherent in the process, from special-cause variation, which indicates a change worth investigating. Rules based on runs, trends, and points beyond control limits detect systematic shifts long before a parameter reaches an alarm threshold. An increase in variance frequently precedes a change in mean and is often the earliest available indicator.
Baseline comparison: Measurements taken during commissioning provide the reference against which later readings are judged. Because most parameters depend on load, temperature, and operating point, useful comparison requires normalization: motor current compared at equal load, converter efficiency compared at equal output power, and thermal resistance computed from the measured power dissipation rather than from temperature alone.
Rate-of-change monitoring: A step change in an otherwise stable parameter indicates an acute event such as a loosened connection, while a slow monotonic drift indicates wear. The two patterns warrant different responses, and separating them prevents a wear trend from triggering an immediate shutdown and an acute fault from being averaged away into a slow trend.
Feature extraction: Raw waveforms are usually reduced to features before trending. Vibration analysis uses root-mean-square level, kurtosis, and the amplitude of specific spectral lines associated with bearing and gear geometry. Electrical monitoring uses harmonic content, ripple amplitude, and switching-edge timing. Good feature selection matters more than sophisticated modeling, since a feature that does not respond to the degradation of interest cannot be rescued by any algorithm.
Component Aging Mechanisms
Understanding how components age enables targeted monitoring of the elements that actually limit service life.
Electrolytic capacitor wear-out: Aluminum electrolytic capacitors lose capacitance and gain equivalent series resistance as electrolyte escapes through the seal, a process strongly accelerated by temperature. Manufacturers commonly specify endurance with a rule of thumb in which useful life roughly doubles for each 10 degrees Celsius of reduction in operating temperature within the rated range, and they commonly define end of life as a capacitance decrease of about 20 percent or a doubling of equivalent series resistance. In a power converter, both effects appear as increased output ripple, which makes ripple amplitude a practical proxy for capacitor health.
Battery capacity fade: Rechargeable cells lose capacity through both cycling and calendar aging. State-of-health algorithms estimate remaining capacity by combining coulomb counting during full charge and discharge, open-circuit voltage measured after relaxation, and internal resistance measured from current steps. Backup batteries in safety systems are commonly retired at 80 percent of rated capacity, since the knee of the fade curve beyond that point makes remaining life difficult to predict.
Connector and relay wear: Contact resistance rises with mating cycles, vibration-induced fretting, and corrosion, particularly in tin-plated contacts where fretting disrupts the thin conductive layer. Monitoring the voltage drop across a connection under known current identifies degrading contacts before intermittent open circuits appear. Relay contacts additionally suffer erosion from arcing, so operation counts weighted by switched current provide a useful wear index.
Semiconductor parameter drift: Bias temperature instability and hot-carrier injection shift transistor threshold voltages over time, slowing circuits; time-dependent dielectric breakdown degrades gate oxides; and electromigration thins metal interconnect under sustained current density. On-chip ring oscillators serve as aging sensors, because their frequency tracks the accumulated threshold shift and can be compared against the value recorded at manufacture. In power semiconductors, bond-wire lift-off and solder-layer fatigue manifest as a rising on-state voltage drop and rising junction-to-case thermal resistance.
Solder joint and interconnect fatigue: Repeated thermal cycling drives crack growth in solder joints through differential expansion. Because damage accumulates per cycle rather than per hour, cycle counting weighted by temperature swing predicts remaining life far better than elapsed operating time.
Environmental Stress Monitoring
Environmental history provides the context that converts a measurement into a life estimate.
Temperature exposure logging: Cumulative time at temperature, and especially excursions above rated limits, correlates strongly with chemically driven aging. Arrhenius models relate temperature history to expected life reduction through an activation energy characteristic of the dominant failure mechanism, and recording a histogram of temperature rather than an average preserves the information those models need, since brief hot excursions dominate the result.
Thermal and mechanical cycle counting: Fatigue mechanisms respond to cycle amplitude, so monitors apply rainflow counting to reduce an irregular temperature or strain history into equivalent cycles. Coffin-Manson relations then convert those cycles into consumed fatigue life. Vibration and shock recorders capture mechanical stress history for the same purpose.
Humidity and contamination: Moisture accelerates corrosion, dendrite growth, and insulation breakdown, and its effect combines with temperature and applied voltage. Combined temperature-humidity sensors inside enclosures detect conditions that approach the dew point, which is the condition under which condensation makes electrochemical migration possible. Sealed enclosures may include desiccants with humidity indicators for periodic visual inspection.
Electrical stress: Supply transients, load current excursions, and power cycling all consume life. Counting power cycles is particularly valuable, because inrush current and thermal shock at start-up frequently cause more damage than continuous running.
Predictive Maintenance
Predictive maintenance uses health monitoring data to forecast maintenance needs, balancing the cost of preventive work against the risk of failure. It differs from preventive maintenance, which acts on a fixed schedule, and from corrective maintenance, which acts after a failure. The economic case rests on avoiding both the unplanned outage and the replacement of components that still had useful life.
Remaining Useful Life Estimation
Estimating when a component will require replacement is the central technical problem of prognostics.
Physics-of-failure models: Mathematical models of the degradation mechanism predict progression from measured operating conditions. Arrhenius relations for chemical aging, Coffin-Manson relations for thermal fatigue, and Black's equation for electromigration are established examples. These models are interpretable and extrapolate to conditions outside the observed data, but they require knowledge of the specific dominant mechanism and of parameters that are often not published.
Data-driven models: Regression and machine learning methods trained on historical run-to-failure data identify patterns that precede failure without requiring a mechanistic model. They adapt readily to complex systems, but they need representative failure data that safety-critical equipment rarely produces, since such equipment is usually replaced before it fails. Models trained on one fleet or duty cycle frequently fail to transfer to another.
Hybrid approaches: Combining a physics-based degradation model with recursive state estimation, using techniques such as particle or Kalman filtering, updates the model parameters from observed data while retaining physical interpretability. This is the prevailing approach where some mechanistic understanding exists and data are limited.
Uncertainty and prognostic performance: A remaining-life estimate without a confidence interval is not actionable, since maintenance planning depends on the lower bound rather than the expected value. Prognostic algorithms are evaluated on how early their predictions enter an acceptable error band around the true remaining life and on whether that accuracy improves as failure approaches. An algorithm whose predictions converge only in the final hours provides little planning value.
Condition-Based Maintenance Triggers
Defining triggers balances early intervention against unnecessary work.
Threshold-based triggers: Threshold crossings initiate action when a monitored parameter exceeds a defined limit. Multiple levels provide warning, alarm, and trip conditions, and hysteresis prevents repeated toggling when a value sits near the limit.
Trend-based triggers: Extrapolating the current trend to predict when a threshold will be crossed converts a limit into a schedule. This allows maintenance to be planned into an existing outage window while consuming most of the component's useful life.
Probabilistic triggers: Bayesian methods combine evidence from multiple indicators into a failure probability and trigger maintenance when risk exceeds an acceptable level. This handles measurement uncertainty explicitly and permits weak evidence from several sources to accumulate into a decision that no single indicator would justify.
Risk-weighted prioritization: Not every degrading component deserves the same urgency. Weighting the predicted failure probability by the consequence of failure and by the cost of intervention ranks competing maintenance demands and matches the criticality analysis already produced for the safety case.
Maintenance Optimization
Coordinating maintenance across components improves overall availability and cost.
Opportunistic maintenance: When one component requires attention, nearby components approaching their limits can be addressed in the same outage. The saving is largest where access is expensive, as with offshore installations, satellites in orbit, and equipment that requires a full plant shutdown to reach.
Spare parts management: Health data shifts inventory policy from statistical demand forecasting toward known upcoming need, reducing both stockouts and carrying cost. This matters most for long-lead-time and obsolescence-prone parts, where advance warning of several months changes the outcome.
Maintenance resource planning: Predicted needs allow scheduling of personnel, tooling, and access equipment, which frequently dominate the duration of the outage more than the repair itself.
Effect on the safety case: In a system certified against a functional safety standard, maintenance intervals are inputs to the failure probability calculation. Replacing a fixed proof-test interval with a condition-based one requires demonstrating that the diagnostic reveals the relevant dangerous failures with adequate coverage, so predictive maintenance interacts directly with the certification argument rather than sitting outside it.
Health Monitoring Architectures
The architecture of a health monitoring system must balance detection capability, resource consumption, and independence from the functions being monitored.
Centralized Monitoring
A dedicated health management unit collects and processes health data from throughout the system.
Advantages: Central processing enables correlation across subsystems, which is essential for distinguishing a root cause from its many downstream symptoms. Monitoring policy, alarm priority, and health reporting remain consistent, and there is a single authoritative view of system state.
Disadvantages: The health manager becomes a single point of failure and must therefore monitor itself or be monitored externally. Communication bandwidth and latency limit how finely it can observe fast phenomena in remote subsystems.
Distributed Monitoring
Health monitoring functions are embedded within individual subsystems.
Advantages: Local monitoring responds within microseconds to local anomalies and transmits only summarized status, which suits systems where the communication bus is a constrained resource. Each subsystem supplier can implement monitoring matched to its own failure modes.
Disadvantages: System-level assessment requires aggregating heterogeneous status from independently developed units, and a fault that manifests only as an interaction between two healthy subsystems may go unreported. Consistency of severity classification across suppliers is a recurring integration problem.
Hierarchical Monitoring
Multi-level architectures combine local monitoring with higher-level aggregation and analysis, and they are the dominant pattern in complex systems.
Local level: Individual components perform self-tests, range checks, and watchdog functions, reporting a compact status to the subsystem monitor. Response time at this level is measured in microseconds to milliseconds.
Subsystem level: Subsystem health managers aggregate component status, run subsystem-specific diagnostics, and apply the first stage of alarm filtering and fault consolidation.
System level: The system health manager correlates across subsystems, resolves cascaded faults into a single root cause, and interfaces with operational displays and maintenance systems. Its time constants are seconds and longer, and it carries the reasoning that requires a whole-system view.
Monitor Independence and the Monitor-Actuator Pattern
Where the highest integrity is required, a simple, thoroughly verified monitor supervises a complex functional channel. The functional channel computes the output, and an independent monitor, often on separate hardware with separately developed software, checks the output against safety constraints and retains the authority to force the safe state. The pattern is valuable because the monitor can be far simpler than the function it supervises, and simplicity is what makes exhaustive verification feasible.
The pattern only works if the monitor is genuinely independent. Shared power supplies, shared clocks, shared sensors, and shared development assumptions are all common-cause paths that a dependent failure analysis must examine explicitly. Where partitioned software rather than separate hardware provides the independence, the partitioning mechanism itself becomes a safety-critical element requiring its own verification.
Diagnostic Data Management
Effective health monitoring requires systematic handling of diagnostic data from collection through analysis and archival.
Data Collection and Storage
Health monitoring generates substantial data volumes on devices with limited resources.
Sampling strategies: Adaptive sampling raises the rate during anomalies and lowers it during steady operation, preserving detail where it matters. Deadband reporting, which records a value only when it deviates from the last recorded value by more than a defined amount, reduces trend storage by an order of magnitude for slowly varying signals.
Data compression: Lossless compression preserves event logs and trend data exactly. Lossy reduction is acceptable for high-frequency waveforms when only derived features are required, but the decision must be made deliberately, because a discarded waveform cannot be reanalyzed when a new failure mode is discovered later.
Circular buffers and snapshots: A continuously overwritten buffer preserves the interval before a trigger, so that when a fault occurs, both the approach to the fault and its aftermath are available. Automotive systems store an equivalent snapshot of operating conditions alongside each fault code.
Non-volatile storage integrity: Diagnostic records must survive the power loss that often accompanies a fault. Wear leveling, atomic update schemes that never leave a partially written record, and reserved space that prevents a fault storm from overwriting the earliest and most informative entries are all necessary, and the storage subsystem itself needs integrity checks.
Fault Logging and Event Recording
Systematic fault recording supports troubleshooting and long-term reliability improvement.
Event timestamping: Accurate, synchronized timestamps allow reconstruction of the event sequence across distributed nodes, which is what separates a cause from its consequences. Where wall-clock time is unavailable at start-up, a monotonic counter with later correlation to real time preserves ordering.
Fault classification: Standardized codes make automated analysis and fleet comparison possible. Distinguishing the fault status, such as active, previously active, or pending confirmation, from the fault identity prevents intermittent faults from being lost when the condition clears.
Fault consolidation: A single root cause commonly produces dozens of secondary indications. Correlation rules that suppress consequential faults and report the underlying cause are essential; without them, maintenance staff face an undifferentiated list and lose confidence in the system.
Context capture: Recording operating mode, configuration version, software version, environmental conditions, and recent command history alongside the fault provides the context without which field faults are frequently impossible to reproduce.
Remote Monitoring and Telemetry
Connected systems enable off-site diagnostics and fleet-scale analysis.
Secure communication: Health data reveals operational patterns and must be protected in transit and at rest through authenticated, encrypted channels. Equally important, a diagnostic interface must never become a control path: remote access should be read-only unless a rigorously authenticated and authorized maintenance mode is engaged, since diagnostic ports have repeatedly proven to be an attack surface on connected equipment.
Bandwidth optimization: Edge processing computes features locally and transmits summaries, with raw data uploaded only on demand or when an anomaly is flagged. Exception-based reporting suits constrained links, while publish-subscribe protocols suit intermittent connectivity by decoupling the producer from the consumer.
Fleet-wide analysis: Comparing a unit against the population distribution rather than a fixed threshold detects units that are drifting even while remaining within specification, and it reveals systematic design or manufacturing issues that no single unit would expose. Population baselines must account for differences in duty cycle and environment, or benign variation will be misread as degradation.
Interoperability: Industrial deployments increasingly expose health data through standardized information models rather than proprietary formats, so that condition data can flow into asset management systems without bespoke integration for every device type.
Application Domains
The same principles take markedly different forms across industries, shaped by the applicable regulations, the available maintenance access, and the consequences of failure.
Automotive: On-board diagnostics began as an emissions-compliance requirement and grew into a general vehicle health framework. Standardized diagnostic trouble codes, freeze-frame snapshots of operating conditions at the moment of detection, and readiness monitors that indicate whether each diagnostic has completed since the last clearing are all part of the regulated interface. Beyond emissions, unified diagnostic services provide a common request-and-response protocol for reading fault memory, running routines, and accessing data identifiers across electronic control units from different suppliers. Safety-related monitoring runs in parallel, structured around the hardware architectural metrics and the fault-tolerant time interval.
Aerospace: Transport aircraft integrate line-replaceable-unit built-in test equipment with a central maintenance computer, so that a technician sees one consolidated fault list correlated with the cockpit effects the crew observed. Design guidance for built-in test equipment and for the onboard maintenance system, including the aircraft condition monitoring function, standardizes how member systems report to that central function. Rotorcraft add health and usage monitoring systems that track drivetrain vibration and record usage spectra for fatigue-life tracking of rotating components, a domain where accumulated cycle counts govern retirement more than elapsed time does.
Process industry: Safety instrumented systems combine online diagnostics with scheduled proof testing, and both intervals appear directly in the calculation of average probability of failure on demand. Field devices report standardized status categories, distinguishing failure, function check, out of specification, and maintenance required, so that an operator can tell an actual process deviation from a device undergoing calibration. This separation of diagnostic status from process alarm is what keeps the alarm system usable.
Medical devices: Regulated devices must define essential performance and demonstrate that a single fault condition does not compromise it. Self-tests at power-on and at defined intervals verify alarm systems, backup power, and measurement accuracy. Records of these checks form part of the device history, and alarm design is itself standardized so that a monitoring alarm conveys priority unambiguously in a clinical environment.
Computing infrastructure: Servers embed a baseboard management controller that operates on standby power and monitors temperatures, voltages, fan speeds, and memory error counters independently of the host operating system, so that a hung or powered-down host can still be diagnosed. Power components report through a dedicated management bus, and modern deployments expose the whole inventory through a structured management interface consumed by data-center automation. Corrected memory error rates are a standard predictive indicator, since a rising rate on a particular module reliably precedes uncorrectable errors.
Spacecraft: Where repair is impossible and communication is intermittent, health monitoring must be autonomous. Onboard fault protection detects anomalies and commands a safe mode that preserves power and thermal control and orients antennas toward Earth, then waits for ground intervention. Telemetry is continuously recorded and downlinked when a link is available, and the flight system relies on watchdogs, memory scrubbing against radiation-induced upsets, and redundancy management that can be reconfigured from the ground.
Implementation Considerations
Designing effective health monitoring requires attention to several practical concerns that determine whether the mechanisms work in the field as intended.
Resource allocation: Monitoring consumes processor time, memory, and bandwidth. Continuous core self-test and memory scrubbing can occupy a noticeable share of the processor budget, and this overhead must be included in the worst-case timing analysis rather than measured after the fact. Diagnostics that run only when the processor is idle provide no guaranteed test interval and cannot support a coverage claim.
False alarm management: Overly sensitive monitoring produces nuisance alarms that erode confidence and lead operators to disable or ignore the system, which is a worse outcome than no monitoring at all. Debouncing, confirmation counters that require a condition to persist across several evaluations, and hysteresis around thresholds reduce false positives. The trade-off against detection latency must be made explicitly, because every confirmation cycle lengthens the detection time that the safety timing analysis assumes.
Monitoring system reliability: The monitor is itself a component that can fail, and a monitor that fails silently in a permanently satisfied state is worse than no monitor. Self-checking mechanisms, comparison against a redundant monitor, and periodic verification through fault injection ensure that a monitoring failure is revealed rather than latent.
Testability and verification: Every diagnostic requires evidence that it detects what it claims to detect. Test hooks that allow controlled fault injection during development and periodic verification in service should be designed in from the start and protected so that they cannot be triggered inadvertently in normal operation.
Proportionate response: The reaction must match the severity of the detected condition. A logged advisory suits a slow degradation trend; a warning to the operator suits an approaching limit; an immediate transition to the safe state suits a dangerous fault. Reacting too aggressively converts recoverable conditions into unnecessary shutdowns, and repeated unnecessary shutdowns create their own hazards.
Human factors: Diagnostic output is consumed by people. Codes that map to a clear corrective action, severity levels that are consistent across subsystems, and displays that show the root cause rather than the full symptom list determine whether monitoring shortens repair time or lengthens it. Fleet data on the rate at which components are replaced without resolving the reported fault is a direct measure of diagnostic quality.
Configuration and lifecycle management: Thresholds, baselines, and models are configuration data that must be version-controlled, validated, and traceable to the analysis that justified them. When hardware is revised or software updated, the associated diagnostic parameters must be revisited, since a baseline captured on an earlier hardware revision may no longer represent healthy behavior.
Standards and Guidelines
Several standards address health monitoring requirements for safety-critical and high-reliability systems.
IEC 61508: The base functional safety standard requires diagnostic coverage appropriate to the target safety integrity level and constrains the diagnostic test interval relative to the process safety time. Part 2 enumerates diagnostic techniques for hardware in its annexes and relates diagnostic coverage to the safe failure fraction used in the architectural constraints, while Part 3 lists the corresponding software techniques, including program-sequence monitoring and defensive programming.
ISO 26262: The automotive adaptation defines the hardware architectural metrics in Part 5, with single-point fault metric targets of at least 90, 97, and 99 percent for ASIL B, C, and D, and latent fault metric targets of at least 60, 80, and 90 percent for the same levels. It also formalizes the fault-tolerant time interval, which bounds the sum of fault detection and fault reaction time.
DO-178C and DO-254: In civil avionics, monitoring software must satisfy the objectives of the design assurance level appropriate to its contribution to safety, and partitioning requirements ensure that monitoring functions cannot be corrupted by the functions they supervise. DO-254 provides the corresponding guidance for complex electronic hardware, where built-in test structures are part of the design assurance evidence.
ARP4754A and ARP4761: These aerospace recommended practices cover development of civil aircraft systems and the associated safety assessment process, including common-cause and common-mode analysis. That analysis is where the independence claimed for redundant monitors and monitored channels must be substantiated rather than assumed.
ARINC 604 and ARINC 624: ARINC 604 provides design guidance for built-in test equipment and describes a centralized fault display system that gathers fault data from individual line-replaceable units. ARINC 624 provides design guidance for the onboard maintenance system, integrating failure monitoring, built-in test access, the aircraft condition monitoring system, and onboard maintenance documentation around a central maintenance computer function.
IEC 62243 (IEEE Std 1232), AI-ESTATE: This dual-logo standard defines information models and software services for intelligent diagnostic reasoners, so that diagnostic knowledge and results can be exchanged between different test and maintenance environments rather than being locked into a single tool chain. The second edition was published in 2012 as IEC 62243:2012(E), corresponding to IEEE Std 1232-2010.
ISO 13374 and ISO 13381: ISO 13374 defines the data processing, communication, and presentation architecture for condition monitoring and diagnostics of machines, establishing the now-familiar chain from data acquisition through manipulation, state detection, health assessment, and prognostic assessment to advisory generation. ISO 13381 addresses prognostics specifically. Together they underpin open condition-based maintenance data architectures.
NAMUR recommendations: NE 43 standardizes the use of out-of-range current-loop signals to indicate transmitter failure, and NE 107 defines four diagnostic status categories for field devices: failure, function check, out of specification, and maintenance required. Both are widely implemented in process instrumentation because they let a control system distinguish a device problem from a process problem automatically.
Test access standards: IEEE 1149.1 defines the boundary-scan architecture used for interconnect testing and, in the field, for structural verification during maintenance. IEEE 1687 defines a network for accessing embedded instruments such as memory BIST engines, temperature sensors, and aging monitors, allowing firmware to operate them without dedicated pins.
Summary
System health monitoring is what allows an embedded system to make a defensible claim about its own condition. Watchdog timers detect the loss of correct execution, built-in self-test verifies that hardware still functions, signal validation confirms that inputs represent reality, degradation detection tracks the consumption of margin, and predictive maintenance converts that knowledge into a schedule. Each mechanism covers failure modes the others miss, which is why credible monitoring is always a combination rather than a single technique.
The quantitative requirements are what separate safety monitoring from general instrumentation. Diagnostic coverage must be established by analysis and confirmed by fault injection, detection plus reaction must complete within the process safety time or the fault-tolerant time interval, and latent faults must be addressed by start-up and periodic tests rather than by runtime monitoring alone. The monitor must be independent of what it monitors, and it must reveal its own failures rather than fail silently.
The practical failure modes of health monitoring are as much organizational as technical. Nuisance alarms lead to disabled monitors, unconsolidated fault lists lead to unnecessary component replacement, and unversioned thresholds drift out of step with the hardware they describe. As machine learning improves anomaly detection, connected fleets enlarge the available population baseline, and digital twin models sharpen remaining-life estimates, the discipline of validating what a monitor actually detects, and how quickly, remains the element that determines whether the added sophistication improves safety or merely adds data.