Electronics Guide

The Purpose of Testing

Testing serves multiple purposes in embedded software development:

Defect detection: The primary purpose of testing is finding defects before software reaches production. Each defect found during testing is one that will not cause field failures.

Requirements verification: Testing demonstrates that software meets its specified requirements. Test cases trace back to requirements, providing evidence of compliance.

Design validation: Testing validates that the software design correctly addresses the intended use cases and performs adequately under expected operating conditions.

Regression prevention: Automated tests guard against regression, ensuring that code changes do not break existing functionality.

Documentation: Well-written tests document expected behavior and serve as executable specifications that remain synchronized with the code.

Testing Limitations

Testing has inherent limitations that must be understood and addressed:

Cannot prove correctness: Testing can reveal the presence of defects but cannot prove their absence. Even exhaustive testing of all inputs is typically infeasible for non-trivial systems.

Test quality matters: Tests that do not exercise meaningful functionality provide false confidence. Test quality is as important as test quantity.

Coverage gaps: No practical test suite covers all possible execution paths, input combinations, and timing scenarios. Risk-based testing prioritizes coverage of critical functionality.

Environment differences: Tests may pass in test environments but fail in production due to environmental differences. Production-like test environments reduce this risk.

Test Types and Levels

Different test types address different aspects of software quality:

Unit tests: Test individual functions or modules in isolation. Unit tests are fast, focused, and provide immediate feedback during development.

Integration tests: Test interactions between components. Integration tests verify that modules work together correctly.

System tests: Test the complete integrated system against requirements. System tests exercise end-to-end functionality.

Acceptance tests: Validate that the system meets user needs and is ready for deployment. Acceptance tests often involve stakeholder participation.

Functional tests: Verify that functionality works according to specifications without regard to internal implementation.

Non-functional tests: Address quality attributes such as performance, reliability, security, and usability.

Unit Testing Principles

Effective unit testing follows established principles:

Isolation: Unit tests exercise individual units in isolation from dependencies. Mocks, stubs, and fakes replace real dependencies to achieve isolation.

Speed: Unit tests should execute quickly, enabling frequent execution during development. Slow tests reduce developer productivity and discourage testing.

Determinism: Unit tests must produce consistent results regardless of execution order or timing. Non-deterministic tests undermine confidence in the test suite.

Independence: Each test should be independent of other tests. Tests should not rely on state left by previous tests or require specific execution order.

Readability: Tests serve as documentation and should clearly express what behavior they verify. Descriptive test names and clear assertions improve maintainability.

Unit Testing Frameworks

Numerous frameworks support unit testing of embedded C code:

Unity: A lightweight, portable test framework designed for embedded systems. Unity provides a minimal footprint and can run on constrained targets.

CppUTest: A C/C++ test framework with support for memory leak detection and mocking. CppUTest is popular for embedded testing and works well with test-driven development.

Google Test: A feature-rich C++ testing framework. While designed for desktop development, Google Test works well for host-based testing of embedded code.

Ceedling: A build system and test automation tool built around Unity and CMock. Ceedling simplifies test setup and execution for embedded C projects.

Criterion: A modern C testing framework with automatic test registration, parameterized tests, and assertion macros.

Check: A unit testing framework for C that supports forking to isolate test failures and provides multiple output formats.

Mocking and Test Doubles

Test doubles replace real dependencies during unit testing:

Stubs: Provide canned responses to function calls without implementing real behavior. Stubs enable testing code that depends on unavailable hardware or external systems.

Mocks: Verify that code under test makes expected calls to dependencies. Mocks fail tests when expected interactions do not occur.

Fakes: Simplified implementations of dependencies that work but use shortcuts unsuitable for production. In-memory databases and file systems are common fakes.

CMock: An automatic mock generation tool that creates mock implementations from C header files. CMock integrates with Unity and Ceedling.

FFF (Fake Function Framework): A lightweight framework for creating fake functions in C. FFF provides simple syntax for defining fake behavior and verifying calls.

Testing Hardware-Dependent Code

Hardware dependencies present unique unit testing challenges:

Hardware abstraction layers: Designing code with clean hardware abstraction enables replacing hardware interfaces with test doubles during unit testing.

Register mocking: Memory-mapped registers can be redirected to RAM during testing, enabling verification of register access patterns.

Peripheral simulation: Simple peripheral simulations enable testing of driver logic without hardware. Simulations can model normal operation and error conditions.

Host-based testing: Running tests on development hosts rather than targets enables faster execution and better tooling support. Platform differences must be managed carefully.

Cross-compilation considerations: Code tested on hosts must also compile and run correctly on targets. Differences in word size, endianness, and compiler behavior require attention.

Integration Testing Approaches

Several approaches organize integration testing:

Big bang integration: Combining all components at once and testing the integrated system. This approach is simple but makes defect localization difficult.

Top-down integration: Starting with high-level components and progressively adding lower-level modules. Requires stubs for unavailable lower-level components.

Bottom-up integration: Starting with low-level components and progressively adding higher-level modules. Requires test drivers to exercise lower-level components.

Sandwich integration: Combining top-down and bottom-up approaches, integrating from both ends toward the middle.

Continuous integration: Integrating changes frequently, often multiple times daily. Automated testing catches integration issues early.

Interface Testing

Integration testing focuses heavily on interfaces between components:

API contract verification: Tests verify that components honor their API contracts, including parameter ranges, return values, and error handling.

Protocol compliance: Communication protocols between components must be implemented consistently. Protocol-level testing verifies message formats and sequences.

Timing and sequencing: Components may have dependencies on initialization order or timing relationships. Integration tests verify these constraints.

Error propagation: Tests verify that errors are correctly propagated across component boundaries and handled appropriately.

Hardware-Software Integration

Integrating software with hardware reveals issues invisible during host-based testing:

Driver integration: Testing device drivers with actual hardware verifies timing, interrupt handling, and register access patterns.

Timing validation: Real-time constraints can only be fully validated with actual hardware. Integration testing measures actual timing behavior.

Resource usage: Memory usage, CPU utilization, and power consumption are validated during hardware integration.

Environmental testing: Testing across temperature, voltage, and electromagnetic interference conditions reveals hardware-software interaction issues.

Functional System Testing

Functional tests verify that the system performs its intended functions:

Requirements-based testing: Test cases derive directly from requirements, providing traceability and demonstrating requirement coverage.

Use case testing: Tests exercise typical user scenarios and workflows to verify that the system supports intended use cases.

Boundary value testing: Tests focus on boundary conditions where defects commonly occur. Input ranges, timing limits, and resource limits are tested at boundaries.

Error handling testing: Tests verify appropriate responses to error conditions including invalid inputs, hardware failures, and communication errors.

Non-Functional Testing

Non-functional tests address quality attributes beyond basic functionality:

Performance testing: Measures response times, throughput, and resource utilization under various load conditions. Performance tests verify that timing requirements are met.

Stress testing: Subjects the system to extreme conditions beyond normal operating parameters to find breaking points and verify graceful degradation.

Reliability testing: Long-duration testing reveals intermittent failures, memory leaks, and degradation over time. Mean time between failures can be estimated from extended testing.

Security testing: Evaluates resistance to security threats including unauthorized access, data tampering, and denial of service attacks.

Usability testing: Assesses ease of use for human operators. For embedded systems with user interfaces, usability affects safety and effectiveness.

Hardware-in-the-Loop Testing

Hardware-in-the-loop (HIL) testing connects the embedded system to simulated environments:

Plant simulation: For control systems, HIL testing connects the controller to a real-time simulation of the controlled system. The controller operates as in production while the plant is simulated.

Sensor simulation: HIL systems inject simulated sensor signals, enabling testing of scenarios difficult or dangerous to create with real sensors.

Actuator loading: Simulated loads on actuator outputs verify that the system behaves correctly under realistic loading conditions.

Fault injection: HIL systems can inject faults into signals and power supplies to verify fault detection and handling.

Automated regression: HIL test benches enable automated execution of comprehensive test suites against actual hardware.

Benefits of Automation

Test automation provides numerous advantages:

Repeatability: Automated tests execute exactly the same way each time, eliminating human variability in test execution.

Speed: Automated tests run much faster than manual testing, enabling more frequent execution.

Coverage: Automation makes it practical to execute comprehensive test suites that would be infeasible manually.

Regression detection: Automated tests catch regressions immediately when code changes, before defects propagate.

Documentation: Automated tests document expected behavior and remain synchronized with the code.

Continuous Integration Testing

Continuous integration (CI) systems automate test execution on code changes:

Build verification: Every code change triggers automated builds and tests. Failures are reported immediately to developers.

Test selection: CI systems may run different test suites based on change scope. Fast tests run on every commit while longer tests run periodically.

Cross-target testing: CI can compile and test code for multiple target configurations, catching platform-specific issues.

Hardware farm integration: CI systems can dispatch tests to pools of hardware targets for on-target test execution.

Test Infrastructure

Effective test automation requires supporting infrastructure:

Test environments: Consistent, reproducible test environments ensure reliable test results. Containerization and virtual machines help manage test environments.

Test data management: Tests require appropriate test data. Data generation, management, and cleanup must be automated.

Result reporting: Test results must be collected, stored, and reported effectively. Trend analysis reveals quality changes over time.

Failure analysis: When tests fail, logs, traces, and other artifacts support root cause analysis. Test infrastructure must capture sufficient information for debugging.

Formal Methods Overview

Formal methods apply mathematical rigor to software development:

Formal specification: Mathematical notation precisely specifies what software should do. Specifications eliminate ambiguity inherent in natural language requirements.

Formal verification: Mathematical proofs demonstrate that implementations satisfy their specifications. Unlike testing, verification can prove absence of certain defect classes.

Model checking: Exhaustively explores all possible states of a finite-state model to verify properties. Model checking automatically finds counterexamples when properties are violated.

Theorem proving: Interactive or automated provers construct mathematical proofs of software properties. Theorem proving handles infinite state spaces but requires more expertise.

Design by Contract

Design by contract is an accessible formal method for everyday development:

Preconditions: Conditions that must be true when a function is called. The caller is responsible for ensuring preconditions are met.

Postconditions: Conditions that must be true when a function returns. The function is responsible for establishing postconditions.

Invariants: Conditions that must remain true throughout execution. Class invariants and loop invariants express consistency requirements.

Runtime checking: Contracts can be checked at runtime using assertions, catching violations during testing.

Static verification: Tools can statically verify that code satisfies its contracts without executing the code.

Static Analysis and Formal Verification

Static analysis tools apply formal methods to detect defects:

Abstract interpretation: Mathematically analyzes program behavior by approximating program states. Abstract interpretation can prove absence of certain defects like buffer overflows.

Data flow analysis: Tracks how values flow through programs to detect issues like uninitialized variables and null pointer dereferences.

Type system verification: Advanced type systems can encode and verify complex properties. Dependent types and refinement types extend verification capabilities.

Commercial tools: Tools like Polyspace, Astree, and FramaC provide formal verification capabilities for industrial embedded software.

Model-Based Verification

Model-based approaches verify software through abstract models:

State machine verification: Modeling software as state machines enables verification of properties like absence of deadlocks and liveness.

Timed automata: Extensions of state machines with timing constraints enable verification of real-time properties.

Process algebras: Mathematical frameworks for modeling concurrent systems and verifying properties like absence of race conditions.

Model extraction: Some tools extract models automatically from code, enabling verification without manual modeling.

Applying Formal Methods

Practical application of formal methods requires pragmatic choices:

Selective application: Formal methods are often applied to critical components rather than entire systems. Safety-critical algorithms and security-sensitive code are prime candidates.

Scalability considerations: Full formal verification may not scale to large codebases. Combining formal methods with testing provides practical coverage.

Tool support: Effective use of formal methods requires appropriate tools. Tool selection depends on the properties to verify and the development language.

Expertise requirements: Formal methods require specialized expertise. Training and potentially hiring specialists may be necessary for serious adoption.

Traceability

Traceability connects requirements to tests and vice versa:

Forward traceability: Links requirements to test cases, ensuring every requirement has associated tests.

Backward traceability: Links tests back to requirements, ensuring tests are justified by requirements and not merely testing implementation details.

Traceability matrices: Tables showing relationships between requirements and tests enable coverage analysis and impact assessment.

Tool support: Requirements management tools like DOORS, Polarion, and Jama provide traceability features integrated with test management.

Test Case Design

Systematic test case design improves coverage and efficiency:

Equivalence partitioning: Dividing input space into partitions where all values in a partition should behave similarly. Testing one value from each partition provides reasonable coverage.

Boundary value analysis: Testing values at and near partition boundaries where defects commonly occur.

Decision tables: Tabular representation of combinations of conditions and their expected outcomes. Decision tables systematically cover condition combinations.

State transition testing: For stateful systems, testing transitions between states and sequences of transitions.

Combinatorial testing: Systematically testing combinations of input parameters. Techniques like pairwise testing provide coverage of parameter interactions with manageable test counts.

Coverage Analysis

Coverage metrics assess how thoroughly tests exercise the software:

Requirements coverage: Percentage of requirements addressed by test cases. Complete requirements coverage is typically required for certification.

Code coverage: Percentage of code executed during testing. Statement, branch, and MC/DC coverage metrics measure different aspects of execution coverage.

Coverage goals: Coverage targets depend on system criticality. Safety standards specify coverage requirements for different integrity levels.

Coverage gaps: Analysis of uncovered requirements and code guides test development priorities.

Safety Standards Requirements

Safety standards prescribe testing requirements:

DO-178C: The aerospace software standard requires structured testing with coverage requirements increasing with software criticality level. Modified condition/decision coverage is required for the most critical software.

ISO 26262: The automotive functional safety standard specifies testing methods based on Automotive Safety Integrity Levels (ASIL). Higher ASIL levels require more rigorous testing methods.

IEC 62304: The medical device software standard requires risk-based testing with intensity matching software safety classification.

IEC 61508: The industrial functional safety standard defines Safety Integrity Levels (SIL) with corresponding testing requirements.

Verification and Validation Activities

Safety-critical development requires specific V&V activities:

Reviews: Requirements, design, and code reviews are mandatory. Independence requirements specify reviewer qualifications and separation from development.

Analysis: Static analysis, timing analysis, and safety analysis complement testing.

Testing: Unit, integration, and system testing with specified coverage levels provide evidence of correct implementation.

Documentation: Test plans, procedures, results, and traceability must be documented to certification standards.

Independent Testing

Independence in testing provides additional assurance:

Independence levels: Standards define independence levels from the same person reviewing their own work to separate organizations performing verification.

Independent test development: Tests developed independently from code are more likely to find defects due to different interpretations of requirements.

Independent test execution: Test execution by independent parties ensures tests are not tailored to pass.

Qualification testing: Independent qualification testing may be required before deployment to demonstrate system readiness.

Test Planning

Test planning establishes the testing approach:

Test strategy: Defines the overall testing approach including test levels, types, and techniques to be applied.

Test plan: Documents specific testing activities, schedules, resources, and entry/exit criteria.

Risk-based prioritization: Testing effort should focus on areas with highest risk. Risk assessment guides test planning priorities.

Resource planning: Test planning must account for test development effort, test execution resources, and hardware availability.

Defect Management

Systematic defect management maximizes the value of testing:

Defect tracking: All defects should be recorded in a tracking system with sufficient information for reproduction and analysis.

Defect analysis: Analyzing defect patterns reveals quality issues and guides process improvement.

Root cause analysis: Understanding why defects occurred helps prevent similar defects in the future.

Metrics: Defect metrics like discovery rate, fix rate, and age provide insight into quality status and trends.

Test Environment Management

Test environments require careful management:

Environment configuration: Test environments must be configured consistently and documented thoroughly.

Hardware management: Physical hardware for testing requires inventory management, maintenance, and scheduling.

Environment isolation: Test activities should not interfere with each other. Isolation mechanisms prevent cross-contamination.

Production similarity: Test environments should match production environments as closely as practical to ensure test validity.

Test Design Best Practices

Test one thing at a time: Each test should verify one specific behavior. Focused tests are easier to understand, maintain, and debug when they fail.

Use descriptive names: Test names should clearly describe what is being tested and expected outcome. Good names serve as documentation.

Keep tests simple: Complex tests are hard to understand and maintain. Simplicity in tests is more important than avoiding code duplication.

Test behavior, not implementation: Tests should verify observable behavior rather than internal implementation details. Implementation-focused tests break when code is refactored.

Design for testability: Consider testability during design. Dependency injection, clean interfaces, and modularity improve testability.

Test Execution Best Practices

Run tests frequently: Execute tests as often as practical to catch issues early. Continuous integration enables frequent automated testing.

Fix failing tests immediately: Failing tests lose value when ignored. Investigate and fix failures promptly to maintain test suite integrity.

Maintain test isolation: Tests should not depend on each other or leave state that affects other tests. Isolation ensures reliable results.

Monitor test performance: Slow tests reduce testing frequency. Monitor and optimize test execution time.

Test Maintenance Best Practices

Treat tests as production code: Tests deserve the same quality standards as production code including code review, refactoring, and documentation.

Remove obsolete tests: Tests for removed functionality or superseded requirements should be removed. Dead tests clutter the test suite.

Refactor tests: As test suites grow, refactoring improves maintainability. Extract common setup, improve assertions, and simplify complex tests.

Review test coverage regularly: Periodically assess whether tests cover current requirements and identify gaps requiring new tests.