Automotive software testing techniques as a defect-escape ledger

Get a safety-critical validation review

The requirement we trace, and why one thread beats a glossary#

Most guides to automotive software testing techniques hand you a definition list. First they define MIL. Then they define SIL, PIL, and HIL in turn. However, a definition never tells you which rung would have caught your bug. Therefore this guide refuses the glossary and threads a single requirement instead.

Our requirement comes from a hazard analysis. In ISO 26262, a HARA rates each hazard and assigns an Automotive Safety Integrity Level. Because an unwanted or missing automatic emergency brake can kill, it lands at ASIL D, the highest level. So the function reads: command braking within 200 ms of a validated obstacle.

Because the requirement carries a number, every rung below can be measured against it. First the model must implement the right control law. Then the generated code must preserve it. Next the target must execute it in time. Finally the integrated vehicle must honour it in the world. In short, one thread turns four acronyms into one accountable chain.

The V-model, actually threaded#

Every ranking page shows an empty V-model. This one threads the AEB requirement through it. The left arm decomposes the safety goal into requirements and design as it descends. Meanwhile the right arm integrates and checks as it climbs, and each check traces back to the requirement it verifies.

Read the dashed edges as the trace links. Because each verification node cites a specific left-arm artefact, a red result is never orphaned. Instead it points at the exact requirement or design it violates. Consequently the V-model stops being a diagram and starts being an accountability structure.

Verification vs validation: where the boundary actually falls#

These two words get used loosely, so pin them down. Verification asks whether you built the thing right, against a spec. Validation asks whether you built the right thing, against the real world and the original intent. The boundary is not decorative. It decides which rung owns which question.

Therefore MIL, SIL, PIL, and HIL are all verification. Each one checks an artefact against the spec above it in the V. In contrast, on-vehicle testing is validation. It asks whether the integrated system actually keeps people safe when a real sensor sees a real obstacle. Because the questions differ, a rung that verifies cleanly can still fail validation, and that gap is where field defects live.

Requirements traceability at every verification node#

Traceability is the glue that makes the ledger work. In practice each rung's pass criteria cite the requirement under test. So when a check fails, it names the requirement it broke, not just a line of code. Because ISO 26262-8 asks for bidirectional traceability, the link runs both ways, from requirement to test and back.

Furthermore, traceability is what lets you prove coverage. A requirement with no linked test is a gap you can see. Consequently the trace matrix doubles as a completeness check, which is exactly what an assessor asks for. The same discipline underpins a full enterprise software development lifecycle, where every requirement earns a test.

Walking the requirement down the automotive software testing techniques chain#

Now walk the requirement down. The chain below states, at each rung, what it CATCHES and what it is BLIND to. Because the blind spot is passed to the next rung, the escape is visible at a glance. The rung names follow the standard vendor taxonomy documented in the MathWorks reference for SIL and PIL simulation. First read the diagram, then read each rung in turn.

MIL: control-logic and algorithm errors, blind to code generation#

Model-in-the-loop runs the control model on the host in floating point. Because it executes the algorithm directly, it catches requirement mismatches and control-logic errors early. So a wrong gain or a missing clamp shows up here, cheaply. However, MIL runs the model, not the code you ship. Therefore it is structurally blind to anything the code generator introduces, and it passes that whole class downstream.

SIL: fixed-point and code-gen defects, blind to target timing#

Software-in-the-loop compiles the production code and runs it on the host. Because it exercises the generated code, it catches integer-scaling, quantization, and code-generation discrepancies MIL never sees. The two tabs below show the same braking law as a floating-point model and as its fixed-point generated form. Notice the overflow the model never exercised.

/* MIL / model view: the braking law in floating point on the host.
   The algorithm is correct here. Timing and word length do not exist yet. */
float brake_command(float closing_speed, float distance) {
    float ttc = distance / closing_speed;      /* time to collision, seconds */
    float demand = GAIN * (TTC_LIMIT - ttc);   /* proportional brake demand   */
    if (demand < 0.0f) demand = 0.0f;
    if (demand > 1.0f) demand = 1.0f;          /* clamp to [0, 1]             */
    return demand;
}
/* On the host, in double-wide floats, this passes every requirement check. */

/* SIL view: the SAME law after fixed-point code generation (Q15).
   SIL compiles this production code for the host and finds what MIL cannot:
   an integer-scaling defect the floating-point model never exercised. */
int16_t brake_command_q15(int16_t closing_speed, int16_t distance) {
    /* distance and closing_speed are Q15. The divide below can overflow
       int16 before the clamp ever runs when closing_speed is near zero. */
    int32_t ttc = ((int32_t)distance << 15) / closing_speed;  /* may saturate */
    int32_t demand = ((int32_t)GAIN_Q15 * (TTC_LIMIT_Q15 - ttc)) >> 15;
    if (demand < 0)      demand = 0;
    if (demand > 32767)  demand = 32767;       /* Q15 one */
    return (int16_t)demand;
}
/* SIL catches the quantization and overflow. It is still BLIND to how the real
   target processor schedules this, which is PIL's job one rung down. */

So SIL earns its place by catching the fixed-point defect. Yet it runs on your host processor, not the target. Because host timing has nothing to do with the electronic control unit's clock, SIL is blind to real target timing. Consequently a timing defect sails straight through to the next rung.

PIL: compiler, timing, and stack behaviour, blind to I/O electrical#

Processor-in-the-loop runs the compiled object code on the real target processor. Because the real compiler and real instruction set are now in play, it exposes compiler-optimization effects, word-length limits, and stack or interrupt behaviour. In particular, it is the first rung that can measure how long the code takes on the actual chip. Therefore PIL is where a target-timing overrun finally becomes visible.

Still, PIL runs the processor on a bench, not the wired-up ECU. Because there is no real sensor signal and no real actuator load, it is blind to I/O electrical behaviour. So a fault in the analogue front end or the network timing is passed on again, down to HIL.

HIL: I/O, fault injection, and ECU integration, blind to real sensor wear#

Hardware-in-the-loop couples the real ECU to a real-time simulated plant. Because the ECU now drives real I/O, real CAN and LIN traffic, and injected faults, it catches electrical interface defects, signal-timing faults, and integration errors. Moreover, HIL can inject a stuck sensor or a shorted line on demand, which is how you verify the safe-state response ISO 26262 requires. As a result, most interface and integration defects die here.

However, a simulated plant is not a worn road sensor in the rain. Because the plant model is still a model, HIL is blind to genuine sensor degradation and to rare real-world perception cases. So a narrow band of defects survives even a clean HIL campaign.

On-vehicle: perception edge cases and emergent behaviour, the last net#

On-vehicle testing is the last net, and it is validation, not verification. Because the whole system now meets the real world, it catches sensor-fusion edge cases and emergent behaviour no earlier rung could reproduce. Yet it is the most expensive place to first find a defect. Therefore the entire point of the rungs above is to leave as little as possible for this one to catch.

The defect-escape ledger#

Here is the table almost nobody publishes. Each row is a rung. The columns state the defect class it catches, the blind spot it passes downstream, the cost of catching that defect one rung later, and the ISO 26262 role it discharges. Read it as the whole argument on one screen.

Notice the two rightmost columns together. First the cost of an escape rises at every rung. Second the ISO 26262 role shifts from verification to validation only at the last step. Because the multipliers compound, a defect that slips from MIL to the field is not a little more expensive. Instead it is orders of magnitude worse. So the automotive software testing techniques are not busywork. Rather they buy down that escape, which the next section makes computable.

The cost of letting a defect escape#

A multiplier column is abstract until it is your defect. So pick the rung where the bug is caught and set a base fix cost. Then watch the model escalate that cost, rung by rung, down to field and recall. This is the instrument the whole post is named for.

Watch how fast the number climbs. Because each escape multiplies, catching a defect at MIL instead of the field can be roughly a thousandfold cheaper. The escalation range comes from published data, including the NASA study on error-cost escalation through the project life cycle and the NIST report on the economics of software testing. Therefore the left-shifted V-model is not a preference for an ASIL D programme. Instead it is the only economically sane way to build one. For the wider picture, our guide to what custom software development costs at this level of rigor puts these figures in budget terms.

Worked example: a 40 ms compiler-induced overrun#

Make the escape concrete with the 200 ms budget. Split it across three stages. First perception validates the obstacle in about 90 ms. Then the decision stage plans the brake in about 60 ms. Finally actuation commands the brake in about 50 ms. Added up, that is 200 ms, exactly at the limit.

Read the example as the ledger in miniature. Because the timing defect is invisible until PIL, skipping PIL does not save time. Instead it ships the overrun to HIL, the vehicle, or the field, where the same 40 ms costs far more to chase down. In short, the rung that catches a defect is set by physics, not by preference.

ASIL sets the rigor: coverage obligations by integrity level#

ISO 26262 does not ask every function for the same rigor. Instead the ASIL sets it. A higher ASIL demands stronger structural coverage and stronger independence. So the ASIL is what wires the automotive software testing techniques to a mandatory coverage target. The software-level obligations live in ISO 26262-6, product development at the software level. The table summarises how the obligation climbs from ASIL A to ASIL D.

Why MC/DC for ASIL D#

Branch coverage proves each branch went both ways. However, it cannot prove that each condition inside a compound decision actually mattered. Because safety logic is full of compound conditions, that gap hides real defects. Therefore ASIL D reaches for MC/DC.

Modified condition/decision coverage demonstrates that each condition independently changes the decision outcome. In short, it proves every input to a safety decision earns its place. So MC/DC is not gold-plating. Instead it is the coverage that matches how catastrophic an ASIL D failure would be.

Confirmation-measure independence#

Coverage is not only about code. It is also about who checks the work. Because a reviewer who reports to the author is compromised, ISO 26262-2 asks for organisational independence on the confirmation measures. At ASIL D that independence reaches the highest level, an independent organisation.

Read independence as a coverage obligation, not a courtesy. Furthermore, it is exactly the kind of control an assessor audits. So a programme that skips it has a gap no line coverage can close.

ASIL decomposition#

Sometimes ASIL D rigor on one monolith is impractical. Therefore ISO 26262-9 allows ASIL decomposition. Because a requirement can be met by two sufficiently independent elements, the ASIL D goal can split into, for example, two ASIL B elements plus a demonstrated independence argument.

However, decomposition is not a discount. Instead it moves rigor from one element to an independence claim you must prove. So the mandatory XiL rigor follows the decomposed ASIL of each element, only if the independence genuinely holds.

The promotion gate: how a real team decides SIL to HIL#

Read the gate as a promotion, not a schedule. Because each rung is costlier than the last, you promote a requirement only when the cheaper rung has nothing left to teach. Consequently a mature team treats coverage and back-to-back agreement as the entry ticket to the next rung, not as paperwork filed afterwards.

When NOT to reach for the full chain#

There is a deeper trap to name too. Sometimes a team runs every rung yet skips traceability, so a green result proves nothing about the requirement. Instead of more rungs, that team needs the trace links this post opened with. Because rigor without traceability is motion without coverage, the fix is discipline, not another test bench.

Where this fits in a larger build#

Safety-critical validation is one demanding corner of a much larger picture. The automotive software testing techniques above are a specialised branch of ordinary delivery discipline, not a separate world. It sits inside the different categories of custom software, where embedded and safety-critical systems are their own class with their own rules. Around it runs a full enterprise software development lifecycle, which carries the same requirements-to-test discipline into every domain. Below it sits the cost question, because what custom software development costs at this level of rigor is driven precisely by the rung depth this post describes.

So treat the ledger as a planning tool. First decide the ASIL from your hazard analysis. Then choose how far down the chain each requirement must travel. Finally cost the escape you are trying to prevent. If you want a partner who works this way, explore Atyantik's software development services. Measure the requirement first, then test only as deep as the safety goal demands.