Mutation Testing

Part 6 · Coverage · Principal (P)

On this page

Throughout this course, every DCRouter testbench has been judged the same way: switch on each of 24 planted bugs in turn and count how many the testbench reports. That number has been more informative than any coverage figure. The directed suite at 10 of 24, the random hunt at 15, the final testbench at 24: each step was measured by bugs caught, not by bins hit. That method has a name, and it can be automated.

How do you measure the quality of a testbench itself?

The obvious answer, and why it breaks

The obvious measures are coverage and pass rate. Coverage shows what was exercised, not what was checked (lesson 6.1); a pass rate shows nothing about the checks at all. The direct question is: if the design had a bug, would the testbench notice? The only way to answer it is to give the design bugs and watch.

The idea: plant bugs, count kills

Term	Meaning
Mutant	A copy of the design with one small change: an operator swapped, a constant moved by one, a condition inverted
Killed	At least one test fails on the mutant
Survived	Every test passes on the mutant: either a missing test or check, or an equivalent mutant
Equivalent mutant	A change that does not alter behaviour, so no test can kill it
Mutation score	Killed mutants divided by all (non-equivalent) mutants

Each survivor is specific: it names a behaviour the testbench would not notice if it were wrong. That makes mutation results more actionable than a coverage percentage.

Build it: a mutation tester

Three DCRouter rules written as Python functions (the error counter, the credit counter and the send condition) are mutated one change at a time: every operator, constant and not goes through a numbered site that a mutant can flip. Each mutant runs against a weak test suite and a strong one.

example.py

"""Mutation testing: measuring a test suite by the bugs it kills.

The "design" is three DCRouter rules written as Python functions: the error counter (R9:
saturate at 255, a clear wins), the credit counter (R7: a send and a return in the same cycle)
and the send condition (R7: only with a lock, a flit and a credit). Every operator, constant
and `not` in the design goes through a numbered site, so a mutant can change exactly one of
them: swap an operator (+ and -, == and !=, and/or), move a constant by one, or drop a `not`.
Each mutant is run against two test suites:
  weak    the cases a quick reading of the rules suggests
  strong  the same plus the boundaries and simultaneous events
A mutant is killed if any test fails. The mutation score is killed / total.
"""
SWAP = {"+": "-", "-": "+", "==": "!=", "!=": "==", "and": "or", "or": "and"}
APPLY = {"+": lambda a, b: a + b, "-": lambda a, b: a - b, "==": lambda a, b: a == b,
         "!=": lambda a, b: a != b, "and": lambda a, b: a and b, "or": lambda a, b: a or b}

def build(mutant=None):
    """The design, with one site changed if mutant = (site, change) is given."""
    def op(site, sym, a, b):
        return APPLY[SWAP[sym] if mutant == (site, "swap") else sym](a, b)
    def k(site, value):
        return value + {"+1": 1, "-1": -1}.get(mutant[1], 0) if mutant and mutant[0] == site else value
    def neg(site, x):
        return x if mutant == (site, "drop") else not x

    def err_counter(count, tail_err, clear):                       # R9
        if clear:
            return k("E1", 0)
        if tail_err:
            return count if op("E2", "==", count, k("E3", 255)) else op("E4", "+", count, k("E5", 1))
        return count

    def credits_next(credits, send, ret):                          # R7
        return op("C1", "-", op("C2", "+", credits, k("C3", 1) if ret else 0), k("C4", 1) if send else 0)

    def may_send(credits, locked, empty):                          # R7
        return op("S1", "and", op("S2", "and", locked, neg("S3", empty)), op("S4", "!=", credits, k("S5", 0)))

    return {"err_counter": err_counter, "credits_next": credits_next, "may_send": may_send}

SITES = {"E1": "k", "E2": "op", "E3": "k", "E4": "op", "E5": "k", "C1": "op", "C2": "op", "C3": "k",
         "C4": "k", "S1": "op", "S2": "op", "S3": "not", "S4": "op", "S5": "k"}
MUTANTS = [(s, c) for s, kind in SITES.items()
           for c in {"op": ["swap"], "k": ["+1", "-1"], "not": ["drop"]}[kind]]

WEAK = [
    ("err_counter", (0, True, False), 1), ("err_counter", (5, False, False), 5),
    ("err_counter", (7, False, True), 0), ("credits_next", (4, True, False), 3),
    ("credits_next", (2, False, True), 3), ("may_send", (4, True, False), True),
]
STRONG = WEAK + [
    ("err_counter", (254, True, False), 255), ("err_counter", (255, True, False), 255),
    ("err_counter", (9, True, True), 0), ("credits_next", (3, True, True), 3),
    ("credits_next", (0, False, False), 0), ("may_send", (0, True, False), False),
    ("may_send", (1, True, False), True), ("may_send", (4, False, False), False),
    ("may_send", (4, True, True), False),
]

def killed(design, suite):
    return any(design[fn](*args) != want for fn, args, want in suite)

assert not killed(build(), STRONG), "the design itself must pass every test"
weak_k = [killed(build(m), WEAK) for m in MUTANTS]
strong_k = [killed(build(m), STRONG) for m in MUTANTS]
n = len(MUTANTS)
print(f"mutants: {n}")
print(f"weak suite   ({len(WEAK):2d} tests): {sum(weak_k):2d} killed, score {100 * sum(weak_k) / n:.0f}%")
print(f"strong suite ({len(STRONG):2d} tests): {sum(strong_k):2d} killed, score {100 * sum(strong_k) / n:.0f}%")
print("survivors of the weak suite that the strong suite kills:")
for m, w, s in zip(MUTANTS, weak_k, strong_k):
    if not w and s:
        print(f"  site {m[0]} {m[1]}")
print("survivors of both:")
for m, s in zip(MUTANTS, strong_k):
    if not s:
        print(f"  site {m[0]} {m[1]}")
if all(strong_k):
    print("  none")

assert sum(strong_k) > sum(weak_k) and all(strong_k)
print(f"PASS: the boundary and simultaneous-event tests raise the mutation score from {100 * sum(weak_k) / n:.0f}% to 100%; each weak-suite survivor names a missing test")

It prints:

Diagram

mutants: 20
weak suite   ( 6 tests): 14 killed, score 70%
strong suite (15 tests): 20 killed, score 100%
survivors of the weak suite that the strong suite kills:
  site E3 +1
  site E3 -1
  site S1 swap
  site S2 swap
  site S5 +1
  site S5 -1
survivors of both:
  none
PASS: the boundary and simultaneous-event tests raise the mutation score from 70% to 100%; each weak-suite survivor names a missing test

How to read it: 20 mutants. The weak suite, six obvious tests, kills 14 (70%). Its six survivors are precise: moving the saturation constant 255 up or down (site E3) survives because no test reaches 254 or 255 (the rule BUG_15 breaks); turning either and of the send condition into or (S1, S2), or moving its zero (S5), survives because no test tries to send with zero credits, a flit missing or the output unlocked (the rules BUG_9 and BUG_10 touch). The strong suite adds exactly those boundary and simultaneous-event tests and kills all 20.

The Bug Zoo as a mutation campaign

	Automatic mutation	DCRouter's Bug Zoo
Bugs	Many small syntactic changes	24 hand-written, realistic bugs, each in one class from Part 1
Score	Killed / total	Caught / 24
Strength	Unbiased, exhaustive over its operators	Realistic: concurrency, configuration and low-power bugs no operator swap produces
Weakness	Equivalent mutants; many trivial bugs; costly for long simulations	Only the bugs someone thought of

Both answer the same question. Automatic mutation is cheap per bug and finds missing boundary tests; seeded realistic bugs test whether whole strategies (stimulus, checkers, coverage) catch whole classes of bug, which is how this course has used them.

Using mutation testing in practice

Start where checks are weakest: control logic, counters, error paths.
Triage survivors: missing stimulus, missing check, or equivalent mutant.
Budget simulation: each mutant is a regression run; sample mutants or use fast tests.
Report the score with coverage at sign-off: one shows what was exercised, the other what would be noticed (Part 14).

How real teams use it

Practice	Purpose
Mutation or fault-injection tools on RTL	Automatic mutants of operators, constants and connections
Bug seeding by experienced engineers	Realistic bugs that test strategies, not just boundaries
Survivor reviews	Each surviving mutant becomes a test, a check or a documented equivalence

What breaks

A team (illustrative) had 98% functional coverage and ran a mutation campaign before sign-off. 30% of the mutants in the interrupt controller survived: the scoreboard had never compared interrupt timing, only that interrupts occurred. Coverage had shown every interrupt source exercised; mutation showed that nobody checked when they fired.

Check yourself Why does the mutant that moves 255 to 254 (site E3 -1) survive the weak suite?

No weak test reaches the saturation boundary, so a counter that saturates one early behaves identically on every weak test.

Check yourself What are the two possible explanations for a mutant that survives every test?

A missing test or check (the behaviour is never exercised or never compared), or an equivalent mutant (the change does not alter behaviour).

Key takeaways

Mutation testing measures what the testbench would notice, not what it exercised.
Each surviving mutant names a missing test or check, or is equivalent.
Seeded realistic bugs, like the Bug Zoo, measure whole verification strategies the same way.

Navigate

The obvious answer, and why it breaks

The idea: plant bugs, count kills

Build it: a mutation tester

The Bug Zoo as a mutation campaign

Using mutation testing in practice

How real teams use it

What breaks

Key takeaways

Related topics

Mutation Testing

The obvious answer, and why it breaks

The idea: plant bugs, count kills

Build it: a mutation tester

The Bug Zoo as a mutation campaign

Using mutation testing in practice

How real teams use it

What breaks

Key takeaways

Related topics

The obvious answer, and why it breaks

The idea: plant bugs, count kills

Build it: a mutation tester

The Bug Zoo as a mutation campaign

Using mutation testing in practice

How real teams use it

What breaks

Key takeaways

Related topics

Up next

The obvious answer, and why it breaks

The idea: plant bugs, count kills

Build it: a mutation tester

The Bug Zoo as a mutation campaign

Using mutation testing in practice

How real teams use it

What breaks

Key takeaways

Related topics

Up next