Your testbench reports errors on every counting edge. You run the build, expecting red, and the screen stays green. The bug is still in the design, the errors are still in the log, and yet the build passed.
How does a failing test make the whole build fail?
The answer is a chain of four verdicts, and your green build has a gap in the middle of it. This lesson walks the chain link by link, finds the gap, and shows the one gate that closes it.
When a command finishes, it reports an exit status: a small number that says how the run went. Zero means success. Any nonzero value means failure, and different values can name different failures. This convention is shared by the simulator, the shell, make, and the automation that runs your checks on every change.
Make treats a nonzero status as an order to stop. When any recipe line fails, make runs nothing further and exits nonzero itself. That stopping behavior is what turns one failing step into a failed build, and it is also what a later automation layer reads: nonzero means red, zero means green.
A simulation result reaches you through four stages: the simulator runs the test, a shell pipe carries its output, make reads the recipe's status, and the automation layer reads make's status. Each stage reports only what it saw, and each one can hide what the stage before it knew.
Look at the picture below first, then read the two boxes under it. The left box is where a failure can disappear: by default a pipe reports the status of its last command. The middle box is a second check: a gate that counts verdict lines in the log.
Here is the project's build file again. Read the three lines of the sim target slowly, because the second and third lines together are the entire verdict mechanism of this course.
SHELL := /bin/bash
IVERILOG ?= iverilog
VVP ?= vvp
YOSYS ?= yosys
VERILATOR ?= verilator
RTL := rtl/mini_controller.sv
TB := tb/tb_mini_controller.sv
BUILD := build
.PHONY: help sim lint synth all clean check-tools
.DEFAULT_GOAL := all
help:
@echo "Targets: sim lint synth all check-tools clean"
all: lint sim synth
$(BUILD):
mkdir -p $(BUILD)
sim: | $(BUILD)
$(IVERILOG) -g2012 -Wall -s tb_mini_controller -o $(BUILD)/sim.vvp $(RTL) $(TB)
set -o pipefail; cd $(BUILD) && $(VVP) sim.vvp | tee sim.log
@grep -c "^PASS:" $(BUILD)/sim.log | grep -qx 1
lint:
$(VERILATOR) --lint-only -Wall --top-module mini_controller $(RTL)
synth: | $(BUILD)
$(YOSYS) -Q -T -p 'read_verilog -sv $(RTL); hierarchy -check -top mini_controller; proc; opt; check -assert; stat' > $(BUILD)/synth.log
check-tools:
@bash scripts/check_tools.sh
clean:
rm -rf $(BUILD)This is the same build file as the previous lesson: the sim target compiles, runs through a pipe with pipefail switched on, and then counts verdict lines.
The run line changes directory into build/, plays the compiled simulation, and pipes a copy of everything into sim.log through tee. By default a pipe reports the status of its last command, which is tee, and tee succeeds whenever it writes, even when the simulation it copied had already failed. The set -o pipefail at the start of the line (and SHELL := /bin/bash at the top, because that option is a bash feature) makes the pipe report the first failure instead, so the simulator's own status reaches make. The next line is a second, separate check: it counts lines starting with PASS: in the log and demands exactly one, so a run that exits cleanly but shows no verdict line, or two, still fails the build. The simulator's status and the verdict line are both required.
Run the simulation target on the good design and ask for its exit status, the number the shell kept:
# captured terminal output
$ make sim > /dev/null 2>&1; echo "make sim exit status: $?"
make sim exit status: 0This output shows the good design exiting with status 0, measured with GNU Make 3.81 on macOS: the testbench showed one verdict line, the gate counted exactly one, and every stage of the chain agreed.
tee, which succeeds, but pipefail is on. What exit status will make sim report?Status 2: failure. With pipefail the pipe reports the simulator's failure, so make stops the recipe and exits 2. The verdict gate would also fail, since no line starts with PASS:. Two independent checks agree.
Swap the mutant design in, so each event adds two instead of one, and run the same command. The testbench reports mismatches on the counting edges and shows no verdict line:
# captured terminal output
$ make sim > /dev/null 2>&1; echo "make sim exit status with a counts_by_two bug: $?"
make sim exit status with a counts_by_two bug: 2This output shows the broken design exiting with status 2, measured with the same GNU Make 3.81 on macOS: the build stopped because the simulator failed, and the verdict gate would have failed too. Run on its own without the pipe, the simulator exits 1 when the testbench ends in a fatal report, measured on macOS with Icarus 13.0. Two different numbers, two different reporters: 1 is the tool saying its run failed, 2 is make saying a recipe step failed.
Restore the original design file and run the command once more to confirm green before continuing.
The gate only works because the testbench keeps a strict contract with it. Show exactly one line starting with PASS: when every check succeeds. Never show that line when any check fails. Fail loudly otherwise: end the run with a fatal report so the log explains itself and the simulator exits nonzero on its own, which pipefail lets make see.
Each half of the contract guards against a different quiet failure. The exact count guards against a run that stopped early and a run that somehow reported twice. The fatal ending guards against a testbench that reaches its last line having checked nothing at all.
Trusting the status of a pipe. Without pipefail, a tool piped through tee or grep reports the last command's status, not the tool's. Switch pipefail on, and also check what the tool wrote.
Showing the verdict before checking. A testbench that logs its verdict line at the top and checks afterwards passes the count on a broken design. Show the line only after the last check has succeeded.
Ending with a plain finish and no verdict. A testbench that stops without reporting anything leaves no line to count and no failure to hear. Every testbench ends with either its one verdict line or a loud fatal report, never silence.
Rule: a passing build counts exactly one verdict line; anything else is red.
make sim fail on the counts_by_two bug even though tee succeeded?Because pipefail makes the pipe report the simulator's failure instead of tee's success. Even without it, the next recipe line counts lines starting with PASS: and finds none, since the failing testbench never shows its verdict line. Either failure makes make stop the recipe and exit 2.
The build would stay green on a broken design. The gate counts verdict lines, and it would find exactly one, even though every counting check failed below it. The contract demands the line only after the last check succeeds, precisely so the count means something.
Because two different layers are speaking. The simulator's 1 says its own run failed, measured on macOS with Icarus 13.0 when the testbench ends in a fatal report. Make's 2 says a recipe step failed and the build stopped. Read each number as a statement about its own layer, not as a code for the bug.
Every command ends with an exit status, make stops at the first nonzero one, and automation reads nonzero as red.
A pipe reports its last command's status unless pipefail is on; the build checks both the simulator's status and one counted verdict line.
A testbench shows exactly one PASS verdict line on success, never on failure, and ends loudly otherwise.