A green check proves little unless the intended verifier actually executed, inspected the intended subject, and can fail on a known-bad case. False green is the condition where automation reports success without evaluating the required boundary. The cure is not another success screenshot; it is evidence that the checker rejects a controlled defect and accepts a valid neighbor.
How can a real command pass without running its check?
One Northsun (formerly Mnemix) repository gate exposed a useful failure pattern. An ESM script used a main-module guard to decide whether to call its main() function. The comparison behaved differently when the repository was reached through a symlink: Node resolved one side to the real path while the process argument retained the symlinked path.
The paths did not match. main() never ran. The module reached the end and exited with status 0, which the surrounding gate interpreted as green.
The checker existed. The command existed. CI could display success. The intended verification still did not happen.
This is an engineering lesson from Northsun's own build system, not a claim that the public Northsun product validates customer actions today.
What are the common shapes of false green?
The symlink case is one member of a broader family:
- an entrypoint guard skips the real program;
- a file glob matches nothing and the loop performs zero assertions;
- a grep pipeline treats “no input” as “no violations”;
- a test fixture never reaches the branch named by the test;
- a mock proves its own assumptions instead of exercising the production boundary;
- a report excludes failures or empty results from its denominator;
- a deploy check inspects a local artifact while the release uses another one.
All of them confuse absence of observed failure with evidence of correctness.
Why is exit 0 insufficient evidence?
An exit code describes how a process ended. It does not, by itself, establish what the process examined.
For a meaningful gate, you need a chain of evidence:
- Entrypoint evidence: the intended verifier started.
- Subject evidence: it inspected the exact artifact, route, configuration, or deployment under review.
- Assertion evidence: at least one relevant assertion executed.
- Negative-control evidence: a known-bad fixture produced failure.
- Positive-neighbor evidence: the corresponding valid fixture produced success.
- Identity evidence: the result belongs to the current commit, build, or deployed version.
Without that chain, a green badge is presentation rather than proof.
How do you prove that a check can say no?
Build the failure observation into the gate's acceptance criteria.
For example, if a docs gate claims to reject a forbidden public route:
- Create a controlled fixture containing the forbidden route.
- Run the same entrypoint CI invokes.
- Assert a non-zero exit and a diagnostic naming the fixture.
- Remove only the forbidden value.
- Run the same command again and assert success.
- Repeat through any path or execution mode that previously changed behavior, such as a symlinked worktree.
The important property is adjacency: the red and green fixtures should differ only in the behavior the gate claims to evaluate.
What should a trustworthy measurement include?
Agent-memory and retrieval evaluations have their own false-green traps. A metric can look healthy while excluding empty packets, failed requests, or cases where irrelevant context displaced the answer.
Prefer measurements that preserve total failures and test counter-hypotheses:
- count empty and degraded packets rather than dropping them;
- add irrelevant material and verify that quality does not improve merely because the packet is larger;
- include contradictory and stale candidates;
- inspect precision as well as recall;
- distinguish retrieval success from correct downstream use;
- bind results to the exact corpus, configuration, and code revision.
A measurement repair is not automatically a product-quality improvement. It may simply reveal the true baseline for the first time.
How does this apply to governed agent actions?
“Every agent proposes. Northsun governs.” is Northsun's public differentiator and product thesis. A future validation boundary cannot graduate merely because allowed examples pass. It needs evidence that prohibited examples are rejected and that rejected high-impact actions do not reach the external effect.
The current Validation gate page distinguishes a Stage-1 evaluator that exists in source from the complete non-public gate design. No validation route belongs to the frozen public REST contract. The red/green discipline is an acceptance requirement for any graduated customer surface, not evidence that one is publicly callable today.
A pre-merge false-green review
Before trusting a gate, ask:
- Can I name the exact function or assertion that executed?
- What known-bad input did it reject in this run?
- What valid neighboring input did it accept?
- Does the command behave the same from a real path and a symlinked path?
- Can an empty file set, empty loop, or missing artifact pass silently?
- Is the report bound to the current commit and actual release artifact?
- Does the checker depend on the same assumption it is supposed to verify?
If the answers are unclear, the next task is to test the instrument—not to publish the green result.
What should you read next?
Read Validation gate for the design boundary and its red/green graduation criteria. Use Production readiness to separate code presence, deployment evidence, and product proof before making a public claim.