---
title: A Check That Can't Fail Is Worse Than None
description: A canary I added passed the exact case it existed to catch, because of one line of shell.
image: https://willcurran.com/hubfs/will-curran-brand/a-check-that-cant-fail-featured.png
---

[Skip to content](https://willcurran.com/blog/a-check-that-cant-fail#main)

[Will Curran](https://willcurran.com/)

[Home](https://willcurran.com/) [Portfolio](https://willcurran.com/portfolio) [Press](https://willcurran.com/media) [Blog](https://willcurran.com/blog) [Favorites](https://willcurran.com/favorites) [Speaking](https://willcurran.com/speaking) [Consulting](https://willcurran.com/consult) 

[Hire Will](https://willcurran.com/hire)

[Writing](https://willcurran.com/blog) Builds — A Check That Can't Fail Is Worse Than None [Subscribe](https://willcurran.com/blog/subscribe)

Builds

# A Check That Can't Fail Is Worse Than None

A canary built to catch one bad state passed exactly that state, because grep -c prints 0 and also exits non-zero. The post ends with copyable self-tests that run a known-bad and a known-good input through a check in the same run.

![](https://willcurran.com/hs-fs/hubfs/will-curran-brand/will-curran-avatar.jpg?width=44&height=44&name=will-curran-avatar.jpg)

Will Curran Sep 25, 2026 · 6 min read

<https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwillcurran.com%2Fblog%2Fa-check-that-cant-fail>

[Part of: Checks that lie](https://willcurran.com/blog/series/checks-that-lie) Part 2 of 7 [Next: A Healthy Container and a Blind DJ App](https://willcurran.com/blog/healthy-but-blind)

On this page

![An abstract dark graphic: two tall cards side by side, one with a solid grey check mark above a circle, the other outlined in orange with a dotted orange check mark above a red X, next to the title A check that can't fail is worse than none.](https://willcurran.com/hs-fs/hubfs/will-curran-brand/a-check-that-cant-fail-featured.png?width=1600&height=900&name=a-check-that-cant-fail-featured.png)

In August I added a canary whose only job was to notice one bad state and complain about it. As first written, it passed exactly that state. The bug was one line of shell, and it got caught for one reason: before trusting the canary, I fed it the bad state.

That is the rule this post is about. A check hasn't earned your trust until you have watched it refuse something.

## One line of shell that always says ok

Here is the shape, rebuilt as a toy. The check counts a line that has to be in a file, and raises an alarm when the count is zero.

```
n=$(grep -c 'required-line' "$file" || echo 0)
if [ "$n" -eq 0 ]; then echo "ALARM: required line missing"; exit 1; fi
echo "ok"
```

The `|| echo 0` looks defensive, in case grep fails. But when `grep -c` finds nothing, it prints `0` and also exits with status 1. So on exactly the file the check exists to catch, both halves run and `n` holds two lines: a 0, a newline, and another 0. The numeric test can't parse that. Bash prints "integer expression expected" to stderr, the test comes back false, and the alarm never runs.

On a file that has the line, the check says ok. On a file that is missing it, the check also says ok. A canary that answers the same way in both cases tells you nothing, while printing a reassuring line on every run.

## Quiet failures point the reassuring way

A check can break in two directions. When it breaks loudly, it crashes, exits non-zero or raises a false alarm, and that costs you an afternoon. When it breaks quietly, it returns what a healthy run returns: zero findings, a green tick, silence. You then read that output as evidence of health.

The quiet kind is worse than having no check at all. With no check, you know you don't know. With a check that can't fail, you have a green line, and nothing will ever file a bug report, because from the outside things have never looked better.

This class of bug produces no symptoms, so waiting won't find it. You find it by testing the checkers on a day when nothing is wrong.

![A two-by-two grid of the four outcomes of any check. Columns: the thing is fine, the thing is broken. Rows: check passes, check fails. Passes and fine: correct. Fails and broken: correct, the reason the check exists. Fails and fine: loud failure, a false alarm that costs an afternoon. Passes and broken, highlighted: quiet failure, green, read as evidence of health. A note says a self-test puts a known-bad input in the broken column and a known-good one in the fine column, and if the check passes both, it can't fail.](https://willcurran.com/hs-fs/hubfs/will-curran-brand/a-check-that-cant-fail-four-outcomes.png?width=1440&height=1954&name=a-check-that-cant-fail-four-outcomes.png)

## Testing the test is its own skill

Don't accept a check on the strength of a green run. Build the bad input, watch the check refuse it, and if you can't make it fail on demand, call the check unknown.

Two separate attempts to test the same guard both went wrong. The guard was supposed to refuse to run when its config was missing.

- **The override nobody read.** The first attempt set an override variable that pointed the config at an empty file. The script never looked at that variable, because its config path was hardcoded. Nothing was removed, the run succeeded, and the guard looked absent. That result was correctly reported as inconclusive rather than as a defect.
- **The edited copy.** The second attempt edited a copy of the script to point at the empty file and tested the copy. That proved the logic. It said nothing about the file that actually runs, because an edited script is a different program.

Both attempts skipped the same question: can this test produce the failure at all? The way out was to give the real script a documented test seam, an override for the config path that exists for exactly this, and then watch the guard refuse twice by independent routes: once through the override, and once in an environment with no config at all.

## A silent result needs a loud control

The same idea one level up is the positive control. If you are checking that something gets rejected, also run something that should be accepted, in the same run, and assert that it was. If both come back silent, you haven't shown the check is strict. A broken harness returns silence for both.

## Shapes that fail green

- **Empty inputs.** A check that loops over a list of things it expects passes forever when the list is empty. Empty config should stop the check from running.
- **Skips.** pytest counts a skipped test toward a green run. In a toy suite I built for this post, a test whose whole body is `assert False` skipped on a missing file, and the run exited 0 with "1 passed, 1 skipped".
- **Zero versus could not look.** A count that turns an error into 0 reads like good news. Report a data error and exit non-zero.
- **Counters that can't move.** A counter that cannot increment is not a counter that read zero.

## What a self-test doesn't buy you

A self-test proves the check can refuse the bad inputs you thought of. It says nothing about the ones you didn't.

## If you are about to try this

- **Pair every negative with a positive.** Run a known-good case in the same run as the known-bad one.
- **Test the file that runs.** If testing it means editing it, add a documented seam instead.
- **Give "could not look" its own outcome,** separate from "found nothing".

## Build it yourself

First, the toy check with the bug fixed. It keeps grep's exit code, so "could not read the file" can't turn into a count.

```
n=$(grep -c 'required-line' "$1"); rc=$?
if [ "$rc" -gt 1 ]; then echo "ERROR: could not read $1"; exit 2; fi
if [ "$n" -eq 0 ]; then echo "ALARM: required line missing"; exit 1; fi
echo "ok"
```

Then a self-test you can wrap around any check that takes a file. It runs a known-good file, a known-bad file and an unreadable path in the same run, and fails if the check gets any of them wrong. Against the buggy version above it failed with "passed bad". Against the fixed one it passed.

```
check=$1
good=$(mktemp); bad=$(mktemp)
printf 'required-line\n' > "$good"; printf 'something else\n' > "$bad"
"$check" "$good" >/dev/null || { echo "FAIL: rejected good"; exit 2; }
"$check" "$bad"  >/dev/null && { echo "FAIL: passed bad"; exit 2; }
"$check" /nonexistent >/dev/null 2>&1 && { echo "FAIL: passed unreadable"; exit 2; }
rm -f "$good" "$bad"; echo "self-test ok"
```

For pytest, a few lines in `conftest.py` turn an undeclared skip into a failed run. In the toy suite, an undeclared skip exited 1, and the same skip exited 0 once declared.

```
ALLOWED_SKIPS = set()  # declared reasons, e.g. {"needs GPU"}

def pytest_sessionfinish(session, exitstatus):
    rep = session.config.pluginmanager.get_plugin("terminalreporter")
    skipped = rep.stats.get("skipped", []) if rep else []
    reasons = [r.longrepr[2].removeprefix("Skipped: ") for r in skipped]
    if any(x not in ALLOWED_SKIPS for x in reasons):
        session.exitstatus = 1
```

Run the self-tests in CI or at startup, so a check that loses its ability to refuse stops the build instead of passing quietly.

Give this to your AIWrite a self-test for each check, monitor, canary or test you write for me. The self-test feeds the check a known-bad input and fails if the check doesn't refuse it. In the same run, it feeds a known-good input and requires a pass, so a broken harness can't make both cases look the same. Add a third case where the check can't read its input, and require a distinct error, never a zero and never a pass. Treat empty config, empty lists and skipped tests as failures unless I have declared them.

If you have found a check of your own that could never fail, I would like to hear how you caught it.

Filed under [Builds](https://willcurran.com/blog/tag/builds)

![](https://willcurran.com/hs-fs/hubfs/will-curran-brand/will-curran-avatar.jpg?width=64&height=64&name=will-curran-avatar.jpg)

Will Curran

I spent sixteen years building one company, and I have been in the event industry since 2007. These days I write about the technology I actually use — trucks, radios, home infrastructure, and the tools that survive past the first invoice.

[All posts by Will →](https://willcurran.com/blog/author/will-curran)

Get the next one by email

One email per post. No digest, no sponsors.

[← Previous Trivia Over LoRa: a game show on a slow, lossy radio mesh](https://willcurran.com/blog/trivia-over-lora) [Next → Party Mode: 49 Ramps Blocked, Zero During a Six-Hour Party](https://willcurran.com/blog/party-mode)

## Related reading

[Builds Trivia Over LoRa: a game show on a slow, lossy radio mesh 8 min read](https://willcurran.com/blog/trivia-over-lora) [Builds "Precise" Is Less Precise 3 min read](https://willcurran.com/blog/precise-is-less-precise) [Builds Terrain Beats Floors 7 min read](https://willcurran.com/blog/terrain-beats-floors)

[Will Curran](https://willcurran.com/)

[LinkedIn — 29,339+ followers](https://www.linkedin.com/in/willcurran) [GitHub](https://github.com/rancur) [YouTube](https://www.youtube.com/channel/UCJgscc27l4fpkWx1SeK05xg/) [Spotify](https://open.spotify.com/user/rancur) [SoundCloud DJ mixes](https://www.soundcloud.com/thewillsee) [Twitch](https://www.twitch.tv/thewillsee) [Last.fm](https://www.last.fm/user/default) [Trakt — movies & TV](https://trakt.tv/users/willcurran) [Goodreads](https://www.goodreads.com/user/show/55302838-will-curran) [Will See Music](https://www.thewillsee.com) [Arizona Meshtastic](https://azmsh.net) [Electric Forest Meshtastic](https://efmesh.com)

© 2026 Will Curran — All rights reserved. [RSS](https://willcurran.com/blog/rss.xml) [Back to top](https://willcurran.com/blog/a-check-that-cant-fail#main)

```json
{
  "@context" : "https://schema.org",
  "@type" : "BlogPosting",
  "author" : {
    "@type" : "Person",
    "name" : "Will Curran",
    "url" : "https://willcurran.com/blog/author/will-curran"
  },
  "dateModified" : "2026-10-22T23:26:42.271Z",
  "datePublished" : "2026-09-25T16:00:00.000Z",
  "headline" : "A Check That Can't Fail Is Worse Than None",
  "image" : [ "https://willcurran.com/hubfs/will-curran-brand/a-check-that-cant-fail-featured.png" ],
  "mainEntityOfPage" : {
    "@id" : "https://willcurran.com/blog/a-check-that-cant-fail",
    "@type" : "WebPage"
  },
  "publisher" : {
    "@type" : "Organization",
    "logo" : {
      "@type" : "ImageObject",
      "url" : "https://willcurran.com/hubfs/image-1.png"
    },
    "name" : "Will Curran LLC"
  }
}
```