30-second answer
Three codes, and the distinction between 1 and 2 is the one that matters in
a pipeline.
| Code | Meaning |
|---|---|
0 | The command did what it was asked |
1 | The command ran, and the result was negative |
2 | The engine refused to run, or refused to answer |
Why 1 and 2 are not the same failure
1 is a result. Rules failed under --fail-on. An assessment did not
verify. doctor found something wrong. A collection reached the tenant and
failed. In every case the engine did its job and the answer was negative.
2 is a refusal. An argument was missing, a document was not what the
command expects, or producing an answer would have meant claiming something the
evidence does not support. Nothing was decided.
A pipeline that treats them the same will page somebody about a governance regression when the real problem is a typo in a path. Worse, it will treat a refusal as a clean run when the exit code is inverted somewhere.
Per command
| Command | 1 means | 2 means |
|---|---|---|
collect | The collection ran and failed. Nothing was written to the tenant | A missing argument, or a preflight problem locally |
evaluate | --fail-on was met | Evidence did not match the schema |
assess | Not used | No evidence documents, bad evidence, or a manifest that could not be assembled honestly |
verify | The assessment does not verify | The file is not an assessment |
diff | A rule left pass, with --fail-on-regression | A document is not an assessment |
validate | Rules have problems | Not used |
doctor | The installation has problems | Not used |
show-rule | Not used | No rule with that identifier |
report | Not used | The document cannot be rendered |
What to set in CI
evaluate --fail-on unresolved is the strict setting: it counts unknown,
invalid-evidence and error alongside fail, so a build cannot go green on
evidence the engine could not read.
--fail-on fail counts only outright failures, and leaves unknown for a
human to chase.
Neither of them makes 2 less serious. A 2 in a pipeline means the run did
not happen.