Wrong-object errors in research pipelines

A computation can execute exactly as specified on the wrong object, such as the wrong document, security or probability. Such defects raise no error, pass every test written for the computation, and return well-formed, plausible output. Our defect register recorded seven over eighteen days; in the case examined here, the only check comparing across the two questions at issue was scoped on the field the defect had corrupted, so it never ran.

Structure of the defect

A pipeline computation selects an object, computes on it, and satisfies a contract describing the output, and verification concentrates almost entirely on the computation. A unit test fixes an input the author chose, so it says nothing about the selection the running system performs. Type and range checks pass because a wrongly selected object is usually of the right type and in range, and the output contract holds because a correct computation on a wrong input produces a well-formed answer.

In every recorded instance the code did what it was written to do, and the number was of the right magnitude and units. Because the answers resemble what the reader expects, the failure survives review by a person as readily as review by a test.

Software testing calls the general obstacle the oracle problem, surveyed by Barr and colleagues in 2015: many systems have no automatic means of telling that an output is correct, so testing reduces to confirming what the programmer already believed. Metamorphic testing, proposed by Chen, Cheung and Yiu in 1998, is the standard response and asserts relations between outputs rather than checking any output against a known truth. The class described here is the oracle problem as quantitative pipelines produce it, and the useful assertions likewise relate independent parts of a system.

Seven recorded instances

Seven entries were recorded between the twelfth and the twenty-ninth of one month. They are described by mechanism rather than by episode.

• A conversion ratio was evaluated against an unrelated listed security whose name overlapped with the correct one, producing a per-share figure of ordinary magnitude.
• The newest filing in a family was taken in preference to the one governing the question, although in that family the most recent document is routinely the least informative about terms, so the selection rule was close to anti-correlated with relevance.
• The repair for that selection preferred the informative form, then reached back to a document of that form describing an unrelated transaction from three years earlier and returned a clean answer about the current one.
• An ancillary fee exhibit of roughly seventeen thousand bytes and a rendering stub of a few thousand characters were read in place of primary documents of roughly eight hundred thousand to 1.3 million characters, and the absence of relevant language in either was recorded as a clean read rather than a failure to read, in the field that exists to flag the difference.
• A sentence describing corporate ownership was parsed as one describing transaction role, so the published statement had the consideration flowing opposite to the direction the document’s own heading states.
• A resource guard governed on a memory figure that includes cached file data the system can discard, as though it measured memory pressure. It refused work at a reported utilisation more than three times the true working set, and because a refusal exited successfully, the scheduled jobs it governed reported success without running; over roughly three days forty-one jobs logged 482 such skips, and one daily data capture went dark for eleven days before the gap was noticed.
• A position was sized on a probability that answers a different question from the one the position takes; the rest of this note examines that instance.

Two probabilities on one artifact

One artifact published two probabilities, one for whether a transaction completes and one for whether a bid arrives at all. The position was sized from the first while the headline carried the second, 84.55 percentage points apart, and nothing on the artifact declared which question the position took. Run against the archived defective artifact and against its repair, the coherence checker returned the identical 55 violations on both. Its source already contained the comparison that would have caught the defect.

Every other pairing stayed within one question, and the single comparison across the two probability families was conditioned on the artifact’s type label, which is the field this defect corrupts, since the wrong label is what puts a bid-arrival probability on an artifact whose governing document says the bid already arrived.

An invariant scoped on a field the defect corrupts cannot see the corruption.

The structures holding the mismatched probabilities were also on the checker’s skip lists for prose scanning, and three further fields carrying the same numbers were read by no check.

Three rules

1. An invariant must not be scoped on a field the defect it hunts can corrupt. It should be scoped on evidence independent of the thing checked, such as the governing document rather than a type label, and should fail closed, treating a missing declaration as a failed one.
2. Comparing a value against another copy of itself is not a check, since agreement between two renderings of one number shows only that the renderer works. Informative checks compare across families and questions, as metamorphic testing relates outputs to one another.
3. A check that cannot fail is not a check. Every invariant should ship with a failing control built from the real defect and be re-run against it on a schedule, demanding a failure on the archived defect and a pass on the archived repair, rather than relying on the existence of the code. The scheduled test should itself report failure when the invariant is deleted, weakened to report everything clean, re-scoped on the corrupted field, or deprived of its inputs.

Recurrence on new axes

After the repair the class recurred on three axes that no existing check read.

An expected completion horizon was taken from a default for that class of transaction rather than from the document stating the expected timing. Because downstream rates are annualised, the substitution rescaled each rate by the ratio of the two horizons, a median of 2.25 times across twenty-one items and up to 19.3 times. Timing checks that compared an artifact’s clock with itself passed, since every figure came from the same default; a horizon can be wrong only relative to a document.

An analysis was correct up to a conversion date and silent after it, when the holder stops owning the analysed security and begins owning a different one. Every existing check passed because each described a position that ceases to exist at that boundary.

The resource guard above was committed inside the module whose documentation records an earlier instance of the same class on the same subsystem. A module can document a defect class and commit it on a different axis, which is why rules generalise where a list of past fixes does not.

Limitations

• The register counts instances found and cannot estimate how many pass unrecorded, so seven in eighteen days measures attention at least as much as incidence.
• The three rules are engineering rules with no statistical content. Showing that a system applying them holds fewer defects of this class would require a controlled comparison that is not available.
• The central case is a single audited episode, reproduced by re-running an unchanged checker against two archived states; its claim to generality rests on the mechanism being structural, not on sample size.
• Correct selection does not make a computation correct: a check on which probability governs says nothing about whether that probability is well estimated.
• Failing closed has a cost. A system that refuses to act when it cannot demonstrate which object governs will sometimes refuse when the object was fine, and the resource guard above, failing closed on a mismeasured quantity, silently withheld work for days.

Further reading

• Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz and Shin Yoo, “The oracle problem in software testing: a survey”, IEEE Transactions on Software Engineering, 2015.
• T. Y. Chen, S. C. Cheung and S. M. Yiu, “Metamorphic testing: a new approach for generating next test cases”, technical report, 1998 — relations between outputs where no oracle exists.
• Nancy G. Leveson, Engineering a Safer World, MIT Press, 2011 — on accidents produced by components each working to specification.
• Charles Perrow, Normal Accidents, 1984 — the older argument that tight coupling turns local correctness into system failure.