Taxonomy / Divergent precedent / Keeping every rater's judgment

Keeping every rater's judgment

Every rater's label is kept as that person's own judgment, often alongside information about who they are, and models are trained and scored on reproducing the whole spread. Datasets and shared tasks here rate each item many times over and grade a system on how well it predicts either the population's distribution of views or a named individual's view, while analysis in the same vein shows that standard Bradley-Terry reward modeling and LLM-as-judge evaluation discard exactly that spread. What this family does not do is worth stating plainly: it preserves divergence in the record, but nothing in it cites an earlier judgment or lets one decision bear on a later one, so it supplies raw material a precedent system would need without any precedent machinery of its own.

The method, against Divergent precedent

Scroll the diagram sideways to see all of it.

Concept Analysis: Theoretical Foundations

Each concept is read twice: whether the approach carries it, and whether the approach's own sources claim it. A concept that is absent and was never claimed is a gap in the field rather than a failure of the work, and is marked out of scope.

Recorded dissent

PartialClaimed · partial

Definition · Appellate practice

When judges disagree, the losing view is not erased — it is published alongside the majority's, signed and reasoned. That does two things at once: it forces the majority to answer a stated objection, which is what makes the majority's own reasoning legible, and it keeps the rejected theory on the shelf, available to become tomorrow's majority.

Analysis

The minority judgment genuinely survives, attributed to the individual who gave it and modelled rather than averaged away, which is the half of the concept about keeping the rejected view on the shelf. What is missing is the reasoned, signed opinion. A differing label carries no argument, so the majority is never made to answer a stated objection and nothing is available to be taken up later as a better theory. Dissent here is a count, not a case.

Overruling with provenance

AbsentNot claimed · out of scope

Definition · Common-law practice

The law can reverse itself when a precedent has become unworkable, unjust or simply overtaken by events — but only in the open: the overruling decision names what it overturns and why, so the change has a traceable history instead of being quietly written out of the record.

Analysis

No rating is ever revisited and nothing names an earlier judgment it overturns. The nearest thing is the LeWiDi series revising its own evaluation openly across editions, moving beyond cross-entropy to new metrics in the third round, but that changes how systems are graded rather than any holding about a case. Diverging Preferences does alter what counts as usable disagreement, and says so plainly, yet it acts at the point of evaluation and training instead of reversing anything on the record.

Incompletely theorized agreement

AbsentNot claimed · out of scope

Definition · Cass R. Sunstein, Legal Reasoning and Political Conflict (1996)

Agreeing on what to do in the case at hand while explicitly leaving the deeper principle unresolved. The agreement can be on an outcome, a rule or a mid-level principle; what gets bracketed is the comprehensive worldview underneath. Courts do this constantly: it lets people who disagree about theory still act together, one case at a time.

Analysis

Nothing here produces an agreed outcome, so there is no case settled while the principle underneath is left open; the output is a distribution or a per-annotator prediction. The taxonomy in Diverging Preferences does sort disagreement into ten categories across four classes and finds most of it comes from task underspecification or response style, which is the sort of diagnostic Sunstein's move needs, but it is put to the opposite use. Divergence is located in order to reduce its influence rather than bracketed so that the case can be decided anyway.

Concept Analysis: Newly Introduced

Dissent measured at population scale

Added

Appellate practice records the dissent of the few judges who happened to sit. DICES rates each item many times over with fine-grained rater demographics attached, and LeWiDi grades systems on how closely they reproduce a population-level distribution of judgments as well as individual annotators' interpretations. Disagreement thereby becomes a quantity that can be estimated in advance and scored against. A court can tell you that someone dissented; this can tell you how far a judgment would split, and which demographic groups the split runs along.

Papers