A nightly pipeline can pass every check it runs and still hand a dashboard the wrong number. The schema matches, no required column is null, row counts fall inside their usual band, and the revenue total is quietly wrong anyway, because an upstream service changed what one field means while leaving its type and shape untouched. The checks were green. The data was not correct.

Running checks is not the same as having data worth trusting, and the distance between the two is seldom one missing rule that a longer list would have caught. It is more often a matter of framing. A validation suite is a model of how data can go wrong, and a suite can be diligent, extensive, and green while modeling the wrong failures. Three different concerns travel under the single name data quality, and they are easiest to keep apart by what it takes to check each one.

The first is organizational: some person or team has to define what correct means for a dataset and answer for it when it is not. The second is deterministic: properties that can be evaluated from a fixed rule, whether the rule applies to one row, a whole table, or a relationship between tables. The third is statistical: properties that exist only in aggregate, such as a distribution or a rate, which can be judged only by estimating them from the data and comparing against what the data used to look like.

The quality dimensions usually catalogued do not belong permanently to one layer: validity, accuracy, completeness, and timeliness each shift between deterministic and statistical depending on whether a single record or a moving aggregate is the worry. The organizational layer is the cheapest of the three to establish, and the most often left implicit. The deterministic one is the best understood, and the layer at which validation effort tends to both begin and stop. The statistical one is the hardest, and where the errors that survive the first two accumulate.

The distinction is easier to see against concrete failure modes:

Keep a simple orders table in view throughout: each order has an order_id, status, amount, currency, and event_time, and the warehouse publishes a daily revenue table derived from it. The contract defines what amount means and how status may change; assertions reject null identifiers and impossible amounts; monitors watch the missing rate, status mix, and freshness; reconciliation checks that accepted orders add up to the daily revenue total; lineage explains which dashboards and models move when any of those definitions change.

None of this means trustworthy data is out of reach. A belief common on teams that have been burned holds that discrepancies are a fact of life, to be watched for and patched after they surface. For genuinely noisy external inputs there is something to it, and the statistical layer exists to bound what cannot be removed. For the pipelines a business actually runs on, most discrepancies are engineering defects rather than noise. The three layers above all catch error once it exists; on the paths that matter, correctness can also be engineered upstream, so that fewer errors arise to be caught at all.

A contract for correctness

Correctness is not a property a dataset holds on its own. It is a claim measured against an expectation, and an expectation belongs to someone. A column of timestamps is neither right nor wrong until a definition says whether it records event time or ingestion time, which time zone it carries, and whether nulls are allowed. When no one owns that definition, each downstream team supplies its own, the versions disagree, and the resulting mismatches are indistinguishable from data errors. Much of what gets filed as a data quality incident is really an undocumented disagreement about meaning.

The costliest data problems are often structural rather than technical. A production dataset is a dependency that other systems build on, and unlike a code dependency it usually exposes no interface and makes no promises. Downstream teams come to depend on an incidental column, a particular encoding, or yesterday’s cardinality, while the upstream team, often unaware those consumers exist, changes the data freely. The failure modes of these unowned, undeclared data dependencies were identified as a leading source of long-term cost in machine learning systems, where one upstream change can silently degrade everything trained on it.¹ The same dynamic holds for any dataset with more than one consumer.

A data contract is the interface a raw dataset lacks: an explicit, versioned agreement, owned by the producer and agreed with its consumers, that states the schema, the meaning and units of each field, the allowed values, the freshness, and how change will be announced. Its value is not really enforcement; it is that the definition has to exist at all. Recording that a field is event time in UTC, and that categories are added only through a documented process, converts a class of silent, expensive, downstream-discovered errors into a cheap disagreement settled once, in the open, between a producer and its consumers. This is the least technical layer of data quality and often the highest leverage: no anomaly detector recovers a meaning that was never agreed on, and no volume of checks stands in for a named owner accountable when the contract breaks.

Encoding what is known

Once a contract fixes what correct means, the deterministic part of it can be made executable. These are the classic checks, and they cover facts that can be decided from fixed rules: that a field is present and of the right type, that a number sits within a possible range, that an identifier is unique within a table, that a foreign key resolves in another table, that a categorical value comes from a known set. Some are row-local, some are table-level, and some are referential, but under a stable contract each has a determinate answer rather than a probabilistic one. This is the most tractable layer, and tools have automated it for years.

Where these checks run matters more than what they assert. A check evaluated at the boundary where data enters, before the record is written into a table others read, can reject or quarantine the bad row while the context that produced it still exists. The same check run days later against data at rest can only report that something is already wrong, after the corruption has spread. Encoding the contract’s assertions as versioned code, run in the pipeline that ingests the data and in continuous integration beside the transformations that produce it, keeps the definition of correct next to the data it governs and turns a violation into a build failure rather than a later discovery in production.

Two mistakes recur here, both from asking assertions to do more than they can. The first is reading a passing schema as a passing dataset. Type and null checks confirm shape, not meaning, and the amount field from the orders example, correctly typed and quietly redefined from dollars to cents, passes every one of them. Shape is necessary and nowhere near sufficient. The second is writing assertions against numbers that move on their own. A rule that a daily orders table holds more than a million rows encodes a fact that was true the day it was written, and turns into a false alarm on the first slow holiday and a missed error the day a duplicated upstream feed doubles the count. Fixed thresholds on quantities that vary with season, growth, and weekday are a standing source of both noise and blindness, and they mark the point where deterministic assertion stops being the right instrument.

What happens on failure matters as much as the predicate. A pipeline that silently coerces or drops the offending rows, casting a malformed value to null or discarding records that fail a constraint, makes the metric that measures quality look better while making the data worse, and it erases the evidence needed to find the cause. Rejecting the batch, or routing bad rows to a quarantine table on a dead-letter path for data, keeps the failure visible and recoverable. Silent repair is the one response that turns a detected problem back into an undetected one.

Every check is a classifier

Adding more assertions eventually stops helping, for a reason that has little to do with data engineering and everything to do with classification. Every check is a binary classifier over the thing it inspects, sorting each record or batch into pass or fail, and like any classifier it can make two kinds of error. A false positive fires on data that was fine; a false negative stays silent on data that was not.

A deterministic constraint over a stable contract rarely errs on its own. Under a fixed contract, a type mismatch, duplicate key, or unresolved foreign key really is a violation, and what looks like a false positive is usually the contract being wrong rather than the check misfiring. The two errors begin to trade off in earnest only once a check estimates or thresholds something, and the million-row bound that misfires on a slow holiday is the first step onto that ground. They dominate at the statistical layer, where every signal is an estimate: no check drives both errors to zero, and tightening a rule to catch more real problems also makes it flag more innocent ones.

The two errors are not paid for in the same currency. False negatives are quiet, and their cost surfaces later and downstream, as a wrong decision no one traces back to the alert that never fired. False positives are loud and immediate, and their cost is paid in attention: a suite that cries wolf teaches the people who own it to ignore it, and a check that is routinely overridden is worse than absent, because it still consumes review while lending false authority to whatever passes. A validation system whose alerts a team has learned to dismiss has failed regardless of how many real problems it also catches; the property that makes data validation useful in practice is that its alarms are trustworthy enough to act on.²

Reading checks as classifiers changes how a suite is run. Not every check should block: the ones tied to a real, immediate consequence stop the pipeline, while weaker signals warn or are logged for review, so the cost of a false positive is matched to the check that raises it. How aggressively each check fires is set by the cost of its two errors on that specific field, not by one uniform default. Alerts go to the owner who can act on them rather than to a shared channel no one reads. And a check that has produced nothing but false positives for months is retired, because a suite is maintained by removing noise as much as by adding coverage. These are the disciplines a mature monitoring system already applies to its alerts, carried over to data.

Estimating what cannot be asserted

The errors that survive a working contract and a disciplined set of assertions are the ones no single record reveals. A field can be valid on every row while the column as a whole drifts: a category that was five percent of traffic climbs to forty, a numeric distribution shifts after an upstream unit change, the rate of missing values creeps upward, a feed that normally lands by two in the morning arrives at nine. Every record passes every predicate. The dataset is still wrong, and wrong in a way visible only in aggregate and only against what the data used to look like.

Judging these properties means estimating them and comparing against a baseline, which is why the static thresholds that fail at the deterministic layer fail worse here. A hand-set bound on a distribution is stale the moment the data evolves. The workable approach learns the expected behavior from the data’s own history, models the normal range of a metric including its weekly and seasonal shape, and treats a departure from that range as an anomaly rather than a violation of a fixed rule. Methods that derive quality constraints automatically from historical measurements, and raise an alert only when a new batch departs from the learned pattern, exist precisely because hand-maintained thresholds do not hold up as the data changes.³

This layer behaves like monitoring, not testing, and the practice built around it usually goes by data observability. A test returns a verdict on one input at one time; monitoring watches a signal continuously and reacts to how it moves. Data quality metrics are recorded over time: the fraction complete, the distribution of a key column, the freshness lag. The question then shifts from whether a batch passed to whether the trend has broken. Framed that way, the properties become the subject of objectives rather than pass-or-fail gates: a freshness objective that 99.9 percent of hourly checks find a table less than an hour stale, a completeness objective that a critical field stays above a target rate, each with an error budget that tolerates rare, small violations and escalates on sustained ones. Here data quality meets reliability engineering, for the same reason it does elsewhere: a signal watched over time catches the slow degradations a point-in-time check cannot.

Discrepancy is not inevitable

Detection, whether by assertion or by monitoring, assumes an error will occur and asks how quickly it is caught. A prior question is how many errors have to occur at all. The fatalism worth confronting here is the assumption that discrepancies can only be cleaned up after the fact, never prevented, and it holds together because two unlike things are filed under one word. One is irreducible: a sensor with real measurement error, an external feed with genuine gaps, a quantity that moves because the world moves. Bounding that variability is the statistical layer’s task, not abolishing it. The other kind is manufactured inside the pipeline: a figure that disagrees between two internal systems, a revenue total that reads $1.02M on one run and $1.03M when the same job is replayed, rows that vanish in a join or double when a load is retried. These have mechanical causes, and unlike the irreducible kind they can be driven to zero.

Driving them to zero begins with making the pipeline deterministic. A transformation that returns the same output every time it sees the same input, and a load that can be re-run without double-counting or dropping records, removes the largest source of unexplained discrepancy, which is a pipeline that disagrees with itself. Idempotent writes, joins whose ordering is fixed or explicitly resolved, and processing keyed so that a retry replaces a partition rather than appending to it are what separate a number that is reproducible from one that depends on when the job happened to run.

On top of determinism sits reconciliation. Many pipelines carry invariants that must survive from one stage to the next: a step that only reshapes data should conserve its row count, a transformation that moves money should preserve the sum of the amount column, and a daily revenue figure computed from accepted orders should agree with the published aggregate to the cent. Checking those invariants across stage boundaries, control totals that are verified rather than assumed, catches the case where data is silently lost or altered in transit, which no per-record check at a single stage can see. Reconciliation is the check that spans the pipeline instead of inspecting one point along it.

What makes the whole thing navigable is lineage: a record, kept by the system rather than reconstructed by hand after an incident, of where each dataset came from, what transformed it, and which downstream tables and reports read it. Column-level lineage turns a wrong figure in a dashboard into a bounded walk backward along a known path, rather than a search of the whole warehouse, and the same map answers the question the ownership layer left open, of which consumers a change will actually affect, by naming them instead of leaving them undeclared. Lineage corrects nothing by itself. It is the substrate that makes correction, reconciliation, and impact analysis tractable rather than heroic.

None of this is free, and not every table earns it. The discipline of deterministic processing, cross-stage reconciliation, and maintained lineage is worth its cost on the paths that carry revenue, regulatory figures, and the numbers executives act on, and hard to justify on a scratch table feeding one exploratory notebook. The realistic goal is modest in scope and strict within it: the critical paths reconcile provably, the low-stakes ones are merely watched, and the boundary between them is drawn deliberately and pushed outward as the tooling to maintain it gets cheaper. Within that boundary, a discrepancy points to a gap in the design, and a gap in the design can be closed.

Deequ and Great Expectations

Metrics against expectations

The deterministic and statistical layers are where teams usually reach for a framework to run their checks. Teams built around a transformation tool like dbt often start with the not_null, unique, and accepted_values tests it ships, which cover the simplest deterministic cases inside the warehouse. Past those built-in checks, Deequ and Great Expectations make a useful open-source comparison because they come from different worlds and are built around different central abstractions.

Deequ was built at Amazon to verify data quality on very large tables, and its design was set out in a 2018 research paper.⁴ It runs on Apache Spark, is written in Scala, and is organized around metrics rather than assertions. A check is a set of constraints, and each constraint is a predicate over a metric the engine computes from the data, such as completeness, an approximate distinct count, or a quantile.

// Deequ: constraints stated over metrics computed in one pass across a Spark DataFrame
val result = VerificationSuite()
  .onData(df)
  .addCheck(
    Check(CheckLevel.Error, "orders")
      .isComplete("order_id")      // completeness metric == 1.0
      .isUnique("order_id")
      .isNonNegative("amount")
      .isContainedIn("status", Array("open", "paid", "shipped")))
  .run()

Separating constraints from metrics is what lets the design scale. Deequ inspects a whole suite, works out the distinct set of metrics its constraints need, and computes them together in as few passes over the data as possible, often a single aggregation, so a second constraint over a metric already being computed costs almost nothing.

Where an exact metric would be expensive, it approximates: distinct counts through HyperLogLog sketches, quantiles through quantile sketches, so a column of billions of values yields an approximate distinctness without a full sort or an enormous hash set. It can also compute a metric incrementally, persisting its partial state and updating it as new data arrives instead of rescanning history, which suits tables that grow by append. Each of these choices is aimed at data too large to scan casually or often.

Great Expectations starts from the other end, with the expression of intent. Its central object is the expectation, a named, declarative, serializable assertion such as ExpectColumnValuesToNotBeNull, and a suite is a collection of them. A Python wrapper for Deequ, PyDeequ, exists, but the engine underneath stays Spark and the Java Virtual Machine (JVM); Great Expectations, by contrast, is a Python framework in its own right, not a wrapper over a JVM engine.

# Great Expectations: named expectations added to a suite, then rendered as human-readable docs
from great_expectations import expectations as gxe

suite.add_expectation(gxe.ExpectColumnValuesToNotBeNull(column="order_id"))
suite.add_expectation(gxe.ExpectColumnValuesToBeUnique(column="order_id"))
suite.add_expectation(gxe.ExpectColumnValuesToBeBetween(column="amount", min_value=0))
suite.add_expectation(gxe.ExpectColumnValuesToBeInSet(
    column="status", value_set=["open", "paid", "shipped"]))

Because an expectation is a portable object rather than a Spark operation, one suite can be run by more than one engine. Great Expectations translates its expectations to execute on Pandas, on Spark, or on a SQL database through SQLAlchemy, and against a warehouse it compiles them into SQL that runs where the data already sits, without extracting it first. It computes metrics internally to evaluate expectations, but minimizing passes over the data is not its organizing principle the way it is for Deequ; meeting the data in place is.

Its results are correspondingly richer for a human reader: a failed expectation reports the observed value, the count and fraction of unexpected rows, and a sample of the offending values, and its Data Docs feature renders suites and their latest results as browsable HTML. That turns a suite into a shared, legible statement of what the data is meant to be, closer to a contract artifact than a test file.

Deequ optimizes the computation of quality signals at scale; Great Expectations optimizes the expression and communication of quality expectations. Deequ can also profile a dataset and suggest constraints from what it finds, a way to bootstrap a suite, though a generated suite is only ever a starting point and not a substitute for the contract.

Fit and its limits

Set against the layers, the two have different centers of gravity. Great Expectations is strongest at the deterministic layer and at carrying the contract to the people who must agree on it, which is what Data Docs are for. Deequ reaches further into the statistical layer, because the metrics it already computes are the natural input to history: it can store them in a repository and, from that record, run anomaly detection that flags a batch whose metrics depart from earlier runs, the learned-baseline approach the statistical layer requires.

Which one fits depends first on where the data lives and what runs it. Deequ presupposes Spark and the JVM; inside a Spark-based pipeline it is a natural fit and scales with the cluster, but a table that lives in a SQL warehouse has to be made available to a Spark job before Deequ can evaluate it. That can be a real operational cost when the warehouse could have run the check itself. That is the case Great Expectations serves, pushing the check down to where the data already sits, and it is a common one for stacks built around a warehouse rather than a cluster. The choice also follows the dominant failure mode: a system whose main risk is silent drift across enormous tables is often better served by Deequ’s metric history and anomaly detection, while one whose main risk is undocumented, shifting expectations across many teams gains more from the legible suites Great Expectations produces.

For all their differences, the two share a boundary. Each checks data within a dataset or a batch, and at its best watches it over time, but neither reaches past the checker into how the pipeline is built or who owns it. A team that adopts either tool before settling who owns a dataset, and before building the paths so its numbers can be traced, has acquired a capable instrument for measuring a standard it has not yet agreed on or secured.

Keeping the definition honest

A validation suite is a maintained model of how a dataset fails, and keeping it honest means keeping its boundaries visible. Deterministic checks catch what a fixed rule can decide; statistical monitors catch what only the aggregate and its history reveal; reconciliation and lineage keep the critical path explainable when a number moves. The usual order of construction, assertions first and the rest deferred, is close to the reverse of the order in which these pay off.

The frameworks help most once that order is set right. Deequ and Great Expectations are strong instruments for the parts of data quality that can be computed, and choosing between them is mostly a question of where the data lives and which kind of error it tends to produce. What neither can supply is ownership, traceability, or proof that two stages agree. Data earns trust the way any dependency does: it has a named promise, executable checks for what can be asserted, monitors for what can only be estimated, and a path a wrong number can be traced through. Checks confirm that the promise is being kept. They are no substitute for the promise.

References