From writing data tests to judging them

Models that can read the shape and lineage of a dbt project now recommend the tests it needs, and the research says a human still has to pick which ones to keep.

The cost of a test

Every test you wrote used to cost someone time. You wrote it by hand, then you spent time keeping it working as the models underneath it changed. So you rationed them: you tested what would hurt most if it broke, and what could be standardized and automated with determinism, and you lived with the gaps.

That cost of ownership has collapsed. An LLM can read a dbt project and propose the tests it needs, then keep them current as the models change. So why not test everything?

Because writing a test was only ever part of its cost. Every test still runs against the warehouse, and every one that fails for no good reason teaches people to ignore the failures. The question used to be which tests we could afford to write. Now it's which ones are worth running, and the number is much much bigger.

Shape and lineage

A rent roll's lineage from a property system and lease admin through units to an asset performance report, with its columns tagged entity, dimension, time or measure. Seven tests an LLM proposed are numbered where they apply: four kept, three cut, each with the reasonA rent roll's lineage from a property system and lease admin through units to an asset performance report, with its columns tagged entity, dimension, time or measure. Seven tests an LLM proposed are numbered where they apply: four kept, three cut, each with the reason

An illustrative rent roll. The LLM reads the shape and lineage and proposes tests. A person keeps the ones worth running.

A model can now read a dbt project the way a senior data engineer would. It sees the shape of each model: its columns, types and SQL. It sees the lineage: which models ref which, and what joins to what. From that it can say what the grain should be, which keys need referential integrity checks, which columns are enums and which numbers need a range.

The models are nondeterministic, but that's OK, we only want the LLM to suggest options. The tests are ordinary deterministic dbt tests, it's not inventing anything new, which pass or fail the same way every run. You get the judgement of a probabilistic model and the guarantees of deterministic test code.

A testing guide I'm using

The best example I've found is the ADAF testing guide, an agent skill for Claude Code, Copilot and Codex, which I'm now incorporating into my own work.

It works from a catalogue of about 33 testing patterns, written up like Martin Fowler's refactorings, each with symptoms, the pattern and the mechanics. I really like how it decides what a column is, not by its warehouse type, but by what it does in SQL:

  • In a JOIN … ON, it's an entity, so it needs key and referential integrity tests.
  • In a GROUP BY or WHERE, it's a dimension, so accepted values and cardinality.
  • Inside SUM, COUNT or AVG, it's a measure, so ranges and distributions.
  • A date or timestamp gets the time tests, such as freshness and bounds.

A column can be more than one of these and then it gets both sets and it always names the model's grain first. Every recommendation comes with why it matters, which package to use, the data quality dimension it defends, what it costs to run, and when not to use it. It also searches the web to check the current syntax before you write anything, because package advice dates fast.

Referential integrity is a good example. dbt doesn't enforce foreign keys, so the guide's answer for every key in a fact table that points at a dimension is a relationships test on the child side:

models:
  - name: orders
    columns:
      - name: customer_id
        data_tests:
          - not_null
          - relationships:
              to: ref('customers')
              field: customer_id

It's the test that catches orders showing up on a dashboard against "Unknown Customer", one of the symptoms the guide lists. Knowing to put it on every foreign key, and to leave out not_null when guest checkouts are allowed, is exactly the judgement the guide encodes.

What my Claude research found

Generate, then filter. That's the same shape as the testing guide: a fixed vocabulary of patterns, a reason for every test, and a "when not to use" for every one.

The job moves

So the work shifts from writing tests to judging them. That's the same move I wrote about in when the whiteboard becomes the thing you build from: defining what correct means, and adjudicating the diffs.

It's also still mostly the first of Georg Klaus's four layers. Grain, keys and ranges are syntactic. Whether an LLM can help with what the data means, and what it points at, is the more interesting question.