---
title: "Data Harness for AI Agents: Why Agents Fail on Data"
description: "A data harness is what lets an AI agent see, touch, verify and remember unstructured data. Why frontier models fail on video and sensor data without one."
author: "MDflow"
date: 2026-08-16
reading_time: "17 min"
canonical_url: https://mdflow.cz/blog/data-harness-for-ai-agents
md_url: https://mdflow.cz/blog/data-harness-for-ai-agents.md
---

# Data Harness for AI Agents: Why Agents Fail on Data

*Published August 16, 2026 · 17 min read*


Everyone has now had the experience of pointing a very capable coding agent at a directory of video files and watching it fall over. Not fail loudly — fail slowly, expensively, and with total confidence, twenty minutes into a script that was never going to work.

At AI Engineer in July 2026, Dmitry Petrov gave a talk called *When Agents Meet Physical Data: The Other Physics of Agent Harnesses* that names the reason precisely. Petrov spent a decade on data tooling — he built [DVC](https://dvc.org/), essentially git for data, and now works on [DataChain](https://datachain.ai/) — and his argument is that the model is not the problem. The physics are different down here, and your agent's intuition is calibrated for the wrong universe.

> **TL;DR** — A **data harness** is the layer between an agent and your data that makes data work possible at all. It has four jobs: let the agent **see** the data through a real schema, **touch** it through a resumable execution engine, **verify** answers fast through a metadata layer, and **remember** what it built so nobody pays for the same compute twice. The frontier labs found the same thing from the structured side — Anthropic's analytics agent went from **21% to over 95%** accuracy on written context alone. And in every one of these systems, the remembering layer turns out to be a folder of markdown. That layer is what [MDflow](https://mdflow.cz) is. [Start free](/login).

## What is a data harness for AI agents?

**A data harness is everything an agent needs around your data in order to work on it — schema, execution, verification and memory — as distinct from the harness that wraps the model itself.** The [agent harness](/blog/ai-agent-harness-era) gives the model a loop, tools, permissions and durability. The data harness gives it a working relationship with bytes it cannot read.

Petrov's framing is anatomical. To do data work an agent needs to *see* the data properly, to have *legs* — an engine that can actually run over it — to have *hands* that can touch it, verify results and run tests, and to *remember* what mattered so the next project reuses it. Take any one away and the whole thing degrades in a way no model upgrade repairs.

The two frontier labs published results in 2026 that point the same direction from the other end of the data universe. Anthropic's writeup of [its internal self-service analytics agent](https://claude.com/blog/how-anthropic-enables-self-service-data-analytics-with-claude) reports that **without skills, Claude's accuracy on its analytics evals never exceeded 21%**; adding skills — which Anthropic defines as *"a folder of markdown the agent reads on demand"* — pushed it *"consistently above 95% in aggregate and regularly around 99% in certain domains."* OpenAI's [in-house data agent](https://openai.com/index/inside-our-in-house-data-agent/) is built on six deliberate layers of context — table usage patterns, human annotations, code-derived definitions, institutional knowledge from Slack and docs, persistent memory of corrections, and live runtime queries — sitting over 600 petabytes and 70,000 datasets for thousands of employees.

Both are impressive, and Petrov's point is that both were played on easy mode. They operate over structured business data in a warehouse, with tables, a schema registry, lineage and a query engine already in place. That is the luxury version. Underneath it sits the messy half of the data world — video, audio, sensor streams, robot telemetry, scans, often several modalities in one project — where none of that infrastructure exists yet.

## The neutron star problem

**The reason unstructured data breaks agents is that its surface area and its mass have nothing to do with each other.** Two thousand video files in an object store look like a small problem. An agent sees two thousand paths and reasons about them the way it reasons about two thousand source files.

Then you open one. A recording contains clips; clips contain frames; frames contain detected objects; each object carries a class, a label, a confidence score and a bounding box. Two thousand files become millions of records. Petrov's image for this is a neutron star: the size of a city on the surface, more massive than the sun.

In his live demo the numbers are modest and still make the point — **91 dashcam clips produced roughly 100,000 records** in about 24 minutes of processing, after which the question *"how many of these clips have people in them?"* became a sub-second query returning 82 of 91. Scale to thousands of clips, or go one level deeper into the objects, and you are at tens of millions.

The two ways teams normally cope both fail:

1. **Write JSON sidecars next to the files in object storage.** You end up with millions of small JSON files, terrible latency, and no consistency guarantee.
2. **Stand up a central metadata database.** Better — and now you have two systems, two languages, and a SQL island in the middle of a Python codebase that most researchers on the team will not go near.

DataChain's answer is to define the schema in [Pydantic](https://docs.pydantic.dev/) and transpile it down to SQL, so the schema, the data model and the processing code are all the same language. The general principle survives the specific tool: **the agent cannot reason about what it cannot see, and a schema is what makes an opaque binary legible.**

## The four jobs of a data harness

### 1. See — a schema over the opaque

Give the agent a typed model of what is inside the files: a video file with a path, checksum, e-tag and size; a frame with an index and timestamp; a detection with a class, confidence and bounding box. Nested objects, real types. Once that exists, "how many clips contain a person" is a filter and a count, not a script that downloads and parses a million JSON files.

### 2. Touch — a resumable execution engine

In the structured world the execution engine is your warehouse and you never think about it. In the unstructured world you have to bring one — Ray, Spark, or something in the [Dask](https://www.dask.org/) shape that binds compute to data. Two properties are non-negotiable, and both are about money:

- **Parallelism you can ask for in one line.** Run this function over these files on forty machines.
- **Checkpoints and incremental update.** If a job dies at 60% because of a bug or an API error, you fix it and catch up — you do not recompute from zero. And when new files land in the bucket, only the new files get processed.

Nobody puts checkpointing in the first version. Everybody adds it after the first time an LLM-enrichment job dies eight hours in.

### 3. Verify — a metadata layer so tests are fast

Running tests is how a coding agent knows anything. It is also how a data agent grinds to a halt, because a test that re-reads raw binaries takes hours instead of seconds — and the agent's whole working style assumes tests are cheap.

Petrov's fix is the oldest idea in the data industry: stop querying raw data, build a dimensional layer over it. Star schemas, one-big-table, pre-aggregated slices — techniques the analytics world has had for decades and the multimodal world mostly hasn't adopted. Wired into the agent, it becomes a habit: before answering, ask *do I already have a dataset that answers this in one query?* If not, build one — and build it general enough to answer the neighbouring questions too.

He also quotes Anthropic's sharper observation about why accuracy matters more here. In software there are many acceptable ways to solve a problem. **In data there is usually one correct answer**, and a plausible wrong one looks identical until it reaches a decision.

### 4. Remember — and this is where it gets interesting

You have now spent real money turning a neutron star into a useful slice of metadata. Without a memory layer, the next person on the team — or the same person in three weeks — spends it again. Petrov is blunt: people pay double, triple, quadruple for the same result, over and over.

So DataChain writes a knowledge base. And the shape of it is the part worth stealing whatever tooling you use:

> **A set of markdown files, one per derived dataset.**

Each document carries the description of the dataset, the **session context** — why it was built, out of the conversation that produced it — the dependency on the source storage location, a preview of the data, the schema, summary statistics, and **the source code that generated it**. Petrov calls the source code the most crucial part, and notes that OpenAI reached the same conclusion in its own data-agent writeup: meaning lives in code, because pipeline logic carries the assumptions.

Together those documents plus the source bucket plus the resulting tables form a lineage graph. Share the knowledge base and your teammates' agents already know the dataset exists, what it cost, and whether it answers the new question — so the recompute simply does not happen.

It is not an accident that Anthropic landed in the same place from the structured side: skills as markdown, kept in the same repository as the transformation models, updated in the same pull request. Two teams, opposite ends of the data universe, both concluded that the durable layer is prose.

## Why a data harness matters

### For developers

**Because "use a better model" has stopped being an available answer.** Everyone is already on a frontier model. The gap between an agent that flails on your video bucket and one that answers in seconds is entirely in the harness — and unlike model quality, the harness is something you control.

There is also a straightforward cost argument. Unstructured processing is the expensive kind of compute: GPU inference, LLM calls per frame, hours of wall time. Every layer of the harness is a mechanism for not paying twice — checkpoints so a crash costs minutes rather than a day, metadata slices so a question costs a query rather than a job, a knowledge base so a colleague costs nothing at all.

### For AI agents

**Because an agent's intuition about data is actively misleading, and only written context corrects it.** A coding agent's priors say: files are small, tests are fast, iteration is free, and if a script fails you just run it again. Every one of those is false here. Nothing in the model's weights knows that your `january/` prefix has a corrupted codec in 3% of clips, or that the previous team already computed person-detections at frame granularity and put them in a table.

That knowledge only exists if a human or an earlier agent wrote it down somewhere a later agent will look. Which is the same conclusion as [context engineering](/blog/context-engineering-for-ai-agents) in general, arriving from an unusually expensive direction: the cost of *not* writing it down is measured in GPU-hours.

## Which applications benefit most

1. **Physical and embodied AI** — robotics telemetry, autonomous-driving footage, drone and dashcam data, where video is the primary modality and every question implies a preprocessing job.
2. **Computer-vision and multimodal model teams** — curation, labelling review, detection and segmentation runs, and the endless "which subset should we train on this time" question.
3. **Media and video platforms** — catalogues where the interesting metadata (scenes, faces, objects, transcripts) has to be derived before it can be searched.
4. **Scientific and biomedical imaging** — microscopy, radiology, genomics pipelines, where reproducibility and knowing exactly which code produced a derived set are not optional.
5. **IoT, sensor and industrial telemetry** — high-volume streams that are cheap to store and expensive to interpret.
6. **LLM evaluation at scale** — running judge models over large sample sets is structurally the same problem: expensive per-item compute whose results must be versioned, sliced and never recomputed by accident.

## How MDflow fits

**MDflow is not a data warehouse, an object store, or an execution engine.** It does not hold your terabytes, run your YOLO model, or execute distributed jobs. DataChain, Ray, Spark, Dask, your cloud storage and your warehouse do all of that, and should.

MDflow is the fourth job — **remember** — and it is the one most teams leave as a folder on one laptop. Look again at what that dataset knowledge base actually is: a set of plain markdown documents with a description, the reasoning behind them, dependencies, and code blocks. That is a document store's job, not a database's.

### What already lines up today

**Markdown documents are the native unit.** No conversion layer, no proprietary block model. A dataset document written by an agent is the same markdown you would commit to a repository, and every document is also served as a raw [`.md` twin](/blog/llms-txt-explained) with YAML frontmatter, so an agent can fetch one over plain HTTP without a client library.

**Folder descriptions carry the intent.** Every folder in MDflow has a description saying what belongs in it, and [`mdflow_get_context`](/docs/mcp) ranks those descriptions **above** folder names and document titles before returning matching bodies. A folder described as *"Derived datasets from the dashcam corpus — schema, stats and generating code for each"* is a retrieval signal you wrote, not one the model inferred. That is [why folder descriptions beat file names](/blog/folder-descriptions-agent-context).

**Every runtime reads the same knowledge base.** The same workspace is reachable from Claude, ChatGPT, Cursor and Codex over the [remote MCP server](/docs/mcp) with OAuth or a Personal Access Token, and from a training job, an orchestration DAG, a cron schedule or an [n8n](/docs/n8n) workflow over the [HTTP API](/docs/api) with a bearer token — plus a local stdio server, a [VS Code extension](/docs/vscode) and an iOS app. The pipeline that generates a dataset and the agent that later asks about it write and read the same document, and neither is bound to a particular checkout.

**Version history shows how a description drifted.** Anthropic's most useful number is not 21% or 95% — it is the drift from ~95% back to ~65% in a month when the written context went stale. Version history captures the previous version on every saved change across every write path, with line-by-line diffs and non-destructive restore, so you can see when a dataset document stopped matching reality and revert the edit that broke it. (Pro, private to the document owner, and deliberately not exposed over the API or MCP.)

**The Document Log answers "which job wrote this?"** A cross-document activity feed records created, edited, shared and deleted events with the actor on every row, shown as `automated · <token name>` for anything arriving through the API or MCP. Give each pipeline its own Personal Access Token and the log tells you exactly which job registered which dataset, with a click-through side-panel diff.

**Sharing without a data platform.** A dataset document can be shared as a read-only public link or a `.md` endpoint, so a collaborator at another organisation can read what a dataset is and how it was built without an account, a VPN, or access to the bucket. Collections group several documents into one shareable set, and [client-side encryption](/blog/client-side-encryption-for-online-notes) keeps chosen documents opaque to any agent without the password.

**Outstanding work as checkbox lines.** Because [`/tasks`](/blog/markdown-task-management) aggregates ordinary `- [ ]` lines out of markdown bodies, a pipeline that records "needs re-run after codec fix" as a checkbox produces something a human can re-order and tick off, with the document body remaining the source of truth.

### Where we are headed

Direction, not a dated commitment. Two things about this use case interest us most: richer structured retrieval over folder descriptions, so an agent asking about a corpus reliably surfaces the three dataset documents that matter out of four hundred; and narrower agent credentials, since a Personal Access Token today reads and writes everything its owner owns — good for attribution, not yet least privilege. Neither turns MDflow into a data platform, and that is deliberate.

## The bottom line

Agents are not bad at data because they are not smart enough. They are bad at data because the laws are different down here — surface area lies about mass, tests are slow instead of fast, and a failed run costs real money instead of ten seconds. The fix is a harness: schema so the agent can see, an engine so it can touch, a metadata layer so it can verify, and a knowledge base so it remembers.

Three of those are infrastructure. The fourth is writing, and it is the one that decides whether your team pays for the same neutron star twice. Anthropic keeps it as markdown in a repo. DataChain keeps it as markdown in a directory. Keep it somewhere every agent and every teammate can reach.

[Start free](/login) · [Connect an AI agent](/docs/mcp) · [Read the API docs](/docs/api)

## Frequently asked questions

### What is a data harness for AI agents?

A data harness is the layer between a coding agent and your data that lets it work on that data reliably: a schema so it can see what is inside opaque files, an execution engine so it can process terabytes without losing work, a metadata layer so it can verify answers in seconds instead of hours, and a knowledge base so the result is remembered rather than recomputed. Dmitry Petrov of DataChain framed it this way at AI Engineer in July 2026: you do not fix data work by reaching for a stronger model, you fix it by building the harness underneath the model.

### Why do AI agents perform badly on unstructured data?

Because their intuition is calibrated for code and text, and unstructured data does not behave like either. Two thousand video files look small on the surface but contain millions of nested objects — clips, frames, detections, bounding boxes, labels, confidence scores — so an agent that treats them like files runs out of context or writes something that takes hours to fail. Processing is also slow and expensive enough that a crash halfway through destroys real money, which is a failure mode agents never face when running a unit test.

### How much does context improve data agent accuracy?

Anthropic reported that without skills, Claude's accuracy on its internal analytics evals never exceeded 21 percent, and that adding skills — folders of markdown the agent reads on demand — pushed it consistently above 95 percent, and around 99 percent in some domains. The same post reports the reverse effect: when the skill documents went stale, accuracy drifted from roughly 95 percent back to about 65 percent within a month. The lever was written context, not a bigger model.

### What belongs in a dataset knowledge base?

One markdown document per derived dataset, containing the description of what it is, the session context explaining why it was built, the dependency on the source storage location, a preview of the data, the schema, summary statistics, and — most important — the source code that produced it. That last item is what lets a later agent judge whether the dataset answers a new question or has to be rebuilt, and it is the item both OpenAI and Anthropic singled out in their own data-agent writeups.

### Is MDflow a data warehouse or a place to store datasets?

No. MDflow does not store binaries, run compute, or hold tables — object storage, a warehouse and an execution engine do that, and DataChain, Spark, Ray and Dask are the right tools for the processing. MDflow is the remember half of a data harness: the markdown knowledge base of dataset documents that agents read over MCP or the HTTP API, with folder descriptions telling the agent what a set of documents is for, version history showing how a dataset description drifted, and the Document Log naming which agent wrote it.

## Further reading

- [When Agents Meet Physical Data: The Other Physics of Agent Harnesses](https://www.youtube.com/watch?v=bUJgirn4_yc) — Dmitry Petrov, DataChain, at AI Engineer (July 2026)
- [How Anthropic enables self-service data analytics with Claude](https://claude.com/blog/how-anthropic-enables-self-service-data-analytics-with-claude) — the 21% → 95% skills result, and the drift back to 65%
- [Inside our in-house data agent](https://openai.com/index/inside-our-in-house-data-agent/) — OpenAI's six layers of context
- [We read OpenAI's and Anthropic's data-agent posts so you don't have to](https://datachain.ai/blog/openai-anthropic-data-agents) — DataChain's side-by-side reading of both
- [DataChain](https://github.com/iterative/datachain) — the open-source data harness, with skills for Claude Code, Codex and Copilot
- [The AI agent harness era](/blog/ai-agent-harness-era) — the harness around the model, as opposed to around the data
- [Context engineering for AI agents](/blog/context-engineering-for-ai-agents) — the general case of the same lesson
- [Provenance for AI agent memory](/blog/provenance-for-ai-agent-memory) — why "where did this come from" is harder than it looks
- [MDflow MCP documentation](/docs/mcp) and [HTTP API documentation](/docs/api)