DataPrep-Bench: Benchmarking LLMs as Training Data Preparators

TL;DR

The first benchmark that scores an LLM on preparing training data end to end — both building supervised data from raw sources and predicting, before any training run, whether a candidate dataset is worth training on. Its headline result is negative: adding synthesized domain data on top of Dolly-15k often hurts downstream performance, across every generator family tested (source).

Authors & Org

Peking University, the Institute for Advanced Algorithms Research (Shanghai), OriginHub Technology and Zhongguancun Academy. Corresponding author Wentao Zhang. The full author list could not be captured — arxiv.org returned HTTP 403 in this run and the record is from search-result summaries (source).

arXiv: 2607.20465. #1 trending on Hugging Face Daily Papers on 2026-07-28.

Method

Data preparation is split into two capabilities and both are measured under one protocol:

1. Data construction — turning raw sources into supervised training data.
2. Data quality evaluation — predicting the training value of a candidate dataset before downstream training.

The load-bearing definition is that "quality" means downstream training utility, not surface-level textual properties. Construction methods consume identical raw sources and are scored by fine-tuning a base model on their outputs jointly with Dolly-15k; the evaluation spans six domains and multiple base models.

The released baseline evaluator is the Distributional Alignment Score (DAS), which measures Maximum Mean Discrepancy (MMD) between a candidate dataset and a domain proxy (source).

Results

The reported finding is that synthesized domain data added on top of Dolly-15k often degrades downstream performance — and that this holds across DataFlow-based, direct-LLM and agent-based generators alike, rather than being an artifact of one generator family.

Per-domain and per-model numbers could not be captured in this run.

Significance

Two things follow if the result holds.

First, it is an argument against the reflex of the last two years — that a capable model can be pointed at a domain and asked to manufacture its own training set. The benchmark's protocol is what makes the claim checkable: by grounding every score in an actual fine-tune rather than in a textual quality proxy, it measures the thing practitioners care about instead of the thing that is easy to compute.

Second, it puts a number on a gap that agentic-RL work has been assuming away. Self-improving pipelines — AREX: Towards a Recursively Self-Improving Agent for Deep Research, Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills — depend on generated data being worth training on. A benchmark saying that generated data frequently is not is a direct constraint on that loop, and on the recursive-self-improvement arguments built above it (Frontier Pacing).

Open Questions

• Does the negative result survive at frontier scale, or is Dolly-15k plus a small base model an unrepresentative regime?
• DAS is a distributional proxy. How well does MMD-against-a-domain-proxy actually predict fine-tuning outcomes relative to just running the fine-tune?
• The six domains are not identified in the captured record — coverage may or may not include the code and agentic domains where synthetic data is most aggressively used.
• The submission date could not be verified; one aggregator reported 2026-05-19, which is inconsistent with a 2607 identifier.

Cite

DataPrep-Bench: Benchmarking LLMs as Training Data Preparators.
arXiv:2607.20465. https://arxiv.org/abs/2607.20465

Related: Agentic Reinforcement Learning · Agents (LLM Agents) · Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills