Announcing AI training data pipelines on Onehouse: feeding open data to open models

TL;DR

• Problem: Improving models with production experience requires joining agent runs to outcomes and turning them into reliable training examples. Manual training exports make those datasets hard to reproduce, refresh and correct.
• Why now: Post-training, or adapting a pretrained model to a task or behavior, is producing compelling results with open models. The examples below show how much those gains depend on carefully prepared data and evaluations.
• What we built: Onehouse prepares validated, versioned training datasets from open lakehouse tables, submits training jobs to Baseten, Fireworks or Together AI, and tracks their status in the Onehouse UI. The providers train the models.
• How it works: OneFlow ingests source data, Quanton curates and exports training examples, and Apache Airflow™ can coordinate the workflow. Lakegres serves context to the agent at inference. We will share more about agent trace logging in the coming weeks.
• Why it matters: The lakehouse can connect training, evaluation and inference through a shared record of source data, examples and outcomes. Engineers can trace what went into a training run and rebuild its inputs as new feedback arrives.

The contract example below walks through the data preparation and provider handoff.

Introduction

Today, models are constantly being trained all around us. Models are driving cars, doing homework and writing software. The trained model depends on the quality of the data pipeline feeding it. A mislabeled conversation becomes a bad training example. A training data dump that misses recent outcomes teaches the model from incomplete data. If the next model performs worse, engineers need to inspect those training inputs and understand what changed.

The training techniques differ, but all need data of different shapes and sizes. Pre-training builds a base model from scratch using a huge data corpus. Post-training adapts that model’s behavior after pre-training. Fine-tuning continues training a model on examples selected by a human or another agent to teach a task or behavior. Distillation uses a stronger model’s outputs to train another model, often a smaller one. If you are a data engineer, you would recognize that these patterns need datasets with different selection rules and lifecycle management.

In most companies, the data lake already holds much of that source training data: support conversations, business transactions, contracts and application events. Companies have spent many years and millions of dollars organizing their vast operational data on object storage in open formats. Very often, the companies that lead their market segments tend to be the ones that leverage this data as a moat to build unique products or lasting competitive differentiation. As AI agents scale within these companies on top of open-weight models, the agent runs produce a new class of invaluable data that includes the user request, retrieved context, tool calls, agent turns and AI response. The eventual outcomes from the agent’s interaction with an end user determine which examples are worth learning from. For example, did a customer service agent actually resolve the issue and make the customer happy?

The current weakness is often in how these records become well-managed, high-fidelity training data. Consider a notebook that exports a set of conversations and labels to train_v3_final.jsonl. The file may be valid and the training job may succeed. Later, a label changes or a model evaluation exposes a missing class of examples. Rebuilding the dataset requires the original query, its data lineage and the source table versions it read. A file lying around in cloud storage supplies none of them.

The best-in-class AI companies are already moving to weekly fine-tuning or post-training runs. Repeated training makes this an ongoing data management problem. Each run consumes inference, annotation and evaluation work that the team has already paid for. Losing the underlying records means paying for GPUs again, or training with less information. Open models give teams a starting point they can reuse; the record of how their own product behaved is what they have to build and maintain.

We are applying established lakehouse workflows to that problem. Onehouse manages the source data, curation and exports. Popular open model training and inference providers like Baseten, Fireworks and Together AI handle training. My bias is toward solving this with the data infrastructure we already operate so engineers can inspect the training examples, trace them to their source records and rebuild a corrected dataset for the next training run.

Background: model improvement depends on usable training data

Before we go further, it might be useful to understand the open model landscape and the emerging approaches to improving model quality. Published post-training results make an extremely compelling case for investing in pipelines that maintain high-quality training data. They also show why a collection of unreviewed production traces is insufficient: the examples need a task, a definition of success and a process for selecting useful behavior.

Open models give teams more control to customize agent behavior

All of us are probably using models from OpenAI or Anthropic every day, and they mostly work out of the box as long as the right context is connected. But an agent operating on the complexity of an entire company’s internal or customer data often needs a more flexible and targeted approach. Closed models keep their learned parameters, or weights, under the provider’s control and expose the model through an application or API. Open weights make it possible to adapt and host a model, subject to its license, while retaining a choice of training and serving infrastructure. Production adoption is growing: Vercel reports that open models handled 29% of AI Gateway tokens in June 2026, up from 11% in April, at under 4% of spend. That reflects one gateway’s workload mix; it does not establish equal quality at a lower price for every task.

Customizing an open model starts with a pretrained model and a dataset for the behavior you want to improve. Teams can update its weights or train an adapter, a smaller set of parameters used alongside the base model, using their own infrastructure or a service such as Together AI’s fine-tuning platform. Compatible trained weights and adapters can then be retained and hosted elsewhere, subject to the model license. Closed-model customization is limited to what the provider exposes: prompts, supplied context and, for some models, managed tuning. The Claude API does not currently offer fine-tuning, while Google supports supervised tuning for selected Gemini models within its managed service.

A reusable dataset makes that evaluation easier across providers. Baseten, Fireworks and Together AI accept overlapping training-data shapes, though their supported models and loading paths differ. Keeping the curation logic independent of a provider lets you export another compatible version without reconstructing the examples from scratch. The provider-specific details appear in the code section.

Post-training works when the task and evaluation are well defined

Harvey’s work with Baseten Research shows how a task-specific evaluation can improve an open model. In May, a reinforcement-learning run using GRPO, an algorithm that compares scores across groups of the model’s attempts, took Qwen3.5-9B from 42.5% to 63.0% criterion pass rate in 40 steps on held-out legal tasks. The reward came from the legal rubric. As the score improved, the model changed how it researched the documents. This measures criteria satisfied, not the fraction of entire matters completed correctly. Harvey’s separate Tenet project with Fireworks improved the fraction of tasks completed correctly. Fireworks’ August report describes post-training Kimi K3: LAB all-pass rose from 10.8% to 19.7%, and Mercor APEX Corporate Law from 58.8% to 74.0%. Cost per LAB task was $5.92 versus $5.62 for the base model. The gain came with a $0.30 increase per task.

The training environments required explicit engineering of the task and its expected outcome. Harvey describes partner-style instructions, a matter containing documents, and expert-written rubrics specifying what a correct work product must contain. It also states that it used no customer data in its post-training efforts. These results support investment in carefully constructed training environments; they do not establish that copying customer traffic into a trainer produces the same gains.

Coding agents make the same dependency on the execution environment clear. Cursor’s technical report says Composer 2 training “occurs in realistic Cursor sessions with the same tools and harness the deployed model uses.” The harness is the environment in which the agent reads files, calls tools and completes its task. Keeping training aligned with that environment is part of constructing the examples. Successful and failed runs can both contain useful training material. Nebius released 80,036 software-engineering agent trajectories, meaning sequences of agent actions. Its filtering removed malformed and unsuccessful steps and selected successful outcomes, but it also retained useful prefixes of some failed trajectories. That distinction matters: a failed run may still demonstrate how to locate the right file or reproduce a bug. Selecting those portions requires the action history and evaluation results together.

Data inspection remains work even if the training run itself is managed for you. Baseten’s supervised fine-tuning study uses datasets whose targets were refined against task evaluations built with customers. That requires a process for deciding what should enter the training set.

Ad hoc data handling loses progress the team has made

The improvement loop needs a durable record of both behavior and outcomes: serve the model, record what happened, select useful examples, train and evaluate the next version. There are several ways a manual handoff breaks that loop.

Expired traces remove potential training examples. Langfuse reports storing petabytes of tracing data and processing 19 times more data through 2025. Its retention feature deletes expired traces nightly when a retention policy is configured; saved dataset items remain available. That is reasonable for an observability store. If it is also your only copy of a useful run, the retention policy becomes your training-history policy. Preserve eligible records and their scores before expiry when you intend to reuse them.

An old data export excludes newly available evidence. A support dataset created before a policy change will contain the old answers until someone refreshes it. Freezing a dataset is necessary for a reproducible experiment. The failure is losing the process that builds its successor from newer records and corrected labels.

Traces without outcomes give an incomplete definition of success. A support agent can produce a plausible reply while leaving the issue unresolved. The conversation may live in a tracing tool and the resolution in the ticket database. Training selection needs the join between them, including enough time for the outcome to become known.

Missing historical state makes model comparisons ambiguous. Replaying the same request against today’s documents can produce a different answer even with an unchanged model. Preserve the relevant data snapshot, tool behavior and evaluation version so the comparison has a known baseline. A current production database cannot supply historical state that was never retained. In its account of an Anthropic engineer’s talk, Arize describes resettable state, versioned tool schemas and detailed environment logs as requirements for reproducible agent evaluations. Retained lakehouse snapshots also let evaluation jobs read that history without adding training reads to the production database.

These failures share dependencies: stable record identifiers, retained history, joins to outcomes and repeatable selection logic. Building each training export separately leaves every team to maintain those dependencies again.

Managing training inputs in the lakehouse

Training data should follow the same lifecycle as other production datasets: ingest, curate, validate, version and consume. Onehouse applies that workflow to open tables and produces the inputs that training providers can then consume.

Ingest source data and the history needed to reuse them

OneFlow continuously ingests data into open Apache Hudi and Apache Iceberg tables in the customer’s cloud storage. Records from databases, documents and agent activity can then participate in the same downstream queries used to craft context or curate training inputs. Low-code transformers also make it possible for the pipeline to attach scores, labels and eventual outcomes to the original run. Record and secondary indexes help these queries locate a particular run or select records by fields such as customer or outcome, while column statistics help skip files outside a date or score range. The same ingestion pipeline can also prepare data for similarity-based curation. Onehouse’s built-in vector generation transformer turns selected fields into embeddings and stores them alongside the source data.

Reproducible queries require granular retention policies and lake storage that can handle a long history of table versions. LSM-optimized table metadata supports a virtually unlimited history of table commits, keeping metadata access efficient as that history grows. The corresponding data files remain subject to cleanup, and savepoints protect the files for selected snapshots. Onehouse allows a training data curation pipeline to programmatically create savepoints and control retention for the datasets that produced the training data, for as long as needed to assess model behavior and performance during and after training. Those savepoints can correspond to the source-table versions read by a particular training dataset, so the next evaluation can revisit the same inputs. The savepoint controls also support choosing a previous commit, scheduling savepoints and setting their expiry.

Onehouse savepoint controls and the commit selector for retaining a point-in-time table snapshot.

Build curation pipelines with existing lakehouse tools

Data pipelines authored on the Quanton engine can crunch massive amounts of operational data and turn source records into training examples. These curation pipelines join agent run traces to outcomes deduced from operational or business data, filter records by policy, select the cases the model handles poorly and reshape the result for the trainer. Quanton’s training dataset exporter checks the schema and row contents and writes a versioned export. A curation query can then use vector search and blob retrieval together to find similar examples and retrieve their original content. Batch vector search extends this to many query vectors at once. For example, a set of failed agent runs can become the starting point for finding related cases that deserve review for the next training dataset.

The lineage tools already used on the lake can capture how those training datasets were built. For example, OpenLineage’s Spark integration records the input and output datasets of a job, connecting source tables, intermediate transformations and the final curated dataset. A mislabeled support conversation may have contributed to training examples, evaluation rows, embeddings used for retrieval, and difficult negative examples used to teach the model what to reject. The existing dependency graph helps identify which curation pipelines are affected when that label changes. The same reasoning applies when feedback changes preferred answers or a parser update alters the chunks extracted from a document.

Versioning and backfills give these curation pipelines a familiar way to apply corrections. A pipeline can rebuild the affected datasets from corrected source records and write a new immutable export, while comparisons with the previous version show which examples changed. Retention policies determine how long the older inputs and exports remain available for evaluation and when they are retired. Deleting a source row does not remove its influence from a model already trained on it; the dataset history helps identify the training runs affected and the inputs for a corrected run.

Hand the dataset to the provider and retain the run record

With this launch, Onehouse submits the training job to Baseten, Fireworks or Together AI and tracks its status right inside the Onehouse UI. The provider performs the training. Teams can also track the workflow and its lineage through open-source orchestration tools such as Apache Airflow, keeping the training run connected to the curation pipelines that produced its inputs. For now, users pass their provider credentials into the jobs. We will soon add platform-wide model provider configuration, so teams can manage these settings centrally and reuse them across jobs.

Lakegres supplies context at inference, when the application uses the trained model. It serves open lakehouse tables through a Postgres-compatible endpoint. The application can record its subsequent runs and outcomes for ingestion into the next training cycle. We will share more about agent trace logging in the coming weeks, including how agent runs and feedback feed back into lakehouse tables for the next round of curation and training. The diagram marks this upcoming part of the workflow with dashed arrows:

The dataset is the contract: its contents, provenance and reproducibility are Onehouse’s responsibility; training compute and optimization belong to the provider. Shared data shapes let teams revisit provider choice. Model compatibility, credentials and execution settings still need to be checked when they move a run.

Quanton LLM training, in code

The Quanton guide builds training examples from contract text and lawyer-assigned clause labels. Its inputs are contract_chunks and contract_annotations, Hudi tables created by the companion retrieval guide from 510 commercial contracts, 6,591 lawyer-assigned clause labels, and 41 clause types. The chunks include text and embeddings; the annotations supply the expert labels. The code below retains that example and its limitations.

The exporter registers as a PySpark 4 data source supplied in the Quanton image:

import quanton_llm_training
quanton_llm_training.register(spark)

The first query finds a chunk containing the start of each labeled span. Here, chunks and annotations are DataFrames loaded from those input tables.

from pyspark.sql import functions as F

norm = lambda c: F.lower(F.regexp_replace(c, r"\s+", " "))

spans = (annotations
         .select("file_name", "clause_type", F.explode("clause_spans").alias("span"))
         .where(F.length("span") > 40))

positives = (
    spans.withColumn("span_key", F.substring(norm(F.col("span")), 1, 120))
         .join(chunks.withColumn("chunk_norm", norm(F.col("chunk"))), "file_name")
         .where(F.expr("instr(chunk_norm, span_key) > 0"))
         .groupBy("file_name", "clause_type", "span")
         .agg(F.first("chunk").alias("chunk"))
)

Matching uses normalized whitespace because the labels and chunks were extracted separately. The prefix permits matches when a longer span crosses chunk boundaries. It also means this illustrative query needs review for full answer coverage: matching the start does not establish that the chunk contains the entire quoted answer. Persist stable source identifiers and use deterministic selection when turning it into a production dataset.

The next query constructs a chat example from each match:

train = positives.select(
    F.array(
        F.struct(F.lit("system").alias("role"),
                 F.lit("You are a contract analyst. Answer strictly from the excerpt; "
                       "quote the contract language that supports your answer.").alias("content")),
        F.struct(F.lit("user").alias("role"),
                 F.concat(F.lit('Does the following contract excerpt contain a "'),
                          F.col("clause_type"),
                          F.lit('" clause? If yes, quote the relevant language.\n\nExcerpt:\n'),
                          F.col("chunk")).alias("content")),
        F.struct(F.lit("assistant").alias("role"),
                 F.concat(F.lit("Yes. This excerpt contains a "), F.col("clause_type"),
                          F.lit(' clause: "'), F.substring(F.col("span"), 1, 800),
                          F.lit('"')).alias("content")),
    ).alias("messages")
)

This dataset needs negative examples before it is suitable for training. Every answer currently begins with “Yes.” The guide proposes an anti-join to find contracts where a clause type is absent, then pairing their excerpts with negative answers. Verify that the labels establish absence; an annotation missing because it was never collected is not a reliable negative. The guide suggests roughly balancing the positive and negative examples.

Export names the provider and training type explicitly. This example uses the guide’s single-object options because Together uploads a file:

(train.coalesce(1).write.format("training_dataset")
   .option("provider", "together")
   .option("training_type", "sft")
   .option("singleFile", "true")
   .mode("append")
   .save("s3://bucket/exports/contract-analyst/v1"))

Validation catches structural defects before the provider upload. The driver checks the schema and supported provider options before writing. Executors check roles, empty content and missing assistant turns as they write. Those checks cannot determine whether a legal label is correct or whether an example belongs in the dataset.

Keep curated rows in their own Hudi table, keyed to the source records that produced them. Comparing those rows across versions identifies which examples changed when model quality moved.

The exporter supports these training-data shapes:

training_type	What the model learns from	Required columns
sft	Supervised fine-tuning: prompts paired with desired answers	messages array<struct<role,content>>, or prompt + completion
preference	A prompt with a preferred and a rejected response	input, preferred_output, non_preferred_output
text	Raw text for continued pre-training	text

Provider adapters support different subsets. The guide documents all three shapes for Together, sft and text for Baseten, and sft with JSONL for Fireworks. Keep provenance columns in the curated table: only the training columns reach the export. The optional rowIdColumn names a column for validation errors, so it must exist in the DataFrame you export.

Each successful export occupies an immutable path. A later write to that populated path fails; write v2 beside v1 for the next dataset. Despite its name, .mode("append") is the Spark invocation required here, not permission to add rows to an existing export. Preserving the input bytes supports reproducibility, alongside the training configuration and execution environment.

The _manifest.json file records the export URI and how to load the exported data. For a Fireworks export, its load section includes dataset registration and job creation commands. This excerpt uses the local-file upload alternative and training command documented in the guide:

{
  "load": {
    "upload_instead": "firectl dataset create <dataset-id> <local dataset.jsonl>",
    "steps": [
      "firectl sftj create --base-model <model> --dataset <dataset-id> --output-model <fine-tuned id>"
    ]
  }
}

Token counts require tokenized output and the chosen model’s tokenizer. Use the guide’s outputFormat=tokenized and tokenizer options with a compatible provider when you need those counts. A JSONL export alone does not establish the training token count.

Fireworks SFT requirements specify a minimum of 3 examples and a maximum of 3 million examples per dataset, using JSONL in the OpenAI chat shape. Check those limits before submitting the export.

The provider loading paths differ:

Provider	Dataset handoff
Baseten	Mount the storage prefix through BDN; the supplied training script reads it. Training docs.
Fireworks	Register a single object with --external-url, documented for a gs:// URI, or upload a local JSONL file. The docs do not establish S3 support for that flag. CLI docs.
Together AI	Upload JSONL or tokenized Parquet and use the returned file ID. The documented file limit is 100 GB; check the installed client’s validation limits too. Data preparation.

Airflow can coordinate data preparation with the provider tasks. The existing Onehouse provider supplies cluster and Spark-job operators plus sensors for completion. This dependency sketch shows where provider submission and completion tracking fit; task definitions and arguments are omitted:

# pip install apache-airflow-providers-onehouse
create_cluster = OnehouseCreateClusterOperator(...)
curate_and_export = OnehouseRunJobOperator(...)
delete_cluster = OnehouseDeleteClusterOperator(...)

create_cluster >> wait_for_cluster >> curate_and_export >> wait_for_export
wait_for_export >> submit_finetune >> poll_finetune
wait_for_export >> delete_cluster

Conclusion

The data feedback loop has supported production machine learning for years. Uber’s ETA system learns from historical trips and real-time signals. LinkedIn’s People You May Know ranks connections using signals that include the likelihood of an invitation being sent and accepted. In both, the model depends on records of what actually happened. These products set industry standards not because they had a 10x better model. The advantages came from nailing this virtuous flywheel: operational data => reliable training data => better models => better products => better outcomes => more operational data.

Post-training makes data management essential to how AI systems improve: each round needs reliable examples, a record of where they came from, and feedback from the last round. The lakehouse can hold that history alongside the enterprise data agents use, connecting training, evaluation and inference to the same foundation. We see it becoming essential infrastructure for AI as teams turn agent runs and their outcomes into data they can learn from. AI training data pipelines are one part of that vision for Onehouse, and we have more announcements coming in the weeks ahead as we build toward powering the data for every part of the agent loop.

Create a free account at quanton.dev and give it a shot, or talk to our engineers →.