AI training data pipelines: curation and post-training

Use Onehouse to build the datasets your models are post-trained on. A pipeline has three steps: curate agent runs and business data into a dataset, validate and version it, and submit it to a training provider such as Baseten, Fireworks, or Together AI, which runs the training. The dataset stays in your tables, where Onehouse tracks its contents and provenance and keeps it reproducible.

Curate

Curation pipelines run on Quanton, which reads structured and unstructured data in one engine: Parquet, Iceberg, CSV, and JSON alongside PDF, DOCX, HTML, Markdown, and plain text. There is no separate parsing stack to run, and curation is native PySpark, so any Python library works. The same jobs run models over the whole lake as batch inference: embedding, classification, and extraction, inline at Spark scale.

A typical pipeline joins agent run traces to outcomes deduced from operational data, filters records by policy, selects the cases the model handles poorly, and reshapes the result for the trainer. Time travel reads the business as it was when each run happened. Vector search over the same tables finds examples similar to a set of failed runs, so related cases can be pulled in for review.

Validate, version, and export

An exporter validates the dataset against the provider’s schema: supervised fine-tuning as message arrays or prompt/completion pairs, preference pairs, or raw text. It versions the export with a savepoint, so the exact training set can always be reproduced, and retention policies decide how long older inputs and exports stay available for evaluation.

The exporter is a Spark write format on every Spark cluster:

import quanton_llm_training
quanton_llm_training.register(spark)

(train.write.format("training_dataset")
  .option("provider", "together")      # together, fireworks, or baseten
  .option("training_type", "sft")      # sft, preference, or text
  .option("rowIdColumn", "chunk_id")   # named in validation errors
  .mode("append")
  .save("s3://bucket/exports/contract-analyst/v1"))

It checks the DataFrame schema against the provider’s contract before writing anything, checks each row on the executors, and writes a _manifest.json that includes the provider’s load command. Each version is a new path; an export is never overwritten, and a failed job removes its partial output. It can also write pre-tokenized Parquet for a given tokenizer and sequence length. Fine-tuning open models from lakehouse tables walks through building a dataset in SQL and exporting it.

When a label or a parser changes, the pipeline rebuilds the affected datasets from corrected source records and writes a new immutable export; comparing versions shows which examples changed. Lineage tools already used on the lake, such as OpenLineage’s Spark integration, record which source tables fed each dataset.

Submit to a training provider

Onehouse submits the training job to Baseten, Fireworks, or Together AI and tracks its status in the console; the provider performs the training. You can also hand the dataset to a training cluster you run yourself, for example on a NeoCloud next to the GPUs. Orchestrators such as Apache Airflow can drive the workflow end to end, keeping each training run connected to the curation pipelines that produced its inputs.

Once the trained model is serving, Lakegres supplies its context and the AI Gateway records its runs, which become the input to the next dataset.

Why training data belongs on open tables

Model improvement is a loop: serving produces runs, runs become training data, training produces a better model, and the better model serves. The loop breaks in four places, all because the data is in the wrong place.

Where it breaks	What happens
Runs are not kept	Tracing tools keep traces for a limited retention window and struggle under large payloads, and scores sit in ad hoc files. Last year’s best runs cannot train this year’s model.
Training reads stale data	A snapshot is copied by hand before each run, and a large corpus takes weeks to move. Each model learns from a business that is months old.
Runs are never tied to outcomes	Traces sit in an observability store and the business outcome sits in the database or lake, often on a different cloud. Models are fine-tuned without knowing which runs actually worked.
Runs cannot be replayed	Production databases keep days of history and cannot take training traffic. Evaluations drift, and results are not reproducible.

Keeping runs, operational data, documents, and media in one set of open tables fixes all four. AI Gateway keeps runs indefinitely at object-storage cost, to be scored after the fact. Flows land every database change continuously, so training reads the business as it is now. Runs and business data sit in the same tables, so a fine-tuning set is one query. Any query can run as of any past moment without touching production.

Agent runs need lakehouse-scale storage: every agent interaction is a multi-step run, which at consumer scale is ten to a hundred times the event volume teams are used to from application logging.

Further reading

• Announcing AI training data pipelines on Onehouse
• Onehouse launches vector embeddings generator
• 2025 market study: modern data architecture in the AI era