SQL pipelines

Use a SQL cluster to run Spark SQL from your own tools: dbt, Airflow, and SQL clients such as beeline, DBeaver, and DbVisualizer. Create a SQL cluster, copy its JDBC endpoint from the console, and connect any client that speaks the Spark JDBC protocol. The endpoint is inside your network, served by a Thrift server on the cluster.

This is the cluster your ETL and transformation pipelines run on: dbt models, scheduled SQL, and DML and DDL against Hudi, Iceberg, or Delta tables, all on the Quanton engine. Every table in the project appears as database.table, whether Onehouse wrote it or you registered it as an external table.

Table maintenance is automatic. Tables your pipelines create are registered for you. Table Optimizer runs clustering, compaction, and cleaning on them in the background and syncs them to your default catalogs, so your dbt project needs no maintenance steps of its own.

Recurring ETL runs incrementally. Quanton recognizes recurring ETL over change feeds and time-partitioned tables and rewrites it to process only the latest changes on each run. See Incremental processing.

Attach a different catalog per cluster. A SQL or Spark cluster uses the Onehouse catalog by default, or an external catalog you choose: an Iceberg REST catalog such as AWS Glue or Snowflake Open Catalog, or a Hive Metastore-compatible catalog. Other cloud warehouses and lakehouses tie their compute to their own catalog. On Onehouse, pipelines get Quanton’s speed while the tables stay registered, and governed, in the catalog they use today.

Settings. Set Spark properties per session; table properties persist on the table. Java and Scala UDFs are supported.

A SQL cluster and its catalog. The catalog is chosen per cluster; Table Optimizer maintains whatever tables the pipelines produce.

A running SQL cluster showing its JDBC endpoint.