Open engines

Open engine clusters deploy open-source engines on the same infrastructure you already run, so each team can use the engine it prefers on the same tables. Open engine clusters are not metered in OCU.

Trino runs with Hudi-specific improvements (column-stats data skipping, record index, and partition pruning) against an external Glue or Dataproc catalog.

Flink runs application JARs submitted as Kubernetes custom resources, against a Hive Metastore or Dataproc catalog.

Ray runs distributed Python across mixed CPU and GPU capacity, reads Hudi tables directly, and takes jobs through the Ray Jobs API.

Endpoints are internal to your network. Trino and Ray query Onehouse tables in place, while Flows, Spark, and SQL clusters handle writes. Tables created by open engines register as external tables under an observed lake (see Lakes, databases, tables).

Further reading

• Announcing Open Engines
• Spark vs Ray vs Dask
• Spark Structured Streaming vs Flink vs Kafka Streams