Lakehouse Glossary

A

ACID transactions

ACID transactions refers to database inserts, updates, and deletions that have four characteristics:Atomicity - each statement in a transaction (to read, write, update, or delete date) is treated as a single unit; that is, either the entire...

A

Apache Hudi

Hudi was originally developed at Uber and was released as an open source project in 2017. Hudi is considered to be the first data lakehouse project and is today one of the three leading data lakehouse projects. Hudi was originally...

A

Apache Kafka

Apache Kafka was originally developed at LinkedIn and was released as an open source project in 2011. Kafka has many capabilities, but it is best known as scalable software for real-time data streaming. Kafka is fully distributed. Streaming...

A

Apache Parquet

Apache Parquet is an open source file format that stores data in column-based format, making it more useful for many analytics operations. This is in contrast to data stored in row-based formats, such as data stored in Avro, which are easier...

A

Artificial Intelligence (AI)

Artificial intelligence (AI) is a field in computer science that seeks to use machines to achieve human-like results on tasks that require intelligence, reasoning, judgment, and other capabilities in which human capability exceeds other living creatures...

A

Avro

Avro is an open source row-based storage format with superior support for write operations, when compared to the column-based storage supported by Apache Parquet. Avro allows columns to be added or modified (schema evolution)...

C

Change data capture (CDC)

Database change data capture (CDC) enables changes to a database to be identified, tracked, and sent as updates in real-time. This allows downstream processes and/or systems to act on the change. A common use case for CDC is to keep a...

C

Compute engine

A compute engine is a specialized framework, system, or platform designed to efficiently process and analyze large volumes of data. Compute engines contribute to scalability of a system and support analytics, ML, and...

C

Copy on write (CoW)

Copy on write (CoW) refers to writing data incrementally; for instance, when using a column-based data storage format such as Apache Parquet, the data is updated only in the Parquet partitions that need to be changed. This avoids rewriting...

D

Data lake

A data lake is a repository for storing structured, semi-structured, and unstructured data, at any scale, without changing the structure of the data. The data lake uses object storage, which is very efficient for a wide range of data types and...

D

Data lakehouse

A data lakehouse provides storage of unstructured, semi-structured, and structured data, like the data lake. However, the data lakehouse adds core functionality and services that enable the data lakehouse to rival the data warehouse...

D

Data warehouse

A data warehouse is a repository originally designed in the 1980s for storing structured (relational) data for use in reporting and business intelligence. It has been widely used for that purpose ever since. A process called extract, transform...

E

ELT

ELT is a variant of ETL. Whereas ETL stands for extract, transform, and load, ELT reverses the last two steps: it stands for extract, load, and transform.In ELT, data is loaded unchanged into the analytics system, such as a...

E

Embedding Models

Embedding models are machine learning models designed to represent data—such as words, sentences, images, or other entities—into vector embeddings, which are dense, numerical representations in a high-dimensional space. These models learn to capture...

E

ETL

ETL stands for extract, transform, and load. The term is very widely used in data management and data engineering.ETL was defined decades ago to describe the process of moving data from transactional systems, such as...

E

Euclidean Distance

Euclidean distance is a measure of the straight-line distance between two points in a multidimensional space. It is calculated by taking the square root of the sum of the squared differences between the coordinates of two points. In the context of...

G

Generative AI

Generative AI refers to a subset of artificial intelligence that focuses on creating new content such as text, images, music, and/or data. It works by learning patterns from existing data and then generating similar content that appears original...

H

Hudi

Apache Hudi was originally developed at Uber and was released as an open source project in 2017. Hudi is considered to be the first data lakehouse project and is today one of the three leading data lakehouse projects. Hudi was originally...

I

Incremental update

Incremental updates avoid rewriting an entire table for each update. Instead, the data management system stores updates in separate change tables, and uses both base tables and change tables in routine operations.

I

Ingest

Ingest, or data ingestion, is the process of extracting data from various sources and writing it into the data warehouse, data lake, data lakehouse, or other data store. Ingested data may be highly processed data from an operational...

K

Kafka

Apache Kafka was originally developed at LinkedIn and was released as an open source project in 2011. Kafka has many capabilities, but it is best known as scalable software for real-time data streaming. Kafka is fully distributed. Streaming...

M

Machine Learning

Machine learning (ML) is a branch of artificial intelligence that enables computers to learn patterns from data and make decisions or predictions without being explicitly programmed to do so. ML models are trained on historical data to identify...

M

Medallion architecture

The medallion architecture is a framework for describing a set of data transformations within a data lakehouse or data warehouse. In the medallion architecture, data moves through several steps - Bronze: ingestion of raw...

M

Merge on read (MoR)

Merge on read (MoR) stores upserts for file groups into a row-based delta log as they arrive, so write performance is high. Queries then check the delta log as well as the base file, which causes a small hit to query performance. A compactor...

M

Metadata

Metadata is literally “data about data.” The column header names in a row-based data table are metadata, as are the data types associated with each column (text, date, integer, etc.) Summary data is often included in data...

O

Onehouse Compute Runtime (OCR)

Onehouse Compute Runtime is the foundation for how Onehouse manages compute resources and performance for its managed data lakehouse service. Whether allocating and managing resources for data ingestion, table optimizations, or extract-transform-load (ETL) operations...

P

Parquet

Parquet is an open source file format that stores data in column-based format, making it more useful for many analytics operations. This is in contrast to data stored in row-based formats, such as data stored in Avro, which are easier...

Q

Query processing

A query is a request for information from a data store. Fast, efficient query processing is vital to many processes in business, science, and other fields. The structured query language (SQL) format was invented, and adopted...

R

Retrieval-Augmented Generation (RAG)

Retrieval-augmented generation (RAG) is a hybrid approach in AI that combines two key techniques: document retrieval and generative modeling. RAG first retrieves relevant information or documents from a large database and then uses a generative model...

S

Similarity Search

Similarity search is the process of finding data points (such as documents, images, or vectors) that are similar to a given query point based on a specific metric, such as cosine similarity or Euclidean distance. It is often used in tasks like image...

S

Streaming data

Streaming data is data that is transferred in real-time or near real-time, without being stored en route. Streaming can be true streaming (new data is sent as it is generated or arrives) or micro-batches (data is aggregated up to a number of...

V

Vector Database

A vector database is a specialized type of database designed to store and manage vector embeddings. It allows for efficient indexing, searching, and querying of high-dimensional vectors, making it ideal for tasks such as similarity search, recommendation...

V

Vector Embeddings

Vector embeddings are numerical representations of data (such as words, images, or other objects) in the form of vectors in a continuous, high-dimensional space. These embeddings capture the semantic meaning or features of the data, enabling machines to...