Metadata

Metadata is literally “data about data.” The column header names in a row-based data table are metadata, as are the data types associated with each column (text, date, integer, etc.) Summary data is often included in data tables, specifically to speed up queries. 

In general, metadata falls into three categories:

• Technical metadata includes the name of the database system in which data is stored, the column names used in a table, table sizes, data types, etc.
• Operational metadata includes the recency of data, its state (active, archived, etc.), and data lineage (storage and processing steps the data has undergone)
• Business metadata describes the ownership of data, applicable policies, and relevant definitions used within the organization; business metadata is important in data governance

Metadata is often very large in size, even when compared to the underlying data it is associated with. Keeping metadata current is an important operational concern; for instance, a database may not consider a write operation to be complete until not only the underlying data, but its associated metadata, is written and confirmed.

Relevant links on the Onehouse website: 

• Introducing Multi-Modal Index for the Lakehouse in Apache …
• Knowing Your Data Partitioning Vices on the Data …
• Exploring New Frontiers: How Apache Flink, Apache Hudi ...