Medallion architecture

The medallion architecture is a framework for describing a set of data transformations within a data lakehouse or data warehouse. In the medallion architecture, data moves through several steps:

• Bronze: ingestion of raw, unfiltered data - which may come from operational databases, from logs, from sensors, or other sources - into bronze data tables
• Silver: initial processing of raw data, including filtering, cleansing, and deduping, as well as combinations of data from multiple tables - into silver data tables
• Gold: further processing of data to produce results that speed up business-specific queries and dashboards - stored in gold tables 

There are many advantages to the medallion architecture, which include the governance advantages of maintaining an original copy of raw data; additional governance advantages from having the results of intermediate processing steps; the performance advantage of re-using tables from one process for repeated or new processes; the ability for open source projects and proprietary products to add value at specific steps in the process; and enhanced comprehensibility of the architecture to all stakeholders. 

The medallion architecture may incur additional costs due to the storage of multiple copies of data as it goes through various transformations, but this may be offset by governance advantages and potential performance advantages from having the copies available. 

Related terms: ingest; query processing

On the Onehouse website: 

• Getting Started: Incrementally process data with Apache Hudi
• Exploring New Frontiers: How Apache Flink, Apache Hudi …
• How Apache Hudi Simplifies MPP Data Warehouse ...