Incremental update

Incremental updates avoid rewriting an entire table for each update. Instead, the data management system stores updates in separate change tables, and uses both base tables and change tables in routine operations. The data management system also reconciles the changes stored into the change tables back into a revised version of the base tables at specified times. 

Incremental updates deliver much better performance when dealing with streaming data, including change data capture (CDC), and other mutable data. Incremental updates are crucial to delivering real-time or near real-time performance in complex systems.

In addition to the benefits of incremental updates for queries in general, some information requests can be handled by an incremental query – a query that only covers information that has been updated since a specified date and time in the past. A system that supports incremental updates can handle incremental queries up to ten times faster than a typical system, because a system that can handle incremental updates can check only records updated since the specific date and time in question.

Such systems may also handle incremental transformations (referred to as incremental ETL).

Related terms: streaming data; change data capture (CDC); Apache Hudi / Hudi; ETL

On the Onehouse website: 

• Apache Hudi vs Delta Lake vs Apache Iceberg
• An Introduction to the Hudi and Flink Integration‍
• Incrementally process data with Apache Hudi