Iceberg CDC Merge on Read for Every Query Engine

Since the Iceberg Target launch, the Iceberg connector now includes a positional deletes only mode, partitioning, and history mode.

Iceberg support varies across query engines. Spec version support differs (V1, V2, V3), and so does support for delete file types. A Merge on Read table that works correctly in Spark may return stale data in Snowflake. This fragmentation is the main obstacle to writing Iceberg tables that work everywhere.

Positional Deletes Only Mode

Equality deletes are the natural choice for CDC. The writer emits the primary key, and query engines match it against data files at read time. But every query must scan all accumulated delete files regardless of what it selects. The Iceberg community has proposed deprecating them.

This is the single biggest blocker for streaming CDC to Iceberg with broad engine compatibility.

Positional deletes flip the complexity from query engines to the writer. Instead of emitting a primary key and leaving query engines to find the match, the writer tracks where each row lives and emits the exact file path and row position. Supermetal takes on this complexity by maintaining a local primary key index that maps each key to its current file and position. When a delete or update arrives, it looks up the position and emits a deletion vector (V3) or position delete file (V2).

The index requires local disk proportional to the number of primary keys in the source database, with object store offload on the roadmap. It updates incrementally as CDC processes each change. When compaction or external writes change the table, Supermetal diffs the manifest and reindexes only what changed.

pk │ file          │ pos
───┼───────────────┼────
42 │ 00001.parquet │  17
43 │ 00001.parquet │  31
44 │ 00002.parquet │   5

s3://bucket/warehouse/db.orders/
├── data/
│   ├── 00001.parquet
│   ├── 00002.parquet
│   ├── dv_00001.puffin   (masks 17, 31)
│   └── dv_00002.puffin   (masks 5)
└── metadata/
    └── v2.metadata.gz.json

In practice, primary key values are hashed and the index schema includes additional bookkeeping fields. The table above is simplified for illustration.

Merge on Read now works with every query engine, including Snowflake, Databricks, Spark, Trino, DuckDB, and StarRocks.

Partitioning

Supermetal supports Iceberg hidden partitioning. Define partition transforms per table, and query engines prune files at scan time without partition columns appearing in the schema.

Available transforms are identity, year, month, day, hour, bucket(N), and truncate(W). Partitioning an orders table by day(created_at) groups rows into daily directories. A query filtering on WHERE created_at > '2026-06-01' reads only the relevant day files instead of scanning the full table.

bucket(N) distributes rows across N hash buckets for high cardinality columns. truncate(W) rounds integers or prefixes strings to width W.

System columns _sm_synced_at, _sm_version, and _sm_deleted are also available as partition sources. Partitioning by day(_sm_synced_at) organizes data by when it was replicated.

History Mode

History mode is an opt-in setting that writes every source event to a parallel append-only table. For a source table orders, Supermetal creates orders (current state, Merge on Read) and orders_history (every event, append-only).

Because the history table is append-only, it has no delete files and no merge overhead at read time. This makes it useful for audit trails and point-in-time replay.

Get started

curl -fsSL https://trial.supermetal.io/install.sh | sh

iwr -useb https://trial.supermetal.io/install.ps1 | iex

Read the Iceberg Target docs or reach out to us.