Announcing Google BigQuery Target

Supermetal now supports snapshot and CDC to Google BigQuery. Replicate Postgres, MySQL, MongoDB, SQL Server, or Oracle to BigQuery in minutes.

A single Rust binary reads the source's transaction log, converts changes to Arrow, writes Parquet to a GCS buffer, and loads into BigQuery.

BigQuery

Supermetal stages Parquet files in GCS, and BigQuery loads snapshots into native tables. For each keyed target in a buffered CDC batch, Supermetal reads the files through one external table, keeps the latest change per key, and applies the batch with MERGE. Tables without a key use append-only loads.

When a source table has a primary key, Supermetal declares the same key on the BigQuery target as NOT ENFORCED. Supermetal also clusters the target table by those columns. Clustering lets BigQuery prune storage blocks during MERGE instead of scanning the full target table, reducing bytes processed and cost. Users can opt into multi-table transactions to apply buffered CDC batches atomically across tables.

Partition-pruned MERGEs and Storage Write API support are on the roadmap.

Performance

Postgres to BigQuery on the TPC-H dataset. The snapshot covers SF10–SF50 (86.6M–433M rows across 8 tables). CDC updates lineitem and orders at 1K–40K ops/sec, with each tier held for 30 seconds followed by a 10-second pause.

Source

Postgres on Cloud SQL

db-custom-8-32768

8 vCPU / 32 GB RAM

400 GB SSD

us-west1 · private VPC

Supermetal

GCP Compute Engine

c3-highcpu-8

4 physical cores / 16 GB RAM

Fedora 44

us-west1-b

Target

Google BigQuery

On-demand compute

Regional GCS buffer

us-west1

Cumulative Row Volume

433M rows in 6m 09s

Duration6m 09s

Rows433M

Snapshot load times range from 2m 26s at SF10 (86.6M rows) to 6m 09s at SF50 (433M rows). On SF50, Supermetal sustains about 1.26M rows/sec (248 MB/sec) from Postgres. BigQuery loads completed Parquet files while Supermetal reads from Postgres. The BigQuery load finishes 24 seconds after the source read completes.

CDC: Latency Under Load

Postgres to BigQuery · 10s flush interval · p100 15–31s at 1K–40K ops/sec

p100 End-to-End15–31s

Target Load

Throughput

Total (p100)

End-to-end p100 latency stays within 25 seconds through 35K ops/sec. Ten seconds of that is the configured flush interval, and BigQuery apply time reaches 13.4 seconds p100. At 40K ops/sec, Postgres saturates and total latency reaches 31 seconds. The Breakdown View shows source read p100 rising from 1.1 seconds to 3.8 seconds.

Get started in minutes

curl -fsSL https://trial.supermetal.io/install.sh | sh

iwr -useb https://trial.supermetal.io/install.ps1 | iex

Direct download

Questions? Check out our docs or reach out to us.