loadgen — a Postgres load + verification harness for replicare
loadgen is a manual, idempotent tool for exercising replicare against broad,
high-volume, realistic Postgres data. You run it whenever you like against a
source database; it seeds a rich schema the first time and applies random
inserts/updates/deletes on every subsequent run. Point replicare at the source and
a target, and use loadgen verify to assert the target converged.
It is a developer/test tool — not part of the shipped binary. Run it from a
checkout with go run ./test/loadgen (or task loadgen:*).
What it generates
One schema (loadgen) of 10 tables, chosen to exercise replicare's hard
paths, not just to hold rows:
| Table | What it stresses |
|---|---|
tenants | hub table many others FK to (giant-component pressure); identity PK, jsonb, text[], timestamptz, bool |
categories | self-referential hierarchy column (self-ref FK under --cyclic) |
users | bytea, inet, numeric(20,4), date, jsonb, many nullable columns; unique (tenant_id,email) |
products | composite text PK (tenant_id, sku); float8[]; timestamp (no tz); target of PK-changing updates |
orders | UUID PK; char(3); nullable FK to users |
order_items | composite PK; composite FK to products ON UPDATE CASCADE (a SKU rename is a PK change that cascades) |
product_categories | M:N join, composite PK, two FKs |
events | the high-volume table (~1M rows at default scale); wide type coverage |
generated_metrics | GENERATED … STORED column + identity PK |
audit_log | no primary/unique key → replicare skips it with a loud warning (verify excludes it too) |
Data is generated server-side (INSERT … SELECT generate_series, random()
made reproducible with setseed, md5(…)::uuid) — fast to a million rows, robust
across old Postgres, no client-side type encoding. The churn mix includes
PK-changing SKU renames and mutation of the nullable FK-cycle edge.
At --scale 1.0 (default) the seed writes ~1.2M rows (events dominate). Scale it
down for quick loops: --scale 0.01 ≈ 12k rows in ~150ms.
Usage
# 1. Create the schema on the TARGET (replicare replicates data only — target
# tables must pre-exist).
go run ./test/loadgen ddl --dsn "$TARGET"
# 2. Seed the SOURCE (first run seeds; later runs churn).
go run ./test/loadgen run --dsn "$SOURCE" # ~1M+ rows
go run ./test/loadgen run --dsn "$SOURCE" --scale 0.05 # smaller, faster
# 3. Start replicare source -> target (your own config), then generate change:
go run ./test/loadgen run --dsn "$SOURCE" --ops 500 # a burst of I/U/D
# 4. Assert the target converged (retries for replication lag):
go run ./test/loadgen verify --source "$SOURCE" --target "$TARGET" --wait 60s
--dsn/--source/--target take a libpq/pgx URL (postgres://user:pw@host:port/db)
or keyword string; empty uses the standard PG* environment variables.
Commands & flags
run --dsn <src> [--scale F] [--ops N] [--seed S] [--cyclic] [--duration D] [--node-id K]— ensure schema, then seed if empty or churnNstatements.--seedmakes a run reproducible.--duration Dchurns continuously forD(repeated--opsbursts) — run it while replicare does its initial copy to exercise the live-source cyclic-copy skew: parent rows (tenants/users) inserted mid-copy are referenced by children copied from a later snapshot, a transient FK the cyclic copy must recover from rather than crash-loop on. Pair with--cyclic.--node-id K(default 0) selects this writer's key slice for active-active testing — see Active-active; 0 is the default single-writer (active-passive) behaviour, unchanged.ddl --dsn <db> [--cyclic]— apply the schema only (prep the target). One shared schema serves every node;--node-idis arun-only concept (it partitions the data, not the schema).verify --source <src> --target <tgt> [--wait D] [--interval D]— per-table row-count + ordered content-checksum comparison. Exits non-zero on drift.--waitpolls until converged or the timeout. GUCs are pinned to match replicare's transport sorow::textrenders identically on both ends. It is a pure equality check, so it doubles as the active-active convergence oracle — run it between any two mesh nodes (they must hold the identical union).reset --dsn <db> --yes—DROP SCHEMA loadgen CASCADE.
verify compares only the 9 keyed tables (it skips audit_log, which replicare
skips). The GENERATED … STORED column is included and matches because both
servers recompute it.
Active-active (multi-master) load testing
The same harness drives a clusters: mesh. The model is disjoint partitioning:
each writer node gets --node-id K, which shifts every generated key and every FK
reference into a private slice [K·1e9+1, K·1e9+1e9), so nodes never collide on a
key and the mesh converges to the clean union of all slices (predictable counts,
matching checksums). --node-id 0 is the default and is byte-for-byte the
single-writer behaviour above, so active-passive is unaffected.
What --node-id K changes on run:
- Seed writes ids
base+g, order UUIDsmd5('order:'||(base+g)), SKUsSKU-(base+g), and offsets every FK reference bybase, so a node's children point only at that node's own parents. Identity sequences are then restarted past the node's seeded rows so churn inserts stay in the slice. - Churn inserts into the node's slice and scopes every UPDATE/DELETE to node-owned rows, so a node only ever mutates what it wrote — no accidental cross-node conflict (resolving concurrent same-key writes is the separate, planned conflict-storm mode; see below).
--node-id 0 and any --node-id K seeded/churned against the same node's DB must
be consistent — always pass the same --node-id for a given node. Keep --scale
well under ~500M rows per node (the harness refuses a scale that approaches the slice).
Three-node mesh example
Assume a running replicare clusters: mesh over three Postgres nodes A, B, C
(config per multi-master.md), with $A/$B/$C their DSNs.
# 1. Schema on every node (data-only replication; targets must pre-exist).
for n in "$A" "$B" "$C"; do go run ./test/loadgen ddl --dsn "$n"; done
# 2. Seed each node into its own disjoint slice.
go run ./test/loadgen run --dsn "$A" --node-id 0
go run ./test/loadgen run --dsn "$B" --node-id 1
go run ./test/loadgen run --dsn "$C" --node-id 2
# 3. Churn every node concurrently (each in its own slice).
go run ./test/loadgen run --dsn "$A" --node-id 0 --ops 800 --seed 1 &
go run ./test/loadgen run --dsn "$B" --node-id 1 --ops 800 --seed 2 &
go run ./test/loadgen run --dsn "$C" --node-id 2 --ops 800 --seed 3 &
wait
# 4. Convergence oracle: every pair must be identical (all hold the union).
go run ./test/loadgen verify --source "$A" --target "$B" --wait 300s
go run ./test/loadgen verify --source "$A" --target "$C" --wait 300s
Convergence is transitive, so A==B and A==C imply all three agree. A pass ending
CONVERGED: all 9 replicated tables match on every pair — with the daemon logs clean
of HALTED / stream pass error — is a passing active-active load test.
Not yet: conflict-storm mode. The disjoint model above never has two nodes write the same key, so it tests convergence of the union. A follow-up
--overlapmode will deliberately overlap node slices to exercise HLC last-write-wins under concurrent same-key writes; thereverifystill asserts convergence (all nodes equal), but not which value wins (the LWW winner is nondeterministic).
Reset the state store per node when re-seeding (see Gotchas) — this applies per node in a mesh: clear each node's
replicareschema and its state-store entries alongsideloadgen reset.
Running a full load test end-to-end
The commands above assume replicare is already running with "your own config". Here is the complete rig — two Postgres instances, a state store, a config, the daemon, and a heavy-churn convergence loop — that this harness is meant to drive. It is the exact flow used to validate replicare's copy + streaming under load.
1. Two Postgres instances + a state-store database. Any two reachable Postgres servers work; the quickest is two containers. replicare's state store is a third logical database (here, one on the target server):
docker run -d --name lg-src -e POSTGRES_PASSWORD=pw -p 55432:5432 postgres:16
docker run -d --name lg-tgt -e POSTGRES_PASSWORD=pw -p 55433:5432 postgres:16
# state store (replicare's own progress/cursors — NOT your data):
docker exec lg-tgt psql -U postgres -c 'CREATE DATABASE replicare_state'
export SOURCE="postgres://postgres:pw@localhost:55432/postgres"
export TARGET="postgres://postgres:pw@localhost:55433/postgres"
2. A minimal replicare config (loadtest.yaml). drain_interval is short so
streaming keeps up under churn:
logging: { level: info, format: text }
observability: { metrics_addr: ":19090", status_addr: ":18080" }
state_store:
engine: postgres
postgres: { host: localhost, port: 55433, database: replicare_state, user: postgres, password: pw, sslmode: disable }
sources:
src: { engine: postgres, postgres: { host: localhost, port: 55432, database: postgres, user: postgres, password: pw, sslmode: disable } }
targets:
tgt: { engine: postgres, postgres: { host: localhost, port: 55433, database: postgres, user: postgres, password: pw, sslmode: disable } }
syncs:
- name: loadtest
source: src
targets: [tgt]
include: ["loadgen.*"]
tuning: { drain_interval: 1s }
3. Seed the source, create the target schema, start replicare. Capture is enabled first, then the chunked copy runs — so seeding before the daemon starts is fine (the initial copy reproduces it):
go run ./test/loadgen run --dsn "$SOURCE" # seed ~1M+ rows on the source
go run ./test/loadgen ddl --dsn "$TARGET" # create the (empty) target schema
go run ./cmd/replicare run loadtest.yaml & # start the daemon
4. Verify the initial copy converged:
go run ./test/loadgen verify --source "$SOURCE" --target "$TARGET" --wait 120s
# => CONVERGED: all 9 replicated tables match
5. Churn-and-verify loop — this is the actual load test. Each round applies a random burst of inserts/updates/deletes (including PK-changing SKU renames) and asserts the target re-converges while the daemon streams deltas:
for r in 1 2 3 4; do
go run ./test/loadgen run --dsn "$SOURCE" --ops 800 --seed $r
go run ./test/loadgen verify --source "$SOURCE" --target "$TARGET" --wait 120s
done
To stress cross-batch dependencies and delta backlog (what shook out the FK livelocks), churn several times back-to-back before verifying, so a large backlog builds while streaming lags, then converges:
for r in $(seq 1 8); do go run ./test/loadgen run --dsn "$SOURCE" --ops 800 --seed $r; done
go run ./test/loadgen verify --source "$SOURCE" --target "$TARGET" --wait 300s
A run finishing with CONVERGED: all 9 replicated tables match — and the daemon log
showing no HALTED / stream pass error — is a passing load test. For the cyclic
variant, add --cyclic to every ddl and run (see below).
This is a manual test, distinct from
task test:integration(the automated Go suite). Those integration tests must run serially (go test -p 1 ...) because they share schemas; the load test above is driven by hand and has no such constraint.
The --cyclic flag
By default the schema is acyclic. --cyclic adds two nullable FK cycles —
the categories self-reference and a users ↔ orders 2-table cycle — to exercise
replicare's cyclic-copy (null_then_fill) path. Apply it consistently to source
and target (ddl --cyclic, run --cyclic).
Both initial copy and streaming of a cyclic component converge under
--cyclic, with standard non-DEFERRABLE target FKs (the common real-world case). Initial copy loads every table with its cyclic FK columns NULL, in an order that respects the non-cyclic edges, then fills those columns. Streaming uses the same NULL-then-fill idea per pass, but per table: each table's upsert, delete, and cyclic-column fill commit independently, so a parent advances through its own delta queue even when a cross-batch child is transiently blocked (the same progress guarantee the acyclic drain gives), and the cycle itself is closed by a final fill phase. This harness's--cyclicschema drains cleanly under heavy churn with noDEFERRABLErequirement.
Gotchas
- Reset the state store when re-seeding from scratch. replicare records
per-table copy-progress in its state store; if you wipe and re-seed the source
without clearing that state, replicare treats already-
Donetables as copied and skips them (their parents go missing → FK errors on children). Clear thereplicareschema in your state-store database (or use a fresh one) alongsideloadgen reset. - The target schema must exist first — run
ddlagainst the target before starting replicare. - Deletes cascade (
orders → order_items,tenants → everything), so adelete_orders/delete_usersop can remove more rows than it names.
Task shortcuts
task loadgen:ddl DSN="$TARGET" # schema on the target
task loadgen:seed DSN="$SOURCE" SCALE=0.1 # seed / scale
task loadgen:churn DSN="$SOURCE" OPS=500 # a churn burst
task loadgen:verify SRC="$SOURCE" TGT="$TARGET" # convergence check