loadgen-redis — a Redis load + verification harness for replicare
loadgen-redis is the Redis sibling of test/loadgen. It
is a manual, idempotent tool for exercising replicare's Redis engine against
broad, multi-type, high-volume data. You run it against a source Redis; it
seeds a rich keyspace the first time and applies random mutations on every
subsequent run. Point replicare at the source and a target, and use
loadgen-redis verify to assert the target converged.
It is a developer/test tool — not part of the shipped binary. Run it from a
checkout with go run ./test/loadgen-redis (or task loadgen-redis:*).
Unlike the Postgres harness there is no ddl step: Redis has no schema, and
replicare RESTOREs keys straight into the target keyspace, which need not
pre-exist.
What it generates
Keys are laid out as lg:{<bucket>}:<type>:<n>. Everything is chosen to
exercise replicare's Redis paths, not just to hold data:
| Cohort | What it stresses |
|---|---|
str | plain strings; ~20% carry a TTL (volatile keys → TTL replication) |
hash / list / set / zset | every collection type, through DUMP→RESTORE |
stream | streams with consumer groups — the trickiest DUMP/RESTORE payload |
big | deliberately large keys (~100 KB hashes) → the big-key path / MEMORY USAGE gate |
lgskip:* | written to the source only; replicare's include: ["lg:*"] must exclude it — the Redis analog of loadgen's keyless audit_log. verify asserts the target holds none of these. |
The {bucket} hash tag is the cluster story: on a standalone/sentinel server
the braces are inert literal bytes, but on a Redis Cluster they spread the keyspace
across slots/shards (exercising replicare's per-master SCAN fan-out) and keep a
RENAME's old+new key in one slot (a cross-slot RENAME is an error).
The churn mix includes the two cases replicare's Redis engine handles specially:
RENAME= delete-old + create-new — the analog of a PK change, and a stress on the delete-reconciliation sweep.- heavy
DEL— Redis has no delete capture, so deletes are only caught by replicare's target-vs-source diff; this is the genuinely hard path.
At --scale 1.0 (default) the seed writes ~430k keys. Scale down for quick loops:
--scale 0.02 ≈ 8.6k keys in well under a second.
Usage
# 1. Seed the SOURCE (first run seeds; later runs churn).
go run ./test/loadgen-redis run --dsn "$SOURCE" # ~430k keys
go run ./test/loadgen-redis run --dsn "$SOURCE" --scale 0.05 # smaller, faster
# 2. Start replicare source -> target (your own config), then generate change:
go run ./test/loadgen-redis run --dsn "$SOURCE" --ops 500 # a burst of mutations
# 3. Assert the target converged (retries for replication lag):
go run ./test/loadgen-redis verify --source "$SOURCE" --target "$TARGET" --wait 60s
Connection specs
- standalone / sentinel: a
redis://URL, e.g.redis://:pw@localhost:6390/0(empty =redis://localhost:6379/0). - cluster: pass
--clusterand a comma-separated seed list, e.g.--dsn "n1:6379,n2:6379,n3:6379"(use--passwordfor AUTH). One tool, both topologies — see the hash-tag note above.
Commands & flags
run --dsn <src> [--scale F] [--ops N] [--seed S] [--cluster] [--node-id K]— seed if empty or churnNops.--seedmakes a run reproducible (op selection + generated values).--node-id K(default 0) selects this writer's key namespace for active-active testing — see Active-active; 0 is the default single-writer (active-passive) behaviour, unchanged.verify --source <src> --target <tgt> [--wait D] [--interval D] [--cluster] [--node-id K]— compares everylg:*key version-independently: a type-aware canonical content hash (GET/ sortedHGETALL/LRANGE/ sortedSMEMBERS/ZRANGE WITHSCORES/XRANGE+ group last-ids) plus TTL presence. Also asserts nolgskip:*leaked to the target. It is a pure equality check over the wholelg:*keyspace, so it doubles as the active-active convergence oracle (run it between any two mesh nodes — both hold the union).--node-id Kis the source's node index; it scopes the skip-leak check to that source's ownlgskip:*namespace (0 = the classic single-writer check).--waitpolls until converged or the timeout. Exits non-zero on drift.reset --dsn <db> --yes [--cluster]—DELeverylg:*,lgskip:*, and thelgmeta:*marker key (all nodes' namespaces). Scoped by prefix — it never runsFLUSHDB.
Why not compare
DUMPbytes directly? The harness runs an old source (6.2) against a modern target (7.4), whose RDB serializations differ even for identical data (and replicare deliberately preserves each side's own encoding). Soverifycompares a canonical rendering of each value, not its serialized bytes. TTL is compared as presence only (has-TTL vs no-TTL) — the exact remaining seconds legitimately differ by the replication delay.
Active-active (multi-master) load testing
The same harness drives a Redis clusters: mesh. The model is disjoint key
namespaces: each writer node gets --node-id K, which moves every key it writes
into a private segment right after the prefix — lg:n<K>:{<bucket>}:<type>:<n> (and
lgskip:n<K>:… for the source-only cohort). Node segments never overlap, so nodes
never collide on a key and the mesh converges to the clean union of all segments.
--node-id 0 is the default and is byte-for-byte the single-writer layout above
(lg:{<bucket>}:…, no segment), so active-passive is unaffected.
The segment sits after the lg:/lgskip: prefix, so a replicare
include: ["lg:*"] selection still matches every node's data keys and still excludes
every node's lgskip:* keys — no per-node config change is needed.
What --node-id K changes on run:
- Seed writes only this node's namespace (
lg:n<K>:…), and the seeded/empty check scans that namespace alone, so a node re-seeds only when its own slice is empty — peers' replicated-in keys don't count as "already seeded". - Churn sets/updates/deletes only keys in this node's namespace, and a
RENAMEkeeps the new key in the same segment (and hash-tag bucket), so a node only ever mutates what it wrote — no accidental cross-node conflict (resolving concurrent same-key writes is the separate, planned conflict-storm mode; see below).
On verify, --node-id K is the source's node index: it scopes the skip-leak
check to that source's own lgskip:n<K>:* namespace, so a mesh target — which is
itself a source with its own lgskip:* — never false-positives on the peer's
excluded cohort. The content comparison itself is a pure equality check over the
whole lg:* keyspace, so it doubles as the convergence oracle: run it between any two
mesh nodes and both must hold the identical union.
Always pass the same --node-id for a given node's DB. Keep --scale modest per node
(Redis holds everything in RAM, and the mesh target holds every node's union).
Three-node mesh example
Assume a running replicare clusters: mesh over three Redis nodes A, B, C
(config per multi-master.md), with $A/$B/$C their
connection specs. There is no ddl step — Redis has no schema.
# 1. Seed each node into its own disjoint namespace.
go run ./test/loadgen-redis run --dsn "$A" --node-id 0 --scale 0.1
go run ./test/loadgen-redis run --dsn "$B" --node-id 1 --scale 0.1
go run ./test/loadgen-redis run --dsn "$C" --node-id 2 --scale 0.1
# 2. Churn every node concurrently (each in its own namespace).
go run ./test/loadgen-redis run --dsn "$A" --node-id 0 --ops 800 --seed 1 &
go run ./test/loadgen-redis run --dsn "$B" --node-id 1 --ops 800 --seed 2 &
go run ./test/loadgen-redis run --dsn "$C" --node-id 2 --ops 800 --seed 3 &
wait
# 3. Convergence oracle: every pair must be identical (all hold the union).
# --node-id names the SOURCE so the skip-leak check scopes to its lgskip namespace.
go run ./test/loadgen-redis verify --source "$A" --target "$B" --node-id 0 --wait 300s
go run ./test/loadgen-redis verify --source "$A" --target "$C" --node-id 0 --wait 300s
Convergence is transitive, so A==B and A==C imply all three agree. A pass ending
CONVERGED: … no skip leak on every pair — with the daemon logs clean of HALTED /
stream pass error — is a passing active-active load test.
Not yet: conflict-storm mode. The disjoint model above never has two nodes write the same key, so it tests convergence of the union. A follow-up
--overlapmode will deliberately overlap node namespaces to exercise HLC last-write-wins under concurrent same-key writes; thereverifystill asserts convergence (all nodes equal), but not which value wins (the LWW winner is nondeterministic).
Reset the state store per node when re-seeding (see Gotchas) — in a mesh this applies per node: clear each node's state-store entries alongside
loadgen-redis reset(which already removes every node'slg:/lgskip:namespace).
Running a full load test end-to-end
The commands above assume replicare is already running with "your own config". Here is the complete rig — two Redis instances, a Postgres state store, a config, the daemon, and a churn-and-verify loop — that this harness is meant to drive. This is the exact flow used to validate replicare's Redis copy + streaming under load.
1. Two Redis instances + a Postgres state store. The bundled harness is the
quickest source/target pair (an old 6.2 source and a modern 7.4 target); the state
store is Postgres (v1's only StateStore backend — the replicated data is Redis,
the daemon's own progress/cursors live in Postgres):
task harness:redis:up # source on :6390, target on :6391
docker run -d --name lg-state -e POSTGRES_PASSWORD=pw -p 55440:5432 postgres:16
docker exec lg-state psql -U postgres -c 'CREATE DATABASE replicare_state'
export SOURCE="redis://localhost:6390/0"
export TARGET="redis://localhost:6391/0"
2. A minimal replicare config (loadtest-redis.yaml). drain_interval is short
so the reconciliation SCAN + delete sweep keep up under churn:
logging: { level: info, format: text }
observability: { metrics_addr: ":19091", status_addr: ":18081" }
state_store:
engine: postgres
postgres: { host: localhost, port: 55440, database: replicare_state, user: postgres, password: pw, sslmode: disable }
sources:
src: { engine: redis, redis: { host: localhost, port: 6390, db: 0 } }
targets:
tgt: { engine: redis, redis: { host: localhost, port: 6391, db: 0 } }
syncs:
- name: redisload
source: src
targets: [tgt]
include: ["lg:*"] # excludes lgskip:* by construction
tuning: { drain_interval: 1s }
3. Seed the source, start replicare. Redis CDC is capture-less; the initial copy SCANs the source, so seeding before the daemon starts is fine:
go run ./test/loadgen-redis run --dsn "$SOURCE" # seed the source
go run ./cmd/replicare run loadtest-redis.yaml & # start the daemon
4. Verify the initial copy converged:
go run ./test/loadgen-redis verify --source "$SOURCE" --target "$TARGET" --wait 120s
# => CONVERGED: N replicated keys match, no skip leak
5. Churn-and-verify loop — this is the actual load test. Each round applies a
random burst of mutations (including RENAMEs and heavy DELs) and asserts the
target re-converges while the daemon streams:
for r in 1 2 3 4; do
go run ./test/loadgen-redis run --dsn "$SOURCE" --ops 800 --seed $r
go run ./test/loadgen-redis verify --source "$SOURCE" --target "$TARGET" --wait 120s
done
To stress the delete-reconciliation sweep and streaming backlog, churn several times back-to-back before verifying, so deletes and renames pile up while the sweep runs, then converge:
for r in $(seq 1 8); do go run ./test/loadgen-redis run --dsn "$SOURCE" --ops 800 --seed $r; done
go run ./test/loadgen-redis verify --source "$SOURCE" --target "$TARGET" --wait 300s
A run finishing with CONVERGED: … no skip leak — and the daemon log showing no
HALTED / stream pass error — is a passing load test. Watch
replicare_deletes_reconciled_total on :19091/metrics climb as DEL/RENAME
churn is reconciled.
Cluster mode
The same tool drives a Redis Cluster target. Bring up the cluster harness and pass
--cluster plus a seed list:
task harness:redis:cluster:up
go run ./test/loadgen-redis run --dsn "127.0.0.1:7000,127.0.0.1:7001,127.0.0.1:7002" --cluster --scale 0.1
go run ./test/loadgen-redis verify --source "…standalone source…" --target "127.0.0.1:7000,…" --cluster --wait 120s
The {bucket} hash tags spread keys across the cluster's slots (so the per-master
SCAN fan-out is genuinely exercised) while keeping each RENAME intra-slot.
Gotchas
- Reset the state store when re-seeding from scratch. replicare records
per-unit copy-progress in its state store; if you wipe and re-seed the source
without clearing that state, it may treat the unit as already copied. Use a fresh
replicare_statedatabase (or clear thereplicareschema in it) alongsideloadgen-redis reset. RENAMEneeds an existing source key. The tool renames within the seeded string range and silently skips a key that a prior op already renamed/deleted — benign for the harness.lgmeta:countsis a small bookkeeping key (per-type seeded counts, so churn can address existing keys without scanning). It is neitherlg:*norlgskip:*, so replicare andverifyboth ignore it;resetremoves it.
Task shortcuts
task loadgen-redis:seed DSN="$SOURCE" SCALE=0.1 # seed / scale
task loadgen-redis:churn DSN="$SOURCE" OPS=500 # a churn burst
task loadgen-redis:verify SRC="$SOURCE" TGT="$TARGET" WAIT=60s
task loadgen-redis:reset DSN="$SOURCE" # add CLUSTER=1 for a cluster