replicare
replicare is an open-source data replication daemon. Point it at a source database and one or more target databases: it makes an initial copy of the selected data, then keeps replicating inserts, updates and deletes so the targets stay in step with the source.
It is written in Go and ships as a single static binary. Postgres, MySQL and Redis are supported, each replicating to the same engine. The source code is on GitHub.
Why it's different
- No privileged change stream required. replicare does not need the Postgres WAL, the MySQL
binlog or a Redis change feed. On Postgres and MySQL it captures changes with ordinary,
grantable privileges using triggers; on Redis it uses
SCANreconciliation and writes nothing to the source. So it works on managed databases and old servers where logical replication isn't available. - Least privilege. On the source it needs
CREATEon the database (or a pre-created schema it owns) plusUSAGE,TRIGGERandSELECT. On the target, justSELECT,INSERT,UPDATEandDELETEon tables that already exist. See the grant scripts. - Faithful transport, never transform. Values move verbatim and are validated by the target database itself. replicare never coerces, truncates or "repairs" data: on a type or constraint error it stops loudly instead of corrupting silently.
- Handles very large and very old sources. Change capture starts first, then a chunked, parallel copy runs while changes queue up and reconcile, so there is no single frozen snapshot and "too big to snapshot" isn't a problem. Source-side SQL stays compatible back to Postgres 9.x.
- Observable. Prometheus metrics, a status HTTP API, structured logs and OpenTelemetry traces,
plus a
statuscommand that works without any monitoring stack.
Where to start
- Getting started — the pre-migration checklist, then from the demo to your own databases.
- Architecture — how the pipeline works end to end, and the four ideas that make it non-obvious.
- Configuration reference — every config field, with types and defaults.
- CLI reference — every command and flag, with examples.
- Operations — the migration checklist, tuning, health signals, vacuum and maintenance, retention and reseed, restarts.
- Kubernetes (Helm) and the Helm chart reference.
- Monitoring — the Grafana dashboard and scraping metrics with an OpenTelemetry Collector or Prometheus.
- Verifying data integrity — standalone source↔target checks for when the
verifycommand can't be run: Postgres · MySQL · Redis. - Troubleshooting — common problems and fixes.
Engines
The shared pages describe the relational baseline using Postgres; each engine page covers that engine's specifics.
- Postgres — the reference engine: trigger capture, faithful
COPY, foreign-key groups, cyclic and self-referencing tables, least-privilege grants. - MySQL — MySQL→MySQL: InnoDB and
local_infilerequirements and the two operational wrinkles. - Redis — Redis→Redis: capture-less
SCANreconciliation, the delete sweep, value-faithfulDUMP/RESTORE, TTL, cluster and ACLs.
Active-active
- Active-active: operating rules — the rules a multi-node setup must follow, in plain terms.
- Multi-master design — the
clusters:config, loop suppression and last-write-wins conflict resolution, shipped for Postgres, MySQL and Redis.
Elsewhere in the repository
Deployment files live in
deploy/: a sample systemd unit, the
least-privilege grant SQL and the Redis ACL presets. Runnable end-to-end demos are in
examples/
(Postgres,
MySQL,
Redis); one config that runs
a Postgres and a Redis pipeline together is
examples/postgres-and-redis.yml.
Load-and-verify harnesses that drive heavy write traffic and check the target converges are
documented for Postgres, MySQL and
Redis; see
Operations → Load & convergence testing.
Getting started
This walks you from zero to a running replication in a few minutes, then explains how to point replicare at your own databases.
What replicare does
You give replicare a source database and one or more target databases and a list of tables. It:
- installs lightweight capture (triggers) on the source,
- copies the selected data to the targets (chunked and parallel), then
- streams every insert/update/delete continuously to keep the targets converged.
No WAL access, no REPLICATION role, no wal_level change — just ordinary,
grantable privileges. Architecture explains why.
Try the demo (2 minutes)
The fastest way to see it work end to end:
task docker # build the replicare image
cd examples/demo && docker compose up
This starts a source Postgres (seeded with public.orders), a target Postgres,
and replicare copying then streaming between them. In another shell:
# initial copy landed
docker compose exec target psql -U postgres -d warehouse -c "SELECT count(*) FROM public.orders;"
# make a live change on the source and watch it propagate
docker compose exec source psql -U postgres -d app \
-c "INSERT INTO public.orders VALUES (101,'live'); UPDATE public.orders SET note='changed' WHERE id=1; DELETE FROM public.orders WHERE id=2;"
docker compose exec target psql -U postgres -d warehouse -c "SELECT count(*), max(id) FROM public.orders;"
# operator surface
curl localhost:8080/status | jq
curl localhost:9090/metrics | grep replicare_
Ctrl-C then docker compose down -v to clean up.
Verifying the metrics
/metrics shows many replicare_* series once a sync is streaming, not just
one. To eyeball the series that only populate at runtime:
curl -s localhost:9090/metrics | grep -E \
'replicare_(rows_copied_total|apply_batch_seconds|throughput_rows_per_second|replication_lag_seconds|table_phase_info|target_up)'
replicare_table_phase_info(one line per table,initial_copythenstreaming),replicare_target_upand the row estimatereplicare_initial_copy_rows_targetappear as soon as the sync starts;replicare_rows_copied_totalclimbs as the initial copy commits rows.replicare_apply_batch_secondsandreplicare_replication_lag_secondsrefresh on every streaming drain pass.replicare_throughput_rows_per_secondis only non-zero on a pass that actually applied deltas — make a change on the source (theINSERT/UPDATE/DELETEabove) and re-scrape; a caught-up idle sync correctly reports0here.
The full list of every metric, its type, and its labels is in operations.md.
Point it at your own databases
0. Run the pre-migration checklist
Before pointing replicare at a production source, gather the facts below and confirm the must-haves. The queries for each item, for Postgres and MySQL, are in operations.md → Before a migration: checklist.
Sizing and traffic
- Total database size, and row counts and sizes per table.
- Write volume over 24 hours (inserts, updates and deletes): every change adds one row to replicare's change tracking on the source.
- Daily and weekly low-traffic windows, for catching up and for the final cutover.
Must be true before starting
-
Every table has a primary key or a
UNIQUEconstraint (tables without one are skipped). -
Postgres: every extension the tables use exists on the target. MySQL: every table is InnoDB,
the source is 5.7 or newer (older sources cannot add replicare's triggers next to a table's
own), and the target has
local_infileon. - Materialized views and views are not replicated: plan to create and refresh them on the target. Postgres large objects are not replicated either.
-
Schema changes and migrations are frozen for the whole replication (including online
schema-change tools such as
pt-online-schema-change/gh-oston MySQL). -
Regular vacuum and maintenance jobs are known: plain
VACUUM/ autovacuum is fine; heavy jobs that lock whole tables (VACUUM FULL,CLUSTER,pg_repack, MySQLOPTIMIZE TABLE) are paused during the initial copy and kept out of peak hours afterwards. -
No long-open (
idle in transaction) sessions on the source. - Source and target versions checked (target the same major version or newer), grants in place, and disk room on the source for the change-tracking tables to grow.
-
A plan for sequence / identity counters at cutover (
replicare reseed-sequences).
1. Prepare the target schema
replicare replicates data only — it never creates or migrates target tables. Create/migrate the target tables yourself first, matching the source columns you want to replicate. (Types must be compatible; pre-flight will tell you if not.)
The quickest way to match the source is to dump its structure only (no data) to a file, then load that file on each target.
# Postgres — 1) dump the source structure (no ownership/privilege noise)
pg_dump --schema-only --no-owner --no-privileges -h SRC_HOST -p SRC_PORT -U USER -d DB > schema.sql
# 2) load it into the target
psql -h TARGET_HOST -p TARGET_PORT -U USER -d DB -f schema.sql
# narrow it: -n public (one schema), -t public.orders (one table)
# MySQL — 1) dump (-P is the PORT; lowercase -p is the password prompt)
mysqldump --no-data -h SRC_HOST -P SRC_PORT -u USER -p DB > schema.sql
# 2) load
mysql -h TARGET_HOST -P TARGET_PORT -u USER -p DB < schema.sql
Credentials. These tools ask for a password when the server requires one: Postgres
prompts interactively (and mysqldump -p means "prompt me"). Don't put the password in
the command — it leaks into your shell history and ps. Instead store it in ~/.pgpass
(Postgres, chmod 600) or ~/.my.cnf (MySQL). If you prefer a one-step pipe
(pg_dump … | psql …), note it cannot prompt — the password step and the data stream
collide — so set up ~/.pgpass / PGPASSWORD first, or stick with the two-step form above.
For active-active, run this against every node before starting (and apply any later schema change to every node first — see active-active.md).
2. Grant least-privilege roles
Run the grant scripts (edit the role/schema/db values, or pass them with -v):
psql -v ON_ERROR_STOP=1 -v role=replicare -v db=app -v app_schema=public \
-f deploy/grants-source.sql "host=SOURCE dbname=app user=admin"
psql -v ON_ERROR_STOP=1 -v role=replicare -v db=warehouse -v app_schema=public \
-f deploy/grants-target.sql "host=TARGET dbname=warehouse user=admin"
The source role needs only CREATE-on-db (or a pre-created owned replicare
schema) plus USAGE/TRIGGER/SELECT; the target role needs only DML. See
configuration.md and
../deploy/ for details.
For MySQL, use ../deploy/grants-source-mysql.sql
and ../deploy/grants-target-mysql.sql
instead: the source role needs SELECT+TRIGGER on the replicated tables and
ownership of the replicare capture database; the target role needs the four DML
verbs plus CREATE TEMPORARY TABLES. See the MySQL engine page.
For Redis, use the ACL presets
../deploy/acl-source-redis.txt and
../deploy/acl-target-redis.txt: the source role
is read-only (+scan +dump +pttl +exists +type +memory|usage); the target role
needs +restore +del +scan +exists +module|list. Note the foot-gun — RESTORE is
@dangerous and must be granted explicitly with +restore. See
the Redis engine page.
3. Write a config
Copy ../examples/replicare.yml and edit the
connection blocks and the sync's table selection. Minimal example:
state_store:
engine: postgres
postgres: { host: db, port: 5432, database: replicare_state, user: replicare, password: "${PW}", sslmode: require }
sources:
app:
engine: postgres
postgres: { host: app-db, port: 5432, database: app, user: replicare, password: "${PW}", sslmode: verify-full }
targets:
warehouse:
engine: postgres
postgres: { host: wh-db, port: 5432, database: warehouse, user: replicare, password: "${PW}", sslmode: require }
syncs:
- name: app-to-warehouse
source: app
targets: [warehouse]
include: ["public.*"]
exclude: ["*_audit"]
Choosing tables. include and exclude are lists of schema.table glob
patterns, and exclude is applied on top of include. To drop one specific table,
name it in full under exclude:
include: ["public.*"] # everything in public…
exclude: ["public.audit_log"] # …except this one table
Patterns work too — *_audit excludes every table whose name ends in _audit. A few
notes: selection is read at daemon start, so change it and restart; excluding a table
only stops replicating it (rows already on the target are left in place, not deleted); and
if another replicated table has a foreign key pointing at the excluded one, replicare warns
about the dangling reference (the target must already satisfy that parent). For Redis,
selection uses key patterns instead of schema.table.
Secrets like ${PW} are read from the environment. Full field reference:
configuration.md.
For a MySQL sync, set engine: mysql with mysql: blocks (see the
MySQL connection block); the state
store stays on Postgres. A ready-to-run example is in
../examples/demo-mysql/config.yml.
For a Redis sync, set engine: redis with redis: blocks (see the
Redis connection block); the state store
still stays on Postgres (Redis writes nothing to the source, so all its durable
state lives there). Selection uses key-pattern globs (include: ["app:*"])
rather than schema.table. See the Redis engine page.
4. Validate, then run
export PW=... # the replicare role's password
replicare validate config.yml # pre-flight: no changes, reports problems
replicare run config.yml # start replicating (Ctrl-C / SIGTERM to stop cleanly)
validate connects read-only and classifies every column pair — fix any BLOCK
findings (incompatible/missing types) before running. Once running, check
progress any time with replicare status config.yml.
A few more commands you'll reach for later: replicare capture install|remove to
pre-provision (or tear down) the source-side trigger capture out of band,
replicare reseed --sync <s> --target <t> to force a laggard or diverged target
to re-copy its data, and — when you promote a one-way (DR) target —
replicare reseed-sequences --sync <s> to advance its identity/sequence counters to
max(id)+1 before it takes writes (otherwise nextval() collides with the copied rows;
passive/DR only — it refuses an active-active mesh). See the CLI reference.
5. Deploy
Install the binary and the sample systemd unit, run the Docker image, or deploy on Kubernetes with the Helm chart — see Kubernetes (Helm). See operations.md for tuning, monitoring, retention/reseed, and troubleshooting.
Next
- Configuration reference — every config field.
- CLI reference — every command.
- Operations — tuning, health signals, retention, restarts.
- Troubleshooting — common problems and fixes.
- Architecture — how the pipeline works end to end (capture → copy → stream → apply → checkpoint).
- Postgres engine — the reference engine: trigger CDC, faithful
COPY, FK components, least-privilege grants. - MySQL engine — MySQL→MySQL specifics, plus a runnable MySQL demo.
- Redis engine — Redis→Redis specifics: SCAN reconciliation, the delete sweep, TTL, cluster, and ACLs.
Architecture
How replicare moves data, end to end: the overview for newcomers and architects.
Everything here is
engine-neutral unless noted — Postgres, MySQL, and Redis all ride the same
pipeline, differing only behind the Source/Sink driver interfaces.
The one non-negotiable
replicare assumes it will never have a privileged, durable change stream — no Postgres WAL, no MySQL binlog, no durable Redis change feed. Change capture must work with ordinary, grantable privileges. That single constraint shapes everything below.
The pipeline
A sync (one source → one or more targets, with a table/key selection) runs this lifecycle. Copy and streaming are the two phases; capture and checkpointing wrap them.
install capture ─► chunked parallel COPY ─► cut over ─► stream deltas ─► (repeat)
(first) (initial copy) (continuous)
└──────────────── checkpoint to the StateStore throughout ─────────────┘
- Capture is installed FIRST (relational engines), so changes begin queuing before and during the copy. For Redis there is nothing to install — capture is replaced by re-reading the live keyspace (see Engines).
- Initial copy streams each table/unit in chunks, in parallel, source →
target through an
io.Pipethat never fully buffers a chunk. Because capture is already running and apply is idempotent, any change during the copy window reconciles automatically — there is no single frozen snapshot, which is how replicare handles sources too large to snapshot. - Cutover flips each table to streaming once its copy completes.
- Streaming drains changes continuously and applies them idempotently, forever, until the daemon is stopped.
- Checkpointing records progress in the StateStore at every step, so a crash resumes from the last confirmed point (applies are idempotent, so replay is harmless).
The four ideas that make it non-obvious
Most of replicare's design collapses into four decisions. Understand these and the rest follows.
1. Capture is a dirty-key set, not an ordered log
Triggers on the source record only the changed row's primary key (PK-only capture) into a per-table delta table — never the full row image. At sync time replicare reads the unconsumed keys, re-reads the current values from the live source table, and applies those. This is self-healing: it always copies the latest value, so multiple changes to one key coalesce into one re-read and transient intermediate states don't matter.
Crucially, consumption is a set-difference (delta MINUS track, per target),
not a WHERE seq > cursor scan. A scalar high-water-mark cursor is broken
here: a value a trigger assigns inside a transaction (a sequence, a txid) is fixed
when the trigger fires, but the delta row only becomes visible when the transaction
commits — and commits happen in a different order than those values were grabbed.
A slow transaction holding a low sequence that commits after the daemon's read
would be skipped forever. The set-difference model can't skip: an uncommitted or
late-committing delta is simply "not yet in track," caught on the next pass. Rows
are deleted by unique delta_id (never by PK) to survive the read-and-clear race.
2. FK connected components are the unit of ordering, parallelism & consistency
Within a sync, replicare auto-partitions the selected tables into FK connected components (from the source catalogs, over the selected tables only). No FK edge crosses a component boundary, which gives three properties for free:
- Ordering is always complete — upserts apply parent→child, deletes child→parent, within a component; no child is ever stranded from its parent. Cycles and self-references can't be topo-sorted, so a retry fallback handles them.
- Distinct components are referentially independent → safe to copy and stream fully in parallel.
- A component's changes for one drain pass apply in a single target transaction, so the target is always referentially consistent within the group.
A "giant component" (everything wired through a users/tenant hub table) erases
parallelism, so replicare warns and lets you override.
3. Faithful transport — never transform
replicare is a pipe, not an interpreter. Values move verbatim: the source server renders them, the target server's own input function validates them. It never coerces, truncates, casts, or "repairs" data — on a type/constraint error it halts loud rather than corrupting silently. This is a hard product promise: no value-transform capability exists, even as an opt-in.
- Relational engines move values as text
COPY(both initial copy and delta apply), pinning formatting-only session settings (e.g.extra_float_digits=3) so text round-trips losslessly across version gaps. - Redis moves values as
DUMP→RESTORE— RDB-serialized bytes, value-faithful (the target re-encodes), never reconstructed. - A pre-flight check classifies every source↔target type pair up front: block on incompatible/missing, warn on risky-lossy, ok on identical/widening — turning mid-stream failures into an actionable report before any sync starts.
4. The StateStore is separate from source-side capture
Two different kinds of state, often confused:
- Source-side capture state — the delta and track tables — lives on the source database (that's where triggers write). It is the correctness-bearing replication state.
- The StateStore — which syncs exist, per-table copy progress, per-target cursors, and the single-active ownership lock — is the daemon's own operational bookkeeping. It lives in a dedicated schema on a Postgres database (v1's only StateStore backend), which may be the target, the source, or a separate server.
They are separate concerns even when both happen to be Postgres. One consequence: every sync needs a Postgres StateStore — including a MySQL→MySQL or Redis→Redis sync (a documented wart for the non-Postgres engines).
How the three engines map on
The pipeline is identical; only the driver behind Source/Sink changes.
| Postgres (reference) | MySQL | Redis | |
|---|---|---|---|
| Capture | trigger CDC → delta/track tables | trigger CDC (same model) | none — capture-less |
| Upsert CDC | re-read dirty PKs | re-read dirty PKs | full-keyspace SCAN reconciliation (re-read every pass) |
| Delete CDC | delta row for the deleted PK | delta row | durable target-vs-source keyspace diff |
| Transport | text COPY | LOAD DATA LOCAL INFILE | DUMP → RESTORE REPLACE |
| Unit | FK component of tables | FK component of tables | one key-namespace unit (redis.db<N>), single-member |
| Apply atomicity | per-component transaction | per-component transaction | per-key (no cross-key atomicity) |
| Delivery | at-least-once; effectively-exactly-once when the StateStore is the target | at-least-once | at-least-once |
Redis is the outlier: with no change stream at all, upserts come from re-reading the whole keyspace each pass and deletes from diffing the target against the source (keyspace notifications are an optional, lossy accelerator, never trusted). See the Redis engine page for the full story, and the Postgres / MySQL pages for the relational specifics.
Delivery, topology & bloat
- Delivery: at-least-once with idempotent apply (upserts keyed on PK/key, deletes by PK/key), checkpointed cursors. A crash replays from the last cursor; replays are harmless.
- Topology: user choice — default single→single; fan-out (one source → many targets) is supported for the relational engines; multi-master and HA/leader election are on the roadmap. A sync is always single-engine.
- Source bloat: the delta tables are a high-churn queue on a source you may not own, so replicare bounds its footprint with per-table delta tables, a batched consumption-gated purge + aggressive autovacuum, and bounded retention with forced reseed of an over-cap target. See operations and the reseed state machine.
Process & observability
One daemon runs many named syncs as goroutine worker pools (one process, central
observability). Everything is visible across four channels at once: a
Prometheus /metrics endpoint, OpenTelemetry traces, structured slog logs, and a
/status HTTP API. A degrading target must surface in metrics, logs, and traces
simultaneously — see operations for the metric/span/event
reference.
Configuration reference
replicare is configured by a single YAML file (passed to every command). The schema is an engine-neutral envelope plus a typed per-engine connection block.
Any string value may reference the environment as ${VAR} (required — errors if
unset) or ${VAR:-default} (falls back to default) — not just secrets. Expanded
scalars are re-typed on decode, so it works for ports, hosts, addresses, and
booleans too (e.g. port: ${PGPORT:-5432}), matching the "env-var overrides for any
field" goal. Keeping passwords out of the file is the common case.
An unset ${VAR} with no default is a hard config-load error (exit 2).
Validate a config without connecting-and-running via replicare validate <config>.
Top-level structure
logging: { ... } # log level/format
observability: { ... } # metrics/status/OTLP endpoints
state_store: { ... } # replicare's own progress store (an endpoint)
sources: { ... } # named source endpoints
targets: { ... } # named target endpoints
syncs: [ ... ] # what replicates where (one-way)
nodes: { ... } # peer endpoints for active-active clusters (optional)
clusters: [ ... ] # active-active (multi-master) groups (optional)
nodes and clusters are optional and additive — a config with neither behaves
exactly like a one-way daemon. See Active-active clusters
and the multi-master design note.
logging
| Field | Type | Default | Notes |
|---|---|---|---|
level | debug|info|warn|error | info | |
format | json|text | json | text is friendlier for local runs |
observability
| Field | Type | Default | Notes |
|---|---|---|---|
metrics_addr | listen addr, e.g. :9090 | off | serves Prometheus /metrics |
status_addr | listen addr, e.g. :8080 | off | serves /status, /healthz, and /metrics |
otlp_endpoint | host:port, e.g. otel:4317 | off | exports OTel traces to an OTLP/gRPC collector |
stall_timeout | duration (2m, 0, negative) | 2m | how long a streaming loop may go without completing a pass before /healthz reports unhealthy (so Kubernetes restarts a wedged pod). Must exceed your slowest drain pass. Unset/0 = 2m; a negative value disables the check |
Leaving an address empty disables that endpoint. When metrics_addr and
status_addr differ, /metrics is served on both.
/healthz reflects streaming liveness, not target reachability: a loop that
is cycling — even one retrying against a down target — stays healthy; only a loop
that stops completing passes (a hung query on a dropped socket) trips
stall_timeout and fails the probe. The daemon also reconnects a dropped
source/target connection on its own between passes, so a transient blip recovers
without a restart; the probe is the backstop for a true wedge.
Endpoints: state_store, sources, targets
An endpoint is an engine name plus a typed connection block. sources and
targets are maps of name → endpoint; state_store is a single endpoint.
sources:
app: # the name referenced by a sync
engine: postgres
postgres: # the engine-specific block
host: app-db
port: 5432
database: app
user: replicare
password: "${APP_PW}"
sslmode: verify-full
params: # optional extra connection params
application_name: replicare
Postgres connection block
Used when an endpoint's engine is postgres. See the Postgres engine page
for the CDC model, FK-component handling, faithful COPY transport, and the
effectively-exactly-once bonus when the state store is the target.
| Field | Type | Notes |
|---|---|---|
host | string | |
port | int | usually 5432 |
database | string | |
user | string | the least-privilege replicare role |
password | string | use ${VAR} |
sslmode | disable|allow|prefer|require|verify-ca|verify-full | TLS mode (libpq semantics) |
params | map | extra key=value connection parameters |
MySQL connection block
Used when an endpoint's engine is mysql. See the MySQL engine page
for the two operational wrinkles (a MySQL sync still keeps its state store on
Postgres; MySQL syncs are strictly at-least-once).
| Field | Type | Notes |
|---|---|---|
host | string | |
port | int | default 3306 |
database | string | the DSN default database only — selection may reference other databases as db.table (a MySQL schema is a database) |
user | string | the least-privilege replicare role |
password | string | use ${VAR} |
tls | disable|allow|prefer|require|verify-ca|verify-full | TLS mode (same spectrum as Postgres sslmode); default prefer |
local_infile | bool | hint that the target permits LOAD DATA LOCAL INFILE, the copy/apply transport (required in v1). replicare probes it at connect and halts loud if off — no INSERT fallback yet. A server system variable, not a grant |
params | map | extra key=value DSN parameters |
Redis connection block
Used when an endpoint's engine is redis. See the Redis engine page
for how Redis CDC works (SCAN reconciliation + a target-vs-source delete sweep, not
triggers) and its two wrinkles (a Redis sync still keeps its state store on
Postgres; copy progress is coarse). Redis is topology-shaped rather than a single
host, and carries engine-specific selection and CDC tuning.
| Field | Type | Notes |
|---|---|---|
mode | standalone|cluster|sentinel | default standalone; sentinel is parsed but experimental (failover not hardened in v1) |
host / port | string / int | standalone seed (port default 6379) |
nodes | list of host:port | seed nodes for cluster/sentinel (also allowed for standalone) |
db | int | logical DB index (standalone only; must be 0 in cluster) |
user / password | string | ACL user (Redis 6+); use ${VAR} for the password |
tls | disable|allow|prefer|require|verify-ca|verify-full | same spectrum as Postgres sslmode; default disable — note this diverges from Postgres/MySQL (prefer), so set it explicitly to avoid plaintext |
sentinel_master | string | required when mode: sentinel |
read_from_replica | bool | offload value reads to replicas; delete detection still reads the master (a lagging replica would cause false deletes) |
types | list | optional type filter: string/list/set/zset/hash/stream (needs Redis 6.0+ for server-side SCAN … TYPE) |
scan_count | int | SCAN COUNT hint per call (default 512); distinct from the neutral drain batch |
reconcile_interval | duration | cadence of the rolling upsert reconciliation scan |
delete_sweep_interval | duration | cadence of the target-vs-source delete sweep (bounds delete-propagation latency) |
big_key_warn_bytes | size | DUMP-and-warn above this (MEMORY USAGE); 0 = off |
big_key_refuse_bytes | size | block loud above this; must be ≥ big_key_warn_bytes |
notifications | bool | enable the lossy keyspace-notification latency accelerator (default off; never trusted for correctness) |
ttl_mode | relative|absttl | relative (default) is clock-skew-safe; absttl is opt-in for trusted clocks |
params | map | extra engine-internal params |
Selection for a Redis sync is key-pattern globs (Redis SCAN … MATCH
semantics), not schema.table — see Selection below.
state_store is where replicare keeps its own operational state (sync
progress, cursors, the ownership lock) — a dedicated replicare_state schema it
creates and owns. It may point at the target DB, the source DB, or a separate
Postgres. (This is distinct from the source-side delta/track tables, which always
live on the source. Redis writes nothing to the source at all, so a Redis sync's
only durable state is here — see the Redis engine page.)
syncs
A sync is one replication job: a source, one or more targets, and a table selection. All targets of a sync must use the same engine as the source (never cross-engine).
Fan-out (a sync with several targets) copies and streams each target
independently: initial-copy progress is checkpointed per (sync, target, table),
so one slow or restarted target never disturbs another's watermark, and each
converges on its own. (This is also the per-node building block of an
active-active cluster.)
syncs:
- name: app-to-warehouse # unique; also the ownership-lock key
source: app # a key in `sources`
targets: [warehouse] # keys in `targets` (one or more; fan-out)
enabled: true # optional; false PAUSES just this sync (see below)
sync_sequences: false # optional; true keeps DR counters warm (see below)
include: ["public.*"] # selection globs (schema.table)
exclude: ["*_audit"] # excluded from the include set
tuning: { ... } # optional (see below)
Keeping DR counters warm (sync_sequences)
sync_sequences is an opt-in, one-way/DR-only switch (default off). With it on, the
daemon advances each replicated table's owned identity/serial/AUTO_INCREMENT counter on
the target to max(id)+1 on a lazy interval during streaming — the same mechanism as the
reseed-sequences
command, kept running — so a promoted DR node has nothing sequence-related to do at cutover.
Enabling it adds a steady-state target grant (Postgres UPDATE on each sequence, MySQL
ALTER on each table); a missing grant is logged and retried, never fatal. It is refused
at config load when the target is also an active-active cluster member (a mesh counter
advance is silent data loss — use UUID/ULID keys there). See
operations → Sequences & identity counters.
Pausing a sync (enabled)
enabled is a per-pipeline pause switch. It is a pure opt-out: omit it (or set
true) and the sync runs — a config written before this field existed is
unchanged. Set enabled: false to pause that one sync; other syncs (and
clusters) in the same config keep running.
enabled | Behavior |
|---|---|
unset / true | The daemon brings the sync up and streams it (default). |
false | The daemon skips it at startup and does not take its ownership lock. No copy, no streaming, no capture install for a never-started sync. |
Pause is data-loss-free, by design. If a sync was already running when you pause
it, its source capture triggers are left in place, so change deltas keep queuing
on the source while it's paused. Setting enabled: true again (and restarting) resumes
it and drains the accumulated backlog — the target catches up with nothing lost.
The tradeoff: while paused, the source-side delta tables keep growing, and the
retention/reseed source-protection does not run (it lives in the
streaming loop, which is skipped). So a long pause on a busy source can grow the
source's replicare schema unbounded. Pause for maintenance/cutover windows, not
indefinitely; if you need to stop for a long time, prefer removing the sync (which you
can teardown so capture is uninstalled) over an open-ended pause.
It takes effect at daemon start, so pausing or resuming is a config change +
restart, not a live toggle. On Kubernetes/Helm, change the value and helm upgrade —
the resulting pod rollout applies it. Set enabled on the sync by name in your
config; if you use Helm --set, note it addresses syncs by array index
(config.syncs[0] = the first sync), so with several pipelines it's easy to pause the
wrong one — see kubernetes.md.
Verify which sync actually paused in the daemon logs — sync disabled (paused); skipping sync=<name> for a paused sync, sync streaming sync=<name> for a running one —
or with replicare status (the [PAUSED] banner). (Clusters have the same enabled
flag; setting it false pauses all of that cluster's mesh edges.)
Pausing every pipeline is fine — the daemon stays up. With all syncs (and clusters)
paused the daemon does not exit; it idles, keeps serving /metrics, /status, and
/healthz, and stays a healthy pod (so a Kubernetes Deployment does not restart-loop it).
replicare status then shows each paused sync with a [PAUSED] banner (and, live,
its still-growing delta backlog) rather than a stale "streaming" phase.
Selection
include/exclude are schema.table globs (* matches within a name segment).
For MySQL, a schema is a database, so these are db.table globs and a single
sync may span several databases. Tables without a primary key or usable unique key
are skipped with a warning (they can't be captured). An FK pointing from a
selected table to an excluded one triggers a dangling-FK warning — the target
must already satisfy that parent.
For Redis, include/exclude are key-pattern globs using Redis SCAN … MATCH semantics (*, ?, […], hash-tags), matched exclude-wins. There are
no tables, keys, or FKs — the "unit" is the logical keyspace (one DB, or DB 0 in
cluster). An optional types filter in the redis: block narrows selection to
specific value types.
tuning (per sync)
| Field | Type | Default | Meaning |
|---|---|---|---|
drain_interval | duration (1s, 500ms) | 1s | time between streaming drain passes; longer = more coalescing (less source load) but higher lag |
drain_batch | int | 1000 | max dirty deltas applied per table per pass; with drain_interval this is the per-table streaming ceiling (~drain_batch/drain_interval rows/s). Raise for a high-volume table that lags |
apply_concurrency | int (≥1) | 1 | how many of a component's tables apply concurrently during streaming. 1 is strictly sequential; higher fans the per-table apply across the copy-worker pool (bounded by pool.max_*_connections). Helps when several tables are backlogged at once |
retention.max_age | duration (24h, 0 = off) | 24h | oldest unconsumed delta before a laggard target is reseeded |
retention.max_bytes | size (512MB, 0 = off) | off | delta-table on-disk size before reseed |
pool.max_source_connections | int | 4 | source connection cap (copy worker pool is sized from it) |
pool.max_target_connections | int | 4 | target connection cap |
Durations use Go syntax (ms, s, m, h); sizes accept KB/MB/GB/TB
(×1000) or KiB/MiB/GiB (×1024), or a bare byte count. Defaults favor low
source pressure — see operations.md for tuning guidance.
For Redis, the neutral retention.* knobs are inert (there is no source-side
delta queue to bound); Redis pacing lives in the redis: block instead
(scan_count, reconcile_interval, delete_sweep_interval).
Active-active clusters (nodes and clusters)
Status: in progress. For Postgres and MySQL, a
clusters:block runs today with full conflict resolution: the daemon brings up a full-mesh active-active cluster with loop suppression (MM3) and HLC last-write-wins (MM4 for Postgres, MM5 mirrors it for MySQL), so writes accepted on any node — including concurrent writes to the same key — converge to the same value on every node, with no config and no user schema change; deletes resolve via GC'd tombstones. Redis clusters are still parse-and-validate only. See the design note for the full status. These keys are optional and additive: omit them and replicare behaves exactly as a one-way daemon. Do not confuse a replicareclusters:entry (a group of active-active peer databases) with a Redis endpoint'sredis.mode: cluster(one sharded Redis) — different scopes.
An active-active cluster keeps N peer databases converged with writes accepted on any node. Conflict resolution is zero-config — replicare manages a hidden per-row version itself (HLC last-write-wins; no column to add, nothing to declare); see the design note. A cluster is single-engine, like a sync.
nodes
A map of peer endpoints. A node has the same shape as a source/target endpoint (an
engine + that engine's connection block) plus an optional node_id. Unlike a
source/target, a node is both read from and written to.
| Field | Type | Default | Notes |
|---|---|---|---|
engine | string | — | postgres | mysql | redis; must match the cluster's engine |
node_id | string | the map key | stable replication-origin identity, unique within a cluster; stamped into each change's version. Usually leave it to default to the key |
<engine>: | block | — | the engine connection block, exactly as for a source/target |
clusters
A list of active-active groups.
| Field | Type | Default | Notes |
|---|---|---|---|
name | string | — | unique across all syncs and clusters |
engine | string | — | the one engine all members share (single-engine rule) |
members | list of strings | — | keys into nodes:; ≥ 2, each a distinct node with a distinct node_id |
topology | string | mesh | v1 supports only mesh (full mesh, any N ≥ 2); ring/partial are deferred |
include / exclude | list of globs | — | table/key selection, interpreted per engine exactly as for a sync |
tuning | block | sync defaults | same knobs as a sync's tuning |
nodes:
us: { engine: postgres, postgres: { host: pg-us, port: 5432, database: app, user: replicare, password: ${PW}, sslmode: require } }
eu: { engine: postgres, postgres: { host: pg-eu, port: 5432, database: app, user: replicare, password: ${PW}, sslmode: require } }
ap: { engine: postgres, postgres: { host: pg-ap, port: 5432, database: app, user: replicare, password: ${PW}, sslmode: require } }
clusters:
- name: global-app
engine: postgres
members: [us, eu, ap] # any N >= 2; node_id defaults to us/eu/ap
enabled: true # optional; false pauses ALL of this cluster's mesh edges
include: ["public.*"]
exclude: ["*_audit"]
# No conflict block: HLC last-write-wins is automatic. Nothing to declare.
A cluster has the same enabled opt-out as a sync (above):
unset/true runs every mesh edge; false pauses the whole cluster (all edges skipped
at startup). Per-edge pausing is not exposed.
Cycle safety for one-way syncs
Independently of clusters, replicare now rejects an un-declared replication cycle
among plain one-way syncs (e.g. A→B and B→A, or a longer ring) at config load —
that topology silently corrupted data before. A genuine active-active setup must be
declared as a clusters: entry, which is exempt.
Full example
See ../examples/replicare.yml for a fully annotated
config. For one config that runs a Postgres pipeline and a Redis pipeline
side by side (the per-sync single-engine model), see
../examples/postgres-and-redis.yml.
CLI reference
Most commands take the config file as an argument. Exit codes: 0 success,
1 runtime error or a pre-flight BLOCK (see validate), 2 usage/config
error (bad flags, or a config that won't load — unknown engine, unknown field,
unset ${VAR}, cross-engine sync).
replicare <command> [args]
help / version
replicare help # also: --help, -h — prints the command usage
replicare version # also: --version, -v — version, commit, build date, Go version
validate <config>
replicare validate config.yml
Loads the config and runs pre-flight for every (sync, target): connects read-only, introspects both sides, and classifies every column pair — identical/widening (ok), risky/lossy (warn), incompatible or missing type (block). Also reports FK components, a giant-component warning, dangling FK edges, and skipped no-key tables. Makes no changes. Fix all BLOCK findings before running.
Exit codes are meaningful for scripting: 0 = clean (ok/warn only), 1 = at
least one BLOCK finding (a clean "incompatible" verdict, not a crash), 2 =
the config didn't load or a connection failed. So replicare validate ... && replicare run ...
gates a start on a block-free pre-flight.
run <config>
replicare run config.yml
Runs the daemon: starts the observability endpoints, then brings up and streams
every sync until SIGINT/SIGTERM. On a signal it shuts down gracefully —
finishes the in-flight drain pass, checkpoints, and exits 0.
Each sync runs under a single-active ownership lock; a second daemon pointed at the same sync stands by. State is checkpointed, so a restarted process resumes from where it left off.
status <config> [--json] [--sync <name>] [--no-live] [--watch <dur>]
replicare status config.yml # human-readable table, LIVE by default
replicare status config.yml --json # machine-readable
replicare status config.yml --sync pg-pipeline # one sync only
replicare status config.yml --no-live # state store only (no source/target connections)
replicare status config.yml --watch 5s # re-render every 5s until Ctrl-C
Each sync prints a one-line health headline followed by a per-(table, target) grid, read from the state store (so it works whether or not a daemon is running):
sync: app-to-warehouse
streaming · last pass 2s ago · last applied 40s ago (1240 rows total) · backlog 0 (caught up)
TABLE COPY ... SEEN ... LAST_SYNC ROWS RESEED
public.orders done ... 2s ... 40s ago 1240 -
Two different time signals — the distinction that makes a caught-up sync legible:
SEEN— time since the last healthy streaming pass (a liveness heartbeat). The daemon refreshes it every pass whether or not data moved, so a smallSEENmeans "running and current"; a largeSEENmeans the daemon is stopped or stalled. (This is what the oldLAGcolumn was, renamed — it was never "how far behind".)LAST_SYNC/ROWS— time since the last pass that actually applied rows to this (target, table), and how many it applied. A healthy, idle sync shows a recentSEENbut a growingLAST_SYNC— that's normal (nothing to apply), not lag.-until the first data-moving pass. (These come from the state store, so they show in--no-livetoo.)
The headline aggregates these across the sync: phase, most-recent SEEN, most-recent
data movement + total rows applied, and (live) the backlog rollup.
Live by default. Unless you pass --no-live, status also connects to the
sync's source and targets to add signals that live only in the databases — the
visibility you'd otherwise get from Grafana:
SRC_ROWS/TGT_ROWS— live source and target row counts (Redis: key counts). During initial copyTGT_ROWSclimbs towardSRC_ROWS; while streaming they track each other.BACKLOG— the per-target unconsumed delta backlog asrows (oldest-age). It distinguishes three states:0= caught up (healthy — the common steady state);N (age)= that many deltas waiting;-= unknown, not zero — the signal couldn't be read (a table not captured, Redis's capture-less engine, or a transient error), and alive: partial (…)note says why.
Live collection is best-effort and read-only (it installs nothing): if the
source or a target is unreachable, the row still renders from the state store and a
live: partial (…) note explains what was missing. --no-live skips all
source/target connections (useful if you only have state-store access, or want the
cheapest possible check).
--watch <dur> re-renders on the given interval (e.g. 10s, 1m) until
SIGINT; combine with --json to stream snapshots.
verify <config> [--json] [--sync <name>] [--watch <dur>]
replicare verify config.yml # all syncs
replicare verify config.yml --sync pg-pipeline # one sync
replicare verify config.yml --json # machine-readable
replicare verify config.yml --watch 30s # re-check every 30s until Ctrl-C
A read-only source↔target convergence spot-check. For every replicated unit it counts and content-fingerprints the source and each target and compares. It writes nothing and installs nothing, so it is safe to run against a source you may not own and against live targets. Per table it reports one of:
ok— row counts and content checksums match.drift-count— row counts differ (the target is missing/extra rows).drift-checksum— counts match but content differs (a value diverged).count-only-ok— counts match; the engine compares membership only (see below).error— a table could not be compared (e.g. missing on the target).
The content fingerprint is an order-independent hash over a name-matched column
projection, so it is robust to physical column-order differences and streams in
constant server memory. Relational engines (Postgres, MySQL) hash row content;
Redis compares the key set (count + membership), which catches missing/extra
keys — the dominant Redis divergence — since transport is value-faithful
DUMP→RESTORE (per-value Redis content diffing is a documented follow-up).
Exit codes: 0 = every selected sync converged; 1 = drift or an endpoint
could not be reached; 2 = usage/config. So replicare verify config.yml is
scriptable as a convergence gate. In --watch mode the exit code reflects the last
completed pass.
Note: replication has lag — a
drift-count/drift-checksumimmediately after a burst of source writes usually just means the target hasn't caught up yet. Re-run (or use--watch); persistent drift is the real signal.
capture install|remove <config> [--sync <name>]
replicare capture install config.yml # all syncs
replicare capture remove config.yml --sync app-to-warehouse
Installs or removes the trigger-based CDC machinery on a sync's source, over its
selected tables. run installs capture automatically, so this is for
pre-provisioning or manual teardown.
Note: install needs only TRIGGER (plus SELECT/USAGE and schema
creation). remove additionally needs table ownership to drop the triggers —
if the replicare role doesn't own the tables, removal of the trigger fails
(reported), though replicare's own delta/track tables are still dropped.
reseed <config> --sync <name> --target <name>
replicare reseed config.yml --sync app-to-warehouse --target warehouse
Flags a target for a full re-copy. It signals the running daemon (via the state store) to re-copy that target from current source state on its next pass, then resume streaming — honoring single-active ownership rather than acting directly. Use it after out-of-band divergence, or to recover a target that fell behind.
If the target has no cursors yet (you ran reseed before the sync's first
run), there is nothing to flag: replicare reports that the target will full-copy on
its first run anyway and exits 0. That is expected, not a failure.
reseed-sequences <config> --sync <name> [--target <name>] [--dry-run]
replicare reseed-sequences config.yml --sync app-to-warehouse # all targets of the sync
replicare reseed-sequences config.yml --sync app-to-warehouse --target warehouse
replicare reseed-sequences config.yml --sync app-to-warehouse --dry-run # report, write nothing
Advances each replicated table's owned identity/serial/AUTO_INCREMENT counter on a
promoted one-way (DR) target to max(id)+1, so after a failover cutover the node
never re-issues an id already present in the replicated data. replicare replicates row
values faithfully but not the sequence object's counter (data-only); for a passive replica that is harmless, but a promoted node's nextval() would
collide with the copied rows. Postgres uses setval, MySQL ALTER TABLE … AUTO_INCREMENT. Tables with no owned counter (UUID / natural / composite PK) are
skipped; --dry-run reports the target values and writes nothing.
- Not the same as
reseed.reseedre-copies a target's data;reseed-sequencesonly bumps its sequence counters. - Run order: stop replication into the node, run this, then open it to writes — a write in the gap could still grab a colliding id.
- Passive / one-way / DR only. It refuses a target that is an active-active
cluster member: there the same
max(id)+1collapses every node onto one counter and silently drops rows via last-write-wins. Active-active id allocation is a schema concern — use globally-unique keys (UUID v7 / ULID); see multi-master. - Privilege: a DR-only grant beyond the DML set — Postgres
UPDATEon each sequence, MySQLALTERon each table (reads only the target's ownmax(id); no source privilege). See operations → Sequences & identity counters. - Exit codes:
0done;1runtime error (unreachable / missing grant / engine without the capability);2usage/config, or a refused mesh target.
Signals
run responds to:
SIGTERM/SIGINT— graceful shutdown (drain in-flight pass, checkpoint, exit 0).
An ungraceful kill is also safe: state is checkpointed and applies are
idempotent, so a fresh run resumes and converges without duplicates.
Operating replicare
Practical guidance for running replicare in production: the operator's view. How it works underneath is in Architecture.
Lifecycle of a sync
- Pre-flight.
replicare validate <config>(and the daemon at startup) introspects the source and target and classifies every column pair: identical/widening → ok, risky/lossy → warn, incompatible or missing type → block (refuses to start). It also reports FK components, a giant-component warning, dangling FK edges, and skipped no-key tables. Fix blocks before running. - Capture-first. The daemon installs trigger-based CDC on the source's selected tables, so changes begin queuing into per-table delta tables immediately.
- Initial copy. A chunked, parallel copy of each FK component (parents first) runs while capture is active. It is resumable — a restart continues from the last checkpoint, not the top.
- Cutover → streaming. Once a component is copied, it flips to continuous streaming: each pass re-reads the current source values of dirty keys and applies them idempotently, FK-ordered.
- Purge / retention / reseed. Consumed deltas are purged; a target that falls too far behind is reseeded (see below).
Before a migration: checklist
Work through this before pointing replicare at a production source. Most items are information to
gather (sizing, traffic, quiet windows); a few must be true before starting. Run the queries on the
source unless stated otherwise. replicare validate <config> then checks the rest (types,
foreign-key loops, keys) and reports anything that would block.
Postgres
Sizing and traffic
- Total database size.
SELECT pg_size_pretty(pg_database_size(current_database())); - Row counts and sizes per table.
row_estimatecomes from planner statistics;-1(or0on older versions) means the table has never been analyzed — runANALYZEfirst for real numbers.SELECT n.nspname AS schema, c.relname AS table_name, c.reltuples::bigint AS row_estimate, pg_size_pretty(pg_total_relation_size(c.oid)) AS total, pg_size_pretty(pg_indexes_size(c.oid)) AS indexes FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE c.relkind IN ('r', 'p') AND n.nspname NOT IN ('pg_catalog', 'information_schema', 'replicare') AND n.nspname NOT LIKE 'pg_toast%' ORDER BY pg_total_relation_size(c.oid) DESC; - Write volume over 24 hours. The counters only ever go up, so run this twice, 24 hours apart,
and subtract. Every row inserted, updated or deleted in a replicated table adds one row to
replicare's change tracking, so
inserts + updates + deletesper day is roughly the daily change-tracking volume (and the work streaming has to keep up with).
For the busiest tables,SELECT now() AS sampled_at, sum(n_tup_ins) AS inserts, sum(n_tup_upd) AS updates, sum(n_tup_del) AS deletes, (SELECT xact_commit + xact_rollback FROM pg_stat_database WHERE datname = current_database()) AS transactions FROM pg_stat_user_tables;SELECT relname, n_tup_ins + n_tup_upd + n_tup_del AS writes FROM pg_stat_user_tables ORDER BY writes DESC LIMIT 20;. - Low-traffic windows (daily and weekly). Busy periods followed by lulls let streaming catch up, and a quiet window is the best time for the final cutover.
Must be true before starting
- Every table has a primary key or a
UNIQUEconstraint. replicare tracks changes by key. It uses the primary key, or else the firstUNIQUEconstraint; a bareCREATE UNIQUE INDEXwithout a constraint does not count. Tables listed here are skipped (with a warning), not replicated:SELECT n.nspname AS schema, c.relname AS table_name FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE c.relkind IN ('r', 'p') AND n.nspname NOT IN ('pg_catalog', 'information_schema', 'replicare') AND n.nspname NOT LIKE 'pg_toast%' AND NOT EXISTS (SELECT 1 FROM pg_constraint k WHERE k.conrelid = c.oid AND k.contype IN ('p', 'u')) ORDER BY 1, 2; - Extensions. List them on the source and confirm the target has every one the replicated
tables use (column types, defaults). replicare never installs extensions; a missing one fails
pre-flight.
SELECT extname, extversion FROM pg_extension ORDER BY 1; - Materialized views (and plain views) are not replicated — only table data is. Create them on
the target from the schema dump, then
REFRESH MATERIALIZED VIEWthere after the initial copy and again just before cutover.SELECT schemaname, matviewname FROM pg_matviews ORDER BY 1, 2; - Large objects are not replicated. Their contents live in
pg_largeobject, not in a replicated table; a table'soidcolumn is copied but the object it points to is not. Anything above 0 here needs a separate plan:SELECT count(*) FROM pg_largeobject_metadata; - Schema changes are frozen for the whole replication. DDL is not replicated, so a schema change
on the source has to be applied to the target by hand and can break the pipeline. During the
initial copy there is an extra reason: an
ALTER TABLEwaits for replicare's copy read of that table to finish, and the application's queries on that table queue behind theALTER. - Regular vacuum and maintenance jobs. Find out what runs and when — autovacuum settings,
scheduled
VACUUM, and especially heavy jobs that lock whole tables (VACUUM FULL,CLUSTER,pg_repack). PlainVACUUM/ autovacuum is fine at any time; the heavy jobs should be paused during the initial copy and kept out of peak hours afterwards. See Vacuum and other maintenance.
Also ask about jobs outside the database (cron, CI pipelines, the hosting provider's maintenance windows).SHOW autovacuum; -- scheduled jobs, if pg_cron is installed: SELECT jobid, schedule, command FROM cron.job; -- recent vacuum activity and the tables with the most dead rows: SELECT relname, last_vacuum, last_autovacuum, n_dead_tup FROM pg_stat_user_tables ORDER BY n_dead_tup DESC LIMIT 20; - No long-open transactions. Sessions left open — especially
idle in transaction— stop vacuum from cleaning up (replicare's change-tracking tables bloat) and make trigger setup wait. This should normally return nothing:SELECT pid, usename, state, now() - xact_start AS open_for, left(query, 80) AS last_query FROM pg_stat_activity WHERE xact_start < now() - interval '5 minutes' AND pid <> pg_backend_pid() ORDER BY xact_start; - Versions.
SELECT version();on both sides. The source can be old; the target should be the same major version or newer. - Privileges and disk. The source role needs the grants in Least-privilege grants, and the source needs room for the change-tracking tables to grow with the write volume from item 3 (more if a target falls behind, up to the retention cap).
- Sequences at cutover. Row values are copied, sequence counters are not. Plan
replicare reseed-sequences(orsync_sequences: true) before the target takes writes — see Sequences & identity counters.
MySQL
Numbers match the Postgres list. Postgres items 4 (quiet windows), 9 (schema freeze), 13 (privileges and disk) and 14 (sequences) apply as written; large objects (8) don't exist in MySQL, and the version check (12) is part of (6). MySQL versions of the rest, plus two MySQL-only items:
- (1) Total size.
SELECT ROUND(SUM(data_length + index_length) / 1024 / 1024) AS size_mb FROM information_schema.TABLES WHERE table_schema = '<DB_NAME>'; - (2) Row counts, sizes and storage engine per table.
table_rowsis InnoDB's estimate.
Every table must be InnoDB — pre-flight blocks MyISAM/MEMORY tables.SELECT table_name, engine, table_rows AS row_estimate, ROUND(data_length / 1024 / 1024) AS data_mb, ROUND(index_length / 1024 / 1024) AS index_mb FROM information_schema.TABLES WHERE table_schema = '<DB_NAME>' AND table_type = 'BASE TABLE' ORDER BY data_length + index_length DESC; - (3) Write volume over 24 hours. Server-wide counters (all databases on the server); run twice,
24 hours apart, and subtract.
SHOW GLOBAL STATUS WHERE Variable_name IN ('Innodb_rows_inserted', 'Innodb_rows_updated', 'Innodb_rows_deleted', 'Com_commit', 'Com_rollback'); - (5) Every table has a primary key or a
UNIQUEkey (tables without one are skipped):SELECT t.table_name FROM information_schema.TABLES t WHERE t.table_schema = '<DB_NAME>' AND t.table_type = 'BASE TABLE' AND NOT EXISTS (SELECT 1 FROM information_schema.TABLE_CONSTRAINTS c WHERE c.table_schema = t.table_schema AND c.table_name = t.table_name AND c.constraint_type IN ('PRIMARY KEY', 'UNIQUE')); - (6) Extensions don't exist in MySQL. Instead check: MySQL 5.7 or newer (
SELECT VERSION();, MariaDB is not supported); on the target,local_infileisON(SHOW GLOBAL VARIABLES LIKE 'local_infile';) and the target tables have no extraUNIQUEkeys beyond the one replicare uses (pre-flight blocks them — see the MySQL page). - (7) Materialized views don't exist in MySQL; plain views are not replicated (they come over with the schema dump).
- (MySQL only) Existing triggers. Tables can keep their own triggers: from MySQL 5.7.2 a table may
have several triggers per timing and event, so replicare's triggers sit alongside the
application's and both fire. Only a source older than 5.7.2 (in practice a best-effort 5.6) allows
a single trigger per timing and event, and there a table with its own
AFTER INSERT/UPDATE/DELETEtrigger blocks capture. Worth listing either way, to know what else runs on each write:SELECT event_object_table, trigger_name, action_timing, event_manipulation FROM information_schema.TRIGGERS WHERE trigger_schema = '<DB_NAME>'; - (9) Schema freeze also covers online schema-change tools (
pt-online-schema-change,gh-ost). They rebuild a table and swap it in, and the swap leaves replicare's triggers behind on the old table;pt-online-schema-changealso adds triggers of its own. - (10) Maintenance jobs (the MySQL counterpart of vacuum). MySQL has no
VACUUM; the jobs that matter are table rebuilds —OPTIMIZE TABLE,ALTER TABLE … FORCE/ENGINE=InnoDB— run from cron or the event scheduler. A rebuild locks the table briefly at the start and end; during the initial copy it waits for replicare's copy read and the application queues behind it, so pause rebuilds during the copy.SELECT event_schema, event_name, interval_value, interval_field, event_definition FROM information_schema.EVENTS; - (11) No long-open transactions — they hold back InnoDB's purge (the history list grows and
replicare's change-tracking tables bloat) and make trigger setup wait:
SELECT trx_mysql_thread_id, trx_started, trx_state, LEFT(trx_query, 80) AS query FROM information_schema.INNODB_TRX ORDER BY trx_started; - (MySQL only) A Postgres database for replicare's state is still required for a MySQL→MySQL sync (see the MySQL page).
Tuning knobs (per sync, under tuning:)
| Knob | Meaning | Default |
|---|---|---|
drain_interval | Time between streaming drain passes. Longer = more per-PK coalescing (less source load) but higher lag. | 1s |
drain_batch | Max dirty deltas applied per table per pass. With drain_interval this is the per-table streaming ceiling. | 1000 |
apply_concurrency | How many of a component's tables apply concurrently per pass. 1 is sequential; higher parallelizes across the copy-worker pool. Raise when several tables backlog at once. | 1 |
retention.max_age | Cap on the oldest unconsumed delta before a laggard target is reseeded. | 24h |
retention.max_bytes | Cap on the delta table's on-disk size before reseed. | off |
pool.max_source_connections / pool.max_target_connections | Connection caps; the copy worker pool is sized from these. | conservative |
Defaults favor low source pressure. Raise the pool caps and shorten drain_interval for throughput;
lengthen drain_interval to reduce re-read load under heavy churn.
A single high-volume table that lags
Streaming applies at most drain_batch deltas per table per pass, so one table's ceiling is
roughly drain_batch / drain_interval rows/s (e.g. 1000 / 1s = ~1000 rows/s). When one big table
(say a high-churn events) falls behind while the rest stay at zero backlog, you've hit that ceiling.
Two levers, in order:
- Shorten
drain_interval(1s→200ms→100ms): more passes/sec, linearly more throughput, at the cost of more frequent re-readSELECTs on the source. Best when the "Apply batch latency (p95)" is well under the interval (interval-bound). - Raise
drain_batch(1000→5000+): more rows per pass. More efficient than a tiny interval for a genuinely large table — one COPY-to-staging + upsert of 5000 rows amortizes round-trips far better than five of 1000 — and the right lever when apply latency is near/above the interval (apply-bound). Costs a larger re-read and apply per pass (more source and target work in one shot).
Watch - rate(replicare_delta_backlog_rows[5m]) per table: positive means the backlog is shrinking
(catching up).
Several tables backlogged at once
The drain_batch/drain_interval levers above are per-table ceilings on a single-threaded drain:
by default a pass applies each component's tables one at a time. When several tables are backlogged
together, raise apply_concurrency — it applies a component's tables concurrently, across a pool
built from the copy workers (idle during streaming), so they no longer queue behind each other. It's
bounded by the connection caps: the pool has one primary pair plus up to apply_concurrency-1 worker
pairs, and worker pairs come from pool.max_source_connections/max_target_connections (so raise
those alongside it). FK order within a component is still honored per pass (a child that lands before
its parent hits a transient FK and retries), and cyclic components on MySQL — which apply as one
whole-component transaction — stay sequential. This does not speed up a single hot table (that's
still drain_batch/drain_interval + target capacity); it removes the cross-table serialization when
many tables are hot. As always, it helps only until the target's write capacity saturates.
Source footprint (the thing to watch)
replicare writes delta and track tables to the source — a database you may not own — so bounding that footprint is a first-class concern. The headline signals (per target/table):
replicare_delta_backlog_rows/replicare_delta_backlog_bytes— unconsumed work.replicare_delta_oldest_unconsumed_age_seconds— how stale the queue is.replicare_target_up— reachability (1/0).replicare_reseed_total— forced reseeds.
Retention & forced reseed. A slow or unreachable target pins the delta queue (nothing purges
until every target has consumed it), so replicare caps retention by age and/or size. A target over
the cap is marked needs-reseed: its consumption record is reset, the unpinned deltas are purged,
and it is brought back by a fresh full copy — trading a re-copy for a protected source. This is on
by default. Watch retention.cap_approaching logs (they escalate INFO→WARN→ERROR toward the cap).
The xmin-horizon caveat. A long idle-in-transaction session (or other held-back xmin) on the
source prevents autovacuum from reclaiming purged delta rows, so the delta table can bloat even
though replicare purges it. replicare bounds the logical backlog regardless, but physical bloat
under a pinned xmin is surfaced, not fixed — hunt down the idle transaction on the source.
MySQL (InnoDB) has the direct analog. InnoDB does not shrink the tablespace on DELETE, and a
long-running source transaction inflates the history list, blocking purge of rows it can still
see — the same shape as the xmin horizon. The same headline metrics apply (the delta/track tables
live in a dedicated replicare database on the MySQL source), retention/reseed protects the source
identically, and the fix is the same: end the long-running source transaction. See
the MySQL engine page.
Redis has no source footprint at all. Its CDC is capture-less — replicare writes nothing to the
source (no delta/track tables, no replicare schema). So the delta-backlog metrics, retention, and
xmin-style bloat simply don't apply. What you watch instead is delete-propagation latency:
deletes are found by the periodic target-vs-source sweep, so a source DEL converges only after the
next sweep. replicare_delete_reconciliation_lag_seconds (sweep duration) and
replicare_deletes_reconciled_total are the Redis-specific signals; tune cadence with
delete_sweep_interval. Upserts converge faster via the rolling reconciliation scan. See
the Redis engine page.
Vacuum and other maintenance
Plain VACUUM and autovacuum never conflict with streaming. While streaming, replicare runs
only ordinary statements on the source — the trigger's INSERT into its change-tracking table on
every application write, SELECTs that read those tables and re-read changed rows, and batched
DELETEs that clear changes every target has applied. None of them takes a lock that a plain
vacuum blocks, or that blocks a plain vacuum. The same holds on the target, where replicare only
inserts, updates and deletes rows.
Trigger setup is the one place a plain vacuum can get in the way — only at startup, and only when a trigger actually has to be created or replaced (a new table, or a trigger someone removed). Adding a trigger needs a lock that a running vacuum holds. replicare waits at most 2 seconds for it, backs off and retries, and after repeated failures stops with an error naming the table, rather than holding the application up. A normal restart, with every trigger already in place, takes no lock on application tables at all.
Heavy maintenance that locks a whole table — VACUUM FULL, CLUSTER, pg_repack,
ALTER TABLE:
- On an application table, it blocks replicare's reads of that table just like the application's queries; replicare adds no extra harm. During streaming replicare's reads are short, so the job waits only a moment for one to finish.
- On one of replicare's change-tracking tables (for example a database-wide
VACUUM FULL), it briefly blocks the application's writes to the matching table, because the trigger cannot record the change until the job finishes. These tables are small, so the pause is short; keep it out of peak hours.
During the initial copy two more things apply:
- Copy reads are long: each chunk, and a table loaded in one statement (a self-referencing table whose link column cannot be emptied), keeps a read open for as long as it runs. While it runs, vacuum cannot remove rows that read might still need to see, anywhere in the database.
- An
ALTER TABLE(orVACUUM FULL/CLUSTER) on a table being copied waits for that read to finish, and the application's queries on the table queue behind it. Pause heavy maintenance and schema migrations during the initial copy, or run them with alock_timeout.
Long-open transactions are the real vacuum problem. replicare's change-tracking tables get
constant inserts and deletes and rely on autovacuum to reclaim the space. A transaction left open on
the source — typically an application session idle in transaction — stops vacuum from cleaning up
after it anywhere in the database, so those tables grow even though replicare keeps clearing them
(see the xmin-horizon caveat). Streaming keeps working; the
cost is disk. Ending the stuck session fixes it.
MySQL has no VACUUM; the equivalents are table rebuilds (OPTIMIZE TABLE,
ALTER TABLE … FORCE), which behave like the heavy jobs above, and long transactions holding back
InnoDB's purge, which behave like the open-transaction problem.
When a target goes down
A drain against an unreachable target fires all three channels at once: an errored trace span
with backlog attributes, an ERROR target.unreachable log, and replicare_target_up=0 plus a
climbing backlog series. The daemon keeps retrying each pass; nothing is lost. If the target stays
down long enough to exceed the retention cap, it is reseeded on recovery.
Whole-DB vs replicated size
replicare emits two size figures for each endpoint, and they answer different questions:
- Whole-database size —
replicare_source_db_bytes/replicare_target_db_bytes(Postgrespg_database_size, MySQL sum overinformation_schema.tables). This is the entire database the connection points at, including everything that is not the replicated data. - Replicated-data size —
replicare_source_replicated_bytes/replicare_target_replicated_bytes(SUM(pg_total_relation_size)— heap + indexes + TOAST — or MySQLdata_length + index_length, over just the sync's selected tables). Tables absent at an endpoint contribute nothing; a sync with no relational selection reports0.
Why the whole-DB gauges usually differ while the replicated-data gauges converge. On a healthy,
caught-up single→single Postgres sync you will typically see source_db_bytes larger than
target_db_bytes, even though the data is identical. That is expected, not drift:
- Trigger CDC writes to the source. replicare's own
replicareschema lives on the source: per-table delta and track tables, plus trigger functions. Under churn those delta tables are a high-write queue that bloats before autovacuum/purge reclaims it. None of this exists on the target. - Unreplicated tables count too.
pg_database_sizecovers the whole database, so any table on the source you did not select (audit logs, staging tables, keyless tables skipped for lacking a PK) inflates the source figure but never the target's. - The state store is a separate database. When the Postgres
StateStorelives on the target host it is its own database (e.g.replicare_state), sopg_database_size(current_database())for the target's data database does not count it.
The replicated-data gauges strip all of that away — same tables, same size function on both
sides — so they are the right series for an apples-to-apples "has the target caught up to the
source?" comparison. Watch source_replicated_bytes vs target_replicated_bytes for convergence;
watch source_db_bytes (against source_replicated_bytes) for capture/bloat footprint on a source
you may not own (see Source footprint).
Metrics reference
Every metric replicare exposes on /metrics (Prometheus text format; port 9090 in the demo,
configurable under observability:). These names, types, and labels are the single source of truth
locked in the observability contract, so a scrape of a running daemon matches this table exactly.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
replicare_rows_copied_total | counter | sync, table | Rows copied during the initial copy, counted once they are committed on the target (a retried chunk counts once). Tables linked in a loop count too; rows re-copied only to catch up with a busy source do not. Restarts from 0 with the daemon. |
replicare_initial_copy_rows_target | gauge | sync, table | Estimated total rows to copy (the progress denominator), from the source's own statistics (Postgres pg_class.reltuples, MySQL TABLE_ROWS; no scan). Set at bring-up for tables this run still has to copy; left unset for a table never analyzed. |
replicare_throughput_rows_per_second | gauge | sync | Current apply/copy throughput. 0 on an idle, caught-up pass. |
replicare_replication_lag_seconds | gauge | sync, target, table, component | Replication lag — age of the oldest unconsumed delta (0 when caught up). |
replicare_delta_backlog_rows | gauge | sync, target, table, component | Unconsumed delta rows (the queue depth). |
replicare_delta_backlog_bytes | gauge | sync, target, table, component | Estimated unconsumed delta bytes. |
replicare_delta_oldest_unconsumed_age_seconds | gauge | sync, target, table, component | Age of the oldest unconsumed delta (how stale the queue is). |
replicare_delta_purged_total | counter | sync, table | Delta rows purged after consumption. |
replicare_reseed_total | counter | sync, target | Forced reseeds triggered by the retention cap. |
replicare_target_up | gauge | sync, target | Target reachability (1=up, 0=down). |
replicare_source_up | gauge | sync | Source reachability (1=up, 0=down). |
replicare_source_db_bytes | gauge | sync | Source whole-database size in bytes (Postgres pg_database_size, MySQL information_schema; unset for engines that don't report it). Includes replicare's own capture schema + delta bloat and any unreplicated tables — see Whole-DB vs replicated size. |
replicare_target_db_bytes | gauge | sync, target | Target whole-database size in bytes (same engine support). |
replicare_source_replicated_bytes | gauge | sync | Size in bytes of just the replicated tables on the source (SUM(pg_total_relation_size) / MySQL data_length+index_length over the selection). The apples-to-apples source↔target figure — see Whole-DB vs replicated size. |
replicare_target_replicated_bytes | gauge | sync, target | Size in bytes of just the replicated tables on the target (same engine support). |
replicare_apply_batch_seconds | histogram | sync, target | Apply-batch duration per drain pass. |
replicare_errors_total | counter | sync, category | Errors by category (e.g. drain, retention). |
replicare_table_phase_info | gauge | sync, table, phase | Table lifecycle phase as an info gauge — value 1 on the active phase label (initial_copy/streaming); exactly one series per table. Present from bring-up, so it is the series to key dashboards on. |
replicare_delete_reconciliation_lag_seconds | gauge | sync, target, table | Redis only — duration of the last completed delete sweep (staleness of capture-less deletes). Unset for Postgres/MySQL. |
replicare_deletes_reconciled_total | counter | sync, target, table | Redis only — keys DELed by the target-vs-source delete sweep. Unset for Postgres/MySQL. |
The per-table backlog/lag gauges also carry a component label — the table's FK connected
component (its first member) — so you can roll lag and backlog up per component
(max by (component) (replicare_replication_lag_seconds)).
The headline signals to watch are replicare_source_up, replicare_target_up, replicare_delta_backlog_rows,
replicare_delta_oldest_unconsumed_age_seconds, and replicare_replication_lag_seconds — together
they answer "is the target healthy, and how far behind is it?" (see Source footprint
and When a target goes down).
A ready-to-import Grafana dashboard over these metrics — per-table backlog/lag, catch-up, initial-copy progress, throughput, and errors, with a configurable Prometheus data source — lives at Grafana dashboard.
Traces (OpenTelemetry)
Traces are exported via OTLP/gRPC when observability.otlp_endpoint is set. The span names are a
fixed registry:
| Span | Covers |
|---|---|
replicare.initial_copy.chunk | one chunk of the initial copy |
replicare.capture.install | installing trigger capture on the source |
replicare.delta.drain | one streaming drain pass for a table/target |
replicare.apply.component | applying one FK component's pass on the target |
replicare.reseed | a forced/retention reseed |
replicare.preflight | the start-up compatibility check |
A drain/apply span against an unreachable target is recorded with error status plus the backlog/age attributes, so the stall and its blast radius are visible in the trace timeline.
Structured logs (slog)
Logs are JSON or text (logging.format), leveled (logging.level), with a stable event field you
can grep/alert on:
| Event | Level | Meaning |
|---|---|---|
capture.installed / capture.removed | INFO | capture lifecycle |
table.cutover_to_streaming | INFO | a table finished initial copy |
retention.cap_approaching | INFO→WARN | delta backlog nearing the retention cap (escalates with proximity) |
target.unreachable / target.recovered | ERROR / INFO | target connectivity, cross-channel with the target_up gauge and the errored span |
target.needs_reseed | ERROR | a target was sacrificed to the retention cap and will re-copy |
preflight.blocked | ERROR | a sync refused to start on a blocking incompatibility |
A degrading target surfaces in metrics, traces, AND logs at once — never just one channel.
The status HTTP API
The daemon serves an operator HTTP surface (bound to observability.status_addr, e.g. :8080):
GET /status— JSON snapshot per sync/target/table: phase (initial_copy/streaming), rows copied vs total, replication lag, and last error, read from the state store.GET /healthz— a liveness probe (200 when the daemon is up), for Kubernetes/load balancers.GET /metrics— the Prometheus scrape. It is served here too, and additionally onobservability.metrics_addrwhen that is set to a different address (the conventional split).
The CLI replicare status reads the same state-store snapshot and adds live signals on top —
see the next section.
Checking progress from the CLI (no Grafana required)
When you can't deploy Grafana next to replicare — e.g. the daemon runs inside the source cluster because that's the only network path to the source databases — the CLI is your dashboard. Two read-only commands answer "what is it doing, and has it caught up?" directly against the config, with no metrics stack:
replicare status config.yml # phase, lag, LIVE row counts + delta backlog
replicare status config.yml --watch 5s # refresh in place until Ctrl-C
replicare verify config.yml # source<->target convergence spot-check (exit 0 = converged)
statusis live by default: each sync prints a one-line health headline (phase · last healthy pass · last data applied + rows · backlog) over a per-(table, target) grid. Two distinct time signals keep a caught-up sync legible:SEEN(time since the last healthy pass — a liveness heartbeat, small when running) andLAST_SYNC/ROWS(when data was last applied to that table and how many rows). A healthy idle sync shows a recentSEENwith a growingLAST_SYNC— normal, not lag. Live mode addsSRC_ROWS/TGT_ROWSand the per-target deltaBACKLOG—0= caught up,N (age)= waiting,-= unknown (couldn't read it; alive: partialnote says why), never conflated with0.--no-livedrops the source/target connections but keeps the state-store signals (phase,SEEN,LAST_SYNC, reseed).verifyis a read-only convergence check: it counts and content-fingerprints every replicated unit on both ends and reportsok/drift-count/drift-checksumper table, exiting non-zero on drift — the same count+checksum convergence logic the load-test harnesses use, folded into the shipped binary. The fingerprint is by value content, including for Redis: it hashes each key's type + canonical logical value (not just which keys exist), so a changed value on an existing key is caught, not only missing/extra keys. (Redis TTL is excluded — replicated relative, so exact remaining differs by design; stream consumer-groups/PEL aren't deep-verified yet.) A paused sync (enabled: false) is tagged[PAUSED]in the output, since convergence of a static source and target says nothing about whether the pipeline is running.
Both take --json (for scripting) and --watch <dur> (repeat on an interval). Both are read-only and
install nothing, so they're safe to run against a source you may not own.
Running the CLI in the pod (no shell needed). The container image is minimal, so don't rely on a
shell being present — invoke the binary directly, which kubectl exec does without any shell:
kubectl exec <pod> -- /usr/local/bin/replicare status /etc/replicare/config.yml
kubectl exec <pod> -- /usr/local/bin/replicare verify /etc/replicare/config.yml
The runtime image is Chainguard wolfi-base, which does include /bin/sh if you
want an interactive session (kubectl exec -it <pod> -- /bin/sh), but the direct-exec form above is
all you need for a progress check.
Running multiple pipelines in one instance
One replicare config — and one daemon process — can run several syncs across different engines at once. The single-engine rule is per sync, not per config: each sync's source and targets must share an engine, but a config may hold a Postgres sync and a Redis sync side by side, and the daemon runs them concurrently (each under its own single-active ownership lock).
A complete, annotated version of the example below is at
../examples/postgres-and-redis.yml.
state_store: # ALWAYS Postgres — even for a Redis-only deployment
engine: postgres
postgres: { host: pg-state, port: 5432, database: replicare_state, user: replicare, password: ${STATE_PW} }
sources:
pg-src: { engine: postgres, postgres: { host: pg-source, port: 5432, database: app, user: replicare, password: ${PG_PW} } }
redis-src: { engine: redis, redis: { mode: standalone, nodes: [redis-source:6379] } }
targets:
pg-dst: { engine: postgres, postgres: { host: pg-target, port: 5432, database: app, user: replicare, password: ${PG_TGT_PW} } }
redis-dst: { engine: redis, redis: { mode: standalone, nodes: [redis-target:6379] } }
syncs:
- name: pg-pipeline
source: pg-src
targets: [pg-dst]
include: ["public.*"]
- name: redis-pipeline
source: redis-src
targets: [redis-dst]
include: ["*"]
replicare run config.yml streams both. replicare status config.yml shows both (the Postgres
sync with row counts + delta backlog, the Redis sync with key counts and - backlog, since Redis has
no source-side queue). replicare verify config.yml checks both. Scope any command to one with
--sync pg-pipeline.
State store is always Postgres (v1's only backend), even for a Redis→Redis pipeline where no data touches Postgres — a documented requirement, not a bug. See redis-statestore.md. If you have no Postgres to spare, that is the one piece of infrastructure a Redis-only deployment still needs.
Pausing one pipeline
Set enabled: false on a sync to pause just that pipeline while the others keep running — e.g.
to work on the Postgres pipeline above while Redis keeps streaming:
syncs:
- name: pg-pipeline
source: pg-src
targets: [pg-dst]
enabled: false # paused; skipped at startup, no ownership lock taken
include: ["public.*"]
- name: redis-pipeline # still runs
source: redis-src
targets: [redis-dst]
include: ["*"]
The pause is data-loss-free: the paused sync's source capture stays installed, so deltas keep
queuing and setting enabled: true again (then restarting) drains the backlog. It is start-time
only — a config change + daemon restart, not a live command. Because it's not a live toggle, a long
pause lets the source-side delta tables grow and suspends the source-footprint
protection (retention runs in the streaming loop, which is
skipped) — so pause for maintenance/cutover, not indefinitely. Full semantics:
configuration.md → Pausing a sync. On Kubernetes, flip it
with helm upgrade (see kubernetes.md).
Forcing a reseed
To re-copy a target from scratch (e.g. after out-of-band divergence):
replicare reseed <config> --sync <name> --target <name>
This flags the target in the state store; the running daemon re-copies it on its next pass and resumes streaming. It signals the owning daemon rather than acting directly, honoring single-active ownership.
Sequences & identity counters (DR / promotion)
replicare replicates row values faithfully — the integer id a sequence produced lands on the
target verbatim (OVERRIDING SYSTEM VALUE on Postgres, the explicit value on MySQL). What it does
not replicate is the sequence object's counter — Postgres last_value, MySQL's per-table
AUTO_INCREMENT — because that is schema-object state and replicare is data-only.
For a passive replica this is harmless: the target never calls nextval() for a replicated
table (the applier always supplies explicit ids), so the stale counter just sits unused. It bites
only when the target starts generating ids — i.e. a DR promotion / failover: the app inserts,
nextval() returns a value at or near 1, and it collides with the replicated rows at id 1..N.
Before you open a promoted node to application writes, advance its counters past the data:
replicare reseed-sequences <config> --sync <name> [--target <name>] [--dry-run]
For every replicated table it sets the owned identity/serial/AUTO_INCREMENT counter to max(id)+1
on that target (--dry-run reports what it would set, writing nothing). Tables with no
locally-allocated counter (UUID / natural / composite PK) are skipped.
- Order matters: run it after replication into the node has stopped and before the app can write — a write in that gap could still grab a colliding id.
- Why
max(id)+1, not the source'slast_value: the promoted node's data is the replicated rows, somax(id)+1can't collide; any ids the source allocated beyond its max (gaps, or rows lost in the DR window) belong to rows that aren't present, so re-issuing them is harmless. - Privilege: this needs a grant beyond the DML set — Postgres
UPDATEon the sequence, MySQLALTERon the table (see Least-privilege grants). A missing grant fails loudly; the command reads only the target's ownmax(id)and never touches the source. - PASSIVE / one-way / DR ONLY.
reseed-sequencesrefuses a target that is an active-active cluster member: there the samemax(id)+1would collapse every node onto the same counter and silently drop rows via last-write-wins. Active-active id allocation is a schema concern — use globally-unique keys (UUID v7 / ULID); see multi-master → active-active id allocation. (Not the same asreseed, which re-copies a target's data.)
Keeping counters warm continuously (sync_sequences)
The promotion-time reseed-sequences run above is the guarantee. If you'd rather have the counters
kept advanced during streaming — so a failover has nothing sequence-related to do — opt the sync
into continuous sequence syncing on the neutral config layer:
syncs:
- name: app-to-dr
source: primary
targets: [dr]
sync_sequences: true # optional; default false
While streaming, the daemon advances each replicated table's owned counter on the target to
max(id)+1 on a lazy interval (the same mechanism as reseed-sequences). Because a passive
replica's applier always supplies explicit ids (OVERRIDING SYSTEM VALUE), which never advance the
sequence, repeatedly setting it is idempotent and simply holds the counter ahead of the data.
- Opt-in and off by default — unset or
falseis exactly the pre-existing behavior. - Steady-state privilege. Enabling it means the daemon's target connection needs the same
DR-only grant the command needs, now continuously: Postgres
UPDATEon each owned sequence, MySQLALTERon each table. Without it the advance fails — but best-effort: the failure is logged (stream.pass_errorwith asequence syncmessage) and retried next interval; it never aborts the streaming pass or stops data replication. - Still PASSIVE / one-way / DR ONLY. Same mesh refusal as the command, enforced at config
load:
sync_sequences: trueon a sync whose target is also an active-active cluster member is rejected byvalidate/runup front (advancing a mesh counter is silent data loss). - Not a substitute for the runbook. It narrows the promotion gap but the cutover order
(stop replication → open to writes) is unchanged; run
reseed-sequencesas the final belt-and- braces step if you want certainty at the instant of promotion.
Runtime type errors
If the target rejects a value (type/constraint), replicare halts the affected component loudly (logs + telemetry) rather than skipping or guessing. The offending row stays dirty and re-applies automatically once you fix the cause (e.g. a missing target type/extension, or a schema mismatch).
Restarts & ownership
Each sync runs under a single-active pg_advisory_lock; a second daemon pointed at the same sync
stands by. State (copy progress, cursors) is checkpointed in the Postgres state store — which is
always Postgres regardless of the data engine, so MySQL syncs restart and resume through the same
mechanism — so a restarted or rescheduled process resumes cleanly from the last checkpoint; an
ungraceful kill replays at-least-once and converges idempotently. One daemon can run Postgres, MySQL,
and Redis syncs side by side (each sync stays single-engine). A Redis sync checkpoints only coarse
phase (snapshot-complete-per-unit; no resumable SCAN cursor), so a crash mid-copy re-runs the whole
SCAN idempotently rather than resuming a chunk — see the Redis engine page.
Connection resilience & the liveness probe
A dropped source or target connection — RDS/managed failover, an idle NAT/LB reap, a brief network
blip — recovers on its own, no restart needed. All connections set a bounded dial timeout and TCP
keepalive so a dead peer surfaces as an error quickly instead of hanging, and after any failed drain
pass the daemon health-checks both endpoints and reconnects whichever is down (Postgres re-dials its
single connection and re-applies session GUCs; MySQL and Redis rebuild their pooled clients). A
reconnect that can't succeed yet (the endpoint is genuinely down) is retried each pass, so the daemon
never crash-loops on an outage — it heals when the endpoint returns. Recovery is visible as a
stream.reconnected event in /status and the logs.
/healthz reports streaming liveness as a backstop for a true wedge (e.g. a hung query no
timeout caught). It is deliberately not target reachability: a loop that keeps cycling — including
one correctly retrying against a down target — stays healthy (that state shows up in
replicare_target_up and the backlog metrics instead). The probe fails only when a streaming loop
completes no pass within observability.stall_timeout (default 2m), so Kubernetes restarts a pod
whose loop has stopped iterating. Set stall_timeout above your slowest expected drain pass (a large
drain_batch on a slow target lengthens a pass); a negative value disables the check. Liveness
tracking starts only once a sync reaches streaming, so a long initial copy is never mistaken for a
wedge.
Least-privilege grants
See ../deploy/grants-source.sql and
../deploy/grants-target.sql. Note the documented asymmetry:
installing capture needs only the TRIGGER privilege, but capture remove (dropping the trigger)
requires table ownership. For the true zero-database-grant path, pre-create the replicare
capture schema (and, if the state store lives there, the replicare_state schema) owned by the
daemon role — the migration runner then issues no CREATE SCHEMA and needs no CREATE-on-database
privilege. See the Postgres engine page.
For MySQL, use ../deploy/grants-source-mysql.sql and
../deploy/grants-target-mysql.sql. MySQL has no such asymmetry
— the TRIGGER privilege covers both CREATE TRIGGER and DROP TRIGGER, so the daemon role can
fully uninstall capture on its own.
For Redis, use the ACL presets ../deploy/acl-source-redis.txt
and ../deploy/acl-target-redis.txt. The source role is read-only
(+ping +info +scan +type +dump +pttl +exists +memory|usage); the target role needs +ping +info +restore +del +scan +exists +module|list. Both roles need +info (pre-flight reads INFO server), and in cluster mode both also need +cluster|shards +cluster|slots and the same user
granted on every master. Watch the foot-gun: RESTORE is @dangerous and must be granted
explicitly with +restore, or every apply fails loud. See the Redis engine page.
reseed-sequences needs one extra target grant beyond the DML set, and only on the node you run
it against (a DR promotion step): Postgres UPDATE on each replicated table's owned sequence (or
ownership of it); MySQL ALTER on each replicated table. It reads only the target's own max(id) —
no source privilege. Grant it as part of the failover runbook, not the steady-state role.
Load & convergence testing
Three manual load-and-verify harnesses drive replicare against broad, high-volume, realistic data and
assert the target converges with the source under continuous churn. They are developer/test
tools, run from a checkout (go run ./test/…), not part of the shipped binary:
- Postgres —
test/loadgen. Seeds a rich 10-table schema chosen to exercise the hard paths (FK components, composite/UUID/text PKs, PK-changingSKUrenames,GENERATEDcolumns, a keyless table it must skip; optional FK cycles under--cyclic), churns random inserts/updates/deletes, andverifys per-table row-count + ordered content checksum. - MySQL —
test/loadgen-mysql. The MySQL sibling of the Postgres harness: the same 10-table schema in MySQL types (AUTO_INCREMENT,JSON, compositeVARCHARPK,GENERATED … STORED, a keyless table), generated with MySQL-5.7-safe SQL (nogenerate_series/CTE/window functions), churned the same way, andverifyd with a cross-version, order-independent content checksum (BIT_XOR-folded per-row hashes). - Redis —
test/loadgen-redis. Seeds every value type (string/hash/list/set/zset/stream + big/TTL'd keys) plus a source-onlylgskip:*cohort the sync must exclude, churns value mutations,RENAMEs, and heavyDELs (the delete-reconciliation stress), andverifys a version-independent, type-aware content hash + TTL presence. Standalone, sentinel, and cluster.
All three support active-active load testing via --node-id (each writer node claims a disjoint
key slice so a mesh converges to the clean union); --node-id 0 is the default single-writer
(active-passive) behaviour. Each README documents the full end-to-end rig (two databases, a Postgres
state store, a config, the daemon, and a churn-and-verify loop) and the active-active workflow.
task loadgen:*, task loadgen-mysql:*, and task loadgen-redis:* are the shortcuts. These are
distinct from the automated Go suite (task test:integration).
Deploying on Kubernetes (Helm)
replicare ships a Helm chart — replicare-controller — that runs the daemon
(replicare run) on Kubernetes. This page is the deployment guide; the chart's own
README is the exhaustive values
reference.
Two GHCR packages
Each vX.Y.Z release publishes two independent OCI artifacts to GHCR, stamped with
the same version:
| Artifact | Package | Pull |
|---|---|---|
| Daemon image | ghcr.io/rudimk/replicare | docker pull ghcr.io/rudimk/replicare:0.1.1 |
| Helm chart | ghcr.io/rudimk/replicare-controller | helm pull oci://ghcr.io/rudimk/replicare-controller --version 0.1.1 |
The chart's appVersion defaults image.tag to the matching daemon image, so a
versioned chart install pulls a matched chart + image pair unless you override
image.tag.
Install
# From the published OCI chart (recommended) — pin the version:
helm install my-replicare oci://ghcr.io/rudimk/replicare-controller \
--version 0.1.1 -f my-values.yaml
# ...or from a checkout of the repo:
helm install my-replicare deploy/helm/replicare-controller -f my-values.yaml
my-values.yaml is where you put your config and point at your image and secrets.
There is no separate config file to manage — the chart renders your config value
into a ConfigMap at /etc/replicare/config.yml and runs the daemon against it.
Configuring the daemon
config is a structured YAML object (the ergonomic default): Helm merges and
overrides it across -f files and --set, so --set config.logging.level=debug
changes just that field and leaves the rest intact. It is the exact
replicare config — any engines, any number of sources, targets,
and syncs.
image:
repository: <your-registry>/replicare
tag: <version>
config:
logging: { level: info, format: json }
observability: { status_addr: ":8080", metrics_addr: ":9090" }
# replicare's own state — Postgres, always OUTSIDE this chart (see below).
state_store:
engine: postgres
postgres: { host: state.rds.amazonaws.com, port: 5432, database: replicare_state,
user: replicare, password: ${STATE_PW}, sslmode: require }
sources:
cache:
engine: redis
redis: { host: my-redis.default.svc.cluster.local, port: 6379, password: ${SRC_PW:-} }
targets:
replica:
engine: redis
redis: { host: master.my-ec.cache.amazonaws.com, port: 6379, tls: require,
user: ${EC_USER}, password: ${EC_PW} }
syncs:
- { name: cache-to-replica, source: cache, targets: [replica], include: ["*"],
tuning: { drain_interval: 1s, drain_batch: 1000 } }
secret:
existingSecret: my-replicare-secrets # provides STATE_PW / SRC_PW / EC_USER / EC_PW
Two caveats for the structured form:
- Quote any all-digit scalar you inline (e.g. a numeric-only password) so it
decodes as a string, not an int.
${VAR}placeholders need no quoting here — they render in block style. - Prefer a verbatim string for byte-exact control: set
configto a block-scalar string and the chart emits it unchanged (Helm can't merge a string, but nothing is re-serialized either).
Secrets
Keep passwords and tokens out of config with ${VAR} placeholders — replicare
expands env vars in any field. Supply them via a Secret, injected into the pod
with envFrom:
secret.existingSecret— reference a Secret you manage (preferred for production; the chart creates none, so secrets never live in Helm state).secret.env— a map the chart renders into a Secret (fine for dev).
Endpoints & monitoring
The pod serves the operator HTTP surface (see operations):
/healthz— liveness/readiness (both probes use it), on the status port./status— per-sync phase, progress, lag (JSON); whatreplicare statusrenders./metrics— Prometheus scrape.
The Service exposes the status and metrics ports. Set
serviceMonitor.enabled: true to scrape /metrics via the Prometheus Operator.
Port alignment:
service.ports.status/.metricsmust match the addresses yourconfigbinds (observability.status_addr,metrics_addr). The defaults line up at:8080/:9090; if you change one, change both.
Checking progress without Prometheus/Grafana
When you can't run a metrics stack next to the pod — e.g. the daemon runs inside the source
cluster for network reasons — exec the CLI directly against the mounted config. The image is
minimal, so target the binary rather than a shell (kubectl exec runs a command without needing
/bin/sh):
kubectl exec deploy/replicare -- /usr/local/bin/replicare status /etc/replicare/config.yml # live counts + delta backlog
kubectl exec deploy/replicare -- /usr/local/bin/replicare verify /etc/replicare/config.yml # source<->target convergence (exit 0 = converged)
status is live by default (source/target row counts + per-target delta backlog); verify is a
read-only convergence spot-check. Both accept --json, --sync <name>, and --watch <dur>. See the
CLI reference and Checking progress from the CLI.
The runtime image is Chainguard wolfi-base, which also ships /bin/sh if you want an interactive
session for ad-hoc debugging (kubectl exec -it deploy/replicare -- /bin/sh) — but the direct-exec
form above needs no shell.
Single-active, rollouts & state
- Keep
replicaCount: 1. replicare is single-active per sync — a second daemon just stands by on the state store's ownership lock, so extra replicas are wasted and never run two writers. The chart uses aRecreaterollout so the old daemon fully exits (releasing its lock) before the new one starts. - Config changes roll the pod automatically: a checksum annotation over the
rendered ConfigMap means
helm upgradeafter editingconfigrestarts the daemon cleanly. - Pause one pipeline without touching the others by setting
enabled: falseon that sync (a per-pipeline switch; omit it ortrueto run). Because it's a config change, the upgrade's pod rollout applies it — it is start-time, not a live command. Prefer editing the sync's block in your values file by name, so you pause the pipeline you mean to:
thenconfig: syncs: - name: pg-pipeline # ... enabled: false # pauses THIS pipeline only - name: redis-pipeline # ... # (no enabled / true) keeps runninghelm upgrade -f values.yaml. You can use--set, but it addresses syncs by array index, not name —config.syncs[0]is the first sync in your list, which may not be the one you want:
Verify which sync actually paused in the pod logs:# WRONG target if redis-pipeline is the 2nd sync — this pauses syncs[0] (e.g. pg-pipeline): helm upgrade ... --set-string 'config.syncs[0].enabled=false' # Pause the second sync instead: helm upgrade ... --set-string 'config.syncs[1].enabled=false'sync disabled (paused); skipping sync=<name>for the paused one, andsync streaming sync=<name>for each still running.replicare statusalso shows[PAUSED]per sync. The paused sync's source capture stays installed, so re-enabling it drains the queued backlog with no data loss. Full semantics and the long-pause caveat (source-side delta growth): configuration.md → Pausing a sync. - The state store lives outside the chart, always. Every sync needs a Postgres
state store (v1's only backend) — including Redis→Redis and MySQL→MySQL. Point
state_storeat your own managed Postgres (e.g. RDS). The chart never bundles it; it is control-plane bookkeeping, separate from your data.
Security
The container runs non-root (uid 65534), with a read-only root filesystem
and all capabilities dropped (a writable emptyDir is mounted at /tmp). Grant
the daemon's database/Redis roles the least privilege documented in
deploy/ — grants-*.sql for Postgres/MySQL and
acl-*-redis.txt for Redis (note the explicit +restore on Redis
targets). Mount a CA bundle via extraVolumes/extraVolumeMounts if you use
tls: verify-full.
Worked example: in-cluster Redis → ElastiCache
A common shape is replicating a self-managed Redis running in EKS to ElastiCache. Run replicare as a pod in the same cluster — with the AWS VPC CNI the pod gets a VPC IP, so it can reach both the in-cluster Redis (via its Service DNS) and ElastiCache (only reachable inside the VPC; open its security group to the node/pod SG). Watch-outs specific to that move:
- Version direction:
RESTORErejects a newer-than-target payload, so the ElastiCache engine version must be ≥ the source version, or pre-flight blocks. RESTOREmust be permitted on the target (it is@dangerous; grant+restorevia RBAC). Runreplicare validateagainst the real endpoint first — some managed tiers restrict commands.- TLS + AUTH: set
tls: require(orverify-full) and the RBACuser/passwordon the target block. Redistlsdefaults todisable, so set it explicitly. - Cluster-mode ElastiCache: use
mode: clusterseeded with the configuration endpoint (DB 0 only); cluster-mode-disabled connects asstandaloneto the primary endpoint. - Deletes lag the sweep interval, and big keys briefly block the source on
DUMP— see the Redis engine page and troubleshooting.
Validate before applying
helm lint deploy/helm/replicare-controller
helm template my-replicare deploy/helm/replicare-controller -f my-values.yaml \
| kubectl apply --dry-run=client -f -
replicare-controller Helm chart
Deploys the replicare daemon (replicare run) on Kubernetes: one replica that
brings up and streams every sync in your config, exposing the /status,
/healthz, and /metrics endpoints.
The chart is published as an OCI artifact at
ghcr.io/rudimk/replicare-controller — a package distinct from the daemon image
(ghcr.io/rudimk/replicare). Both are stamped with the same version on each
vX.Y.Z release.
Install
# From the published OCI chart (recommended) — pin the version:
helm install my-replicare oci://ghcr.io/rudimk/replicare-controller \
--version 0.1.1 \
-f my-values.yaml
# ...or from a checkout of the repo:
helm install my-replicare deploy/helm/replicare-controller \
-f my-values.yaml
The chart's appVersion defaults image.tag to the matching daemon image
(ghcr.io/rudimk/replicare:<version>), so a versioned OCI install pulls a matched
chart + image pair. Override image.tag to decouple them.
How it's configured
replicare is driven by one config file. This chart renders the config value
into a ConfigMap mounted at /etc/replicare/config.yml — you get replicare's full
expressiveness (any engines, any number of sources/targets/syncs). See the
configuration reference.
config is a structured YAML object by default, so Helm merges and overrides
it the normal way — layer -f files or --set config.logging.level=debug and only
that field changes:
config:
logging: { level: info, format: json }
observability: { status_addr: ":8080", metrics_addr: ":9090" }
state_store:
engine: postgres # always OUTSIDE this chart — point at your own Postgres
postgres: { host: state.rds, port: 5432, database: replicare_state, user: replicare, password: ${STATE_PW}, sslmode: require }
sources:
cache: { engine: redis, redis: { host: my-redis, port: 6379, password: ${SRC_PW:-} } }
targets:
ec: { engine: redis, redis: { host: my-elasticache, port: 6379, tls: require, user: ${EC_USER}, password: ${EC_PW} } }
syncs:
- { name: cache-to-ec, source: cache, targets: [ec], include: ["*"], tuning: { drain_interval: 1s } }
Two small caveats for the structured form: quote any all-digit scalar you inline
(e.g. a numeric-only password) so it decodes as a string, not an int; ${VAR}
placeholders need no quoting (they render in block style). If you'd rather hand the
daemon a byte-exact config, set config to a block-scalar string instead and
it is emitted verbatim.
Pausing a pipeline. Each sync (and cluster) takes an optional enabled: true|false
— set it false to pause just that pipeline; the others keep running. Because it's a
config change, the helm upgrade pod rollout applies it (it's start-time, not live).
Prefer editing the sync's block by name in your values so you pause the right one:
config:
syncs:
- name: pg-pipeline
enabled: false # pauses THIS pipeline only
- name: redis-pipeline
# ... # keeps running
--set works too but addresses syncs by array index, not name — config.syncs[0]
is the first sync in your list, which may not be the one you intend
(--set-string 'config.syncs[1].enabled=false' for the second, etc.). After the
rollout, confirm which sync paused in the pod logs (sync disabled (paused); skipping sync=<name> vs sync streaming sync=<name>) or via replicare status ([PAUSED]).
The paused sync's source capture stays installed, so re-enabling it drains the queued
backlog with no data loss (a long pause grows the source-side delta tables). Full
semantics: configuration.md → Pausing a sync.
Secrets stay out of config using ${VAR} placeholders (replicare expands env vars
in any field), supplied via a Secret:
secret:
# Option A — let the chart create a Secret (fine for dev):
env: { EC_USER: replicare, EC_PW: the-token }
# Option B — reference a Secret you manage (preferred for prod; `env` is ignored):
existingSecret: my-replicare-secrets
The pod loads the Secret via envFrom, so every key becomes an env var the config
can reference.
Key values
| Value | Default | Notes |
|---|---|---|
image.repository / image.tag | ghcr.io/rudimk/replicare / chart appVersion | the daemon image (a separate GHCR package from this chart) |
replicaCount | 1 | keep at 1 — single-active per sync; more just stand by |
config | a Redis→Redis sample | your full replicare config as a structured map (mergeable), or a string for a verbatim config |
secret.existingSecret | "" | reference an existing Secret for ${VAR} values |
secret.env | {} | chart-managed Secret (dev) |
service.ports.status / .metrics | 8080 / 9090 | must match observability.status_addr/metrics_addr in config |
serviceMonitor.enabled | false | scrape /metrics via the Prometheus Operator |
resources, nodeSelector, tolerations, affinity | — | standard scheduling knobs |
extraVolumes / extraVolumeMounts | [] | e.g. a CA bundle for tls: verify-full |
Notes & gotchas
- Single active. The rollout is
Recreateso the old daemon exits (releasing its ownership lock) before the new one starts — never two writers. Scaling above 1 replica is unsupported. - Port alignment. The chart's Service/probe ports must match the addresses your
configbinds (observability.status_addr,metrics_addr). Defaults line up at:8080/:9090. - Scraping metrics. With the Prometheus Operator, set
serviceMonitor.enabled=true. Without it (e.g. an OpenTelemetry Collector or plain Prometheus), use the scrape job in the metrics-scraping guide. - Grafana dashboard. Once
/metricsis scraped (viaserviceMonitor.enabledor your own scrape job), importdeploy/grafana/replicare-dashboard.json— it has a configurable Prometheus data source and covers per-table backlog/lag, catch-up, initial-copy progress, throughput, and errors. - State store. Every sync needs a Postgres state store (v1's only backend),
including Redis→Redis and MySQL→MySQL. Point
state_storeat a reachable Postgres. - Config changes roll the pod automatically (a checksum annotation), so
helm upgradeafter editingconfigrestarts the daemon cleanly. - Least privilege. The container runs non-root (uid 65534), read-only root FS,
all capabilities dropped. Grant the daemon roles per
deploy/(grants-*.sql,acl-*-redis.txt).
Validate before applying
helm lint deploy/helm/replicare-controller
helm template my-replicare deploy/helm/replicare-controller -f my-values.yaml | kubectl apply --dry-run=client -f -
Grafana dashboard for replicare
replicare-dashboard.json is a ready-to-import Grafana dashboard for replicare's
Prometheus metrics. It has a configurable data source — you
pick which Prometheus data source it queries at import time (and can switch it
later from the dashboard's Data source dropdown), so it is not tied to any one
Prometheus.
It assumes replicare's /metrics endpoint is being scraped into Prometheus. See
deploy/helm/replicare-controller (the serviceMonitor option, or a raw scrape
job) for how to wire that up.
Import
Grafana UI
- Dashboards → New → Import.
- Upload
replicare-dashboard.json(or paste its contents). - When prompted, select your Prometheus data source for the
Data sourcevariable, then Import.
Provisioning (file-based)
Drop the JSON into a provisioned dashboards folder and point a provider at it, e.g.:
# /etc/grafana/provisioning/dashboards/replicare.yaml
apiVersion: 1
providers:
- name: replicare
type: file
options:
path: /var/lib/grafana/dashboards/replicare
The dashboard's data source is a template variable (${datasource}), so
provisioning does not hard-code one; Grafana resolves it from the current value
(default: your default Prometheus). To pin it, set the datasource variable's
current value or pass DS_PROMETHEUS via the import API.
Variables
| Variable | Meaning |
|---|---|
Data source (datasource) | Which Prometheus data source to query. Choose at import; change any time. |
Sync (sync) | Filter to one or more syncs (label_values(replicare_table_phase_info, sync) — a metric present from the moment a sync starts, so the filter works during the initial copy too). Defaults to All. |
Target (target) | Filter to one or more targets within the selected syncs. Defaults to All. |
Table (table) | Filter the per-table panels to one or more tables (from replicare_table_phase_info, so every replicated table is listed from the start). Defaults to All. |
Component (component) | Filter the per-component panels to one or more FK components. Defaults to All. |
Panels
Organized into sections (Grafana rows):
Initial copy (top, since it is the first thing a new sync does):
| Panel | Query | Reading |
|---|---|---|
| Initial copy (overall) | sum(replicare_rows_copied_total) / sum(replicare_initial_copy_rows_target) | Percent of the rows this run has to copy. The target is the source database's own row estimate (statistics, no scan), so the end can read a little over or under 100% |
| Tables still copying / Tables streaming | count(replicare_table_phase_info{phase=…}) | How many tables are in each phase; copying reaches 0 when the whole sync is streaming |
| Copy rate (rows/s) | rate(replicare_rows_copied_total[2m]) | Current copy speed |
| Progress % / Rows copied vs target (per table) | the same, by (table) | Which tables are done and which are still moving |
Rows copied count only rows that have been committed on the target, so a retried chunk is never counted twice. A table that loads in one statement (a self-referencing table whose link column cannot be emptied) jumps from 0 to done in one step. replicare_rows_copied_total restarts from 0 when the daemon restarts, and a restart re-copies whatever was not finished, so the percentage restarts with it.
Source / Target — the controller checks both endpoints and emits their reachability and two size figures — the whole database and just the replicated tables:
| Panel | Query (source metric) | Reading |
|---|---|---|
| Source up / Targets up | min(replicare_source_up) / min(replicare_target_up) | UP (green) / DOWN (red) — the controller's health-check of that endpoint |
| Source / Target DB size | replicare_source_db_bytes / replicare_target_db_bytes | Whole-database on-disk size (Postgres/MySQL). The source usually runs larger — it carries replicare's capture schema + delta bloat and any unreplicated tables |
| Source / Target replicated size | replicare_source_replicated_bytes / replicare_target_replicated_bytes | Size of just the replicated tables — the apples-to-apples comparison; on a caught-up sync the two converge even when the whole-DB sizes differ |
Why two size figures? The whole-DB gauge counts everything in the database — including replicare's own
replicarecapture schema on the source and any tables you didn't select — so source > target is normal, not drift. The replicated-size gauge sums only the selected tables with the same function on both sides, so it's the series to watch for convergence. See Whole-DB vs replicated size.
Replication lag — the headline lag section:
| Panel | Query | Reading |
|---|---|---|
| Max replication lag (SLO) | max(replicare_replication_lag_seconds) | Worst lag; green < 30s, amber < 5m, red beyond |
| Catch-up ETA | sum(delta_backlog_rows) / clamp_min(sum(throughput),1) | Estimated seconds to converge at the current apply rate |
| Inflow vs drain | throughput vs throughput + rate(sum(delta_backlog_rows)[5m]) | Inflow above drain ⇒ the backlog is growing |
| Replication lag / Total backlog | replicare_replication_lag_seconds, sum(replicare_delta_backlog_rows) | Per-table lag; total rows still to apply |
Per FK component — rolled up by the component label:
| Panel | Query | Reading |
|---|---|---|
| Replication lag (per component) | max by (component) (replicare_replication_lag_seconds) | Which FK component is behind |
| Delta backlog (per component) | sum by (component) (replicare_delta_backlog_rows) | Queue depth per component |
Backlog detail (per table), Throughput & apply latency, Errors & maintenance — per-table backlog rows/bytes/oldest-age; throughput and apply p95; error rate, purge rate, and reseeds. (Metrics as named.)
The Source / Target health and size panels fill in as soon as a sync starts. The lag, backlog and throughput panels fill in once tables reach streaming — during an initial copy they are expected to read No data.
Notes
- Metrics refresh once per drain pass (your
drain_interval, default 1s), so series update at that cadence — the dashboard auto-refreshes every 30s by default. - The backlog and lag series are the catch-up view: after a churn burst they
rise, then fall back to baseline as streaming drains the queue. A backlog that
climbs while
Targets upreads DOWN is the intended "target is unreachable and falling behind" signal. - The dashboard
uidisreplicare-overview; re-importing the same file updates the existing dashboard in place rather than creating a copy.
Scraping replicare metrics (OpenTelemetry Collector / Prometheus)
replicare exposes Prometheus metrics on an HTTP endpoint:
| Default | Set by | |
|---|---|---|
| Path | /metrics | fixed |
| Port | 9090, named metrics | chart service.ports.metrics / config observability.metrics_addr |
| Metric names | replicare_* | the observability contract (see docs/operations.md) |
The chart's Service/Pod also exposes /status and /healthz on the status port
(8080), so a scrape must target the metrics port specifically — not just "the
Service".
There are two ways to get these metrics into your monitoring stack.
Option A — Prometheus Operator (simplest, if you run it)
The Helm chart ships a ServiceMonitor. Just enable it:
helm upgrade --install replicare oci://ghcr.io/rudimk/replicare-controller \
-f your-values.yaml \
--set serviceMonitor.enabled=true
That wires the operator to scrape the metrics port for you; nothing below is needed.
Option B — Prometheus scrape job (OTel Collector prometheus receiver, or plain Prometheus)
If you scrape with an OpenTelemetry Collector (its prometheus receiver embeds a
Prometheus scrape config) or a plain Prometheus, add the scrape job below. It uses
Kubernetes endpointslice service discovery and keeps only replicare's metrics
endpoint, selected by the chart's stable label app.kubernetes.io/name=replicare-controller
(release-independent — works regardless of your Helm release name).
- job_name: replicare
honor_labels: true
scrape_interval: 30s
scrape_timeout: 10s
kubernetes_sd_configs:
- role: endpointslice
# Optional: scope discovery to replicare's namespace to cut SD churn.
# namespaces:
# names: [replicare]
relabel_configs:
# Keep only endpoints of the replicare Service (stable chart label, any release name).
- action: keep
regex: replicare-controller
source_labels:
- __meta_kubernetes_endpointslice_label_app_kubernetes_io_name
# Keep only the metrics port (the Service also exposes status:8080).
- action: keep
regex: metrics
source_labels:
- __meta_kubernetes_endpointslice_port_name
# Nice-to-have target labels.
- action: replace
source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- action: replace
source_labels: [__meta_kubernetes_pod_name]
target_label: pod
metric_relabel_configs:
# Optional: drop anything that isn't a replicare metric (e.g. go_/process_ noise).
- action: keep
regex: replicare_.*
source_labels:
- __name__
Where it goes in an OpenTelemetry Collector
The job slots into the prometheus receiver's embedded config, then feeds your normal
pipeline:
receivers:
prometheus:
config:
scrape_configs:
- job_name: replicare
# ... the job from above ...
service:
pipelines:
metrics:
receivers: [prometheus]
exporters: [prometheusremotewrite] # or otlp, etc. — whatever you already use
For a plain Prometheus, the same block goes under scrape_configs: in prometheus.yml
unchanged.
Notes
- Port name over number. The job keys on the port name (
metrics), not9090, so it keeps working if you changeservice.ports.metrics. If you match on the number instead, use__meta_kubernetes_endpointslice_portwith your configured value. - No scrape annotations. The chart sets no
prometheus.io/scrapeannotations by default, so annotation-based discovery won't find replicare — use the label-based relabeling above (or add your own annotations viapodAnnotationsif your collector discovers by annotation). - RBAC. The collector's ServiceAccount needs
list/watchonendpointslices(andpodsif you set thepodtarget label) in the target namespace(s). Most Kubernetes Collector deployments already grant this. - Scrape interval. replicare refreshes its gauges once per drain pass
(
tuning.drain_interval, default1s), so a 15–30s scrape interval captures the movement without over-sampling. - What to build on top. Once these metrics land in Prometheus, import the Grafana dashboard — it has a configurable data source and covers per-table backlog/lag, catch-up, initial-copy progress, throughput, and errors.
Troubleshooting
The config won't load (exit code 2)
Before any pre-flight, the config is parsed and structurally validated. These are exit-2 failures with a message naming the problem — the earliest thing a new user hits:
${VAR}is unset and has no default → set the env var, or give a fallback (${VAR:-default}). Env expansion applies to any field, not just secrets.- unknown YAML field → strict decoding rejects typos/unknown keys; fix the field name against the configuration reference.
- unknown engine, or a missing engine block →
engine: postgres|mysql|redismust match a present typed block (postgres:/mysql:/redis:). - a sync references an undefined source/target, or crosses engines → a sync is single-engine; its source and all targets must share one engine.
replicare validate <config> reports these without connecting. Exit 2 is
"config/usage"; exit 1 is a clean pre-flight BLOCK or a connection failure
(see CLI reference).
validate reports BLOCK findings / the daemon refuses to start
Pre-flight blocks on an incompatible or missing target type — e.g. a source column whose type doesn't exist on the target, a missing enum/extension, or a narrowing that would lose data. replicare never coerces data to fit, so it stops before starting.
Fix: create the missing type/extension on the target, or widen the target
column, so the pair is compatible. Re-run validate. Risky-but-allowed
narrowings (e.g. int8→int4) are warnings, not blocks — heed them.
A table isn't being replicated
- No primary key / usable unique key → the table is skipped with a warning (it can't be captured). Add a PK/unique key, or exclude it deliberately.
- Not matched by the selection → check the sync's
include/excludeglobs (schema.table).replicare validatelists the replicable and skipped tables.
capture remove fails to drop a trigger
Dropping a trigger requires table ownership (Postgres allows CREATE TRIGGER
with just the TRIGGER privilege, but not DROP TRIGGER). Either run remove as
the table owner, or have the owner drop the replicare_trg_* trigger. replicare's
own delta/track tables and functions are dropped fine by the daemon role.
The source's disk is filling up (delta/replicare schema growth)
The source-side delta tables are a high-churn queue. If a target is down or slow,
deltas can't purge and the queue grows. replicare bounds this with retention:
past retention.max_age/max_bytes, the laggard target is reseeded and the
over-cap deltas are purged. Check:
replicare status config.yml # look for needs-reseed / stale cursors
curl :9090/metrics | grep -E 'delta_backlog|oldest_unconsumed|target_up'
If the backlog is bounded (retention firing) but the physical delta table
still won't shrink, an idle transaction is pinning the source's xmin horizon so
autovacuum can't reclaim. Find and end it:
SELECT pid, state, xact_start, query FROM pg_stat_activity
WHERE state = 'idle in transaction' ORDER BY xact_start;
replicare bounds logical growth regardless, but only ending the idle transaction lets the source reclaim the space.
A target is unreachable
A drain against a down target fails and surfaces across all channels
(replicare_target_up=0, an ERROR target.unreachable log, an errored trace
span). Nothing is lost — the daemon retries each pass and converges when the
target returns. If it stays down past the retention cap, it's reseeded on
recovery. The source is still protected meanwhile (retention runs even while the
target is down).
A sync halted with a type/constraint error
If the target rejects a value at apply time, replicare halts that component
loudly rather than skipping or guessing (a hard product promise — it never
mangles data). The offending row stays dirty and re-applies automatically once
you fix the cause: usually a missing target type/extension or a schema mismatch.
Check the ERROR logs / replicare_errors_total for the specific failure.
"sync owned by another daemon; skipping"
Each sync runs under a single-active lock (v1). Another process already owns it. This is expected if you run two daemons over the same config — only one drives each sync; the other stands by. If it's stale, ensure the previous daemon actually exited (the advisory lock releases on disconnect).
Traces aren't showing up
Traces export only when observability.otlp_endpoint is set, and to an
OTLP/gRPC collector (port 4317 by convention) over plaintext — point it at a
local collector or sidecar. Metrics (/metrics) and the status API work
independently of tracing.
MySQL-specific issues
- The daemon refuses to connect to a MariaDB server. MariaDB is not supported in v1; it is detected and refused at connect. Use a real MySQL (5.7+) source and target.
validateblocks on a non-InnoDB table. MyISAM/MEMORY tables makeBEGIN/COMMITa no-op, breaking per-component atomicity (target) and the atomic delta consume (source). Convert the table to InnoDB.validateblocks on a target table with a secondary UNIQUE key. MySQLINSERT ... ON DUPLICATE KEY UPDATEhas no conflict target, so a secondary unique could silently rewrite the wrong row — pre-flight blocks it rather than risk corruption. Remove the extra unique key or replicate a table without one.- A copy halts with "target requires local_infile=ON". The copy/apply transport
is
LOAD DATA LOCAL INFILE, which needslocal_infile=ONon the target server (a system variable, not a grant). replicare probes it at connect and halts loud when it is off (v1 has no INSERT fallback). Enable it —SET GLOBAL local_infile=1orlocal-infile=ONin my.cnf — and setlocal_infile: truein the target block. - The source's
replicaredatabase keeps growing under a long transaction. See Source footprint — InnoDB's history list is thexmin-horizon analog; end the long-running source transaction.
See the MySQL engine page for the full requirements and the two operational wrinkles.
Redis-specific issues
validateblocks: newer source → older target (RDB version).RESTORErejects a payload whose RDB version exceeds the target's max, so replicare blocks a newer→older pair up front (the error mentions the version gap). Upgrade the target to at least the source's version, or replicate the other direction.validateblocks: a source module is missing on the target. Module-defined types (RedisJSON, RedisBloom, …) produce opaqueDUMPpayloads only the identical module canRESTORE. Load the module on the target, or exclude those keys.- Every apply fails loud with a
RESTOREpermission error.RESTOREis an@dangerousACL command and is not in the usual read-write presets — grant it explicitly with+restoreon the target role (see../deploy/acl-target-redis.txt). - A copy refuses on a big key. Above
big_key_refuse_bytesreplicare blocks loud rather than doing torn incremental transport (which would be unfaithful). Raise the cap if the key genuinely must replicate, or exclude it. Below the cap but abovebig_key_warn_bytesyou'll see a big-key WARN but the copy proceeds. - A deleted key still exists on the target. Deletes are not captured —
they're found by the periodic target-vs-source sweep, so a source
DELconverges only after the next sweep completes (bounded bydelete_sweep_interval). Watchreplicare_delete_reconciliation_lag_seconds; shortendelete_sweep_intervalfor faster delete propagation (at more source/target scan load). Upserts converge faster via the rolling reconciliation scan. - Keyspace notifications missed a change. Notifications are a lossy accelerator only (fire-and-forget, lost on disconnect, node-local in cluster). A missed event is always caught by the next reconciliation/sweep; on a subscription gap replicare forces a full pass. Never rely on them for correctness — they only cut latency.
- "Redis→Redis but it wants a Postgres?" Yes — replicare's own state store is
Postgres regardless of the data engine (v1's only
StateStorebackend). Point it at any small Postgres; it's just control-plane bookkeeping, and Redis writes nothing to its own source. See the Redis engine page. - The daemon refuses to connect to Dragonfly. Dragonfly is blocked in v1 (its
RDB
DUMP/RESTOREcompatibility is unverified). Valkey and (best-effort) KeyDB are accepted; a non-Redis-family server is refused at connect.
See the Redis engine page for the full requirements, the two pre-flight gates, and the operational wrinkles.
Getting more detail
Set logging.level: debug and logging.format: text for verbose, readable
output while diagnosing. replicare status --json gives the full machine-readable
state.
Verifying data integrity — Postgres
Standalone psql queries to confirm a Postgres source and target are
converged, for when you can't reach the replicare pod to run
replicare status / replicare verify. Run each block on
both databases and compare the output.
These mirror what verify does internally: columns are matched by name,
GENERATED … STORED columns are excluded (the target regenerates them), and the
content hash is order-independent (physical row order, index order, and text
collation don't affect it).
The authoritative check is still
replicare verify <config>. This is the no-pod fallback, and its results should agree withverify.
0. Pin session GUCs (run on both, every session)
Essential — replicare canonicalizes these on every connection. Row text must render
identically on both sides (an old source and a newer target) or you get false
mismatches on dates, floats, and bytea. Run this first in every session before
hashing.
SET DateStyle = 'ISO, YMD';
SET TimeZone = 'UTC';
SET extra_float_digits = 3;
SET IntervalStyle = 'postgres';
SET bytea_output = 'hex';
SET client_encoding = 'UTF8';
1. Row counts per table (run on both)
Exact counts for every base table in a schema, via a session-local helper
(pg_temp.* is dropped automatically at disconnect). A hot table reads a few rows
ahead on the source by whatever is in flight — a gap that shrinks toward 0 is
normal streaming, not drift.
CREATE OR REPLACE FUNCTION pg_temp.rc_counts(sch text)
RETURNS TABLE(table_name text, rows bigint) AS $$
DECLARE r record;
BEGIN
FOR r IN SELECT tablename FROM pg_tables WHERE schemaname = sch ORDER BY tablename LOOP
RETURN QUERY EXECUTE format('SELECT %L::text, count(*)::bigint FROM %I.%I',
r.tablename, sch, r.tablename);
END LOOP;
END $$ LANGUAGE plpgsql;
SELECT * FROM pg_temp.rc_counts('public') ORDER BY table_name;
2. Content digest per table (run on both — digests must match)
The real integrity check: order-independent, column-name-matched, excludes
GENERATED … STORED. Identical digest per table = logically converged. Best for
static / lookup tables; for actively-written tables see step 3.
CREATE OR REPLACE FUNCTION pg_temp.rc_verify(sch text)
RETURNS TABLE(table_name text, rows bigint, digest text) AS $$
DECLARE r record; cols text;
BEGIN
FOR r IN SELECT tablename FROM pg_tables WHERE schemaname = sch ORDER BY tablename LOOP
-- name-sorted, generated-excluded column list: robust to column-order differences
SELECT string_agg(quote_ident(column_name), ',' ORDER BY column_name)
INTO cols
FROM information_schema.columns
WHERE table_schema = sch AND table_name = r.tablename AND is_generated = 'NEVER';
IF cols IS NULL THEN CONTINUE; END IF;
RETURN QUERY EXECUTE format(
'SELECT %L::text, count(*)::bigint,
md5(coalesce(string_agg(md5(ROW(%s)::text), '''' ORDER BY md5(ROW(%s)::text)), ''empty''))
FROM %I.%I',
r.tablename, cols, cols, sch, r.tablename);
END LOOP;
END $$ LANGUAGE plpgsql;
SELECT * FROM pg_temp.rc_verify('public') ORDER BY table_name;
Diff the two sides cleanly inside psql, then compare the files in a shell:
-- in psql, on EACH side:
\pset format unaligned
\o /tmp/rc_src.txt -- use /tmp/rc_tgt.txt on the target
SELECT * FROM pg_temp.rc_verify('public') ORDER BY table_name;
\o
# then in a shell:
diff /tmp/rc_src.txt /tmp/rc_tgt.txt # no output = fully converged
3. Hot-table digest over a stable key range (run on both)
Actively-written tables won't match whole-table (the source runs ahead). Hash an immutable historical window instead — a key range below the active tail — which must match exactly. Adjust the table, key column, and range to a span you know isn't being updated.
SELECT count(*),
md5(string_agg(md5(t::text), '' ORDER BY id)) AS digest
FROM public.user_answer t
WHERE id BETWEEN 1000000 AND 1010000;
4. Referential integrity (run on target)
Catches any FK breakage from the copy. The query below generates one anti-join
per foreign key; run it, then paste and run the emitted SELECT … UNION ALL …. Every
orphans count should be 0.
SELECT string_agg(
format('SELECT %L AS fk, count(*) AS orphans FROM %I.%I c LEFT JOIN %I.%I p ON %s WHERE %s AND %s',
conname,
cn.nspname, cc.relname, pn.nspname, pc.relname,
(SELECT string_agg(format('c.%I = p.%I', a.attname, af.attname), ' AND ')
FROM unnest(con.conkey) WITH ORDINALITY k(attnum,ord)
JOIN unnest(con.confkey) WITH ORDINALITY fk(attnum,ord) ON k.ord = fk.ord
JOIN pg_attribute a ON a.attrelid = con.conrelid AND a.attnum = k.attnum
JOIN pg_attribute af ON af.attrelid = con.confrelid AND af.attnum = fk.attnum),
(SELECT string_agg(format('c.%I IS NOT NULL', a.attname), ' AND ')
FROM unnest(con.conkey) k(attnum)
JOIN pg_attribute a ON a.attrelid = con.conrelid AND a.attnum = k.attnum),
format('p.%I IS NULL',
(SELECT af.attname FROM pg_attribute af
WHERE af.attrelid = con.confrelid AND af.attnum = con.confkey[1]))
), E'\nUNION ALL\n' ORDER BY conname)
FROM pg_constraint con
JOIN pg_class cc ON cc.oid = con.conrelid JOIN pg_namespace cn ON cn.oid = cc.relnamespace
JOIN pg_class pc ON pc.oid = con.confrelid JOIN pg_namespace pn ON pn.oid = pc.relnamespace
WHERE con.contype = 'f' AND cn.nspname = 'public';
Caveats
- Live drift. On hot tables use the step 3 stable-range check, or run during a quiet window. The whole-table digest (step 2) is for static / lookup tables, which must match exactly.
- Schema must match. If the target is missing or has an extra column, step 2's digest differs — that's correct, it's flagging a real schema mismatch, not spurious drift.
- Order & collation don't matter. Step 2 orders rows by their hash, so different heap / index order or text collation never cause a false diff.
- Version gap is handled. With the step 0 GUCs pinned, values render identically
across a source/target major-version gap (floats, timestamps,
bytea, intervals). - Nothing is left behind. The
pg_temp.*helpers are session-local and vanish at disconnect — safe to run against a source you don't own.
See also: MySQL integrity checks · Redis integrity checks.
Verifying data integrity — MySQL
Standalone SQL to confirm a MySQL source and target are converged, for when
you can't reach the replicare pod to run replicare status /
replicare verify. Run each block on both databases and compare.
Like the Postgres guide, these match columns by
name, exclude generated columns, and use an order-independent content hash —
here BIT_XOR over a 64-bit slice of each row's MD5, which folds regardless of row
order and has no GROUP_CONCAT length ceiling.
The authoritative check is still
replicare verify <config>. This is the no-pod fallback.
0. Pin the session (run on both, every session)
replicare moves MySQL values byte-faithfully (LOAD DATA … CHARACTER SET binary), so
the main cross-version rendering risk in a manual hash is the time zone. Pin it to
UTC on both sides before hashing.
SET SESSION time_zone = '+00:00';
SET SESSION group_concat_max_len = 1048576; -- so the generator below isn't truncated
1. Row counts per table (run on both)
information_schema.TABLES.TABLE_ROWS is only an estimate for InnoDB, so generate
exact COUNT(*)s. Run this to emit one SELECT … UNION ALL …, then run the emitted
statement:
SELECT GROUP_CONCAT(
CONCAT('SELECT ''', TABLE_NAME, ''' AS table_name, COUNT(*) AS `rows` ',
'FROM `', TABLE_SCHEMA, '`.`', TABLE_NAME, '`')
ORDER BY TABLE_NAME SEPARATOR '\nUNION ALL\n') AS sql_to_run
FROM information_schema.TABLES
WHERE TABLE_SCHEMA = DATABASE() AND TABLE_TYPE = 'BASE TABLE';
A hot table reads a few rows ahead on the source by whatever is in flight — a gap that shrinks toward 0 is normal streaming, not drift.
2. Content digest per table (run on both — digests must match)
Generate a single statement that hashes every table: name-matched columns, generated
columns excluded, BIT_XOR-folded so row order doesn't matter. Run the generator,
then run the emitted SELECT … UNION ALL … on each side and compare table_name,
rows, digest.
SELECT GROUP_CONCAT(
CONCAT(
'SELECT ''', tab.TABLE_NAME, ''' AS table_name, COUNT(*) AS `rows`, ',
'LPAD(CONV(BIT_XOR(CAST(CONV(SUBSTRING(MD5(CONCAT_WS(''#'', ',
tab.collist,
')), 1, 16), 16, 10) AS UNSIGNED)), 16, 16), 16, ''0'') AS digest ',
'FROM `', DATABASE(), '`.`', tab.TABLE_NAME, '`'
)
ORDER BY tab.TABLE_NAME SEPARATOR '\nUNION ALL\n'
) AS sql_to_run
FROM (
SELECT c.TABLE_NAME,
GROUP_CONCAT(CONCAT('COALESCE(CAST(`', c.COLUMN_NAME, '` AS CHAR), ''\\0N'')')
ORDER BY c.COLUMN_NAME SEPARATOR ', ') AS collist
FROM information_schema.COLUMNS c
JOIN information_schema.TABLES t
ON t.TABLE_SCHEMA = c.TABLE_SCHEMA AND t.TABLE_NAME = c.TABLE_NAME
AND t.TABLE_TYPE = 'BASE TABLE'
WHERE c.TABLE_SCHEMA = DATABASE()
AND c.EXTRA NOT LIKE '%GENERATED%' -- exclude generated columns
GROUP BY c.TABLE_NAME
) tab;
Each column is COALESCEd to a sentinel (\0N) so a real NULL never collides with
an empty string, and CONCAT_WS('#', …) separates columns so ('a','bc') and
('ab','c') don't hash alike.
3. Hot-table digest over a stable key range (run on both)
Actively-written tables won't match whole-table (the source runs ahead). Hash an immutable historical window — a key range below the active tail — which must match exactly. Adjust table, key column, and range to a span you know isn't being updated.
SELECT COUNT(*),
LPAD(CONV(BIT_XOR(CAST(CONV(SUBSTRING(
MD5(CONCAT_WS('#', COALESCE(CAST(id AS CHAR),'\0N'),
COALESCE(CAST(note AS CHAR),'\0N'))),
1,16),16,10) AS UNSIGNED)),16,16),16,'0') AS digest
FROM `rc_it`.`orders`
WHERE id BETWEEN 1000000 AND 1010000;
4. Referential integrity (run on target)
Generate one anti-join per foreign key, then run the emitted SELECT … UNION ALL ….
Every orphans count should be 0.
SELECT GROUP_CONCAT(q SEPARATOR '\nUNION ALL\n') AS sql_to_run FROM (
SELECT CONCAT(
'SELECT ''', k.CONSTRAINT_NAME, ''' AS fk, COUNT(*) AS orphans ',
'FROM `', k.TABLE_SCHEMA, '`.`', k.TABLE_NAME, '` c ',
'LEFT JOIN `', k.REFERENCED_TABLE_SCHEMA, '`.`', k.REFERENCED_TABLE_NAME, '` p ON ',
GROUP_CONCAT(CONCAT('c.`', k.COLUMN_NAME, '` = p.`', k.REFERENCED_COLUMN_NAME, '`')
ORDER BY k.ORDINAL_POSITION SEPARATOR ' AND '),
' WHERE ',
GROUP_CONCAT(CONCAT('c.`', k.COLUMN_NAME, '` IS NOT NULL')
ORDER BY k.ORDINAL_POSITION SEPARATOR ' AND '),
' AND p.`', MIN(k.REFERENCED_COLUMN_NAME), '` IS NULL'
) AS q
FROM information_schema.KEY_COLUMN_USAGE k
WHERE k.CONSTRAINT_SCHEMA = DATABASE()
AND k.REFERENCED_TABLE_NAME IS NOT NULL
GROUP BY k.CONSTRAINT_NAME, k.TABLE_SCHEMA, k.TABLE_NAME,
k.REFERENCED_TABLE_SCHEMA, k.REFERENCED_TABLE_NAME
) fks;
Caveats
- Live drift. On hot tables use the step 3 stable-range check, or run during a quiet window. The whole-table digest (step 2) is for static / lookup tables, which must match exactly.
BIT_XORfolding. Two identical rows XOR to zero and cancel. Every replicated MySQL table has a primary/unique key (replicare requires one), so rows are distinct and this is safe — but it's why the digest is a drift indicator, withverifythe authority. This is the same fold replicare's own Redis verify uses.- Schema must match. A missing or extra column on the target changes step 2's digest — correctly flagging a real schema mismatch.
- Float rendering. MySQL has no
extra_float_digitsequivalent;FLOAT/DOUBLEtext can differ across major versions. If a float-heavy table shows a digest diff with matching counts, spot-check a few rows before treating it as real drift. - Charsets.
CAST(... AS CHAR)renders under the connection charset; use the same client charset on both sides (replicare reads undercharacter_set_results = binary).
See also: Postgres integrity checks · Redis integrity checks.
Verifying data integrity — Redis
Standalone redis-cli checks to confirm a Redis source and target are
converged, for when you can't reach the replicare pod to run
replicare status / replicare verify.
Redis has no query language, so these are shell one-liners and a small script. They
mirror what verify does: hash key + type + canonical logical value, using
type-aware logical reads (GET / LRANGE / SMEMBERS / ZRANGE WITHSCORES /
HGETALL / XRANGE) — never raw DUMP bytes, which are not byte-identical
across Redis versions for equal values. TTL is excluded (replicated relative →
skew-safe, so the exact remaining time differs by design).
The authoritative check is still
replicare verify <config>. This is the no-pod fallback.
1. Key count (run on both)
redis-cli -h "$SRC_HOST" -p "$SRC_PORT" -n 0 DBSIZE
redis-cli -h "$TGT_HOST" -p "$TGT_PORT" -n 0 DBSIZE
The source may read a few keys ahead by whatever the current reconciliation sweep hasn't applied yet — a gap that shrinks toward 0 is normal, not drift. (Deletes on the target lag by the sweep interval by design.)
2. Keyspace content digest (run on both — digests must match)
Save this as rc-redis-digest.sh. It SCANs the keyspace, computes a type-aware
canonical value per key, and XOR-folds each key's MD5 so iteration order doesn't
matter. Run it against source and target; keys and digest must match.
#!/usr/bin/env bash
# rc-redis-digest.sh <host> <port> [db]
# Order-independent content digest of a Redis DB, mirroring `replicare verify`:
# key + type + canonical logical value (NOT DUMP bytes); TTL excluded.
set -euo pipefail
host=${1:?host}; port=${2:?port}; db=${3:-0}
r(){ redis-cli -h "$host" -p "$port" -n "$db" "$@"; }
acc=0 count=0
while IFS= read -r key; do
[ -z "$key" ] && continue
t=$(r type "$key")
case "$t" in
string) v=$(r get "$key") ;;
list) v=$(r lrange "$key" 0 -1) ;; # order is significant, preserved
set) v=$(r smembers "$key" | LC_ALL=C sort) ;; # unordered → sort for canonical form
zset) v=$(r zrange "$key" 0 -1 withscores) ;; # ordered by score, deterministic
hash) v=$(r hgetall "$key" | paste - - | LC_ALL=C sort) ;; # field/value pairs, sorted
stream) v=$(r xrange "$key" - +) ;; # ordered by entry id
*) v="" ;;
esac
h=$(printf '%s\x00%s\x00%s' "$key" "$t" "$v" | md5sum | cut -c1-16)
acc=$(( acc ^ 0x$h ))
count=$(( count + 1 ))
done < <(r --scan)
printf 'keys=%d digest=%016x\n' "$count" "$acc"
chmod +x rc-redis-digest.sh
./rc-redis-digest.sh "$SRC_HOST" "$SRC_PORT" 0
./rc-redis-digest.sh "$TGT_HOST" "$TGT_PORT" 0
Use the same redis-cli binary for both runs so value formatting is identical.
Point the source read at the master, not a replica (a lagging replica reads stale).
3. Per-key spot checks
To eyeball a specific key on both sides, read it with its type's logical command and
diff:
# string
diff <(redis-cli -h "$SRC_HOST" -p "$SRC_PORT" GET mykey) \
<(redis-cli -h "$TGT_HOST" -p "$TGT_PORT" GET mykey)
# hash (sort so field order doesn't matter)
diff <(redis-cli -h "$SRC_HOST" -p "$SRC_PORT" HGETALL myhash | paste - - | sort) \
<(redis-cli -h "$TGT_HOST" -p "$TGT_PORT" HGETALL myhash | paste - - | sort)
# zset (with scores)
diff <(redis-cli -h "$SRC_HOST" -p "$SRC_PORT" ZRANGE myzset 0 -1 WITHSCORES) \
<(redis-cli -h "$TGT_HOST" -p "$TGT_PORT" ZRANGE myzset 0 -1 WITHSCORES)
4. Cluster
--scan and DBSIZE are per-node. On a cluster, run steps 1–2 against each
master and sum/compare per shard (replicare scans, subscribes, and delete-sweeps
per master; delete detection is master-pinned). Only DB 0 is replicated.
Caveats
- Never hash
DUMPbytes. They differ across Redis versions for equal values, so a byte hash would false-positive across a version gap. The script above uses logical reads, which match because transport is value-faithful. - TTL is excluded by design (relative replication → skew-safe; exact remaining
time differs). Don't compare
TTL/PTTL. - XOR folding. Distinct keys fold safely (key is part of each hash); the fold is a
drift indicator, with
verifythe authority. It's the same fold replicare's own Redis verify uses. - Big keys. A multi-million-element value serialized through
redis-cliis slow; spot-check those individually rather than in the full-keyspace script, or accept the runtime. - Streams. Entries are compared; consumer groups / PEL are not deep-verified here
(nor by
verifyin v1). - Value reads may use a replica; delete/keyspace comparisons must use the master — a lagging replica would show phantom missing/extra keys.
See also: Postgres integrity checks · MySQL integrity checks.
Postgres engine
Postgres is replicare's first and reference engine — the engine the engine-neutral copy / apply / delta / reseed / pipeline / observability layers were designed around. It replicates Postgres → Postgres with trigger-based, no-WAL CDC. MySQL and Redis reuse those same neutral layers; this page documents the Postgres specifics. The shared docs (getting-started, configuration, operations, troubleshooting) describe the baseline behavior using Postgres, so treat them as this engine's extended manual.
Status: shipped. Trigger capture, chunked parallel initial copy (text-faithful
COPY), FK-ordered streaming apply (including cyclic components), delta bloat/retention/reseed, full observability, and the daemon lifecycle all run end to end, and Postgres syncs coexist with MySQL and Redis syncs in one daemon. Try it with the Postgres demo.
Supported versions
- Source floor: PostgreSQL 9.6 (best-effort). Target: a modern major (12+). The
CI matrix runs source 9.6 → target 17. See
postgres-version-support.mdfor the rationale (float fidelity on pre-12, no-native-partition path, old PL/pgSQL portability). - Source SQL — trigger functions, capture DDL, introspection — stays conservative
and version-tolerant so it runs against genuinely old servers.
The target is assumed modern, so the apply path uses
INSERT … ON CONFLICT.
How CDC works (the relational baseline)
Postgres has a durable change stream (the WAL), but replicare deliberately does
not use it — logical replication needs a REPLICATION role and a wal_level
change that managed and locked-down servers won't grant. Instead:
- Capture — in a dedicated
replicareschema on the source, replicare installs per-table delta and track tables andAFTER INSERT/UPDATE/DELETEtriggers. The trigger records only the changed row's primary key (PK-only capture), not the full row image. A key-changingUPDATEenqueues both the old and new PK (= delete-old + upsert-new). - Reconcile — at sync time replicare reads the unconsumed delta rows, re-reads
the current values of those PKs from the live source table, and applies them to
the target. This is self-healing: it always copies the latest value, so
overlapping copy/stream windows converge naturally. Consumption is a
delta MINUS trackset-difference (a dirty-key set, not an ordered log), deleting bydelta_id— which sidesteps the commit-order-≠-assignment-order hazard that a scalar high-water-mark cursor would fall into. - Apply — idempotent upserts keyed on the PK (
INSERT … ON CONFLICT DO UPDATE) and deletes by PK, FK-dependency-ordered within each FK component (upserts parent→child, deletes child→parent), with a retry fallback for cycles. - Checkpoint — progress lives in the Postgres StateStore; a crash safely replays from the last confirmed cursor (applies are idempotent).
Capture is enabled first, then a chunked parallel initial copy runs while changes queue — so there is no single frozen snapshot and "too big to snapshot" is not a problem.
Faithful transport (never transform)
Values move verbatim as text COPY and are validated by the target's own input
functions — replicare never coerces, truncates, casts, or "repairs" data; on a
type/constraint error it halts loud rather than corrupting silently. To keep text round-trippable it pins formatting-only session GUCs on
every connection (no privilege needed): DateStyle=ISO,YMD, TimeZone=UTC,
extra_float_digits=3 (critical for float fidelity on pre-12 sources),
IntervalStyle=postgres, bytea_output=hex, client_encoding=UTF8. A pre-flight
compatibility check classifies every source↔target column pair: block on
incompatible/missing types, warn on risky-lossy (e.g. int8→int4), ok on
identical/widening.
Requirements
- Every replicated table needs a primary key or usable unique key. Tables without one are skipped with a loud warning surfaced in logs and telemetry.
- The target schema must pre-exist. replicare replicates data only — no live DDL. You create/migrate target tables with your own tooling; it introspects them to build correct, name-matched apply statements and never installs or alters target types/extensions/enum labels.
- Cyclic / self-referential FKs are handled by case: a nullable
cycle uses a NULL-then-fill two-pass; a
DEFERRABLEcycle loads underSET CONSTRAINTS … DEFERRED; aNOT NULLnon-deferrable cycle is genuinely unloadable without disabling constraints (which replicare won't do), so pre-flight fails loud and asks you to make the FKDEFERRABLEor break the cycle. - A link column inside a unique index where empty values count as equal (for
example
UNIQUE NULLS NOT DISTINCT (email, parent), or a unique index on an expression such asCOALESCE(parent, 0), or one with aWHEREfilter) cannot be emptied during the two-pass load — rows that differ only by that column would clash and the copy would stop with a duplicate-key error. replicare spots these indexes on both the source and the target (bareCREATE UNIQUE INDEXincluded) and handles them without any schema change:- Table that points at itself (an account tree, a category tree): the column is kept at its real value and the whole table is copied in a single statement, which Postgres accepts because it checks this kind of FK only when the statement ends. Live changes keep the real value too. The trade-off: that one table is copied in one piece rather than in parallel chunks, and a restart re-copies it from the start.
- Loop across two or more tables: there is no safe load order, so pre-flight
stops the sync before anything is copied and asks for the FK to be made
DEFERRABLE.
Installing capture on a busy source
Adding a trigger locks the table for an instant, but if something else already holds a
conflicting lock — a long VACUUM, a migration, or an application session left idle in
transaction — the trigger change has to wait, and while it waits Postgres queues every new
query on that table behind it. replicare therefore:
- leaves an already-correct trigger alone on restart, so a restart takes no lock on any application table;
- waits at most 2 seconds for a lock when it does need to add or replace a trigger, then steps back, retries with a growing pause, and after repeated failures stops with an error that names the table, rather than holding the application up.
If that error appears, look for the session holding the table (pg_stat_activity /
pg_locks), let it finish, and restart replicare.
The Postgres advantage: effectively-exactly-once
replicare's delivery is at-least-once with idempotent apply. For a Postgres target, there is a bonus: when the StateStore is the target database, a cursor advance can ride the same transaction as its apply batch — yielding effectively-exactly-once apply per batch. This is unique to Postgres-target syncs: MySQL and Redis targets are strictly at-least-once because their StateStore is a different engine/server, so no single transaction spans both. (At-least-once is still correct and convergent everywhere — a crash may re-apply the last batch, which is harmless because applies are idempotent upserts.)
Delta bloat, retention & reseed
Because capture writes delta tables to a source you may not own, bounding the
footprint is first-class: per-table delta tables (isolated
bloat/contention), a batched, consumption-gated DELETE purge plus aggressive
per-table autovacuum reloptions set at install time (no superuser, no global
change), and bounded retention (age/size) with forced reseed of an over-cap
target to protect the source. The reseed handoff — marking a target needs-reseed,
resetting its track, re-copying, and resuming without gaps — is specified in
reseed-state-machine.md. Per-target delta backlog,
oldest-unconsumed age, purge rate, and reseed events are all surfaced as telemetry
(see operations).
Least privilege
replicare needs no superuser, no REPLICATION attribute, and no wal_level
change. Ready-to-edit grant SQL ships in
../deploy/grants-source.sql and
../deploy/grants-target.sql.
- Source:
USAGEon each replicated schema;SELECT+TRIGGERon each replicated table; and, to hold the capture machinery, eitherCREATEon the database or a pre-createdreplicareschema owned by the daemon role (the no-database-grant option). Teardown asymmetry:CREATE TRIGGERneeds only theTRIGGERprivilege, butDROP TRIGGER(part ofreplicare capture remove) requires table ownership — a least-privilege role can install and run capture but needs ownership (or the owner's help) to fully uninstall the trigger. Dropping replicare's own delta/track tables and functions works with the daemon role since it owns them. - Target: just
SELECT, INSERT, UPDATE, DELETEon the pre-existing target tables. If the StateStore lives on the target database, also grant itCREATEon that database (or pre-create thereplicare_stateschema owned by the role).
Configuration
A Postgres sync is the engine-neutral envelope with a typed postgres: block per
endpoint. The state store is Postgres too (v1's only StateStore backend) and may be
the target, the source, or a separate database:
state_store: # replicare's own bookkeeping (Postgres)
engine: postgres
postgres: { host: state-db, port: 5432, database: replicare_state, user: replicare, password: "${PW}", sslmode: require }
sources:
app:
engine: postgres
postgres: { host: app-db, port: 5432, database: app, user: replicare, password: "${PW}", sslmode: verify-full }
targets:
warehouse:
engine: postgres
postgres: { host: wh-db, port: 5432, database: warehouse, user: replicare, password: "${PW}", sslmode: require }
syncs:
- name: app-to-warehouse
source: app
targets: [warehouse]
include: ["public.*"] # schema.table globs
exclude: ["*_audit"]
- The Postgres block uses
sslmode(libpq'sdisable | allow | prefer | require | verify-ca | verify-full), matching Postgres's own vocabulary — MySQL and Redis use atlsfield with the same spectrum. databaseis the connection default; selection globs areschema.tableand may span other schemas. FK components are computed over the selected tables and copied/streamed with intra-component ordering and cross-component parallelism; a dominating "giant component" is warned about with a config override available.- Extra libpq parameters can be passed via
params. - The single-engine rule is enforced: a Postgres source's targets must all be Postgres — never cross-engine.
- All neutral knobs (worker/connection caps, drain interval, batch sizes, retention, observability endpoints, logging, env-var overrides) are documented in configuration.
Validating
replicare validate config.yml
connects to both sides, classifies every column pair (block on incompatible,
warn on risky-lossy), reports the FK components with giant-component and
dangling-FK (FK→excluded table) warnings, flags tables without a usable key, and
detects unloadable NOT NULL cyclic FKs — all before any sync starts. See
cli.md for the full command surface (run, validate, status,
capture, reseed).
Postgres version support
This records replicare's committed Postgres version support floor and the rationale. It is load-bearing: several design decisions hinge on genuinely old source behavior, so the floor is a decision, not an afterthought.
Decision
| Role | Version in the test harness | Support statement |
|---|---|---|
| Source | PostgreSQL 9.6 | Best-effort support back to 9.6. replicare reads from old sources using conservative, version-tolerant SQL. |
| Target | PostgreSQL 17 | Target should be a modern major (12+); apply uses modern features such as INSERT ... ON CONFLICT. |
The CI integration matrix runs source 9.6 → target 17.
Why 9.6 as the source floor
- Float fidelity.
extra_float_digits = 3is critical for round-trippablefloat4/float8text on servers older than 12. A floor of 9.6 actually exercises that regime; a floor of 12+ would leave the most important float-fidelity claim untested. - No declarative partitioning. Native partitioning arrived in PG 10, so 9.6 exercises the "no native partitions" path and the ctid/keyset chunking fallbacks.
- Old PL/pgSQL. Trigger functions and capture DDL must run on old servers; 9.6 is a realistic old target for that portability.
- Availability.
postgres:9.6is still pullable from Docker Hub for CI.
Caveats
- Apple Silicon (arm64): the
postgres:9.6image is amd64-only, so it runs under emulation locally (platform: linux/amd64in the harness). CI runners are amd64, so this only affects local dev speed. - Even older sources (9.4 / 9.5 and earlier): not covered by the CI matrix.
If we later need them, add matrix entries; until then, behavior on pre-9.6
servers is unverified. (Notably, pre-9.0
byteadefaults toescapeoutput, which the modern target still parses — but this is not exercised.)
What is verified vs unverified
- Verified by CI: everything against source 9.6 → target 17.
- Unverified (flagged): sources older than 9.6, and target majors other than the one(s) in the matrix. Revisit if a real deployment needs them.
MySQL engine
replicare's second engine replicates MySQL → MySQL using the same trigger-based, no-binlog CDC model as Postgres and reuses the engine-neutral copy/apply/reseed/pipeline/observability layers unchanged.
Status: shipped. The MySQL engine runs end to end — trigger capture, chunked initial copy (byte-faithful
LOAD DATA), FK-ordered streaming apply (including cyclicNOT NULLcomponents), bloat control (purge/retention/reseed), observability, and the full daemon lifecycle — and coexists with Postgres syncs in one daemon. This page documents the engine's shape and its two operational wrinkles. Try it with the MySQL demo.
Supported versions
- Source floor: MySQL 5.7. Target: 8.4 (the committed CI pair). 5.6 is
best-effort only. See
mysql-version-support.md. - MariaDB is not supported in v1: its dialect has forked from MySQL's, so the engine detects it at connect and refuses it loudly rather than mis-driving it.
Requirements
- InnoDB tables only. Non-transactional engines (MyISAM/MEMORY) break the per-component atomic apply and the source consume/track atomicity, so pre-flight blocks them.
local_infile=ONis required on the target — it is the copy/apply transport (LOAD DATA LOCAL INFILE). replicare probes it at connect and, if it is off, halts loud with an actionable error rather than failing cryptically mid-copy. Enable it (SET GLOBAL local_infile=1, orlocal-infile=ONin my.cnf) and set thelocal_infile: truehint in the target'smysql:block. (A batched-INSERTfallback forlocal_infile-off targets is deferred to a future release — doing it byte-faithfully across column charsets needs careful design, and a hardening release favors a loud requirement over a risky fallback.)- Every replicated table needs a primary key or a usable unique key. Tables without one are skipped with a loud warning.
- No secondary UNIQUE keys on target tables (beyond the replication key):
MySQL's
INSERT … ON DUPLICATE KEY UPDATEfires on any unique key and would silently mutate the wrong row, so pre-flight blocks such target tables. This is a hard faithful-transport guarantee. - Least privilege (source):
SELECT+TRIGGERon the replicated tables,CREATE/owner on thereplicarecapture database — noSUPER, noREPLICATION SLAVE, no binlog access. (Exact grant SQL ships with MM8.)
The two operational wrinkles
Both follow from the v1 architecture and are worth knowing up front:
1. A MySQL sync still needs a Postgres for state
replicare's own operational state — which syncs exist, per-table copy progress,
per-target cursors, and the single-active ownership lock — lives in a
Postgres state store (state_store.engine: postgres), regardless of the data
engine. This state is small control-plane bookkeeping, entirely
separate from your MySQL data: the correctness-bearing replication state (the
delta/track tables) lives on the MySQL source, exactly as Postgres puts them on
a Postgres source.
So to replicate MySQL→MySQL you also point replicare at a small Postgres for its
state. A MySQL-backed state store is a future item behind the pluggable
StateStore interface; it is not in v1.
2. MySQL syncs are strictly at-least-once
replicare's delivery is at-least-once with idempotent apply. For a Postgres target, when the state store is that target database, a cursor advance can ride the same transaction as its apply batch — yielding effectively-exactly-once apply per batch.
That optimization is not available for MySQL syncs: the state store is a different engine/server (Postgres) than the MySQL target, so no single transaction can span both. MySQL syncs are therefore strictly at-least-once — correct and convergent (applies are idempotent upserts keyed on the PK), but a crash may re-apply the last batch. This is harmless by design; it's noted only so you don't expect exactly-once parity with a Postgres-target sync.
Installing capture on a busy source
Adding or replacing a trigger needs an exclusive metadata lock on the table. If an
application transaction that touched the table is still open, the trigger change has to
wait, and while it waits MySQL queues every new query on that table behind it
(lock_wait_timeout defaults to a year). replicare therefore:
- leaves a trigger alone on restart when its stored body already matches, so a restart takes no lock on any application table;
- creates its
replicaredatabase only when it is missing (evenCREATE DATABASE IF NOT EXISTSwaits behind open application transactions once capture is installed); - waits at most 2 seconds for a lock when it does need to change a trigger, then retries with a growing pause, and after repeated failures stops with an error naming the table.
MySQL cannot replace a trigger in one step (drop and create are separate), so changes to that table are not captured during the instant between them. That only happens when a trigger actually changes (an upgrade that alters its body), never on a plain restart.
Configuration
A MySQL sync is an engine-neutral envelope with a typed mysql: block per
endpoint, plus the Postgres state store:
state_store: # replicare's own bookkeeping (always Postgres)
engine: postgres
postgres: { host: state-db, database: replicare_state, user: replicare, password: "${PW}", sslmode: require }
sources:
app:
engine: mysql
mysql: { host: app-db, port: 3306, database: app, user: replicare, password: "${PW}", tls: verify-full }
targets:
warehouse:
engine: mysql
mysql: { host: wh-db, port: 3306, database: warehouse, user: replicare, password: "${PW}", tls: require, local_infile: true }
syncs:
- name: app-to-warehouse
source: app
targets: [warehouse]
include: ["app.*"] # db.table globs (a MySQL schema IS a database)
exclude: ["*_audit"]
tlsaccepts the full spectrumdisable | allow | prefer | require | verify-ca | verify-full(mapped onto the driver, including a customverify-ca).databaseis the DSN default; selection globs may span other databases, and cross-database FK components are handled.- The single-engine rule is enforced: a MySQL source's targets must all be MySQL — never cross-engine.
Validating
replicare validate config.yml
introspects both sides and classifies every column pair (block on incompatible, warn on risky), reports FK components and giant-component/dangling-FK warnings, and enforces the InnoDB / secondary-unique / no-key rules above — all before any sync starts.
MySQL version support (the MF3 floor decision)
This records replicare's committed MySQL version floor and the MariaDB scope decision for the v1 MySQL engine. It is the MySQL counterpart to Postgres version support, and it fixes the CI test pair every MySQL acceptance test runs against.
The committed pair: MySQL 5.7 source → MySQL 8.4 target
CI runs a single concrete pair — mysql:5.7 as the old source, mysql:8.4 as
the modern target — not a broad matrix. This mirrors the Postgres engine's single
9.6 → 17 pair: one integration-mysql job, amd64 (5.7 has no arm64 image, so it
runs under emulation locally on Apple Silicon; CI is amd64 natively).
Why 5.7 is the source floor
5.7 is the oldest source we support, chosen because it is the oldest release that still has the features replicare's introspection and capture rely on, while being old enough to genuinely exercise the "old source → modern target" path:
| Capability | Introduced | Consequence if the floor were older (5.6) |
|---|---|---|
Generated columns (VIRTUAL/STORED in information_schema.COLUMNS.EXTRA) | 5.7 | 5.6 has none; the generated-column exclusion path (MM4) would be untestable there. |
| JSON type | 5.7 | 5.6 lacks it; a headline fidelity-corpus type would be missing. |
Usable information_schema | 5.7 (materially faster than 5.6) | 5.6's information_schema is notoriously slow, degrading introspection. |
sys schema, spatial indexes, etc. | 5.7 | Not load-bearing, but part of why 5.7 is the practical floor. |
5.6 is best-effort only: the engine may work against it, but no acceptance test is committed to it, and the features above are unverified there. If a future need arises, 5.6 support is additive.
Why 8.4 is the target
8.4 is the current LTS. It exercises the modern-target concerns the plan calls
out: utf8mb4 defaults, ROW_NUMBER() window
functions for boundary discovery (MM4a), and the strict-sql_mode/zero-date
interaction the faithful-transport path depends on. The
5.7→8.4 gap is where the old-source fidelity work earns its keep.
MariaDB: out of scope for v1
replicare v1 does not support MariaDB. Although MariaDB began as a MySQL fork,
its dialect, system catalogs, sql_mode semantics, and JSON/sequence behavior have
diverged enough that silently driving it as MySQL would risk exactly the value
corruption faithful transport forbids. The engine therefore detects MariaDB at connect time
(the server advertises MariaDB in VERSION()) and refuses it with a loud
error rather than mis-driving it. MariaDB is a candidate for a future engine
variant behind the same interface, not a v1 goal.
What this pins
- The CI job builds and tests only against
mysql:5.7andmysql:8.4. - Every MySQL milestone's acceptance is verified on this pair.
- If CI cannot run 5.7 (e.g. image unavailability), the plan requires listing which fidelity claims become unverified rather than silently raising the floor.
Redis engine
replicare's third engine replicates Redis → Redis with the same
no-privileged-change-stream philosophy as Postgres and MySQL, but
a very different mechanism: Redis has no WAL/binlog, no triggers, and no way for us
to write capture state onto the source, so CDC is full-keyspace SCAN
reconciliation rather than trigger deltas. It reuses the engine-neutral
copy/apply/state/config/observability layers, plus one additive delete-
reconciliation seam that the relational engines don't need.
Status: shipped. The Redis engine runs end to end — value-faithful
DUMP/RESTOREcopy, reconciliation-driven streaming upserts, a durable target-vs-source delete sweep, the optional keyspace-notification accelerator, standalone and Cluster topologies, TTL fidelity, least-privilege ACLs, and the full daemon lifecycle — and coexists with Postgres and MySQL syncs in one daemon.
How Redis CDC works (it's different)
There is no change stream to tail, so replicare re-reads the source keyspace:
- Upserts — rolling reconciliation
SCAN. Every streaming passSCANs the source,DUMPs the present keys, andRESTORE … REPLACEs them on the target. This is idempotent and convergent: a rapid rewrite simply converges to the last value on the next pass. Nothing is written to the source — no delta tables, noreplicareschema, no source bloat. - Deletes — a durable target-vs-source keyspace diff. A deleted key just stops
appearing in the source scan; there is nothing to "capture." So replicare runs a
separate delete sweep: enumerate the target keyspace,
EXISTS-check each key against the source master, andDELthe ones the source no longer has. This is the correctness mechanism for deletes — durable and self-healing, but its latency is bounded by the sweep interval, not instant (see Delete-propagation latency). - Keyspace notifications — an optional accelerator only. When enabled they shorten latency by flagging changed keys ahead of the rolling scan, but they are fire-and-forget and lossy, so correctness never depends on them (see Keyspace notifications).
Transport — value-faithful DUMP → RESTORE
replicare moves each value as its RDB-serialized DUMP payload and applies it
with RESTORE … REPLACE. It never interprets, coerces, or reconstructs a value —
the same faithful-transport promise as the relational text COPY path. All six native types (string, list, set, zset, hash, stream) plus TTLs move
this way; a stream's consumer groups and PEL are preserved in the payload.
Transport is value-faithful, not byte-faithful: RESTORE re-encodes on the
target and picks its own optimal internal representation, so a target DUMP need
not byte-match the source's across versions. Fidelity is verified by
type-appropriate logical comparison, not DUMP-byte equality across a version gap.
There is no reconstruction fallback. When a payload can't be RESTOREd
(version gate, missing module, oversize), replicare fails loud — it never falls
back to GET/HGETALL/… rebuilding, which would be the forbidden value transform
(and non-idempotent besides). See the two pre-flight gates.
Supported versions & forks
- Source floor: Redis 3.0 (
RESTORE … REPLACE). Target floor: 5.0. Recommended 6.2+ both ends. CI-tested pair: 6.2 source → 7.4 target plus a 3-node 7.4 cluster. - Valkey is supported (shares Redis's RDB format + protocol); KeyDB is
best-effort; Dragonfly is blocked in v1 (its RDB
DUMP/RESTOREcompat is unverified — refused loudly rather than risk corruption). A non-Redis-family server is refused at connect, the same way MariaDB is for MySQL.
Full detail — including the RDB-version directional gate and the module gate — is
in redis-version-support.md.
The two pre-flight BLOCKS
Redis has no type-coercion axis like the relational engines, but it has two
value-transport gates that pre-flight (validate and daemon startup) blocks on:
-
RDB-version directional gate.
RESTORErejects a payload whose embedded RDB version exceeds the target's maximum. So older/equal source → newer/equal target works; newer source → older target fails loud. Pre-flight readsINFO serveron both ends and blocks a newer→older pair up front rather than failing mid-copy. Fix: raise the target to at least the source's version. -
Module gate. A module-defined type (RedisJSON, RedisBloom, …) produces an opaque
DUMPpayload only the identical module canRESTORE. Pre-flight comparesMODULE LISTon both ends and blocks when a source module type is absent on the target — there is no faithful native reconstruction.Managed Redis that disables
MODULE(ElastiCache returnsERR unknown command 'MODULE'; an ACL may deny it withNOPERM) is handled gracefully: replicare treats a blockedMODULE LISTas "no modules" and proceeds, rather than failing to start. Such servers can't load modules throughMODULEanyway, so this only softens the pre-flight gate — a module-typed value, if somehow present, still fails loudly atRESTORE, never silently.+module|listin the target ACL preset is therefore best-effort: grant it where the provider supports it; where it's blocked, replicare degrades on its own.
The operational wrinkles
Both follow from the v1 architecture and are worth knowing up front.
1. A Redis sync still needs a Postgres for state
replicare's own operational state — which syncs exist, per-unit copy progress,
per-target cursors, and the single-active ownership lock — lives in a Postgres
state store (state_store.engine: postgres), regardless of the data engine. This is a real wart for an all-Redis shop: to replicate Redis→Redis
you also point replicare at a small Postgres for its bookkeeping.
Unlike the relational engines, though, Redis writes nothing to the source — its
CDC is capture-less, so there are no source-side delta/track tables. All durable
state a Redis sync keeps is the coarse control-plane state in that Postgres. A
Redis-backed state store is a future item behind the pluggable StateStore
interface; it is not in v1. See redis-statestore.md for
exactly how a Redis sync's rows are represented.
2. Copy progress is coarse (no resumable SCAN cursor)
A SCAN cursor is an opaque reverse-binary index, not durably resumable across
rehashes or restarts. So a Redis unit's copy progress is deliberately coarse:
snapshot-complete-per-unit, with no watermark. A crash mid-snapshot re-runs the
whole SCAN, which is idempotent (RESTORE REPLACE), so nothing is lost — but a
half-finished large copy restarts from the top rather than resuming a chunk.
Configuration
A Redis sync is an engine-neutral envelope with a typed redis: block per
endpoint, plus the Postgres state store:
state_store: # replicare's own bookkeeping (always Postgres)
engine: postgres
postgres: { host: state-db, database: replicare_state, user: replicare, password: "${PW}", sslmode: require }
sources:
cache:
engine: redis
redis: { mode: standalone, host: src-redis, port: 6379, db: 0, user: replicare, password: "${PW}", tls: require }
targets:
replica:
engine: redis
redis: { mode: standalone, host: dst-redis, port: 6379, db: 0, user: replicare, password: "${PW}", tls: require }
syncs:
- name: cache-to-replica
source: cache
targets: [replica] # v1: exactly one Redis target (fan-out is a non-goal)
include: ["app:*"] # key-pattern globs (Redis SCAN MATCH semantics)
exclude: ["app:tmp:*"] # exclude-wins
The full redis: block reference is in
configuration.md. The most-used fields:
| Field | Notes |
|---|---|
mode | standalone (default) | cluster | sentinel (parsed but experimental) |
host / port | standalone seed; nodes (a host:port list) for cluster/sentinel |
db | logical DB index (standalone only; forced 0 in cluster) |
tls | full disable|allow|prefer|require|verify-ca|verify-full spectrum |
read_from_replica | offload value reads to replicas; delete detection still hits the master |
types | optional type filter (string/list/set/zset/hash/stream; needs 6.0+) |
scan_count | SCAN COUNT hint (per-call; distinct from the neutral drain_interval/batch) |
reconcile_interval / delete_sweep_interval | cadence of the upsert scan and the delete sweep |
big_key_warn_bytes / big_key_refuse_bytes | DUMP-and-warn / block-loud thresholds (via MEMORY USAGE) |
notifications | enable the lossy keyspace-notification accelerator (default off) |
ttl_mode | relative (default, skew-safe) | absttl (opt-in, trusted clocks) |
The single-engine rule is enforced: a Redis source's targets must all be Redis — never cross-engine.
Selection — key-pattern globs
include/exclude are key-pattern globs with Redis SCAN … MATCH semantics
(*, ?, […], hash-tags), applied client-side, exclude-wins. There is one
full SCAN per pass regardless (matching is post-scan), except when a types
filter lets replicare prune server-side with SCAN … TYPE (6.0+). A key that
matches no include glob is simply not replicated.
Topology & Cluster
- Standalone picks one logical DB (
db: 0–15). - Cluster runs everything per master node: the reconciliation
SCAN, the delete sweep, and (optionally) the notification subscription each fan out across masters; topology is discovered viaCLUSTER SHARDS/SLOTSand refreshed onMOVED/ASK. The parallelism unit is the shard. Only DB 0 exists in cluster mode (validated). Per-keyDUMP/RESTORE/DEL/EXISTSsidestep cross-slot limits. - Sentinel is parsed and the client is constructed, but v1 does not harden or test failover — treat it as experimental.
Delete detection always reads the master. read_from_replica may offload value
reads to replicas for politeness, but EXISTS/MissingAtSource are pinned to the
master: a lagging replica could report a just-written key absent and cause a false
DEL. This is enforced.
TTL fidelity
TTL is captured with PTTL and, by default, applied relative (RESTORE key <pttl> …) so the target evaluates expiry from its own clock — immune to
daemon/target clock skew. absttl mode is opt-in for trusted-clock topologies and
is computed from the server's own TIME, not the daemon clock. PTTL -1 (no
expiry) maps to no expiry; PTTL -2/expired keys are handled by the delete sweep.
Big keys
replicare never does incremental/torn transport of a large value (that would be
unfaithful). Instead, above big_key_warn_bytes (measured with MEMORY USAGE) it
still DUMPs the whole key and logs a loud big-key warning (accepting the
transient source blocking); above big_key_refuse_bytes it blocks loud. This
applies to both initial copy and streaming re-reads.
Keyspace notifications (optional accelerator)
Setting notifications: true turns on a latency accelerator: one PSUBSCRIBE per
master to __keyevent@<db>__:*, feeding flagged keys into a priority set drained
ahead of the rolling scan. It is never trusted for correctness — events are
fire-and-forget, value-less, lost on disconnect, and node-local in cluster. A
value-less event only marks a key dirty; the current value is always re-read, so a
set and a del both flow through the same re-read → RESTORE-or-DEL path. On
any subscription gap/reconnect, replicare forces a full reconciliation + delete
sweep for that shard. With notifications off, the engine falls back cleanly to pure
reconciliation. Enabling notifications on the server needs CONFIG SET
(notify-keyspace-events) or, on a managed tier like ElastiCache, a parameter
group set out-of-band.
Delete-propagation latency (the thing to know)
Because deletes are not captured — they're found by the periodic target-vs-
source sweep — a source DEL converges on the target only after the next sweep
completes, bounded by delete_sweep_interval (and the time to SCAN the target).
Upserts converge faster (the rolling reconciliation, accelerated by notifications
when on). Two Redis-specific metrics make this observable:
| Metric | Type | Labels | Meaning |
|---|---|---|---|
replicare_delete_reconciliation_lag_seconds | gauge | sync, target, table | Duration of the last completed delete sweep — staleness of capture-less deletes. |
replicare_deletes_reconciled_total | counter | sync, target, table | Keys DELed by the target-vs-source sweep. |
These are unset for Postgres/MySQL (whose deletes flow through the delta path). The
reused F2 signals apply too, with table carrying the Redis unit (redis.db0);
note that replicare_delta_backlog_* stay ~empty for Redis (there is no source-side
queue to bound).
Least-privilege ACLs (and the +restore foot-gun)
Redis 6+ ACLs; no @admin/@all/DEBUG/CONFIG/KEYS/FLUSH* in the baseline.
Ready-to-edit presets ship in the repo:
- Source (
../deploy/acl-source-redis.txt): read-only —+scan +type +dump +pttl +exists +memory|usage(plus+ping +info). replicare writes nothing to the source. - Target (
../deploy/acl-target-redis.txt):+restore +del +scan +exists +module|list(plus+ping +info).
Foot-gun:
RESTOREis an@dangerouscommand, so it is excluded from the usual read-write presets and must be granted explicitly (+restore). Without it, every apply fails loud. Granting it lets the role overwrite any key in the ACL's key pattern, so scope~…tightly.
The optional notification accelerator additionally needs +psubscribe/+subscribe
(and CONFIG SET where the server isn't preconfigured) on the source. In cluster
mode, grant the same user on every master and add +cluster|shards/|slots.
Active-active mesh (multi-master)
Redis participates in an active-active clusters: mesh (writes accepted on any
node, converging on all under HLC last-write-wins) — same config surface as the
relational engines, detailed in multi-master.md. Because Redis is
capture-less, the (hlc, node_id) version register lives in a parallel metadata
keyspace: for each data key K, a sibling hash \x00rc:m:{…}K (co-located in K's
cluster slot) holding its version + a logical-value fingerprint. The value key is
never wrapped — faithful transport holds. Everything below is cluster-mode-only;
a one-way Redis sync is byte-for-byte unchanged:
- Convergence & loop suppression fall out of one comparison. A re-read RESTOREs/DELs
on a peer only when the incoming
(hlc, node)strictly beats the local register — so replicare's own applied writes lose (no echo storm) and concurrent same-key writes resolve to one value everywhere. Value+register are written in one atomic Lua. - Deletes are version-guarded tombstones, not the stateless sweep (which is disabled
in a mesh — a key absent at one node but present here is a peer's legitimate write, never
an orphan). Tombstones GC by age (
rc_tombstone_retention, default 24h). - Extra ACLs: a mesh member is both source and target and additionally needs the
metadata keyspace RW + logical readers +
+eval/+evalsha— see the mesh sections indeploy/acl-*-redis.txt. - Cost: the metadata keyspace ~doubles key count; correctness runs on the local gate
(
REPLICARE_REDIS=1), not the Postgres-only CI.
Limitations / non-goals (v1)
- Redis fan-out (one source → multiple Redis targets) — a single source
SCANcursor can't serve two targets at different rates without per-target scan state. Single source → single target only (a mesh is different: each edge is its own job). - Sentinel-hardened failover — wired but not hardened/tested.
- Dragonfly — blocked (RDB compat unverified).
- Command-reconstruction / big-key incremental transport — refused (it would transform values; transport is faithful only).
- Cross-key/group atomicity — apply is per-key idempotent, not wrapped in
MULTI/EXEC(a deliberate departure from the relational per-component atomicity). This holds in mesh mode too: each key's value+register pair is atomic, but there is no cross-key transaction. - Cross-engine and point-in-time consistency — never / deferred, as for the other engines. Redis converges eventually and self-heals. Multi-master IS supported (see above); pre-mesh divergent data / pre-mesh deletes resolve by LWW at join.
Validating
replicare validate config.yml
connects to both ends, runs the RDB-version and module gates, reports each replication unit as a single-member component, and (for cluster) validates per-shard — all before any sync starts.
Try it end-to-end
examples/demo-redis/ is a runnable demo: an old 6.2
source, a modern 7.4 target, a Postgres state-store sidecar, and replicare —
replicating app:* keys. Build the image (task docker), then
cd examples/demo-redis && docker compose up and watch the seeded keys, a live
SET, and a DEL propagate. The config is
examples/demo-redis/config.yml.
Redis version & fork support
replicare's Redis engine replicates Redis → Redis using ordinary, grantable
commands (SCAN, DUMP, PTTL, RESTORE, DEL) — no replication link, no
RDB/AOF access, no PSYNC. Transport is DUMP → RESTORE
(value-faithful; the value's RDB-serialized bytes are moved and never
transformed). This page records the supported version range and fork stance.
Version range
| Role | Best-effort floor | Recommended | CI-tested |
|---|---|---|---|
| Source | 3.0 (RESTORE … REPLACE) | 6.2+ | 6.2 |
| Target | 5.0 (RESTORE … ABSTTL opt-in) | 7.x | 7.4 |
The engine keeps source-side commands conservative (SCAN 2.8+, DUMP/RESTORE
2.6+, PTTL 2.6+); SCAN … TYPE (6.0+), ACLs (6.0+), and RESTORE … ABSTTL
(5.0+) are modern-only niceties with older fallbacks.
The RDB-version directional gate (important)
RESTORE rejects a payload whose embedded RDB version exceeds the target's
maximum. Therefore:
- older/equal source → newer/equal target: works (backward compatible).
- newer source → older target: fails loud (
DUMP payload version or checksum are wrong).
replicare's pre-flight (RM2) reads INFO server on both ends and blocks a
newer-source → older-target pair with an actionable error, rather than failing
mid-copy. Upgrade the target to at least the source's version, or replicate the
other direction. This is the Redis analog of the relational "block on
incompatible type."
Forks
Detected from INFO server:
| Server | Status | Rationale |
|---|---|---|
| Redis | Supported | The reference implementation. |
| Valkey | Supported | Shares Redis's RDB format and RESP protocol, so DUMP/RESTORE transport is compatible. |
| KeyDB | Best-effort | Redis-protocol compatible; RDB compatibility generally holds but is not CI-verified. |
| Dragonfly | Blocked (v1) | Its DUMP/RESTORE RDB-format compatibility with Redis is unverified; since transport is DUMP/RESTORE, replicare refuses it loudly rather than risk silent corruption. |
A non-Redis-family server is refused at connect, the same way MariaDB is refused for the MySQL engine.
Modules
Module-defined types (RedisJSON ReJSON-RL, RedisBloom, RedisTimeSeries, …)
produce module-specific opaque DUMP payloads that only RESTORE on a target
loading the identical module. Pre-flight (RM2) compares MODULE LIST on both ends
and blocks when a source module type is absent on the target — there is no
faithful native reconstruction of an opaque module value.
Redis on the Postgres StateStore
replicare's operational state — which syncs exist, per-unit initial-copy
progress, per-target streaming cursors, ownership, and reseed flags — lives in a
dedicated schema on a Postgres database (the target, the source, or a separate
Postgres; see Architecture). This backend is
engine-neutral: the Redis engine ships no new StateStore code. It reuses
the identical Postgres backend by overloading the neutral vocabulary. This
document records those overloads so an operator reading the replicare_state
schema knows how to interpret a Redis sync's rows.
The StateStore is the daemon's own operational state. It is not where Redis change-capture lives — Redis has no source-side delta/track tables; its CDC is SCAN reconciliation (see the Redis engine page).
The overloads
| Neutral concept | Relational meaning | Redis meaning |
|---|---|---|
TableRef (a copy_progress / cursors row's table) | schema.table | a replication unit — the connection's logical DB, rendered redis.db<N> (cluster = db0). One unit per sync. |
SyncDef.Selection | schema.table include/exclude globs | key-pattern include/exclude globs (Redis SCAN … MATCH semantics). |
CopyProgress.Watermark / .Completed | keyset high-water mark + sparse completed PK-ranges | always nil — see coarse checkpointing below. |
CopyProgress.Done | all chunks of a table copied | the unit's SCAN snapshot completed. |
Cursor.LastDelta (engine.DeltaID) | last confirmed source delta-table row id | a per-unit change-id (a reconciliation-pass counter used only for lag/observability). |
Cursor.NeedsReseed | retention-cap forced reseed | identical — retention/reseed is engine-neutral. |
Acquire (advisory lock) | single-active per sync | identical — a Redis sync is single-active too. |
One unit per sync (not one per shard)
A Redis sync replicates a single logical keyspace. Even against a cluster, the
persisted identity is one TableRef (redis.db0); the per-master SCAN
fan-out happens inside the Source (CopyChunk / ReadDirtyKeys), never as
N-per-shard rows. This keeps a single, stable unit ref that round-trips cleanly
through the neutral copy/apply/state layers regardless of topology.
Coarse checkpointing (nil watermark)
A relational table's initial copy is chunked by PK range, so its progress is a
keyset watermark plus a sparse set of out-of-order completed ranges, letting a
restart skip finished chunks. A Redis unit has no ordered key space to chunk
against — its snapshot is a full SCAN pass that is either complete or not. So
its CopyProgress is deliberately coarse: Watermark and Completed stay
nil, and Done flips false → true when the unit's snapshot finishes. A
restart mid-snapshot re-runs the whole SCAN, which is idempotent (RESTORE
REPLACE), so no watermark is needed.
Why no new code
Because the store never interprets these fields — it persists TableRef as
two opaque strings, KeyValues/DeltaID as JSON/bigint, and the advisory lock
by sync name — a Redis unit flows through the exact same read/write path as a
Postgres table. The RM3 tests
(internal/state/postgres/redis_unit_test.go) assert this by round-tripping
Redis-shaped SyncDef, CopyProgress, and Cursor values, and by proving the
ownership lock excludes a second daemon for a Redis sync.
Running active-active replication — the rules
Active-active means several databases (called nodes) that an application can write to on any of them, with replicare keeping them all carrying the same data. This is more demanding than one-way replication (one source, one or more read-only copies), and two things decide whether it works:
- IDs must never collide across nodes.
- Schema changes must reach every node before anything uses them.
Following the two rules below keeps active-active safe. Ignoring them risks silently losing data. replicare is built to stop loudly rather than corrupt, but it cannot fix either problem on its own — both are decisions that live in the database and in the deployment process, before replicare ever sees a row.
For how any of this works under the hood, see the design note,
multi-master.md. This page is just the operating rules.
Rule 1 — Use globally-unique IDs (UUID or ULID), always
The problem. Many tables give each row an id that the database counts up
automatically — 1, 2, 3, 4… (Postgres serial / IDENTITY, MySQL AUTO_INCREMENT). In
active-active this breaks, because each node counts on its own. Two nodes can, at the
same moment, create two different rows that both get id 100. When those rows sync to
each other, the system sees two rows claiming the same id, keeps one, and the other is
silently lost.
The same trap applies to any other column that is forced to be unique and filled in by
the database — for example an auto-generated order_number. It is not only about the
primary key.
The rule. Every id (and every other auto-filled unique value) in an active-active replicated table must be globally unique by design, so any node can create one with no coordination and they will never clash. In practice that means UUID v7 or ULID — large time-based ids that are effectively impossible to collide. Plain auto-counting integers are not allowed in an active-active cluster.
A few clarifications:
- Natural keys are fine. A key that is already unique in the real world (an email address, an ISO country code, an id assigned by the application itself) has no collision problem — the rule is specifically about letting the database hand out sequential numbers.
- It is a schema decision, not a setting. replicare does not and will not invent ids or "sync the counters" between nodes — that approach cannot be made safe. The fix lives in the table design.
- One-way replication is unaffected. This rule is only for active-active. A normal one-way copy can keep using auto-counting integer ids. (See operations → Sequences & identity counters for the separate, one-way concern of keeping a failover copy's counter in step.)
How replicare helps.
- At startup (built today): before it will run an active-active cluster, replicare inspects every table it is about to replicate. If it finds an auto-counting integer used as a primary key or a unique key, it refuses to start, names the offending table and column, and points at this rule. This stops the collision setup from ever going live.
- Continuously (planned, not built yet): the startup check only runs at startup. A schema change made while the cluster is already running — a new table, or a new auto-counting column — could introduce the problem with no restart to catch it. The plan is to run the same check on a schedule (and on demand) so a risky change is flagged the day it ships, not the next time something restarts. Until that exists, it remains the operator's responsibility: do not add auto-counting integer keys to replicated tables in a running cluster.
Rule 2 — Push schema changes to every node first, then deploy the app
replicare never replicates schema changes. It copies data (the rows) between nodes. It does not copy changes to the structure of a table — adding a column, dropping one, renaming one, changing a column's type, or adding a new table. (These structure changes are often called DDL.) This is deliberate and permanent: replicare is a data replicator, not a schema manager.
So a structure change made on one node does not reach the other nodes automatically. It must be applied to each node separately.
What happens if the nodes disagree. Suppose nodes A and B both have a users table,
and a phone column is added to A only. Then a new user is written on A with a phone
number. replicare tries to copy that row to B, B has no phone column, and the copy
fails. replicare does not guess, does not drop the phone, and does not corrupt
anything — it stops copying in that direction and says so loudly in its logs and
status, holding the row. Once the phone column is added to B as well, replicare
resumes on its own and delivers everything it was holding. It heals once the structures
match.
Two cautions about that stall:
- While a node is behind, data for it is piling up, not flowing. replicare waits only so long (about a day by default) before giving up and re-copying that node from scratch. So every node should get the change promptly — nodes must not stay mismatched for long.
- A stall is safe but not invisible — it is a real interruption to that node staying current. The rule below avoids it entirely.
The rule. For every schema change:
- Apply the change to every node first, while nothing is using it yet. Adding an empty new column (or a new table) to all nodes breaks nothing on its own.
- Then deploy the app code that reads or writes the new structure.
Because step 1 finishes everywhere before step 2 begins, no node is ever asked to store something its structure cannot hold. Nothing stalls.
The mirror image and the special cases:
- Dropping a column: reverse the order. Stop using it in the app everywhere first, then drop it from every node last.
- Renaming a column: never rename in place in a running cluster. Instead: add the new column, copy the values across, switch the app to the new column, then drop the old one. (A bare rename looks like "drop one column, add another" and is the easiest way to lose data.)
- Changing a column's type: only safe if the old values still fit the new type (for example widening a number). An incompatible change stalls that node loudly — do it the add-new-column-and-migrate way instead.
- Adding a new table: besides creating it on every node (with globally-unique ids, per Rule 1), replicare must be told to start replicating it — a config change plus a restart on each node. It is a coordinated step, not a one-line migration.
A note on online-schema-change tools (e.g.
gh-ost,pt-online-schema-change): these rebuild a table behind the scenes and swap it in, which does not play well with how replicare captures changes. Prefer the database's native, in-placeALTERfor simple additive changes, and test carefully before using an online-schema-change tool against a replicated cluster.
The short version
- IDs: UUID or ULID on every replicated table. No auto-counting integer keys in a cluster. replicare refuses to start if it finds one.
- Schema changes: replicare will not carry them. Apply every change to all nodes first, then deploy the app that uses it. Drops last; never rename in place.
- If structures ever disagree, replicare stalls loudly and heals once the structures match — but that is a safety net, not a plan.
Multi-master replication — design note
Status: IN PROGRESS. This document is the design; milestones are landing against it. Shipped for Postgres, MySQL, AND Redis: the
nodes:/clusters:config surface (MM0–MM1), loop suppression (MM3), and HLC last-write-wins conflict resolution + tombstones + GC (MM4 Postgres, MM5 MySQL, MM6 Redis) — a full active-active mesh runs on all three engines today: writes accepted on any node converge on all, replicare's own applies are not re-captured and echoed, and concurrent writes to the same key converge to the same value on every node, resolved by a hidden(hlc, node_id)version (no user schema change), with deletes handled by GC'd tombstones (§5.2, §5.3, §5.4, §6.2, §8 status). On Redis the register is a parallel metadata keyspace rather than a side table, since Redis is capture-less (§5.4). Not yet shipped: HA leader election (MM8). The hard constraint on all of this work is that the existing one-way path (source → target(s), changes never flow back) keeps behaving exactly as it does today — see §7 Backward compatibility.
This note covers Postgres, MySQL, and Redis. It records what exists now, why naively wiring a bidirectional topology breaks today, the mechanisms a real multi-master mode would need per engine, the config-schema changes, and a phased plan. It is written to sit alongside the per-engine pages.
Direction set by the project owner (v3): full active-active is required on all
three engines, Redis included (not deferred); N-node full mesh (no fixed node
count); conflict resolution must need no user schema change and no user-specified
column (replicare manages a hidden version itself — §5.3); no priority-fencing;
tombstones live in the replicare schema and are GC'd; it must work with no
superuser anywhere; the config surface is clusters:.
Operator-facing rules. The plain-language rules for safely running an active-active
cluster — the globally-unique-ID requirement and the schema-change procedure — live in
active-active.md. This note is the engineering design behind them; if
the two ever disagree, the mechanisms described here are the source of truth and
active-active.md should be corrected to match.
1. Goal
Keep N database instances (across different clouds) converged, where writes may land on any node — i.e. active-active / multi-master, not just one writer fanned out to read replicas. The target topology is a full mesh of any N (every node ↔ every node — no fixed node count; 3 is only a common example). Ring / partial-mesh forwarding is a deferred efficiency variant (§5.2); the version register (§5.3) already makes such topologies safe.
This is distinct from what replicare does today: one authoritative source fanned out to one or more read-only targets (§3) — which becomes the per-node building block of the mesh rather than the end state.
2. Current behaviour (grounded in the code)
replicare is single-master by construction on every engine. The relevant facts, with file references, because the design has to work with them:
Topology / config (internal/config/config.go, internal/daemon/daemon.go)
- A
Syncis oneSource→ NTargets(config.go:65-72); at runtime the daemon spawns oneSyncerper (sync, target) (daemon.go:129-158), and aSyncerholds a single source, single sink, single target (internal/pipeline/syncer.go) — deliberately one-directional. Config.Validate()(config.go:221-275) validates each sync in isolation. There is no cross-sync graph analysis and no cycle detection: two syncsA→BandB→A, or a ring, pass validation silently.- Parsing is strict — unknown YAML keys are rejected (
config.goLoad), which is important for §6: new keys are additive and old configs are unaffected. - Per-sync single-engine is enforced (
config.go:254-258). - Fan-out (one source → many targets) exists but is explicitly "present, not
hardened" (
syncer.godocstring) and has no end-to-end test; initial-copy progress is keyed(sync, table)without target (internal/state/postgres/progress.go), so two targets on one sync currently share copy progress.
Postgres capture/apply (internal/engine/postgres/)
- Capture triggers are
AFTER INSERT/UPDATE/DELETE ... FOR EACH ROWwith noWHENguard; the trigger function unconditionally enqueues the changed PK (capture_ddl.go). Nocurrent_setting/session check. - The apply connection sets only text-formatting GUCs (
conn.gosessionGUCs) — notsession_replication_role='replica'or anything that suppresses triggers. - Apply is blind source-wins:
INSERT … ON CONFLICT (pk) DO UPDATE SET col = EXCLUDED.colwith noWHERE(sink.go), plus delete-to-match (apply.go). - Delta rows are PK-only +
delta_id+rc_op+ seq/txid/rc_at— no origin column.
MySQL capture/apply (internal/engine/mysql/)
- Triggers are three separate
AFTER I/U/Dtriggers (MySQL has noWHENclause); the bodies unconditionally insert the PK (capture_ddl.go). The UPDATE body'sIF NOT (NEW.pk <=> OLD.pk)is only a PK-change detector, not an origin guard. - MySQL has no
session_replication_roleand no per-session trigger-skip. The apply path sets onlyFOREIGN_KEY_CHECKS(for cyclic components); no user variable, no origin marker. - Apply is blind source-wins:
INSERT … ON DUPLICATE KEY UPDATE col = VALUES(col)- delete-to-match (
apply.go,apply_tx.go).ON UPDATE CURRENT_TIMESTAMPcolumns are carried verbatim from the source (faithfulness) — which removes the one timestamp signal a naive LWW might have used.
- delete-to-match (
- Delta rows are PK-only +
delta_id+rc_op+k1..kn+rc_at— no origin, no txid. - Constraints that matter later: before MySQL 5.7.2 a table allows only one trigger per
(timing,event), so install blocks there if a non-replicare AFTER trigger exists (from 5.7.2
they coexist); InnoDB has no
deferrable FKs;
FOREIGN_KEY_CHECKSis session-level and non-transactional.
Redis capture/apply (internal/engine/redis/, internal/pipeline/delete.go)
- Capture-less. Upserts = rolling
SCAN+RESTORE … REPLACE(idempotent, value-faithful). Deletes = a target-vs-source keyspace diff:SCANthe target,EXISTS-check the source master,DELevery target key missing at the source (delete.go,stream.goMissingAtSource). - The transport record is
{key, ttl, flags, dump}(copy.go,framing.go) — no origin, no version. The per-unitchangeIDis a local lag hint only; it never crosses nodes. - No conflict resolution, no origin, no loop suppression anywhere.
State / ownership (internal/state/)
- Ownership is a per-sync Postgres advisory lock keyed by the sync name; cursors
are namespaced
(sync, target, table). RunningA→BandB→Ameans two separate sync names → two independent locks → nothing coordinates the two directions. - HA / leader election is stubbed (a plain
pg_try_advisory_lock; a second daemon is skipped with a warning), not implemented.
3. The fan-out building block (a prerequisite, not an alternative)
A single-writer fan-out — one authoritative primary → N read-only replicas (one source → N targets) — keeps N instances converged when all writes go to the primary. The owner requires symmetric active-active (writes on any node), so fan-out is not the endpoint here — but it is the building block the mesh is made of: each cluster node fans its local changes out to its N−1 peers. So hardening fan-out is the first substantive milestone (§8 phase 2), on the critical path.
Caveats it must close (they matter for the mesh too):
- Fan-out is present, not hardened and untested end-to-end; initial-copy progress must be re-keyed to include the target before it is trustworthy for 2+ targets.
- Nothing enforces "read-only" on a plain fan-out target — but in a cluster every node is intentionally writable, and correctness there comes from the version register + HLC-LWW (§5.3), not from read-only discipline.
4. Why naive bidirectional wiring breaks today
Wiring A→B and B→A (or a ring) with today's code does not "sort of work with
lag" — it fails destructively:
- Postgres / MySQL — infinite replication loop. A node in a bidirectional
topology is both a source (has triggers) and a target (receives applied writes).
Because no trigger distinguishes replicare's own apply writes, an applied write is
re-captured and shipped back, forever. (PG has
session_replication_roleas a potential fix but it needs superuser, which the project forbids; MySQL has no equivalent at all.) - Redis — data loss + ping-pong. A key freshly written to
Band not yet onAis, to theA→Bdelete sweep, "present on target, missing at source" → it isDEL'd onBbeforeB→Acan propagate it. New writes are indistinguishable from orphans. Upserts also ping-pong via repeatedRESTORE … REPLACE. - All three — no conflict resolution. Concurrent writes to the same key/row on two nodes clobber each other non-deterministically (whichever apply pass runs last wins).
Because Config.Validate() does no cycle detection, a user can enter this
configuration today and get silent corruption. The first concrete change this
note recommends (independent of full multi-master) is to make the config layer
detect and refuse an un-declared cycle — see §6 and §8.
5. Proposed design — the mechanisms multi-master needs
Three capabilities are missing everywhere and must be built. They are engine-neutral in concept, engine-specific in mechanism.
5.1 Node identity & replication origin
Every participating instance gets a stable node_id (configured, persisted with
the daemon's state so it survives restarts). Every replicated change is associated
with the node_id where it originated. This is the foundation for both loop
suppression (5.2) and conflict attribution (5.3).
5.2 Loop suppression (origin filtering)
Shipped for Postgres (MM3). Implemented exactly as described below for the apply-suppression sub-problem (1). Concretely:
- Origin-aware capture.
Source.InstallOriginCaptureinstalls the capture trigger with theWHEN (current_setting('replicare.apply', true) IS NULL)guard (internal/engine/postgres/capture_ddl.go). One-wayInstallCaptureinstalls the byte-identical unguarded trigger — the only difference is that oneWHENclause, asserted by a test — so the one-way path is untouched (§7).- Marked apply/copy. A cluster member's
Sinkis switched to origin marking (EnableOriginMarking); every apply and copy transaction then runsSET LOCAL replicare.apply = '1'(internal/engine/postgres/{apply,sink}.go).SET LOCALis transaction-scoped, so the marker can never leak onto an ordinary connection. (The value is'1'in v1 — presence is all a full mesh needs; it becomes the origin id only when ring/forwarding lands, see below.)- Wiring. A
clusters:block expands into one directed edge per ordered pair of members (full mesh); each edge runs as an independent single-active job in cluster mode — origin-aware capture on its source, marked sink — under its own ownership lock (internal/daemon/cluster.go). A config with no clusters runs exactly the one-way daemon.- Mesh initial copy. Because every member is simultaneously copied-from and written-to by its reciprocal edges, a member's live table transiently carries rows the other edge just applied; a direct
COPYof those bounced rows would collide on the target PK. Cluster edges therefore copy with the idempotent merge path (INSERT … ON CONFLICT DO UPDATE,copy.WithLoadMode(LoadMerge)) and skip the DELETE-range resume. This makes a mesh converge on non-conflicting data (e.g. node-partitioned keys); resolving concurrent same-key writes is MM4 (§5.3).Not yet shipped: the version register + HLC (§5.3), so cross-node writes are applied last-writer-by-arrival, not by
(hlc, node). Same-key conflict resolution is MM4.
Two sub-problems:
- Don't re-capture replicare's own apply writes. When the daemon applies a change to a node, that node's capture must not treat it as a fresh local write.
- Don't forward a change back toward where it came from. In a ring/partial mesh a change is relayed hop-to-hop and must stop after one lap. In a full mesh with direct edges, (1) alone prevents loops (every node receives each change directly and never re-captures it); origin tagging is still needed for (3) and for rings.
Per-engine mechanism for (1):
- Postgres. Add a
WHEN (current_setting('replicare.apply', true) IS NULL)clause to the capture trigger. The apply connection doesSET LOCAL replicare.apply = '<origin_node_id>'inside the apply transaction. Application writes (no GUC set) still capture; replicare's apply writes are skipped. This is a custom GUC (a namespacedreplicare.*setting) — it needs no superuser and no privilege beyond what the daemon already has, unlikesession_replication_role. (When origin forwarding is needed, the trigger instead records the GUC's value as the origin rather than skipping unconditionally — see below.) - MySQL. MySQL has no
WHENand nosession_replication_role. Use a user session variable: the apply connection issuesSET @replicare_apply = '<origin_node_id>'atBeginApply, and each of the three trigger bodies wraps its delta-insert inIF @replicare_apply IS NULL THEN … END IF. User variables are strictly connection-scoped; the apply pool is alreadyMaxOpenConns=1with a pinned connection, so the marker is isolated — but it must be reset on connection release, exactly asFOREIGN_KEY_CHECKSalready is. - Redis. Capture-less, so there is nothing to suppress at "capture" time — the
problem moves to the reconciliation passes (5.4). Loop suppression requires per-key
origin+version metadata that a reconcile pass consults to decide "I already have
the newest version; don't re-RESTORE, don't delete." Redis has no room in the value
(
DUMP/RESTOREis verbatim, and faithful transport forbids wrapping it), so this metadata lives in a parallel metadata keyspace (see 5.4).
v1 is full-mesh only, so sub-problem (2) does not arise — every node ships directly to every other, so skip-suppression alone prevents loops, and the version register (§5.3) is the correctness backstop: any change that is re-delivered but not newer is a no-op, so a stray loop is self-limiting (this is also what makes ring/partial topologies safe to add later). The source-side origin column and hop-to-hop forward-suppression (skip deltas whose origin is the destination; preserve origin across hops) are only needed to make ring/partial meshes efficient and are deferred post-v1 — which keeps the one-way delta table DDL literally unchanged in v1.
Bootstrapping a new member uses the same marker: the initial copy / reseed into a
member runs with replicare.apply / @replicare_apply set (so its cold-copy writes are
not self-captured into an echo storm) and seeds the version register from the source's
register, so copied rows carry their true (hlc, node) rather than a fresh local stamp.
5.3 Conflict resolution — zero-config, replicare-managed
Shipped for Postgres (MM4), exactly as described below. Implementation:
- Version register — a per-table
reg_<hash>table in the sourcereplicareschema keyed by the primary key, holding(rc_hlc_phys, rc_hlc_log, rc_node, rc_deleted); named by a stable hash ofschema.tableso every member computes the same name (internal/engine/postgres/mesh.go). Mesh-only — a one-way source has no register.- HLC — a single-row
replicare.hlc_state(physical, logical, node_id) withreplicare.hlc_tick()(advance for a local write) andreplicare.hlc_observe()(advance past an incoming version on apply). The cluster capture trigger stamps the register with a fresh tick on every local change; a delete (and the old key of a PK-change) writes a tombstone (rc_deleted=true).- Version-guarded apply — the re-read carries each row's version (
rereadVersioned), and apply upserts the value + register, or deletes + tombstones, only when the incoming(hlc, node)strictly beats the target register (internal/engine/postgres/apply_mesh.go); otherwise a no-op. The target HLC is then advanced past the max applied version.- GC — a tombstone is reclaimed once its delete's delta has been consumed by every peer (the delta is purged), reusing the consume+purge machinery as the distributed watermark (
GCTombstones, run each streaming pass).Not yet: seeding the target register from the source's during the initial COPY (the bootstrap-vs-concurrent caveat in §8); MySQL/Redis (MM5–MM6).
replicare deliberately has no commit-order signal (the entire trigger-CDC premise), so it cannot do true causal ordering. And a hard product requirement: conflict resolution must need no user schema change and no user-specified column. The primary key cannot be the tiebreaker — it is the conflict key (two colliding writes share it), so it says which rows conflict, never which value wins. Resolution needs something ordered, and we will not ask the user to add it.
So replicare maintains the version itself, invisibly: for every replicated row/key it
keeps a hidden record PK → (hlc, origin_node) in its own replicare schema — a
side "version register" table (Postgres/MySQL) or the metadata keyspace (Redis), never a
column on the user's tables. hlc is a hybrid logical clock (max(physical_now, last_seen_hlc) + logical_tick): monotonic, wall-clock-tracking, causality-respecting.
Every change is stamped (hlc, node_id); the node_id breaks equal-HLC ties, so the
order is total, with no ties.
- The one policy is HLC last-write-wins, zero-config. On apply of an incoming change
for
PK, compare(hlc_in, node_in)against the register's stored(hlc, node): apply the value and update the register iff strictly greater, else no-op; then advance the local HLC pasthlc_in. This is a LWW-register CRDT (a join-semilattice max) → it converges for any number of nodes regardless of message order. - No priority-fencing, no user version column, no
customresolver in v1. The policy interface stays open for a future custom resolver, but v1 ships exactly this default and there is nothing to configure beyond declaring cluster membership.
Cost, stated honestly: the register is ~one small row per replicated row (PK + ~16 bytes) — the size of an extra index. That is the price of touching none of the user's schema (the rejected alternative was a version column on user tables).
Deletes are tombstones carrying (hlc, node_id) in the same register/keyspace, so a
delete competes with a concurrent update under the same total order. Tombstones and
version records for deleted keys are GC'd once every peer has observed a version ≥
them (a per-cluster min-observed watermark across peers' cursors), keeping the source
footprint bounded.
5.4 Redis specifics (the hardest engine) — SHIPPED (MM6)
Redis needed the most new machinery because it is capture-less and value-opaque. As
implemented (internal/engine/redis/mesh*.go), gated entirely on cluster mode so one-way
Redis is byte-for-byte unchanged:
- Metadata keyspace (the version register). For each replicated data key
Ka sibling metadata hash\x00rc:m:{<K's slot tag>}Kholds{p, l, n, d, v}=(hlc phys, hlc log, origin node, deleted flag, vhash). The reserved control-char prefix keeps it out of the user keyspace and it is excluded from every data operation (copy, verify, reconcile) at one choke point (selection.match). The value key is never wrapped (faithful transport holds); the metadata co-locates inK's cluster slot (a CRC16-derived hash tag) so value + register are written together in one atomic Lua. - Capture-less "capture" (loop suppression + LWW in one comparison). There is no
trigger. The version-aware re-read (
ClusterReadSource) detects a local write by comparing each key's current logical-value fingerprint (vhash, type-aware reads — never DUMP bytes, so it is version-gap-stable) against the stored register; a change stamps a fresh(hlc, node)off the key's own stored version (per-key monotonic, so no shared clock is needed). Apply (OriginMarkingSink) RESTOREs/DELs only when the incoming(hlc, node)strictly beats the local register — the same comparison is both loop suppression (an echoed write already has a version the peer holds → loses) and HLC-LWW. - Tombstone-based deletion replaces the stateless diff. The one-way "missing at source
⇒ DEL" sweep is disabled in cluster mode (a key absent at one source but present
here is a legitimate peer write, not an orphan). A local delete is detected by scanning
the metadata keyspace for a live entry whose data key is gone; it stamps a tombstone that
propagates and version-guard-deletes on peers. Tombstones are GC'd by age
(
TombstoneGC, default 24h — a peer lagged beyond that is a reseed case, MM7). - Conflict resolution on opaque bytes. LWW uses the register
(hlc, node); the value is never merged. Sub-key merges (hash-field CRDTs, à la Redis Enterprise Active-Active) are out of scope.
Bootstrap. No separate protocol: capture-first + idempotent versioned apply reconciles the mesh, so the first streaming pass lazily seeds the register (a key with no metadata is treated as a fresh local write) without spurious tombstones (no pre-existing register to misread), mirroring Postgres's deferred copy-time seeding. A consequence, documented: mesh formation over two nodes that already hold divergent data converges each key to one value under LWW, and a delete made before a node joined the mesh is not preserved (there is no pre-mesh history to tombstone from).
Cost, honestly: the metadata keyspace ~doubles key count, and the change-detector adds a logical value read per key per reconcile pass (big keys are already DUMP-and-warn). A same-version DUMP-hash fast path is a documented future optimization.
5.5 Ownership, cursors, topology expansion
- A mesh expands to a set of directed edges; each edge keeps its own cursor
namespace (extend the existing
(sync, target, table)keying with the mesh/edge and origin). The per-sync advisory lock generalizes to a per-edge (or per-mesh-member) lock. - HA/leader-election is still deferred, but multi-master raises the stakes (a
split-brain daemon in a mesh is worse than in one-way), so the ownership interface
must be ready for
pg_advisory_lock-based leader election before mesh is declared production-ready.
5.6 Active-active id allocation — globally-unique keys
Conflict resolution (§5.3) decides which value wins when two nodes write the same key. It says nothing about how two nodes avoid minting the same key for different rows in the first place — that is a separate, upstream problem, and in a mesh it is the user's schema's job, not replicare's. This section is the decided guidance; a pre-flight guardrail enforces it (§6.3 item 3 — replicare refuses to bring up a mesh member whose replicated tables use a locally-allocated integer key).
The blessed strategy: globally-unique keys — UUID v7 / ULID. Every node mints its own keys with zero coordination — no sequence to sync, no node windows, no interleave, no engine asymmetry — and replicare moves them verbatim (faithful transport already handles a UUID). Id allocation belongs upstream of replication, and UUIDs put the fix exactly there, keeping replicare a pure replicator with nothing to build on the write path.
Do NOT try to sync integer counters across a mesh. This is the single most dangerous
anti-pattern here, and it is why reseed-sequences refuses a cluster member
and why sync_sequences
is one-way only. In a converged mesh every node holds the union of all rows, so
max(id) is the same global maximum everywhere; setting every node's counter to
max(id)+1 collapses them onto one shared counter. Two nodes then both allocate
global_max+1, insert different rows under the same id, replicate, and HLC-LWW keeps
one and silently discards the other — the cardinal sin (silent data loss),
delivered by an operation that exits 0. On a correctly partitioned mesh it is worse: it
yanks a node out of its disjoint window into another's space, destroying the very invariant
that was keeping the cluster collision-free. replicare cannot de-collide two independently
minted ids after the fact — allocation happened before capture ever saw the row — so the
only safe fix is to not collide in the first place.
Fine print — carry these into any "we'll use UUIDs" decision:
- It solves allocation, not conflict resolution. UUIDs guarantee two nodes never mint the same key for different rows; two concurrent updates to the same row are still HLC-LWW's job (§5.3). UUIDs retire one of the two mesh hazards, not both.
- It is a schema migration. Conceptually "one move", but on a live system it means every surrogate-int PK → UUID, every referencing FK, a reindex, and app code that assumes integer ids. Greenfield: trivial. An established production schema: a real project — plan it as one.
- Use time-ordered UUID v7 / ULID, stored binary —
uuidon Postgres;BINARY(16)viaUUID_TO_BIN(…, 1)on MySQL. Random v4 as a PK scatters inserts (on MySQL/InnoDB the PK is the clustered index, so v4 causes page splits and write amplification); v7/ULID restore insert locality, and binary storage halves the width of the 36-char text form. - It must cover every locally-allocated unique value, not just the PK. A secondary
UNIQUEcolumn fed by a sequence (a human-facingorder_number, say) still collides across nodes even with a UUID PK. "We use UUIDs" has to mean every locally-allocated unique value — which is exactly what the §6.3 guardrail checks.
Why not any of the counter schemes real databases use? Production active-active systems
never sync counters — they make allocations disjoint or globally-unique: interleaved
sequences (offset + increment = N; MySQL Group Replication, Galera), coordinated block
allocation (Postgres BDR galloc, CockroachDB per-node blocks — but the app must
allocate through the coordinator), or globally-unique-by-construction (snowflake ids,
unique_rowid(), UUID/ULID). The first two either need an engine feature replicare does
not control or push allocation into the app's write path; the third is the one that needs
nothing from replicare and no coordination at all. That is why UUID/ULID is the blessed
answer and not merely an option.
A transparent integer-PK auto-config (replicare assigning node-disjoint sequence windows on
Postgres) is a deferred, lower-priority fallback for schemas that genuinely cannot move
to UUIDs; it is Postgres-only in practice (MySQL's AUTO_INCREMENT floats to the global max
and cannot be pinned to a low window) and is out of scope here. For now, replicare's only
mesh-side obligation is to police the chosen strategy (§6.3), never to allocate.
6. Config-schema changes
Design principle: additive and opt-in. Existing configs (no new keys) must be
byte-for-byte valid and behave identically. Multi-master is expressed by new, optional
constructs; the existing sources / targets / syncs schema is untouched, and a config
with no clusters: block is exactly today's one-way daemon.
Naming note. A replicare
clusters:entry is a set of active-active peer nodes — do not confuse it with a Redis endpoint'smode: cluster(one sharded Redis). Different scopes; the docs disambiguate.
6.1 New: a nodes: map + node_id (optional)
Cluster members are endpoints that are both read from and written to, so they live in
their own top-level nodes: map (separate from sources:/targets:, which stay
one-way-only). A node has the same shape as any endpoint (an engine + that engine's
connection block) plus an optional node_id — its stable replication-origin identity,
stamped into every change's (hlc, node_id) version. node_id defaults to the node's
map key, so it is rarely written explicitly.
nodes:
us:
engine: postgres
# node_id defaults to "us"; override only if you need a different origin id
postgres: { host: pg-us, port: 5432, database: app, user: replicare, password: ${PW}, sslmode: require }
eu:
engine: postgres
postgres: { host: pg-eu, port: 5432, database: app, user: replicare, password: ${PW}, sslmode: require }
nodes: is optional and ignored on the one-way path.
6.2 New: a clusters: block (optional, top-level)
A cluster names its member nodes (by key into nodes:), the topology, and the
selection/tuning — mirroring a sync but multi-directional. There is no conflict policy
to configure: resolution is the zero-config, replicare-managed HLC last-write-wins of
§5.3 (no version column, no priority). The daemon expands the cluster into the full set of
directed edges internally.
# Existing one-way syncs keep working, unchanged, alongside clusters.
syncs:
- name: analytics-fanout
source: pgprod # a `sources:` key, as always
targets: [warehouse]
include: ["public.*"]
clusters: # NEW, entirely optional
- name: global-app
engine: postgres # single-engine, like a sync
members: [us, eu, ap] # keys into nodes:; each is source AND target; any N >= 2
topology: mesh # v1: full mesh (ring/partial deferred)
include: ["public.*"]
exclude: ["*_audit"]
tuning: { drain_interval: 1s }
# No conflict block: HLC-LWW is automatic (§5.3). Nothing to declare.
Notes:
membersreference endpoint definitions; each must carry anode_idand be reachable as both read (capture+snapshot) and write (apply). Any N ≥ 2 — no 3-node limit.topology: meshis the only v1 value (full mesh, N·(N−1) edges);ring/explicit(partial) are deferred with the origin-column forwarding work (§5.2).- The engine registry still owns per-engine connection parsing;
clustersadds only neutral wiring, consistent with thesyncmodel.
6.3 New validation (and a safety fix for the one-way path)
Config.Validate() gains:
- Cluster validation — single-engine members; every member has a
node_id;topology: meshonly (v1); no member endpoint reused in a conflicting plain sync. (No conflict-policy validation — there is no policy to configure.) Engine-capability guard:validateCluster(internal/config/config.go) rejects a cluster on any engine that can't do active-active, at config-load time with an actionable error, rather than letting it fail opaquely later at capture install. All three shipped engines (Postgres, MySQL, Redis) now qualify, so the blocklist is empty; the guard is retained for a future engine that cannot do active-active. - Cycle detection for plain
syncs— refuse an un-declared cycle among one-way syncs (A→B+B→A, or a ring) not part of aclusters:block. Closes the silent-corruption footgun in §4, worth doing independently. It only adds a rejection for a config already broken today, so no valid one-way config is affected. - Mesh id-allocation policing (SHIPPED — introspection-based pre-flight). At cluster
bring-up, replicare introspects every replicated table's primary key and every unique
key and classifies each locally-allocated integer key (serial / identity /
AUTO_INCREMENT). A key that is not locally DB-allocated — a UUID/text/natural key, or an app-assigned globally-unique integer where the DB does not generate the value — is allowed (those are the collision-free schemes). A locally-allocated integer PK or secondary unique key → refuse to start, loudly, naming the offending table/column and pointing at the UUID/ULID guidance (§5.6), with narrow (32-bit-or-smaller) keys flagged especially. This converts the silent data-loss setup (§5.6) into a loud startup refusal — the same "block on incompatible" philosophy as the type pre-flight. The classifier is pure over the introspected schema (engine.MeshIDAllocationFindings,internal/engine/preflight_mesh.go) → engine-neutral and CI-tested; the daemon runs it only on a cluster edge (internal/daemon/build.go), so one-way syncs are unaffected. Postgresserial(a plain int with anextval(…)default, not reported as identity) is caught via a dedicatedColumn.DefaultSequenceintrospection flag, kept distinct fromIdentityso the apply path'sOVERRIDING SYSTEM VALUEis unchanged. This is the whole active-active id-allocation code deliverable; replicare polices the strategy, it never allocates or syncs counters. (replicare validatedoes not yet walkclusters:, so the refusal currently surfaces atrun/bring-up, not invalidate— a documented follow-up.) Planned — continuous re-check: the classifier runs only at bring-up today, so a mesh-unsafe key introduced by a mid-stream migration (a new table, or a new auto-counting column added to a running cluster) is not caught until the next restart. The plan is to run the same pure classifier on a schedule and on demand, so a risky change is flagged when it ships rather than at the next restart. This is the continuous check referenced inactive-active.mdRule 1.
6.4 State-store & source-schema changes
- The one-way delta table is unchanged — v1 adds no column to it (a full mesh
needs no origin column; §5.2). The version register (
PK → (hlc, node_id, deleted?)) and tombstones are new, mesh-only tables in thereplicareschema, created only for cluster tables. Applied via the existing in-place, idempotent migration runner with aschema_versionbump that preserves in-flight deltas/cursors — never drop-recreate. - Cursors extend their key with cluster/edge for cluster members; the existing
(sync, target, table)shape is unchanged for one-way. - Redis adds the metadata keyspace (the register + tombstones, §5.4); nothing changes for one-way Redis.
7. Backward compatibility — the non-negotiable
Every change above is designed so the one-way path is provably unchanged. The invariants:
- Config. No
clusters:and nonode_id:⇒ identical parse and identical behaviour. New keys are optional; strict parsing still rejects genuine typos. The only new rejection is an un-declared cycle among one-way syncs — a config that is already corrupt today, never a working one. - Triggers (PG/MySQL). The origin guard is a no-op for one-way:
- A one-way source is never applied to by replicare, so the apply marker
(
replicare.applyGUC /@replicare_apply) is never set on it → local writes are captured exactly as today. - A one-way target has no capture triggers (targets aren't sources), so the guard is irrelevant there.
- The guard defaults to "capture" when the marker is unset, so any non-replicare write is still captured. Origin-aware triggers are only installed for tables that participate in a cluster; one-way tables keep today's trigger DDL.
- A one-way source is never applied to by replicare, so the apply marker
(
- Apply path. The apply marker + HLC-LWW version comparison are only engaged for cluster edges. One-way apply remains blind source-wins overwrite + delete-to-match, unchanged, and writes no version register.
- Delta schema. v1 adds no column to the delta table — the version register and tombstones are separate, mesh-only tables. One-way delta DDL is literally unchanged; migrations preserve in-flight state.
- Redis. The metadata keyspace, tombstones, and delete-diff redesign are gated to cluster mode. One-way Redis keeps the stateless SCAN-reconcile + target-vs-source delete sweep verbatim.
- Tests. The entire existing one-way integration suite (PG/MySQL/Redis, plus the
test/loadgenandtest/loadgen-redisconvergence harnesses) must pass unchanged. Multi-master gets its own harness (an N-node cluster convergence + conflict test) rather than modifying the one-way tests.
Rollout is feature-flagged by the presence of a clusters: block: a daemon with none
compiles and runs exactly as before.
8. Implementation status
| Milestone | Scope | Status |
|---|---|---|
| MM0–MM1 | nodes:/clusters: config surface; one-way cycle guardrail; fan-out per-target progress | Shipped |
| MM2 | Per-target initial-copy progress (fan-out hardening) | Shipped |
| MM3 | Postgres loop suppression — origin-aware capture, marked apply/copy, cluster→edge wiring, idempotent mesh copy | Shipped |
| MM4 | Postgres HLC-LWW — version register + HLC, version-guarded apply, tombstones, GC | Shipped |
| MM5 | MySQL mesh — mirror of MM3+MM4 for MySQL (@replicare_apply guard, inline HLC, version-guarded apply, GC) | Shipped |
| MM6 | Redis mesh — metadata-keyspace version register, version-guarded Lua apply, tombstone deletes, age-based GC, slot co-location, config guard lifted (§5.4) | Shipped |
| Id-alloc guardrail | Pre-flight refusal of a mesh member whose replicated tables use a locally-allocated integer PK/secondary-unique key — the §5.6 policing (§6.3 item 3), pure+CI-tested classifier run at cluster bring-up | Shipped |
| MM7–MM11 | Cluster retention/reseed, HA, observability, E2E gate, release | Not started |
What works today (Postgres, MySQL, AND Redis): define members under nodes:, group them
in a clusters: block, and the daemon runs a full-mesh active-active cluster — writes
accepted on any node converge on all, initial copy is bidirectional and idempotent,
replicare's own applies are not re-captured and echoed (loop suppression), and
concurrent writes to the same key converge to the same value on every node under HLC
last-write-wins, with deletes resolved by GC'd tombstones. Proven end-to-end for Postgres by
internal/daemon.TestDaemonTwoNodeMeshConverges + TestDaemonMeshSameKeyConflictConverges,
for MySQL by internal/daemon.TestDaemonMySQLMeshSameKeyConflictConverges, and for Redis by
the engine-level mesh suite internal/engine/redis.TestMesh* (disjoint-key + same-key
conflict convergence, delete propagation, and the peer-write-not-destroyed safety property),
which drive the exact neutral apply path (DrainComponent); Redis correctness runs on the
local gate (REPLICARE_REDIS=1), not the Postgres-only CI. Pure mesh logic (slot hashing,
version codec, LWW order, HLC) is unit-tested in CI (internal/engine/redis.TestSlotTag,
TestCRC16KeySlot, TestVersionBeats, TestHLC*, TestFramingMesh*).
MySQL specifics (MM5). The design mirrors Postgres; three things differ because the engine forces them:
- Loop suppression uses a
@replicare_applysession user variable (MySQL has no triggerWHENclause): each of the three trigger bodies is wrapped inIF @replicare_apply IS NULL THEN … END IF. The variable is CONNECTION-scoped (not transaction-scoped like PostgresSET LOCAL), so it is reset on connection release alongsideFOREIGN_KEY_CHECKS(the apply pins one connection,MaxOpenConns=1), and set + reset around the pinned bootstrap-copy connection. - HLC is a table (
replicare.hlc_state) + inline SQL (a tick in the trigger body, an observe in the apply), NOT a stored function — a data-modifying MySQL function needs SUPER/CREATE ROUTINEunder the managed-provider default, which would break the no-superuser mandate. - Version-guarded apply cannot read the register inside an
INSERT … SELECTthat also targets it (MySQL error 1093), so the winner set is computed once into a staging column and every value/register statement reads that flag.
How MM4 resolves conflicts. For every replicated row the source's capture trigger
stamps a hidden version register PK → (hlc, node_id[, tombstone]) in the replicare
schema — no user column. hlc is a hybrid logical clock (replicare.hlc_tick()), advanced
past every incoming version on apply (replicare.hlc_observe()), so it tracks wall-clock
and tolerates skew. The re-read carries each row's version; apply upserts (or, for a
tombstone, deletes) only when the incoming (hlc, node) strictly beats the target
register, so a stale write arriving out of order loses and a delete competes with a
concurrent update under one total order (node_id breaks equal-HLC ties → no ties). This is
a last-write-wins register CRDT: it converges for any N regardless of message order.
Tombstones are reclaimed once their delete has been consumed by every peer (the delete's
delta is purged), so the register stays bounded.
Bootstrap-vs-conflict caveat. The register is stamped on local writes and updated on apply, but the initial COPY does not yet seed the target register from the source's (a small follow-up). So a value placed purely by initial copy, never re-written and never applied-over, has no local register entry until it is next written or received — a remote write for such a key always wins even if the copied value was newer. This only affects a key written concurrently during a member's initial copy; steady-state and post-bootstrap same-key conflicts resolve correctly (each side stamps its own write).
8.1 Usage — running a Postgres active-active cluster
Define each member under nodes: and group them in a clusters: block. The target schema
must pre-exist on every member (data-only), and the daemon needs a Postgres
state_store (it may live on any member or a separate DB):
state_store:
engine: postgres
postgres: { host: pg-us, port: 5432, database: replicare_state, user: replicare, password: ${PW}, sslmode: require }
nodes:
us:
engine: postgres
postgres: { host: pg-us, port: 5432, database: app, user: replicare, password: ${PW}, sslmode: require }
eu:
engine: postgres
postgres: { host: pg-eu, port: 5432, database: app, user: replicare, password: ${PW}, sslmode: require }
clusters:
- name: global-app
engine: postgres
members: [us, eu] # any N >= 2 (full mesh)
include: ["public.*"]
tuning: { drain_interval: 1s }
- Grants (no superuser). Each member needs the combined cluster-member grant set: the
source-side grants (
TRIGGER+SELECTon replicated tables,CREATE/USAGEon thereplicareschema — or pre-create it owned by the daemon role) and the target-side grants (INSERT/UPDATE/DELETEon the replicated tables), because every member is both. No superuser, noREPLICATION, nowal_levelchange. - What the daemon does. It expands the cluster into one directed edge per ordered pair of members and runs each edge as an independent single-active job: origin-aware capture on the edge's source, a marked (loop-suppressing) sink on its target, and HLC-LWW version-guarded apply. Each edge takes its own ownership lock, so several daemon replicas can share a cluster's edges.
- Conflict resolution is automatic. Writes may land on any node, including the same key concurrently — HLC last-write-wins converges every node to the same value, with no config and no user schema change (see §5.3). The only remaining caveat is the bootstrap-vs-concurrent edge documented above (initial-copy register seeding is a follow-up).
9. Phased plan
Ordered so each phase is independently shippable and low-risk-first:
- Cycle-detection guardrail (small, do first). Make
Config.Validate()refuse an un-declared one-way cycle. Pure safety, no behaviour change for valid configs. - Harden fan-out (small–medium; a mesh prerequisite). Re-key initial-copy progress by target; add a 2+-target test. Each cluster node fans out to its N−1 peers, so this is on the critical path (and independently delivers single-writer fan-out).
- Postgres loop suppression + version register + bootstrap (medium).
node_id; the HLC + version register in thereplicareschema;replicare.applyGUC + triggerWHENguard; apply and cluster-bootstrap copy set the marker and seed the register. Prove no-loop + no-echo on a 2-node PG mesh, no superuser. - Postgres HLC-LWW + tombstones + GC (medium). Version-guarded upsert over
(hlc, node_id); tombstones; coordinated GC. N-node PG convergence + conflict harness. - MySQL mesh (medium). Mirror 3–4 with the
@replicare_applyuser-variable guard in all three trigger bodies; reset-on-release alongsideFOREIGN_KEY_CHECKS. - Redis mesh (large / highest-risk, but co-equal — NOT deferred). Metadata keyspace
as the version register + tombstones; tombstone/version-aware delete reconciliation;
HLC-LWW via the metadata
(hlc, node). Sequenced after 4 (reuses its abstractions). - Cluster lifecycle (retention/reseed). Reseed runs marked, seeds the register, and never resurrects a tombstoned key.
- HA / leader election (before "production-ready"). Per-edge
pg_advisory_lockleader election + cursor fencing, so a cluster can't split-brain.
10. Open questions / risks
- Version-register cost. ~1 row per replicated row (PK +
(hlc, node_id)) — the accepted price of zero user-schema change (the rejected alternative was a version column on user tables). Index-sized but non-trivial for very large tables; measured as milestone acceptance. - HLC clock skew. The
node_idtie in(hlc, node_id)guarantees a total order, so nodes always converge to the same winner regardless of skew; skew only affects which of two truly-concurrent writes is deemed "latest" (a bounded wobble, never divergence). Document the assumed max skew and surface an HLC-skew metric. - Tombstone/register GC coordination. A per-cluster min-observed-version watermark across peers (no global coordinator) — sub-design before the GC milestone ships. Live keys keep their register row; only tombstones + deleted-key records are collected.
- Redis metadata atomicity & cost. A metadata entry per key ~doubles key count and
needs Lua/
MULTIatomicity; big-key and cluster-slot interactions need care (the metadata key must hash-tag to its value key's slot). The single biggest risk item. - Least-privilege, everywhere. The PG
replicare.applyGUC and MySQL user-variable guards are grantable (no superuser); re-verified against managed offerings (RDS/Cloud SQL/etc.) as milestone acceptance, and the grants docs updated. No superuser is required anywhere. - Schema drift across members. One-way assumes the target schema pre-exists; a cluster assumes all members share a compatible schema. Pre-flight must check every member pair, not just source→target.
- Full-mesh edge scaling. N·(N−1) directed edges — fine for small-to-moderate N; beyond ~tens of nodes a hub/ring (the deferred forwarding work) would be preferable.
- Faithful transport is preserved — none of this transforms values; the HLC version/origin are metadata about a change, never a mutation of the replicated value.
11. Summary
| Capability | PG today | MySQL today | Redis today | Needed for multi-master |
|---|---|---|---|---|
| Loop suppression | ✗ (no trigger guard) | ✗ (no guard, no session_replication_role) | ✗ (re-RESTORE loop) | replicare.apply GUC guard / @replicare_apply var guard + version-comparison backstop |
| Conflict resolution | ✗ (blind overwrite) | ✗ (blind overwrite) | ✗ (blind RESTORE) | replicare-managed HLC-LWW over (hlc, node_id) — zero user schema change; tombstones for deletes |
| Version register / origin | ✗ | ✗ | ✗ | node_id + a hidden PK → (hlc, node) register in the replicare schema / Redis metadata keyspace |
| Topology / cycle safety | ✗ (silent cycles) | ✗ | ✗ | clusters: block (full mesh, any N) + cycle-detection validation |
| Delete handling in a cluster | delete-to-match | delete-to-match | destructive diff | tombstone-based, (hlc, node)-aware, GC'd |
| One-way path | ✓ | ✓ | ✓ | must remain unchanged (§7) |
Multi-master is a substantial, multi-phase project across all three engines (Redis
included, not deferred) — heaviest on Redis. Conflict resolution is zero-config
(replicare-managed HLC-LWW; no user column, no schema change, clusters: just declares
membership). The one-way, source-authoritative path stays the default and is protected by
the §7 invariants throughout. If a single writer happens to suffice, hardening fan-out
(§3, phase 2) delivers "same data in N clouds" even sooner — but it is a prerequisite of
the mesh, not an alternative to it.
Delta purge, retention & the reseed state machine
Status: implemented and shipped. This began as the design artifact for the
purge/retention/reseed work and is now the reference for the code that ships it:
internal/reseed (the state machine) and internal/pipeline/stream.go (the daemon
loop that enforces retention and drives reseeds every pass). It specifies the purge
mechanics, the retention caps, and — the load-bearing part — the reseed state
machine, with a stated invariant for why no source change is lost across the
handoff.
This re-derives the two guarantees M5c has to hold together: capture-first initial copy and crash-safe at-least-once consumption.
1. What M5c protects against
The source-side delta table is a high-churn queue we write to a database we may not own. A slow or down target pins it: a delta row cannot be purged until every target has consumed it, so one laggard makes the source grow without bound. M5c bounds that growth in three layers:
- Consumption-gated purge — the baseline: a delta consumed by all targets is deleted. Batched + rate-limited; paired with the aggressive per-table autovacuum already set at capture install.
- Bounded retention (age + size) — a cap on how far the queue may grow because of a laggard. When a target pushes the queue past the cap, that target is sacrificed rather than the source.
- Forced reseed — the sacrificed target is marked
needs_reseed, its pinning deltas are dropped, and it is brought back to convergence by a fresh full copy + streaming resume. This is the state machine specified in §4.
The xmin horizon is the residual risk the batched-DELETE path cannot
beat on old sources: autovacuum cannot reclaim dead tuples older than the
source's oldest running transaction, so a long idle-in-transaction session makes
the delta table bloat regardless of purge. M5c cannot fix that on an old
source — it surfaces it loudly (telemetry) and still bounds logical growth
via retention/reseed. On PG ≥ 10 the partition+DROP path avoids the trap; see §5.
2. Objects involved (grounding)
Per captured source table (keyed by a stable rel_id, in the source replicare
schema):
replicare.delta_<rel_id>— the dirty-key set. Columns:delta_id bigserialPK,rc_op, key columnsk1..kn, andrc_txid/rc_seq/rc_at(lag/ordering hints only, never a resume cursor).replicare.track_<rel_id>— per(target, delta_id)consumption record. A row means that target has applied that delta. PK(target, delta_id).
Daemon state (the StateStore, a separate Postgres schema, not to be
confused with the source-side tables):
cursors(sync, target, table)carriesphase(initial_copy/streaming),last_delta(lag only), andneeds_reseed— the flag M5c drives.copy_progress(sync, table)— the resumable initial-copy watermark.
Separation of concerns: Source.Purge touches only source objects
(delta/track). It never writes the StateStore. It returns which targets it
sacrificed; the engine-neutral orchestration (internal/reseed) is what writes
needs_reseed, records events, and runs the re-copy. This keeps the delta
lifecycle self-contained on the source and the daemon state on the StateStore.
3. Purge mechanics
Source.Purge(ctx, table, targets, retention) → (PurgeStats, error)
targets is the full configured target set for the sync — required because
"consumed by all targets" is only decidable against the complete set (a target
that has consumed nothing has no track rows, yet still pins the queue).
3.1 Consumption-gated batched purge (baseline, all versions)
A delta is purgeable once every configured target has a track row for it. Batched
by delta_id to bound transaction churn, looped until a pass deletes < batch:
WITH purgeable AS (
SELECT d.delta_id
FROM replicare.delta_<rel> d
WHERE (SELECT count(*) FROM replicare.track_<rel> tr
WHERE tr.delta_id = d.delta_id AND tr.target = ANY($targets)) = $ntargets
ORDER BY d.delta_id
LIMIT $batch
)
DELETE FROM replicare.delta_<rel>
WHERE delta_id IN (SELECT delta_id FROM purgeable);
count(*) is exact because track PK is (target, delta_id) — each target
tracks a delta_id at most once, so the count is the number of distinct targets
that consumed it. When it equals $ntargets, all have. Deletes are always
by delta_id (never by PK), preserving the read-and-clear discipline.
3.2 Retention evaluation (per target)
For each target T, its pinned backlog is delta MINUS track_T. From it:
- oldest-unconsumed age =
now() - min(rc_at)over the backlog. (rc_atis a hint, never a cursor — it is fine as a retention signal because we only ever compare it to a wall-clock cap, never resume from it.) - size signal =
pg_total_relation_size('replicare.delta_<rel>')— the actual on-disk footprint of the queue, which is what "protect the source" means. Attributed to the laggard: the target with the oldest unconsumed delta.
T is over-cap when MaxAge > 0 && age > MaxAge, or
MaxBytes > 0 && size > MaxBytes && T pins the oldest delta. Both caps are
opt-outable (0 = unbounded); age defaults on (24h), size defaults off.
3.3 Retention-forced purge (sacrifice the laggard)
When a set R of targets is over-cap, Purge, in one pass:
- Records
RinPurgeStats.TargetsReseeded(the orchestration will mark themneeds_reseed). - Resets each
R's track:DELETE FROM track_<rel> WHERE target = ANY($R). A reseeded target is about to be re-derived from scratch, so its partial consumption record is void. (We reset to empty, never to a high-water mark — a fast-forward would re-introduce the commit-order hazard.) - Purges the now-unpinned deltas — those consumed by all remaining
(non-reseeding) targets, treating
Ras already satisfied:
WITH purgeable AS (
SELECT d.delta_id
FROM replicare.delta_<rel> d
WHERE (SELECT count(*) FROM replicare.track_<rel> tr
WHERE tr.delta_id = d.delta_id AND tr.target = ANY($remaining)) = $nremaining
ORDER BY d.delta_id
LIMIT $batch
)
DELETE FROM replicare.delta_<rel>
WHERE delta_id IN (SELECT delta_id FROM purgeable);
For the single-target case (the v1 common path) remaining is empty and
$nremaining = 0, so count(*) = 0 matches every row → the queue drains fully.
That is exactly the intended behaviour: the one laggard is being reseeded, so all
its deltas are discardable and the source is fully unpinned.
Why dropping R's unconsumed deltas is safe: because R gets a full
re-copy of current source state. A delta is a pointer to "PK X changed";
re-copy re-reads X's current value regardless. Discarding the pointer loses
nothing the re-copy does not re-derive. See the invariant in §4.3.
PurgeStats{ DeltasPurged, TargetsReseeded } is returned for telemetry
(replicare_delta_purged_total, replicare_reseed_total) and orchestration.
4. The reseed state machine
Driven by internal/reseed over the Source / Sink / StateStore interfaces,
reusing the M4 copy driver and the M5a/M5b drain. Per reseeding target T, over
the tables of its FK component:
┌─────────────┐
│ STREAMING │ (normal drain: delta MINUS track_T)
└──────┬──────┘
│ Purge reports T over-cap (source unpinned, track_T reset)
▼
┌─────────────┐
│ MARKED │ StateStore: cursor.needs_reseed=true, phase=initial_copy
│ │ event target.needs_reseed (ERROR); metric reseed_total++
└──────┬──────┘
│ reset copy_progress(T-tables) to fresh
▼
┌─────────────┐
│ RECOPYING │ per table: DeleteRange(nil,nil) → copy.Component (LoadDirect)
│ │ checkpointed by copy_progress watermark
└──────┬──────┘
│ copy done for the component
▼
┌─────────────┐
│ RESUMING │ drain delta MINUS track_T with track_T EMPTY:
│ │ re-consume every retained delta (idempotent re-read+upsert)
└──────┬──────┘
│ backlog drained; phase=streaming; needs_reseed=false
▼
┌─────────────┐
│ STREAMING │
└─────────────┘
4.1 Why DeleteRange(nil,nil) before re-copy
The reseed target is non-empty (stale data). A fresh copy_progress
(Watermark=nil) makes the copy driver plan every chunk and LoadDirect (COPY
FROM) — which would collide with existing rows and, worse, leave orphans for
source rows deleted since the last copy. So reseed first clears the table with
DeleteRange(ctx, ref, nil, nil) (predicate TRUE, i.e. delete-all — within the
DML-only target grant, no TRUNCATE privilege needed), then copies into the now-
empty table. This re-derives target = current source exactly, deletes included.
4.2 Why track stays empty through RESUMING
After re-copy, track_T is empty, so the resume drain re-consumes all
currently-retained deltas. Each is an idempotent re-read of the key's current
value + upsert (self-healing) — re-applying a delta the copy already covered
is a no-op-shaped upsert, never corruption. We deliberately do not fast-
forward the track: emptiness is both correct (idempotent) and hazard-free
(no commit-order skew). It re-consumes redundant work once; convergence is exact.
4.3 Invariant — no source change is lost across a reseed
**Capture is never uninstalled during reseed, and initial copy is capture-first
- idempotent; therefore for every key the target ends at the key's current source value, regardless of when changes landed relative to the handoff.**
Proof sketch. Triggers stay installed for the whole machine, so every source
change during reseed enqueues a fresh delta_id. Take any key X and any change
C to it during the reseed window. C reaches the target by at least one path:
- (a) copy path — if
Ccommitted before RECOPYING readX's chunk, the re-copy loadedX's current value directly. - (b) delta path —
Cleft a delta row (retained; not purged, since the §3.3 purge only removes rows consumed by remaining targets, andtrack_Tis empty during resume). RESUMING re-readsX's current value and upserts.
Because re-read always fetches the current value (self-healing), intermediate
states are irrelevant and the two paths cannot disagree on the final value. A
change after copy but before resume falls in (b). A delete of X at the source
is path (a) (absent from the re-copy → not loaded) or path (b) (re-read finds X
absent → delete-absent). Thus X converges to its current source state — which
is the one-way guarantee (source authoritative, target overwritten to match).
The only thing discarded is R's historical intermediate deltas, which reseed
intentionally supersedes with a current-state copy. ∎
4.4 Crash safety across the handoff (at-least-once, no 2PC)
Every transition is idempotent and independently checkpointed, so a crash at any point resumes correctly — the same delete-by-id discipline as consumption, applied to the machine:
| Crash point | On restart |
|---|---|
| after track reset, before MARKED persisted | track_T empty; Purge is idempotent; next purge pass re-detects over-cap → re-marks. No loss (deltas already unpinned). |
| after MARKED, before/into RECOPYING | needs_reseed=true + phase=initial_copy in StateStore → orchestration re-invokes Run, which is idempotent from the top: empty-target + fresh-progress reset re-run harmlessly and the re-copy restarts (its per-chunk watermark resumes within a single Run; a Run-crash restarts the copy). |
| mid RESUMING | track_T still (partially) empty; drain re-reads current values; idempotent upserts replay harmlessly. |
after resume drained, before clearing needs_reseed | flag still set → one more (no-op) resume pass, then cleared. |
needs_reseed is cleared only after the resume backlog is drained and
phase set to streaming, so the flag conservatively over-approximates
"reseed in flight." At-least-once throughout; no distributed transaction.
4.5 Ordering guard (the trap we must not fall into)
The reseed must reset track_T to empty, never to max(delta_id) or any
high-water mark. A high-water reset would declare late-committing low-delta_id
rows "already consumed" and skip them forever — the exact commit-order hazard
the dirty-key consumption model was built to defeat. Emptiness re-consumes them idempotently instead. This
is the single most important correctness rule of the machine.
5. Partition+DROP decision (Momus M-4)
Decision: the batched-DELETE + aggressive-autovacuum path is the
universal v1 default, on every supported source version. Time-partitioned
delta tables purged by DROP/TRUNCATE of fully-consumed partitions remain the
documented, recommended optimization for high-churn deployments on PG ≥ 10
(it is bloat-proof and dodges the xmin horizon), but it is not auto-enabled
in v1 and is not the default even on modern sources.
Rationale: partition routing changes the delta-table DDL and the trigger's insert
target and requires a partition-maintenance loop (create-ahead, drop-behind) — a
larger surface that would delay a correct, universal M5c. Keeping one code path
(batched DELETE) correct on the 9.6 floor and PG 17 is worth more in v1 than
the bloat win, especially since retention/reseed already bounds logical growth
and telemetry makes the xmin-horizon physical-bloat case loud. Partition+DROP
is revisited as a tuning pass in M9 (partition granularity/cadence numbers are
still open). Until then it is an opt-in extension point, not a
promise.
This settles whether partition+DROP should be the default on PG≥10: no, not default in v1; opt-in optimization, tune in M9.
6. Telemetry (F2 contract, emitted by M5c, verified by M6)
Purge/retention/reseed emit against the frozen F2 registry
(internal/observability) — no new names:
replicare_delta_backlog_rows/_bytes,replicare_delta_oldest_unconsumed_age_seconds— per(sync, target, table); the "is the source footprint healthy?" headline.replicare_delta_purged_total— per(sync, table); purge progress.replicare_reseed_total— per(sync, target); forced reseeds.- span
replicare.reseed— wraps a reseed; error status on failure, backlog/age attributes. - events
retention.cap_approaching(WARN, escalating withretention_cap_proximitytoward 1.0) andtarget.needs_reseed(ERROR).
The cross-channel rule holds: a target pushing the queue toward the cap is visible in metrics (climbing backlog/age + proximity), logs (escalating WARN→ERROR), and traces (reseed span) simultaneously.
7. Scope / non-goals for M5c
- Fan-out purge is implemented correctly (consumed-by-all across the full target set) but only lightly exercised — v1 hardens single-target (plan goals).
- No partition+DROP implementation (see §5).
- Retention-cap default numbers are set conservatively here (age 24h on, size off) and tuned in M9 against the at-scale fault matrix.
- The daemon loop that schedules purge/reseed passes is M7; M5c delivers the reusable mechanics + primitives and proves them under integration tests.
loadgen — a Postgres load + verification harness for replicare
loadgen is a manual, idempotent tool for exercising replicare against broad,
high-volume, realistic Postgres data. You run it whenever you like against a
source database; it seeds a rich schema the first time and applies random
inserts/updates/deletes on every subsequent run. Point replicare at the source and
a target, and use loadgen verify to assert the target converged.
It is a developer/test tool — not part of the shipped binary. Run it from a
checkout with go run ./test/loadgen (or task loadgen:*).
What it generates
One schema (loadgen) of 10 tables, chosen to exercise replicare's hard
paths, not just to hold rows:
| Table | What it stresses |
|---|---|
tenants | hub table many others FK to (giant-component pressure); identity PK, jsonb, text[], timestamptz, bool |
categories | self-referential hierarchy column (self-ref FK under --cyclic) |
users | bytea, inet, numeric(20,4), date, jsonb, many nullable columns; unique (tenant_id,email) |
products | composite text PK (tenant_id, sku); float8[]; timestamp (no tz); target of PK-changing updates |
orders | UUID PK; char(3); nullable FK to users |
order_items | composite PK; composite FK to products ON UPDATE CASCADE (a SKU rename is a PK change that cascades) |
product_categories | M:N join, composite PK, two FKs |
events | the high-volume table (~1M rows at default scale); wide type coverage |
generated_metrics | GENERATED … STORED column + identity PK |
audit_log | no primary/unique key → replicare skips it with a loud warning (verify excludes it too) |
Data is generated server-side (INSERT … SELECT generate_series, random()
made reproducible with setseed, md5(…)::uuid) — fast to a million rows, robust
across old Postgres, no client-side type encoding. The churn mix includes
PK-changing SKU renames and mutation of the nullable FK-cycle edge.
At --scale 1.0 (default) the seed writes ~1.2M rows (events dominate). Scale it
down for quick loops: --scale 0.01 ≈ 12k rows in ~150ms.
Usage
# 1. Create the schema on the TARGET (replicare replicates data only — target
# tables must pre-exist).
go run ./test/loadgen ddl --dsn "$TARGET"
# 2. Seed the SOURCE (first run seeds; later runs churn).
go run ./test/loadgen run --dsn "$SOURCE" # ~1M+ rows
go run ./test/loadgen run --dsn "$SOURCE" --scale 0.05 # smaller, faster
# 3. Start replicare source -> target (your own config), then generate change:
go run ./test/loadgen run --dsn "$SOURCE" --ops 500 # a burst of I/U/D
# 4. Assert the target converged (retries for replication lag):
go run ./test/loadgen verify --source "$SOURCE" --target "$TARGET" --wait 60s
--dsn/--source/--target take a libpq/pgx URL (postgres://user:pw@host:port/db)
or keyword string; empty uses the standard PG* environment variables.
Commands & flags
run --dsn <src> [--scale F] [--ops N] [--seed S] [--cyclic] [--duration D] [--node-id K]— ensure schema, then seed if empty or churnNstatements.--seedmakes a run reproducible.--duration Dchurns continuously forD(repeated--opsbursts) — run it while replicare does its initial copy to exercise the live-source cyclic-copy skew: parent rows (tenants/users) inserted mid-copy are referenced by children copied from a later snapshot, a transient FK the cyclic copy must recover from rather than crash-loop on. Pair with--cyclic.--node-id K(default 0) selects this writer's key slice for active-active testing — see Active-active; 0 is the default single-writer (active-passive) behaviour, unchanged.ddl --dsn <db> [--cyclic]— apply the schema only (prep the target). One shared schema serves every node;--node-idis arun-only concept (it partitions the data, not the schema).verify --source <src> --target <tgt> [--wait D] [--interval D]— per-table row-count + ordered content-checksum comparison. Exits non-zero on drift.--waitpolls until converged or the timeout. GUCs are pinned to match replicare's transport sorow::textrenders identically on both ends. It is a pure equality check, so it doubles as the active-active convergence oracle — run it between any two mesh nodes (they must hold the identical union).reset --dsn <db> --yes—DROP SCHEMA loadgen CASCADE.
verify compares only the 9 keyed tables (it skips audit_log, which replicare
skips). The GENERATED … STORED column is included and matches because both
servers recompute it.
Active-active (multi-master) load testing
The same harness drives a clusters: mesh. The model is disjoint partitioning:
each writer node gets --node-id K, which shifts every generated key and every FK
reference into a private slice [K·1e9+1, K·1e9+1e9), so nodes never collide on a
key and the mesh converges to the clean union of all slices (predictable counts,
matching checksums). --node-id 0 is the default and is byte-for-byte the
single-writer behaviour above, so active-passive is unaffected.
What --node-id K changes on run:
- Seed writes ids
base+g, order UUIDsmd5('order:'||(base+g)), SKUsSKU-(base+g), and offsets every FK reference bybase, so a node's children point only at that node's own parents. Identity sequences are then restarted past the node's seeded rows so churn inserts stay in the slice. - Churn inserts into the node's slice and scopes every UPDATE/DELETE to node-owned rows, so a node only ever mutates what it wrote — no accidental cross-node conflict (resolving concurrent same-key writes is the separate, planned conflict-storm mode; see below).
--node-id 0 and any --node-id K seeded/churned against the same node's DB must
be consistent — always pass the same --node-id for a given node. Keep --scale
well under ~500M rows per node (the harness refuses a scale that approaches the slice).
Three-node mesh example
Assume a running replicare clusters: mesh over three Postgres nodes A, B, C
(config per multi-master.md), with $A/$B/$C their DSNs.
# 1. Schema on every node (data-only replication; targets must pre-exist).
for n in "$A" "$B" "$C"; do go run ./test/loadgen ddl --dsn "$n"; done
# 2. Seed each node into its own disjoint slice.
go run ./test/loadgen run --dsn "$A" --node-id 0
go run ./test/loadgen run --dsn "$B" --node-id 1
go run ./test/loadgen run --dsn "$C" --node-id 2
# 3. Churn every node concurrently (each in its own slice).
go run ./test/loadgen run --dsn "$A" --node-id 0 --ops 800 --seed 1 &
go run ./test/loadgen run --dsn "$B" --node-id 1 --ops 800 --seed 2 &
go run ./test/loadgen run --dsn "$C" --node-id 2 --ops 800 --seed 3 &
wait
# 4. Convergence oracle: every pair must be identical (all hold the union).
go run ./test/loadgen verify --source "$A" --target "$B" --wait 300s
go run ./test/loadgen verify --source "$A" --target "$C" --wait 300s
Convergence is transitive, so A==B and A==C imply all three agree. A pass ending
CONVERGED: all 9 replicated tables match on every pair — with the daemon logs clean
of HALTED / stream pass error — is a passing active-active load test.
Not yet: conflict-storm mode. The disjoint model above never has two nodes write the same key, so it tests convergence of the union. A follow-up
--overlapmode will deliberately overlap node slices to exercise HLC last-write-wins under concurrent same-key writes; thereverifystill asserts convergence (all nodes equal), but not which value wins (the LWW winner is nondeterministic).
Reset the state store per node when re-seeding (see Gotchas) — this applies per node in a mesh: clear each node's
replicareschema and its state-store entries alongsideloadgen reset.
Running a full load test end-to-end
The commands above assume replicare is already running with "your own config". Here is the complete rig — two Postgres instances, a state store, a config, the daemon, and a heavy-churn convergence loop — that this harness is meant to drive. It is the exact flow used to validate replicare's copy + streaming under load.
1. Two Postgres instances + a state-store database. Any two reachable Postgres servers work; the quickest is two containers. replicare's state store is a third logical database (here, one on the target server):
docker run -d --name lg-src -e POSTGRES_PASSWORD=pw -p 55432:5432 postgres:16
docker run -d --name lg-tgt -e POSTGRES_PASSWORD=pw -p 55433:5432 postgres:16
# state store (replicare's own progress/cursors — NOT your data):
docker exec lg-tgt psql -U postgres -c 'CREATE DATABASE replicare_state'
export SOURCE="postgres://postgres:pw@localhost:55432/postgres"
export TARGET="postgres://postgres:pw@localhost:55433/postgres"
2. A minimal replicare config (loadtest.yaml). drain_interval is short so
streaming keeps up under churn:
logging: { level: info, format: text }
observability: { metrics_addr: ":19090", status_addr: ":18080" }
state_store:
engine: postgres
postgres: { host: localhost, port: 55433, database: replicare_state, user: postgres, password: pw, sslmode: disable }
sources:
src: { engine: postgres, postgres: { host: localhost, port: 55432, database: postgres, user: postgres, password: pw, sslmode: disable } }
targets:
tgt: { engine: postgres, postgres: { host: localhost, port: 55433, database: postgres, user: postgres, password: pw, sslmode: disable } }
syncs:
- name: loadtest
source: src
targets: [tgt]
include: ["loadgen.*"]
tuning: { drain_interval: 1s }
3. Seed the source, create the target schema, start replicare. Capture is enabled first, then the chunked copy runs — so seeding before the daemon starts is fine (the initial copy reproduces it):
go run ./test/loadgen run --dsn "$SOURCE" # seed ~1M+ rows on the source
go run ./test/loadgen ddl --dsn "$TARGET" # create the (empty) target schema
go run ./cmd/replicare run loadtest.yaml & # start the daemon
4. Verify the initial copy converged:
go run ./test/loadgen verify --source "$SOURCE" --target "$TARGET" --wait 120s
# => CONVERGED: all 9 replicated tables match
5. Churn-and-verify loop — this is the actual load test. Each round applies a random burst of inserts/updates/deletes (including PK-changing SKU renames) and asserts the target re-converges while the daemon streams deltas:
for r in 1 2 3 4; do
go run ./test/loadgen run --dsn "$SOURCE" --ops 800 --seed $r
go run ./test/loadgen verify --source "$SOURCE" --target "$TARGET" --wait 120s
done
To stress cross-batch dependencies and delta backlog (what shook out the FK livelocks), churn several times back-to-back before verifying, so a large backlog builds while streaming lags, then converges:
for r in $(seq 1 8); do go run ./test/loadgen run --dsn "$SOURCE" --ops 800 --seed $r; done
go run ./test/loadgen verify --source "$SOURCE" --target "$TARGET" --wait 300s
A run finishing with CONVERGED: all 9 replicated tables match — and the daemon log
showing no HALTED / stream pass error — is a passing load test. For the cyclic
variant, add --cyclic to every ddl and run (see below).
This is a manual test, distinct from
task test:integration(the automated Go suite). Those integration tests must run serially (go test -p 1 ...) because they share schemas; the load test above is driven by hand and has no such constraint.
The --cyclic flag
By default the schema is acyclic. --cyclic adds two nullable FK cycles —
the categories self-reference and a users ↔ orders 2-table cycle — to exercise
replicare's cyclic-copy (null_then_fill) path. Apply it consistently to source
and target (ddl --cyclic, run --cyclic).
Both initial copy and streaming of a cyclic component converge under
--cyclic, with standard non-DEFERRABLE target FKs (the common real-world case). Initial copy loads every table with its cyclic FK columns NULL, in an order that respects the non-cyclic edges, then fills those columns. Streaming uses the same NULL-then-fill idea per pass, but per table: each table's upsert, delete, and cyclic-column fill commit independently, so a parent advances through its own delta queue even when a cross-batch child is transiently blocked (the same progress guarantee the acyclic drain gives), and the cycle itself is closed by a final fill phase. This harness's--cyclicschema drains cleanly under heavy churn with noDEFERRABLErequirement.
Gotchas
- Reset the state store when re-seeding from scratch. replicare records
per-table copy-progress in its state store; if you wipe and re-seed the source
without clearing that state, replicare treats already-
Donetables as copied and skips them (their parents go missing → FK errors on children). Clear thereplicareschema in your state-store database (or use a fresh one) alongsideloadgen reset. - The target schema must exist first — run
ddlagainst the target before starting replicare. - Deletes cascade (
orders → order_items,tenants → everything), so adelete_orders/delete_usersop can remove more rows than it names.
Task shortcuts
task loadgen:ddl DSN="$TARGET" # schema on the target
task loadgen:seed DSN="$SOURCE" SCALE=0.1 # seed / scale
task loadgen:churn DSN="$SOURCE" OPS=500 # a churn burst
task loadgen:verify SRC="$SOURCE" TGT="$TARGET" # convergence check
loadgen-mysql — a MySQL load + verification harness for replicare
loadgen-mysql is the MySQL sibling of test/loadgen. It is a
manual, idempotent tool for exercising replicare's MySQL engine against broad,
high-volume, realistic data. You run it whenever you like against a source MySQL;
it seeds a rich schema the first time and applies random inserts/updates/deletes on
every subsequent run. Point replicare at the source and a target, and use
loadgen-mysql verify to assert the target converged.
It is a developer/test tool — not part of the shipped binary. Run it from a checkout
with go run ./test/loadgen-mysql (or task loadgen-mysql:*).
Everything lives in a dedicated loadgen database (MySQL's analog of the Postgres
loadgen schema), created by ddl/run, so replicare can select it wholesale
(include: ["loadgen.*"]) and the harness never collides with anything else on the
server. Every statement is schema-qualified, so the DSN's default database is irrelevant
and need not exist.
What it generates
One database (loadgen) of 10 tables, chosen to exercise replicare's hard MySQL
paths, not just to hold rows (the same 10 as the Postgres harness, in MySQL types):
| Table | What it stresses |
|---|---|
tenants | hub table many others FK to (giant-component pressure); AUTO_INCREMENT PK, JSON, DATETIME, TINYINT(1) bool |
categories | self-referential hierarchy column (self-ref FK under --cyclic) |
users | VARBINARY (avatar), IP text, DECIMAL(20,4), DATE, JSON, many nullable columns; unique (tenant_id,email) |
products | composite VARCHAR PK (tenant_id, sku); JSON dims; target of PK-changing updates |
orders | CHAR(32) id PK (MD5 hex, the UUID analog); CHAR(3); nullable FK to users |
order_items | composite PK; composite FK to products ON UPDATE CASCADE (a SKU rename is a PK change that cascades) |
product_categories | M:N join, composite PK, two FKs |
events | the high-volume table (~1M rows at default scale); wide type coverage incl. DOUBLE |
generated_metrics | GENERATED … STORED column + AUTO_INCREMENT PK |
audit_log | no primary/unique key → replicare skips it with a loud warning (verify excludes it too) |
Data is generated server-side (INSERT … SELECT over a cross-joined digit-table row
generator, RAND() seeded with RAND(seed), MD5(…) ids) — fast to a million rows and,
crucially, MySQL-5.7 safe: 5.7 is the oldest supported source and has no
generate_series, no CTEs, and no window functions, so the harness uses none of them.
The churn mix includes PK-changing SKU renames (cascaded) and mutation of the nullable
FK-cycle edge.
At --scale 1.0 (default) the seed writes ~1.2M rows (events dominate). Scale it down
for quick loops: --scale 0.01 ≈ 12k rows.
Usage
# 1. Create the schema on the TARGET (replicare replicates data only — target tables
# must pre-exist).
go run ./test/loadgen-mysql ddl --dsn "$TARGET"
# 2. Seed the SOURCE (first run seeds; later runs churn).
go run ./test/loadgen-mysql run --dsn "$SOURCE" # ~1M+ rows
go run ./test/loadgen-mysql run --dsn "$SOURCE" --scale 0.05 # smaller, faster
# 3. Start replicare source -> target (your own config), then generate change:
go run ./test/loadgen-mysql run --dsn "$SOURCE" --ops 500 # a burst of I/U/D
# 4. Assert the target converged (retries for replication lag):
go run ./test/loadgen-mysql verify --source "$SOURCE" --target "$TARGET" --wait 60s
--dsn/--source/--target take a go-sql-driver DSN, e.g.
root:replicare@tcp(127.0.0.1:3340)/ (source) and …@tcp(127.0.0.1:3341)/ (target) for
the bundled harness. The harness pins time_zone='+00:00' on every connection so
NOW()/DATETIME values render deterministically in UTC, matching replicare's own MySQL
session canonicalization.
Commands & flags
run --dsn <src> [--scale F] [--ops N] [--seed S] [--cyclic] [--duration D] [--node-id K]— ensure schema, then seed if empty or churnNstatements.--seedseeds both client op-selection and serverRAND()(server-side value reproducibility is best-effort on MySQL; op selection is exact).--duration Dchurns continuously forD(repeated--opsbursts) — run it while replicare does its initial copy to exercise the live-source skew path: parent rows inserted mid-copy referenced by children copied from a later snapshot (a transient FK the copy must recover from). Pair with--cyclic.--node-id K(default 0) selects this writer's key slice for active-active testing — see Active-active; 0 is the default single-writer (active-passive) behaviour, unchanged.ddl --dsn <db> [--cyclic]— apply the schema only (prep the target). One shared schema serves every node;--node-idis arun-only concept (it partitions the data, not the schema).verify --source <src> --target <tgt> [--wait D] [--interval D]— per-table row-count + order-independent content-checksum comparison (per row, MD5 over a NULL-safe, type-aware rendering of every column; folded withBIT_XOR). Exits non-zero on drift.--waitpolls until converged or the timeout. It is a pure equality check, so it doubles as the active-active convergence oracle — run it between any two mesh nodes (they must hold the identical union).reset --dsn <db> --yes—DROP DATABASE loadgen.
verify compares only the 9 keyed tables (it skips audit_log, which replicare skips).
The GENERATED … STORED column is included and matches because both servers recompute
it.
Why not compare a table checksum directly? The harness runs an old source (5.7) against a modern target (8.4).
verifyrenders each value server-side into a version-stable form before hashing: binary/blob viaHEX,FLOAT/DOUBLEnormalized viaCAST … AS DECIMAL(65,15)(replicare moves the value bit-faithfully, so both ends hold the identical double; normalizing removes any raw double-to-string formatting difference between server versions), everything else viaCAST … AS CHARunder a pinnedtime_zone. It compares that canonical rendering, not raw storage bytes (which differ across versions for identical data).
Active-active (multi-master) load testing
The same harness drives a MySQL clusters: mesh. The model is disjoint partitioning:
each writer node gets --node-id K, which shifts every generated key and every FK
reference into a private slice [K·1e9+1, K·1e9+1e9), so nodes never collide on a key and
the mesh converges to the clean union of all slices (predictable counts, matching
checksums). --node-id 0 is the default and is byte-for-byte the single-writer behaviour
above, so active-passive is unaffected.
What --node-id K changes on run:
- Seed writes ids
base+g, order idsMD5(CONCAT('order:', base+g)), SKUsSKU-(base+g), and offsets every FK reference bybase, so a node's children point only at that node's own parents.AUTO_INCREMENTcounters are then restarted past the node's seeded rows so churn inserts stay in the slice. - Churn inserts into the node's slice and scopes every UPDATE/DELETE to node-owned rows, so a node only ever mutates what it wrote — no accidental cross-node conflict (resolving concurrent same-key writes is the separate, planned conflict-storm mode; see below).
--node-id 0 and any --node-id K seeded/churned against the same node's DB must be
consistent — always pass the same --node-id for a given node. Keep --scale well under
~500M rows per node (the harness refuses a scale that approaches the slice).
Three-node mesh example
Assume a running replicare clusters: mesh over three MySQL nodes A, B, C (config
per multi-master.md), with $A/$B/$C their DSNs.
# 1. Schema on every node (data-only replication; targets must pre-exist).
for n in "$A" "$B" "$C"; do go run ./test/loadgen-mysql ddl --dsn "$n"; done
# 2. Seed each node into its own disjoint slice.
go run ./test/loadgen-mysql run --dsn "$A" --node-id 0
go run ./test/loadgen-mysql run --dsn "$B" --node-id 1
go run ./test/loadgen-mysql run --dsn "$C" --node-id 2
# 3. Churn every node concurrently (each in its own slice).
go run ./test/loadgen-mysql run --dsn "$A" --node-id 0 --ops 800 --seed 1 &
go run ./test/loadgen-mysql run --dsn "$B" --node-id 1 --ops 800 --seed 2 &
go run ./test/loadgen-mysql run --dsn "$C" --node-id 2 --ops 800 --seed 3 &
wait
# 4. Convergence oracle: every pair must be identical (all hold the union).
go run ./test/loadgen-mysql verify --source "$A" --target "$B" --wait 300s
go run ./test/loadgen-mysql verify --source "$A" --target "$C" --wait 300s
Convergence is transitive, so A==B and A==C imply all three agree. A pass ending
CONVERGED: all 9 replicated tables match on every pair — with the daemon logs clean of
HALTED / stream pass error — is a passing active-active load test.
Not yet: conflict-storm mode. The disjoint model above never has two nodes write the same key, so it tests convergence of the union. A follow-up
--overlapmode will deliberately overlap node slices to exercise HLC last-write-wins under concurrent same-key writes; thereverifystill asserts convergence (all nodes equal), but not which value wins (the LWW winner is nondeterministic).
Reset the state store per node when re-seeding (see Gotchas) — this applies per node in a mesh: clear each node's source-side
replicareschema and its state-store entries alongsideloadgen-mysql reset.
Running a full load test end-to-end
The commands above assume replicare is already running with "your own config". Here is the complete rig — an old 5.7 source, a modern 8.4 target, a Postgres state store, a config, the daemon, and a heavy-churn convergence loop — that this harness is meant to drive. It is the exact flow used to validate replicare's MySQL copy + streaming under load.
1. Two MySQL instances + a Postgres state store. The bundled harness is the quickest
source/target pair (an old 5.7 source on :3340 and a modern 8.4 target on :3341); the
state store is Postgres (v1's only StateStore backend — the replicated data is MySQL,
the daemon's own progress/cursors live in Postgres):
task harness:mysql:up # source on :3340, target on :3341
docker run -d --name lg-state -e POSTGRES_PASSWORD=pw -p 55450:5432 postgres:16
docker exec lg-state psql -U postgres -c 'CREATE DATABASE replicare_state'
export SOURCE="root:replicare@tcp(127.0.0.1:3340)/"
export TARGET="root:replicare@tcp(127.0.0.1:3341)/"
2. A minimal replicare config (loadtest-mysql.yaml). drain_interval is short so
streaming keeps up under churn:
logging: { level: info, format: text }
observability: { metrics_addr: ":19092", status_addr: ":18082" }
state_store:
engine: postgres
postgres: { host: localhost, port: 55450, database: replicare_state, user: postgres, password: pw, sslmode: disable }
sources:
src: { engine: mysql, mysql: { host: 127.0.0.1, port: 3340, database: loadgen, user: root, password: replicare, tls: disable } }
targets:
# local_infile: true opts into the LOAD DATA fast path (the target server must also
# permit it — the bundled harness sets --local-infile=ON).
tgt: { engine: mysql, mysql: { host: 127.0.0.1, port: 3341, database: loadgen, user: root, password: replicare, tls: disable, local_infile: true } }
syncs:
- name: loadtest-mysql
source: src
targets: [tgt]
include: ["loadgen.*"]
tuning: { drain_interval: 1s }
3. Seed the source, create the target schema, start replicare. Capture (triggers) is installed first, then the chunked copy runs — so seeding before the daemon starts is fine (the initial copy reproduces it):
go run ./test/loadgen-mysql run --dsn "$SOURCE" # seed ~1M+ rows on the source
go run ./test/loadgen-mysql ddl --dsn "$TARGET" # create the (empty) target schema
go run ./cmd/replicare run loadtest-mysql.yaml & # start the daemon
4. Verify the initial copy converged:
go run ./test/loadgen-mysql verify --source "$SOURCE" --target "$TARGET" --wait 120s
# => CONVERGED: all 9 replicated tables match
5. Churn-and-verify loop — this is the actual load test. Each round applies a random burst of inserts/updates/deletes (including PK-changing SKU renames) and asserts the target re-converges while the daemon streams deltas:
for r in 1 2 3 4; do
go run ./test/loadgen-mysql run --dsn "$SOURCE" --ops 800 --seed $r
go run ./test/loadgen-mysql verify --source "$SOURCE" --target "$TARGET" --wait 120s
done
To stress cross-batch dependencies and delta backlog, churn several times back-to-back before verifying, so a large backlog builds while streaming lags, then converges:
for r in $(seq 1 8); do go run ./test/loadgen-mysql run --dsn "$SOURCE" --ops 800 --seed $r; done
go run ./test/loadgen-mysql verify --source "$SOURCE" --target "$TARGET" --wait 300s
A run finishing with CONVERGED: all 9 replicated tables match — and the daemon log
showing no HALTED / stream pass error — is a passing load test. For the cyclic
variant, add --cyclic to every ddl and run (see below).
This is a manual test, distinct from the automated Go integration suite. MySQL correctness is a local gate (
REPLICARE_INTEGRATION=1with the MySQL harness up); CI is Postgres-only.
The --cyclic flag
By default the schema is acyclic. --cyclic adds two nullable FK cycles — the
categories self-reference and a users ↔ orders 2-table cycle — to exercise replicare's
cyclic-copy (null_then_fill) path. Apply it consistently to source and target
(ddl --cyclic, run --cyclic). MySQL has no ADD CONSTRAINT IF NOT EXISTS, so the
harness adds each cyclic FK only when it is absent, keeping --cyclic idempotent.
Gotchas
- Reset the state store when re-seeding from scratch. replicare records per-table
copy-progress in its state store; if you wipe and re-seed the source without clearing
that state, replicare treats already-
Donetables as copied and skips them. Clear thereplicareschema in your state-store database (or use a fresh one) alongsideloadgen-mysql reset. - The target schema must exist first — run
ddlagainst the target before starting replicare. local_infileon the target. replicare's MySQL initial copy usesLOAD DATA LOCAL INFILE, so the target server must permit it (the bundledharness-mysqlsets--local-infile=ON). The harness itself does not need it (it seeds via ordinaryINSERT … SELECT).- Deletes cascade (
orders → order_items,tenants → everything), so adelete_orders/delete_usersop can remove more rows than it names.
Task shortcuts
task loadgen-mysql:ddl DSN="$TARGET" # schema on the target
task loadgen-mysql:seed DSN="$SOURCE" SCALE=0.1 # seed / scale
task loadgen-mysql:churn DSN="$SOURCE" OPS=500 # a churn burst
task loadgen-mysql:verify SRC="$SOURCE" TGT="$TARGET" WAIT=60s
task loadgen-mysql:reset DSN="$SOURCE" # DROP DATABASE loadgen
loadgen-redis — a Redis load + verification harness for replicare
loadgen-redis is the Redis sibling of test/loadgen. It
is a manual, idempotent tool for exercising replicare's Redis engine against
broad, multi-type, high-volume data. You run it against a source Redis; it
seeds a rich keyspace the first time and applies random mutations on every
subsequent run. Point replicare at the source and a target, and use
loadgen-redis verify to assert the target converged.
It is a developer/test tool — not part of the shipped binary. Run it from a
checkout with go run ./test/loadgen-redis (or task loadgen-redis:*).
Unlike the Postgres harness there is no ddl step: Redis has no schema, and
replicare RESTOREs keys straight into the target keyspace, which need not
pre-exist.
What it generates
Keys are laid out as lg:{<bucket>}:<type>:<n>. Everything is chosen to
exercise replicare's Redis paths, not just to hold data:
| Cohort | What it stresses |
|---|---|
str | plain strings; ~20% carry a TTL (volatile keys → TTL replication) |
hash / list / set / zset | every collection type, through DUMP→RESTORE |
stream | streams with consumer groups — the trickiest DUMP/RESTORE payload |
big | deliberately large keys (~100 KB hashes) → the big-key path / MEMORY USAGE gate |
lgskip:* | written to the source only; replicare's include: ["lg:*"] must exclude it — the Redis analog of loadgen's keyless audit_log. verify asserts the target holds none of these. |
The {bucket} hash tag is the cluster story: on a standalone/sentinel server
the braces are inert literal bytes, but on a Redis Cluster they spread the keyspace
across slots/shards (exercising replicare's per-master SCAN fan-out) and keep a
RENAME's old+new key in one slot (a cross-slot RENAME is an error).
The churn mix includes the two cases replicare's Redis engine handles specially:
RENAME= delete-old + create-new — the analog of a PK change, and a stress on the delete-reconciliation sweep.- heavy
DEL— Redis has no delete capture, so deletes are only caught by replicare's target-vs-source diff; this is the genuinely hard path.
At --scale 1.0 (default) the seed writes ~430k keys. Scale down for quick loops:
--scale 0.02 ≈ 8.6k keys in well under a second.
Usage
# 1. Seed the SOURCE (first run seeds; later runs churn).
go run ./test/loadgen-redis run --dsn "$SOURCE" # ~430k keys
go run ./test/loadgen-redis run --dsn "$SOURCE" --scale 0.05 # smaller, faster
# 2. Start replicare source -> target (your own config), then generate change:
go run ./test/loadgen-redis run --dsn "$SOURCE" --ops 500 # a burst of mutations
# 3. Assert the target converged (retries for replication lag):
go run ./test/loadgen-redis verify --source "$SOURCE" --target "$TARGET" --wait 60s
Connection specs
- standalone / sentinel: a
redis://URL, e.g.redis://:pw@localhost:6390/0(empty =redis://localhost:6379/0). - cluster: pass
--clusterand a comma-separated seed list, e.g.--dsn "n1:6379,n2:6379,n3:6379"(use--passwordfor AUTH). One tool, both topologies — see the hash-tag note above.
Commands & flags
run --dsn <src> [--scale F] [--ops N] [--seed S] [--cluster] [--node-id K]— seed if empty or churnNops.--seedmakes a run reproducible (op selection + generated values).--node-id K(default 0) selects this writer's key namespace for active-active testing — see Active-active; 0 is the default single-writer (active-passive) behaviour, unchanged.verify --source <src> --target <tgt> [--wait D] [--interval D] [--cluster] [--node-id K]— compares everylg:*key version-independently: a type-aware canonical content hash (GET/ sortedHGETALL/LRANGE/ sortedSMEMBERS/ZRANGE WITHSCORES/XRANGE+ group last-ids) plus TTL presence. Also asserts nolgskip:*leaked to the target. It is a pure equality check over the wholelg:*keyspace, so it doubles as the active-active convergence oracle (run it between any two mesh nodes — both hold the union).--node-id Kis the source's node index; it scopes the skip-leak check to that source's ownlgskip:*namespace (0 = the classic single-writer check).--waitpolls until converged or the timeout. Exits non-zero on drift.reset --dsn <db> --yes [--cluster]—DELeverylg:*,lgskip:*, and thelgmeta:*marker key (all nodes' namespaces). Scoped by prefix — it never runsFLUSHDB.
Why not compare
DUMPbytes directly? The harness runs an old source (6.2) against a modern target (7.4), whose RDB serializations differ even for identical data (and replicare deliberately preserves each side's own encoding). Soverifycompares a canonical rendering of each value, not its serialized bytes. TTL is compared as presence only (has-TTL vs no-TTL) — the exact remaining seconds legitimately differ by the replication delay.
Active-active (multi-master) load testing
The same harness drives a Redis clusters: mesh. The model is disjoint key
namespaces: each writer node gets --node-id K, which moves every key it writes
into a private segment right after the prefix — lg:n<K>:{<bucket>}:<type>:<n> (and
lgskip:n<K>:… for the source-only cohort). Node segments never overlap, so nodes
never collide on a key and the mesh converges to the clean union of all segments.
--node-id 0 is the default and is byte-for-byte the single-writer layout above
(lg:{<bucket>}:…, no segment), so active-passive is unaffected.
The segment sits after the lg:/lgskip: prefix, so a replicare
include: ["lg:*"] selection still matches every node's data keys and still excludes
every node's lgskip:* keys — no per-node config change is needed.
What --node-id K changes on run:
- Seed writes only this node's namespace (
lg:n<K>:…), and the seeded/empty check scans that namespace alone, so a node re-seeds only when its own slice is empty — peers' replicated-in keys don't count as "already seeded". - Churn sets/updates/deletes only keys in this node's namespace, and a
RENAMEkeeps the new key in the same segment (and hash-tag bucket), so a node only ever mutates what it wrote — no accidental cross-node conflict (resolving concurrent same-key writes is the separate, planned conflict-storm mode; see below).
On verify, --node-id K is the source's node index: it scopes the skip-leak
check to that source's own lgskip:n<K>:* namespace, so a mesh target — which is
itself a source with its own lgskip:* — never false-positives on the peer's
excluded cohort. The content comparison itself is a pure equality check over the
whole lg:* keyspace, so it doubles as the convergence oracle: run it between any two
mesh nodes and both must hold the identical union.
Always pass the same --node-id for a given node's DB. Keep --scale modest per node
(Redis holds everything in RAM, and the mesh target holds every node's union).
Three-node mesh example
Assume a running replicare clusters: mesh over three Redis nodes A, B, C
(config per multi-master.md), with $A/$B/$C their
connection specs. There is no ddl step — Redis has no schema.
# 1. Seed each node into its own disjoint namespace.
go run ./test/loadgen-redis run --dsn "$A" --node-id 0 --scale 0.1
go run ./test/loadgen-redis run --dsn "$B" --node-id 1 --scale 0.1
go run ./test/loadgen-redis run --dsn "$C" --node-id 2 --scale 0.1
# 2. Churn every node concurrently (each in its own namespace).
go run ./test/loadgen-redis run --dsn "$A" --node-id 0 --ops 800 --seed 1 &
go run ./test/loadgen-redis run --dsn "$B" --node-id 1 --ops 800 --seed 2 &
go run ./test/loadgen-redis run --dsn "$C" --node-id 2 --ops 800 --seed 3 &
wait
# 3. Convergence oracle: every pair must be identical (all hold the union).
# --node-id names the SOURCE so the skip-leak check scopes to its lgskip namespace.
go run ./test/loadgen-redis verify --source "$A" --target "$B" --node-id 0 --wait 300s
go run ./test/loadgen-redis verify --source "$A" --target "$C" --node-id 0 --wait 300s
Convergence is transitive, so A==B and A==C imply all three agree. A pass ending
CONVERGED: … no skip leak on every pair — with the daemon logs clean of HALTED /
stream pass error — is a passing active-active load test.
Not yet: conflict-storm mode. The disjoint model above never has two nodes write the same key, so it tests convergence of the union. A follow-up
--overlapmode will deliberately overlap node namespaces to exercise HLC last-write-wins under concurrent same-key writes; thereverifystill asserts convergence (all nodes equal), but not which value wins (the LWW winner is nondeterministic).
Reset the state store per node when re-seeding (see Gotchas) — in a mesh this applies per node: clear each node's state-store entries alongside
loadgen-redis reset(which already removes every node'slg:/lgskip:namespace).
Running a full load test end-to-end
The commands above assume replicare is already running with "your own config". Here is the complete rig — two Redis instances, a Postgres state store, a config, the daemon, and a churn-and-verify loop — that this harness is meant to drive. This is the exact flow used to validate replicare's Redis copy + streaming under load.
1. Two Redis instances + a Postgres state store. The bundled harness is the
quickest source/target pair (an old 6.2 source and a modern 7.4 target); the state
store is Postgres (v1's only StateStore backend — the replicated data is Redis,
the daemon's own progress/cursors live in Postgres):
task harness:redis:up # source on :6390, target on :6391
docker run -d --name lg-state -e POSTGRES_PASSWORD=pw -p 55440:5432 postgres:16
docker exec lg-state psql -U postgres -c 'CREATE DATABASE replicare_state'
export SOURCE="redis://localhost:6390/0"
export TARGET="redis://localhost:6391/0"
2. A minimal replicare config (loadtest-redis.yaml). drain_interval is short
so the reconciliation SCAN + delete sweep keep up under churn:
logging: { level: info, format: text }
observability: { metrics_addr: ":19091", status_addr: ":18081" }
state_store:
engine: postgres
postgres: { host: localhost, port: 55440, database: replicare_state, user: postgres, password: pw, sslmode: disable }
sources:
src: { engine: redis, redis: { host: localhost, port: 6390, db: 0 } }
targets:
tgt: { engine: redis, redis: { host: localhost, port: 6391, db: 0 } }
syncs:
- name: redisload
source: src
targets: [tgt]
include: ["lg:*"] # excludes lgskip:* by construction
tuning: { drain_interval: 1s }
3. Seed the source, start replicare. Redis CDC is capture-less; the initial copy SCANs the source, so seeding before the daemon starts is fine:
go run ./test/loadgen-redis run --dsn "$SOURCE" # seed the source
go run ./cmd/replicare run loadtest-redis.yaml & # start the daemon
4. Verify the initial copy converged:
go run ./test/loadgen-redis verify --source "$SOURCE" --target "$TARGET" --wait 120s
# => CONVERGED: N replicated keys match, no skip leak
5. Churn-and-verify loop — this is the actual load test. Each round applies a
random burst of mutations (including RENAMEs and heavy DELs) and asserts the
target re-converges while the daemon streams:
for r in 1 2 3 4; do
go run ./test/loadgen-redis run --dsn "$SOURCE" --ops 800 --seed $r
go run ./test/loadgen-redis verify --source "$SOURCE" --target "$TARGET" --wait 120s
done
To stress the delete-reconciliation sweep and streaming backlog, churn several times back-to-back before verifying, so deletes and renames pile up while the sweep runs, then converge:
for r in $(seq 1 8); do go run ./test/loadgen-redis run --dsn "$SOURCE" --ops 800 --seed $r; done
go run ./test/loadgen-redis verify --source "$SOURCE" --target "$TARGET" --wait 300s
A run finishing with CONVERGED: … no skip leak — and the daemon log showing no
HALTED / stream pass error — is a passing load test. Watch
replicare_deletes_reconciled_total on :19091/metrics climb as DEL/RENAME
churn is reconciled.
Cluster mode
The same tool drives a Redis Cluster target. Bring up the cluster harness and pass
--cluster plus a seed list:
task harness:redis:cluster:up
go run ./test/loadgen-redis run --dsn "127.0.0.1:7000,127.0.0.1:7001,127.0.0.1:7002" --cluster --scale 0.1
go run ./test/loadgen-redis verify --source "…standalone source…" --target "127.0.0.1:7000,…" --cluster --wait 120s
The {bucket} hash tags spread keys across the cluster's slots (so the per-master
SCAN fan-out is genuinely exercised) while keeping each RENAME intra-slot.
Gotchas
- Reset the state store when re-seeding from scratch. replicare records
per-unit copy-progress in its state store; if you wipe and re-seed the source
without clearing that state, it may treat the unit as already copied. Use a fresh
replicare_statedatabase (or clear thereplicareschema in it) alongsideloadgen-redis reset. RENAMEneeds an existing source key. The tool renames within the seeded string range and silently skips a key that a prior op already renamed/deleted — benign for the harness.lgmeta:countsis a small bookkeeping key (per-type seeded counts, so churn can address existing keys without scanning). It is neitherlg:*norlgskip:*, so replicare andverifyboth ignore it;resetremoves it.
Task shortcuts
task loadgen-redis:seed DSN="$SOURCE" SCALE=0.1 # seed / scale
task loadgen-redis:churn DSN="$SOURCE" OPS=500 # a churn burst
task loadgen-redis:verify SRC="$SOURCE" TGT="$TARGET" WAIT=60s
task loadgen-redis:reset DSN="$SOURCE" # add CLUSTER=1 for a cluster
Changelog
All notable changes to replicare are documented here. The format follows Keep a Changelog, and the project aims to follow Semantic Versioning.
0.1.0 — first release
The initial release: a complete, single-engine Postgres → Postgres data replicator. Feature-complete and verified end to end on a Postgres 9.6 → 17 integration matrix with fault-injection, perf, and fidelity gates.
Replication engine
- Trigger-based CDC (Bucardo-style) — no WAL access, no
REPLICATIONrole, nowal_levelchange. Works on managed and old Postgres. - PK-only capture with re-read at sync time — self-healing dirty-key-set model; PK changes enqueue both old and new keys.
- Resumable, parallel initial copy — chunked keyset/ctid ranges, text
COPYstreamed source→target, parents-first within FK components, components in parallel; resumes from checkpoints after a crash. - Continuous streaming apply — FK-ordered per-component transactions (parent→child upserts, child→parent deletes) with a bounded retry fallback for cycles/self-references; idempotent, at-least-once.
- Cyclic / self-referential FK initial copy — NULL-then-fill, deferred constraints, or a loud pre-flight block for the genuinely-unloadable case.
Correctness & safety
- Faithful transport, never transform — values move verbatim (text
COPY, both copy and delta paths) with session-GUC canonicalization; a type/constraint error halts loudly instead of corrupting. Verified across the 9.6→17 gap, including enum and range types, on both copy and delta paths. - Bounded source footprint — batched consumption-gated purge, aggressive
per-table autovacuum, and bounded retention (age/size) with forced reseed of a
laggard target. Protects the source even while a target is down, and under a
stalled
xminhorizon. - Crash-safe checkpointing — Postgres state store; a restarted or rescheduled
process resumes cleanly; single-active ownership via
pg_advisory_lock.
Operations
- Daemon managing multiple named syncs concurrently, with graceful (SIGTERM → drain + checkpoint) shutdown.
- CLI —
run,validate,status,capture install|remove,reseed,version. - Pre-flight — classifies every source↔target column pair; blocks on incompatible types, warns on risky ones, skips no-key tables.
- Least-privilege — documented, verified source and target grants; no superuser.
Observability
- Prometheus
/metrics, a/status+/healthzHTTP API, structuredsloglogs, and OpenTelemetry traces (OTLP/gRPC export when configured). - A degrading target (unreachable, or backlog nearing the retention cap) is surfaced across metrics, logs, and traces simultaneously.
Packaging
- Single static, CGO-free binary; ~5 MB
scratch-based Docker image; sample systemd unit; grant SQL; a runnable docker-compose demo; and full docs.
Not in this release (roadmap)
- MySQL and Redis engines (the core is engine-agnostic and designed for them).
- HA / leader election, multi-master conflict resolution, live DDL replication, Helm chart, hardened fan-out.