replicare

replicare is an open-source data replication daemon. Point it at a source database and one or more target databases: it makes an initial copy of the selected data, then keeps replicating inserts, updates and deletes so the targets stay in step with the source.

It is written in Go and ships as a single static binary. Postgres, MySQL and Redis are supported, each replicating to the same engine. The source code is on GitHub.

Why it's different

  • No privileged change stream required. replicare does not need the Postgres WAL, the MySQL binlog or a Redis change feed. On Postgres and MySQL it captures changes with ordinary, grantable privileges using triggers; on Redis it uses SCAN reconciliation and writes nothing to the source. So it works on managed databases and old servers where logical replication isn't available.
  • Least privilege. On the source it needs CREATE on the database (or a pre-created schema it owns) plus USAGE, TRIGGER and SELECT. On the target, just SELECT, INSERT, UPDATE and DELETE on tables that already exist. See the grant scripts.
  • Faithful transport, never transform. Values move verbatim and are validated by the target database itself. replicare never coerces, truncates or "repairs" data: on a type or constraint error it stops loudly instead of corrupting silently.
  • Handles very large and very old sources. Change capture starts first, then a chunked, parallel copy runs while changes queue up and reconcile, so there is no single frozen snapshot and "too big to snapshot" isn't a problem. Source-side SQL stays compatible back to Postgres 9.x.
  • Observable. Prometheus metrics, a status HTTP API, structured logs and OpenTelemetry traces, plus a status command that works without any monitoring stack.

Where to start

Engines

The shared pages describe the relational baseline using Postgres; each engine page covers that engine's specifics.

  • Postgres — the reference engine: trigger capture, faithful COPY, foreign-key groups, cyclic and self-referencing tables, least-privilege grants.
  • MySQL — MySQL→MySQL: InnoDB and local_infile requirements and the two operational wrinkles.
  • Redis — Redis→Redis: capture-less SCAN reconciliation, the delete sweep, value-faithful DUMP/RESTORE, TTL, cluster and ACLs.

Active-active

Elsewhere in the repository

Deployment files live in deploy/: a sample systemd unit, the least-privilege grant SQL and the Redis ACL presets. Runnable end-to-end demos are in examples/ (Postgres, MySQL, Redis); one config that runs a Postgres and a Redis pipeline together is examples/postgres-and-redis.yml. Load-and-verify harnesses that drive heavy write traffic and check the target converges are documented for Postgres, MySQL and Redis; see Operations → Load & convergence testing.

Getting started

This walks you from zero to a running replication in a few minutes, then explains how to point replicare at your own databases.

What replicare does

You give replicare a source database and one or more target databases and a list of tables. It:

  1. installs lightweight capture (triggers) on the source,
  2. copies the selected data to the targets (chunked and parallel), then
  3. streams every insert/update/delete continuously to keep the targets converged.

No WAL access, no REPLICATION role, no wal_level change — just ordinary, grantable privileges. Architecture explains why.

Try the demo (2 minutes)

The fastest way to see it work end to end:

task docker                        # build the replicare image
cd examples/demo && docker compose up

This starts a source Postgres (seeded with public.orders), a target Postgres, and replicare copying then streaming between them. In another shell:

# initial copy landed
docker compose exec target psql -U postgres -d warehouse -c "SELECT count(*) FROM public.orders;"

# make a live change on the source and watch it propagate
docker compose exec source psql -U postgres -d app \
  -c "INSERT INTO public.orders VALUES (101,'live'); UPDATE public.orders SET note='changed' WHERE id=1; DELETE FROM public.orders WHERE id=2;"
docker compose exec target psql -U postgres -d warehouse -c "SELECT count(*), max(id) FROM public.orders;"

# operator surface
curl localhost:8080/status | jq
curl localhost:9090/metrics | grep replicare_

Ctrl-C then docker compose down -v to clean up.

Verifying the metrics

/metrics shows many replicare_* series once a sync is streaming, not just one. To eyeball the series that only populate at runtime:

curl -s localhost:9090/metrics | grep -E \
  'replicare_(rows_copied_total|apply_batch_seconds|throughput_rows_per_second|replication_lag_seconds|table_phase_info|target_up)'
  • replicare_table_phase_info (one line per table, initial_copy then streaming), replicare_target_up and the row estimate replicare_initial_copy_rows_target appear as soon as the sync starts; replicare_rows_copied_total climbs as the initial copy commits rows.
  • replicare_apply_batch_seconds and replicare_replication_lag_seconds refresh on every streaming drain pass.
  • replicare_throughput_rows_per_second is only non-zero on a pass that actually applied deltas — make a change on the source (the INSERT/UPDATE/DELETE above) and re-scrape; a caught-up idle sync correctly reports 0 here.

The full list of every metric, its type, and its labels is in operations.md.

Point it at your own databases

0. Run the pre-migration checklist

Before pointing replicare at a production source, gather the facts below and confirm the must-haves. The queries for each item, for Postgres and MySQL, are in operations.md → Before a migration: checklist.

Sizing and traffic

  • Total database size, and row counts and sizes per table.
  • Write volume over 24 hours (inserts, updates and deletes): every change adds one row to replicare's change tracking on the source.
  • Daily and weekly low-traffic windows, for catching up and for the final cutover.

Must be true before starting

  • Every table has a primary key or a UNIQUE constraint (tables without one are skipped).
  • Postgres: every extension the tables use exists on the target. MySQL: every table is InnoDB, the source is 5.7 or newer (older sources cannot add replicare's triggers next to a table's own), and the target has local_infile on.
  • Materialized views and views are not replicated: plan to create and refresh them on the target. Postgres large objects are not replicated either.
  • Schema changes and migrations are frozen for the whole replication (including online schema-change tools such as pt-online-schema-change / gh-ost on MySQL).
  • Regular vacuum and maintenance jobs are known: plain VACUUM / autovacuum is fine; heavy jobs that lock whole tables (VACUUM FULL, CLUSTER, pg_repack, MySQL OPTIMIZE TABLE) are paused during the initial copy and kept out of peak hours afterwards.
  • No long-open (idle in transaction) sessions on the source.
  • Source and target versions checked (target the same major version or newer), grants in place, and disk room on the source for the change-tracking tables to grow.
  • A plan for sequence / identity counters at cutover (replicare reseed-sequences).

1. Prepare the target schema

replicare replicates data only — it never creates or migrates target tables. Create/migrate the target tables yourself first, matching the source columns you want to replicate. (Types must be compatible; pre-flight will tell you if not.)

The quickest way to match the source is to dump its structure only (no data) to a file, then load that file on each target.

# Postgres — 1) dump the source structure (no ownership/privilege noise)
pg_dump --schema-only --no-owner --no-privileges -h SRC_HOST -p SRC_PORT -U USER -d DB > schema.sql
# 2) load it into the target
psql -h TARGET_HOST -p TARGET_PORT -U USER -d DB -f schema.sql

# narrow it: -n public (one schema), -t public.orders (one table)
# MySQL — 1) dump  (-P is the PORT; lowercase -p is the password prompt)
mysqldump --no-data -h SRC_HOST -P SRC_PORT -u USER -p DB > schema.sql
# 2) load
mysql -h TARGET_HOST -P TARGET_PORT -u USER -p DB < schema.sql

Credentials. These tools ask for a password when the server requires one: Postgres prompts interactively (and mysqldump -p means "prompt me"). Don't put the password in the command — it leaks into your shell history and ps. Instead store it in ~/.pgpass (Postgres, chmod 600) or ~/.my.cnf (MySQL). If you prefer a one-step pipe (pg_dump … | psql …), note it cannot prompt — the password step and the data stream collide — so set up ~/.pgpass / PGPASSWORD first, or stick with the two-step form above.

For active-active, run this against every node before starting (and apply any later schema change to every node first — see active-active.md).

2. Grant least-privilege roles

Run the grant scripts (edit the role/schema/db values, or pass them with -v):

psql -v ON_ERROR_STOP=1 -v role=replicare -v db=app -v app_schema=public \
  -f deploy/grants-source.sql "host=SOURCE dbname=app user=admin"
psql -v ON_ERROR_STOP=1 -v role=replicare -v db=warehouse -v app_schema=public \
  -f deploy/grants-target.sql "host=TARGET dbname=warehouse user=admin"

The source role needs only CREATE-on-db (or a pre-created owned replicare schema) plus USAGE/TRIGGER/SELECT; the target role needs only DML. See configuration.md and ../deploy/ for details.

For MySQL, use ../deploy/grants-source-mysql.sql and ../deploy/grants-target-mysql.sql instead: the source role needs SELECT+TRIGGER on the replicated tables and ownership of the replicare capture database; the target role needs the four DML verbs plus CREATE TEMPORARY TABLES. See the MySQL engine page.

For Redis, use the ACL presets ../deploy/acl-source-redis.txt and ../deploy/acl-target-redis.txt: the source role is read-only (+scan +dump +pttl +exists +type +memory|usage); the target role needs +restore +del +scan +exists +module|list. Note the foot-gun — RESTORE is @dangerous and must be granted explicitly with +restore. See the Redis engine page.

3. Write a config

Copy ../examples/replicare.yml and edit the connection blocks and the sync's table selection. Minimal example:

state_store:
  engine: postgres
  postgres: { host: db, port: 5432, database: replicare_state, user: replicare, password: "${PW}", sslmode: require }
sources:
  app:
    engine: postgres
    postgres: { host: app-db, port: 5432, database: app, user: replicare, password: "${PW}", sslmode: verify-full }
targets:
  warehouse:
    engine: postgres
    postgres: { host: wh-db, port: 5432, database: warehouse, user: replicare, password: "${PW}", sslmode: require }
syncs:
  - name: app-to-warehouse
    source: app
    targets: [warehouse]
    include: ["public.*"]
    exclude: ["*_audit"]

Choosing tables. include and exclude are lists of schema.table glob patterns, and exclude is applied on top of include. To drop one specific table, name it in full under exclude:

    include: ["public.*"]            # everything in public…
    exclude: ["public.audit_log"]    # …except this one table

Patterns work too — *_audit excludes every table whose name ends in _audit. A few notes: selection is read at daemon start, so change it and restart; excluding a table only stops replicating it (rows already on the target are left in place, not deleted); and if another replicated table has a foreign key pointing at the excluded one, replicare warns about the dangling reference (the target must already satisfy that parent). For Redis, selection uses key patterns instead of schema.table.

Secrets like ${PW} are read from the environment. Full field reference: configuration.md.

For a MySQL sync, set engine: mysql with mysql: blocks (see the MySQL connection block); the state store stays on Postgres. A ready-to-run example is in ../examples/demo-mysql/config.yml.

For a Redis sync, set engine: redis with redis: blocks (see the Redis connection block); the state store still stays on Postgres (Redis writes nothing to the source, so all its durable state lives there). Selection uses key-pattern globs (include: ["app:*"]) rather than schema.table. See the Redis engine page.

4. Validate, then run

export PW=...                          # the replicare role's password
replicare validate config.yml          # pre-flight: no changes, reports problems
replicare run config.yml               # start replicating (Ctrl-C / SIGTERM to stop cleanly)

validate connects read-only and classifies every column pair — fix any BLOCK findings (incompatible/missing types) before running. Once running, check progress any time with replicare status config.yml.

A few more commands you'll reach for later: replicare capture install|remove to pre-provision (or tear down) the source-side trigger capture out of band, replicare reseed --sync <s> --target <t> to force a laggard or diverged target to re-copy its data, and — when you promote a one-way (DR) target — replicare reseed-sequences --sync <s> to advance its identity/sequence counters to max(id)+1 before it takes writes (otherwise nextval() collides with the copied rows; passive/DR only — it refuses an active-active mesh). See the CLI reference.

5. Deploy

Install the binary and the sample systemd unit, run the Docker image, or deploy on Kubernetes with the Helm chart — see Kubernetes (Helm). See operations.md for tuning, monitoring, retention/reseed, and troubleshooting.

Next

Architecture

How replicare moves data, end to end: the overview for newcomers and architects. Everything here is engine-neutral unless noted — Postgres, MySQL, and Redis all ride the same pipeline, differing only behind the Source/Sink driver interfaces.

The one non-negotiable

replicare assumes it will never have a privileged, durable change stream — no Postgres WAL, no MySQL binlog, no durable Redis change feed. Change capture must work with ordinary, grantable privileges. That single constraint shapes everything below.

The pipeline

A sync (one source → one or more targets, with a table/key selection) runs this lifecycle. Copy and streaming are the two phases; capture and checkpointing wrap them.

 install capture ─► chunked parallel COPY ─► cut over ─► stream deltas ─► (repeat)
   (first)             (initial copy)                     (continuous)
        └──────────────── checkpoint to the StateStore throughout ─────────────┘
  1. Capture is installed FIRST (relational engines), so changes begin queuing before and during the copy. For Redis there is nothing to install — capture is replaced by re-reading the live keyspace (see Engines).
  2. Initial copy streams each table/unit in chunks, in parallel, source → target through an io.Pipe that never fully buffers a chunk. Because capture is already running and apply is idempotent, any change during the copy window reconciles automatically — there is no single frozen snapshot, which is how replicare handles sources too large to snapshot.
  3. Cutover flips each table to streaming once its copy completes.
  4. Streaming drains changes continuously and applies them idempotently, forever, until the daemon is stopped.
  5. Checkpointing records progress in the StateStore at every step, so a crash resumes from the last confirmed point (applies are idempotent, so replay is harmless).

The four ideas that make it non-obvious

Most of replicare's design collapses into four decisions. Understand these and the rest follows.

1. Capture is a dirty-key set, not an ordered log

Triggers on the source record only the changed row's primary key (PK-only capture) into a per-table delta table — never the full row image. At sync time replicare reads the unconsumed keys, re-reads the current values from the live source table, and applies those. This is self-healing: it always copies the latest value, so multiple changes to one key coalesce into one re-read and transient intermediate states don't matter.

Crucially, consumption is a set-difference (delta MINUS track, per target), not a WHERE seq > cursor scan. A scalar high-water-mark cursor is broken here: a value a trigger assigns inside a transaction (a sequence, a txid) is fixed when the trigger fires, but the delta row only becomes visible when the transaction commits — and commits happen in a different order than those values were grabbed. A slow transaction holding a low sequence that commits after the daemon's read would be skipped forever. The set-difference model can't skip: an uncommitted or late-committing delta is simply "not yet in track," caught on the next pass. Rows are deleted by unique delta_id (never by PK) to survive the read-and-clear race.

2. FK connected components are the unit of ordering, parallelism & consistency

Within a sync, replicare auto-partitions the selected tables into FK connected components (from the source catalogs, over the selected tables only). No FK edge crosses a component boundary, which gives three properties for free:

  • Ordering is always complete — upserts apply parent→child, deletes child→parent, within a component; no child is ever stranded from its parent. Cycles and self-references can't be topo-sorted, so a retry fallback handles them.
  • Distinct components are referentially independent → safe to copy and stream fully in parallel.
  • A component's changes for one drain pass apply in a single target transaction, so the target is always referentially consistent within the group.

A "giant component" (everything wired through a users/tenant hub table) erases parallelism, so replicare warns and lets you override.

3. Faithful transport — never transform

replicare is a pipe, not an interpreter. Values move verbatim: the source server renders them, the target server's own input function validates them. It never coerces, truncates, casts, or "repairs" data — on a type/constraint error it halts loud rather than corrupting silently. This is a hard product promise: no value-transform capability exists, even as an opt-in.

  • Relational engines move values as text COPY (both initial copy and delta apply), pinning formatting-only session settings (e.g. extra_float_digits=3) so text round-trips losslessly across version gaps.
  • Redis moves values as DUMP → RESTORE — RDB-serialized bytes, value-faithful (the target re-encodes), never reconstructed.
  • A pre-flight check classifies every source↔target type pair up front: block on incompatible/missing, warn on risky-lossy, ok on identical/widening — turning mid-stream failures into an actionable report before any sync starts.

4. The StateStore is separate from source-side capture

Two different kinds of state, often confused:

  • Source-side capture state — the delta and track tables — lives on the source database (that's where triggers write). It is the correctness-bearing replication state.
  • The StateStore — which syncs exist, per-table copy progress, per-target cursors, and the single-active ownership lock — is the daemon's own operational bookkeeping. It lives in a dedicated schema on a Postgres database (v1's only StateStore backend), which may be the target, the source, or a separate server.

They are separate concerns even when both happen to be Postgres. One consequence: every sync needs a Postgres StateStore — including a MySQL→MySQL or Redis→Redis sync (a documented wart for the non-Postgres engines).

How the three engines map on

The pipeline is identical; only the driver behind Source/Sink changes.

Postgres (reference)MySQLRedis
Capturetrigger CDC → delta/track tablestrigger CDC (same model)none — capture-less
Upsert CDCre-read dirty PKsre-read dirty PKsfull-keyspace SCAN reconciliation (re-read every pass)
Delete CDCdelta row for the deleted PKdelta rowdurable target-vs-source keyspace diff
Transporttext COPYLOAD DATA LOCAL INFILEDUMP → RESTORE REPLACE
UnitFK component of tablesFK component of tablesone key-namespace unit (redis.db<N>), single-member
Apply atomicityper-component transactionper-component transactionper-key (no cross-key atomicity)
Deliveryat-least-once; effectively-exactly-once when the StateStore is the targetat-least-onceat-least-once

Redis is the outlier: with no change stream at all, upserts come from re-reading the whole keyspace each pass and deletes from diffing the target against the source (keyspace notifications are an optional, lossy accelerator, never trusted). See the Redis engine page for the full story, and the Postgres / MySQL pages for the relational specifics.

Delivery, topology & bloat

  • Delivery: at-least-once with idempotent apply (upserts keyed on PK/key, deletes by PK/key), checkpointed cursors. A crash replays from the last cursor; replays are harmless.
  • Topology: user choice — default single→single; fan-out (one source → many targets) is supported for the relational engines; multi-master and HA/leader election are on the roadmap. A sync is always single-engine.
  • Source bloat: the delta tables are a high-churn queue on a source you may not own, so replicare bounds its footprint with per-table delta tables, a batched consumption-gated purge + aggressive autovacuum, and bounded retention with forced reseed of an over-cap target. See operations and the reseed state machine.

Process & observability

One daemon runs many named syncs as goroutine worker pools (one process, central observability). Everything is visible across four channels at once: a Prometheus /metrics endpoint, OpenTelemetry traces, structured slog logs, and a /status HTTP API. A degrading target must surface in metrics, logs, and traces simultaneously — see operations for the metric/span/event reference.

Configuration reference

replicare is configured by a single YAML file (passed to every command). The schema is an engine-neutral envelope plus a typed per-engine connection block.

Any string value may reference the environment as ${VAR} (required — errors if unset) or ${VAR:-default} (falls back to default) — not just secrets. Expanded scalars are re-typed on decode, so it works for ports, hosts, addresses, and booleans too (e.g. port: ${PGPORT:-5432}), matching the "env-var overrides for any field" goal. Keeping passwords out of the file is the common case. An unset ${VAR} with no default is a hard config-load error (exit 2).

Validate a config without connecting-and-running via replicare validate <config>.

Top-level structure

logging:        { ... }   # log level/format
observability:  { ... }   # metrics/status/OTLP endpoints
state_store:    { ... }   # replicare's own progress store (an endpoint)
sources:        { ... }   # named source endpoints
targets:        { ... }   # named target endpoints
syncs:          [ ... ]   # what replicates where (one-way)
nodes:          { ... }   # peer endpoints for active-active clusters (optional)
clusters:       [ ... ]   # active-active (multi-master) groups (optional)

nodes and clusters are optional and additive — a config with neither behaves exactly like a one-way daemon. See Active-active clusters and the multi-master design note.

logging

FieldTypeDefaultNotes
leveldebug|info|warn|errorinfo
formatjson|textjsontext is friendlier for local runs

observability

FieldTypeDefaultNotes
metrics_addrlisten addr, e.g. :9090offserves Prometheus /metrics
status_addrlisten addr, e.g. :8080offserves /status, /healthz, and /metrics
otlp_endpointhost:port, e.g. otel:4317offexports OTel traces to an OTLP/gRPC collector
stall_timeoutduration (2m, 0, negative)2mhow long a streaming loop may go without completing a pass before /healthz reports unhealthy (so Kubernetes restarts a wedged pod). Must exceed your slowest drain pass. Unset/0 = 2m; a negative value disables the check

Leaving an address empty disables that endpoint. When metrics_addr and status_addr differ, /metrics is served on both.

/healthz reflects streaming liveness, not target reachability: a loop that is cycling — even one retrying against a down target — stays healthy; only a loop that stops completing passes (a hung query on a dropped socket) trips stall_timeout and fails the probe. The daemon also reconnects a dropped source/target connection on its own between passes, so a transient blip recovers without a restart; the probe is the backstop for a true wedge.

Endpoints: state_store, sources, targets

An endpoint is an engine name plus a typed connection block. sources and targets are maps of name → endpoint; state_store is a single endpoint.

sources:
  app:                       # the name referenced by a sync
    engine: postgres
    postgres:                # the engine-specific block
      host: app-db
      port: 5432
      database: app
      user: replicare
      password: "${APP_PW}"
      sslmode: verify-full
      params:                # optional extra connection params
        application_name: replicare

Postgres connection block

Used when an endpoint's engine is postgres. See the Postgres engine page for the CDC model, FK-component handling, faithful COPY transport, and the effectively-exactly-once bonus when the state store is the target.

FieldTypeNotes
hoststring
portintusually 5432
databasestring
userstringthe least-privilege replicare role
passwordstringuse ${VAR}
sslmodedisable|allow|prefer|require|verify-ca|verify-fullTLS mode (libpq semantics)
paramsmapextra key=value connection parameters

MySQL connection block

Used when an endpoint's engine is mysql. See the MySQL engine page for the two operational wrinkles (a MySQL sync still keeps its state store on Postgres; MySQL syncs are strictly at-least-once).

FieldTypeNotes
hoststring
portintdefault 3306
databasestringthe DSN default database only — selection may reference other databases as db.table (a MySQL schema is a database)
userstringthe least-privilege replicare role
passwordstringuse ${VAR}
tlsdisable|allow|prefer|require|verify-ca|verify-fullTLS mode (same spectrum as Postgres sslmode); default prefer
local_infileboolhint that the target permits LOAD DATA LOCAL INFILE, the copy/apply transport (required in v1). replicare probes it at connect and halts loud if off — no INSERT fallback yet. A server system variable, not a grant
paramsmapextra key=value DSN parameters

Redis connection block

Used when an endpoint's engine is redis. See the Redis engine page for how Redis CDC works (SCAN reconciliation + a target-vs-source delete sweep, not triggers) and its two wrinkles (a Redis sync still keeps its state store on Postgres; copy progress is coarse). Redis is topology-shaped rather than a single host, and carries engine-specific selection and CDC tuning.

FieldTypeNotes
modestandalone|cluster|sentineldefault standalone; sentinel is parsed but experimental (failover not hardened in v1)
host / portstring / intstandalone seed (port default 6379)
nodeslist of host:portseed nodes for cluster/sentinel (also allowed for standalone)
dbintlogical DB index (standalone only; must be 0 in cluster)
user / passwordstringACL user (Redis 6+); use ${VAR} for the password
tlsdisable|allow|prefer|require|verify-ca|verify-fullsame spectrum as Postgres sslmode; default disable — note this diverges from Postgres/MySQL (prefer), so set it explicitly to avoid plaintext
sentinel_masterstringrequired when mode: sentinel
read_from_replicabooloffload value reads to replicas; delete detection still reads the master (a lagging replica would cause false deletes)
typeslistoptional type filter: string/list/set/zset/hash/stream (needs Redis 6.0+ for server-side SCAN … TYPE)
scan_countintSCAN COUNT hint per call (default 512); distinct from the neutral drain batch
reconcile_intervaldurationcadence of the rolling upsert reconciliation scan
delete_sweep_intervaldurationcadence of the target-vs-source delete sweep (bounds delete-propagation latency)
big_key_warn_bytessizeDUMP-and-warn above this (MEMORY USAGE); 0 = off
big_key_refuse_bytessizeblock loud above this; must be ≥ big_key_warn_bytes
notificationsboolenable the lossy keyspace-notification latency accelerator (default off; never trusted for correctness)
ttl_moderelative|absttlrelative (default) is clock-skew-safe; absttl is opt-in for trusted clocks
paramsmapextra engine-internal params

Selection for a Redis sync is key-pattern globs (Redis SCAN … MATCH semantics), not schema.table — see Selection below.

state_store is where replicare keeps its own operational state (sync progress, cursors, the ownership lock) — a dedicated replicare_state schema it creates and owns. It may point at the target DB, the source DB, or a separate Postgres. (This is distinct from the source-side delta/track tables, which always live on the source. Redis writes nothing to the source at all, so a Redis sync's only durable state is here — see the Redis engine page.)

syncs

A sync is one replication job: a source, one or more targets, and a table selection. All targets of a sync must use the same engine as the source (never cross-engine).

Fan-out (a sync with several targets) copies and streams each target independently: initial-copy progress is checkpointed per (sync, target, table), so one slow or restarted target never disturbs another's watermark, and each converges on its own. (This is also the per-node building block of an active-active cluster.)

syncs:
  - name: app-to-warehouse    # unique; also the ownership-lock key
    source: app               # a key in `sources`
    targets: [warehouse]      # keys in `targets` (one or more; fan-out)
    enabled: true             # optional; false PAUSES just this sync (see below)
    sync_sequences: false     # optional; true keeps DR counters warm (see below)
    include: ["public.*"]     # selection globs (schema.table)
    exclude: ["*_audit"]      # excluded from the include set
    tuning: { ... }           # optional (see below)

Keeping DR counters warm (sync_sequences)

sync_sequences is an opt-in, one-way/DR-only switch (default off). With it on, the daemon advances each replicated table's owned identity/serial/AUTO_INCREMENT counter on the target to max(id)+1 on a lazy interval during streaming — the same mechanism as the reseed-sequences command, kept running — so a promoted DR node has nothing sequence-related to do at cutover. Enabling it adds a steady-state target grant (Postgres UPDATE on each sequence, MySQL ALTER on each table); a missing grant is logged and retried, never fatal. It is refused at config load when the target is also an active-active cluster member (a mesh counter advance is silent data loss — use UUID/ULID keys there). See operations → Sequences & identity counters.

Pausing a sync (enabled)

enabled is a per-pipeline pause switch. It is a pure opt-out: omit it (or set true) and the sync runs — a config written before this field existed is unchanged. Set enabled: false to pause that one sync; other syncs (and clusters) in the same config keep running.

enabledBehavior
unset / trueThe daemon brings the sync up and streams it (default).
falseThe daemon skips it at startup and does not take its ownership lock. No copy, no streaming, no capture install for a never-started sync.

Pause is data-loss-free, by design. If a sync was already running when you pause it, its source capture triggers are left in place, so change deltas keep queuing on the source while it's paused. Setting enabled: true again (and restarting) resumes it and drains the accumulated backlog — the target catches up with nothing lost.

The tradeoff: while paused, the source-side delta tables keep growing, and the retention/reseed source-protection does not run (it lives in the streaming loop, which is skipped). So a long pause on a busy source can grow the source's replicare schema unbounded. Pause for maintenance/cutover windows, not indefinitely; if you need to stop for a long time, prefer removing the sync (which you can teardown so capture is uninstalled) over an open-ended pause.

It takes effect at daemon start, so pausing or resuming is a config change + restart, not a live toggle. On Kubernetes/Helm, change the value and helm upgrade — the resulting pod rollout applies it. Set enabled on the sync by name in your config; if you use Helm --set, note it addresses syncs by array index (config.syncs[0] = the first sync), so with several pipelines it's easy to pause the wrong one — see kubernetes.md. Verify which sync actually paused in the daemon logs — sync disabled (paused); skipping sync=<name> for a paused sync, sync streaming sync=<name> for a running one — or with replicare status (the [PAUSED] banner). (Clusters have the same enabled flag; setting it false pauses all of that cluster's mesh edges.)

Pausing every pipeline is fine — the daemon stays up. With all syncs (and clusters) paused the daemon does not exit; it idles, keeps serving /metrics, /status, and /healthz, and stays a healthy pod (so a Kubernetes Deployment does not restart-loop it). replicare status then shows each paused sync with a [PAUSED] banner (and, live, its still-growing delta backlog) rather than a stale "streaming" phase.

Selection

include/exclude are schema.table globs (* matches within a name segment). For MySQL, a schema is a database, so these are db.table globs and a single sync may span several databases. Tables without a primary key or usable unique key are skipped with a warning (they can't be captured). An FK pointing from a selected table to an excluded one triggers a dangling-FK warning — the target must already satisfy that parent.

For Redis, include/exclude are key-pattern globs using Redis SCAN … MATCH semantics (*, ?, […], hash-tags), matched exclude-wins. There are no tables, keys, or FKs — the "unit" is the logical keyspace (one DB, or DB 0 in cluster). An optional types filter in the redis: block narrows selection to specific value types.

tuning (per sync)

FieldTypeDefaultMeaning
drain_intervalduration (1s, 500ms)1stime between streaming drain passes; longer = more coalescing (less source load) but higher lag
drain_batchint1000max dirty deltas applied per table per pass; with drain_interval this is the per-table streaming ceiling (~drain_batch/drain_interval rows/s). Raise for a high-volume table that lags
apply_concurrencyint (≥1)1how many of a component's tables apply concurrently during streaming. 1 is strictly sequential; higher fans the per-table apply across the copy-worker pool (bounded by pool.max_*_connections). Helps when several tables are backlogged at once
retention.max_ageduration (24h, 0 = off)24holdest unconsumed delta before a laggard target is reseeded
retention.max_bytessize (512MB, 0 = off)offdelta-table on-disk size before reseed
pool.max_source_connectionsint4source connection cap (copy worker pool is sized from it)
pool.max_target_connectionsint4target connection cap

Durations use Go syntax (ms, s, m, h); sizes accept KB/MB/GB/TB (×1000) or KiB/MiB/GiB (×1024), or a bare byte count. Defaults favor low source pressure — see operations.md for tuning guidance.

For Redis, the neutral retention.* knobs are inert (there is no source-side delta queue to bound); Redis pacing lives in the redis: block instead (scan_count, reconcile_interval, delete_sweep_interval).

Active-active clusters (nodes and clusters)

Status: in progress. For Postgres and MySQL, a clusters: block runs today with full conflict resolution: the daemon brings up a full-mesh active-active cluster with loop suppression (MM3) and HLC last-write-wins (MM4 for Postgres, MM5 mirrors it for MySQL), so writes accepted on any node — including concurrent writes to the same key — converge to the same value on every node, with no config and no user schema change; deletes resolve via GC'd tombstones. Redis clusters are still parse-and-validate only. See the design note for the full status. These keys are optional and additive: omit them and replicare behaves exactly as a one-way daemon. Do not confuse a replicare clusters: entry (a group of active-active peer databases) with a Redis endpoint's redis.mode: cluster (one sharded Redis) — different scopes.

An active-active cluster keeps N peer databases converged with writes accepted on any node. Conflict resolution is zero-config — replicare manages a hidden per-row version itself (HLC last-write-wins; no column to add, nothing to declare); see the design note. A cluster is single-engine, like a sync.

nodes

A map of peer endpoints. A node has the same shape as a source/target endpoint (an engine + that engine's connection block) plus an optional node_id. Unlike a source/target, a node is both read from and written to.

FieldTypeDefaultNotes
enginestring—postgres | mysql | redis; must match the cluster's engine
node_idstringthe map keystable replication-origin identity, unique within a cluster; stamped into each change's version. Usually leave it to default to the key
<engine>:block—the engine connection block, exactly as for a source/target

clusters

A list of active-active groups.

FieldTypeDefaultNotes
namestring—unique across all syncs and clusters
enginestring—the one engine all members share (single-engine rule)
memberslist of strings—keys into nodes:; ≥ 2, each a distinct node with a distinct node_id
topologystringmeshv1 supports only mesh (full mesh, any N ≥ 2); ring/partial are deferred
include / excludelist of globs—table/key selection, interpreted per engine exactly as for a sync
tuningblocksync defaultssame knobs as a sync's tuning
nodes:
  us: { engine: postgres, postgres: { host: pg-us, port: 5432, database: app, user: replicare, password: ${PW}, sslmode: require } }
  eu: { engine: postgres, postgres: { host: pg-eu, port: 5432, database: app, user: replicare, password: ${PW}, sslmode: require } }
  ap: { engine: postgres, postgres: { host: pg-ap, port: 5432, database: app, user: replicare, password: ${PW}, sslmode: require } }

clusters:
  - name: global-app
    engine: postgres
    members: [us, eu, ap]     # any N >= 2; node_id defaults to us/eu/ap
    enabled: true             # optional; false pauses ALL of this cluster's mesh edges
    include: ["public.*"]
    exclude: ["*_audit"]
    # No conflict block: HLC last-write-wins is automatic. Nothing to declare.

A cluster has the same enabled opt-out as a sync (above): unset/true runs every mesh edge; false pauses the whole cluster (all edges skipped at startup). Per-edge pausing is not exposed.

Cycle safety for one-way syncs

Independently of clusters, replicare now rejects an un-declared replication cycle among plain one-way syncs (e.g. A→B and B→A, or a longer ring) at config load — that topology silently corrupted data before. A genuine active-active setup must be declared as a clusters: entry, which is exempt.

Full example

See ../examples/replicare.yml for a fully annotated config. For one config that runs a Postgres pipeline and a Redis pipeline side by side (the per-sync single-engine model), see ../examples/postgres-and-redis.yml.

CLI reference

Most commands take the config file as an argument. Exit codes: 0 success, 1 runtime error or a pre-flight BLOCK (see validate), 2 usage/config error (bad flags, or a config that won't load — unknown engine, unknown field, unset ${VAR}, cross-engine sync).

replicare <command> [args]

help / version

replicare help          # also: --help, -h — prints the command usage
replicare version       # also: --version, -v — version, commit, build date, Go version

validate <config>

replicare validate config.yml

Loads the config and runs pre-flight for every (sync, target): connects read-only, introspects both sides, and classifies every column pair — identical/widening (ok), risky/lossy (warn), incompatible or missing type (block). Also reports FK components, a giant-component warning, dangling FK edges, and skipped no-key tables. Makes no changes. Fix all BLOCK findings before running.

Exit codes are meaningful for scripting: 0 = clean (ok/warn only), 1 = at least one BLOCK finding (a clean "incompatible" verdict, not a crash), 2 = the config didn't load or a connection failed. So replicare validate ... && replicare run ... gates a start on a block-free pre-flight.

run <config>

replicare run config.yml

Runs the daemon: starts the observability endpoints, then brings up and streams every sync until SIGINT/SIGTERM. On a signal it shuts down gracefully — finishes the in-flight drain pass, checkpoints, and exits 0.

Each sync runs under a single-active ownership lock; a second daemon pointed at the same sync stands by. State is checkpointed, so a restarted process resumes from where it left off.

status <config> [--json] [--sync <name>] [--no-live] [--watch <dur>]

replicare status config.yml                    # human-readable table, LIVE by default
replicare status config.yml --json             # machine-readable
replicare status config.yml --sync pg-pipeline # one sync only
replicare status config.yml --no-live          # state store only (no source/target connections)
replicare status config.yml --watch 5s         # re-render every 5s until Ctrl-C

Each sync prints a one-line health headline followed by a per-(table, target) grid, read from the state store (so it works whether or not a daemon is running):

sync: app-to-warehouse
  streaming · last pass 2s ago · last applied 40s ago (1240 rows total) · backlog 0 (caught up)
  TABLE           COPY  ...  SEEN  ...  LAST_SYNC   ROWS   RESEED
  public.orders   done  ...  2s    ...  40s ago     1240   -

Two different time signals — the distinction that makes a caught-up sync legible:

  • SEEN — time since the last healthy streaming pass (a liveness heartbeat). The daemon refreshes it every pass whether or not data moved, so a small SEEN means "running and current"; a large SEEN means the daemon is stopped or stalled. (This is what the old LAG column was, renamed — it was never "how far behind".)
  • LAST_SYNC / ROWS — time since the last pass that actually applied rows to this (target, table), and how many it applied. A healthy, idle sync shows a recent SEEN but a growing LAST_SYNC — that's normal (nothing to apply), not lag. - until the first data-moving pass. (These come from the state store, so they show in --no-live too.)

The headline aggregates these across the sync: phase, most-recent SEEN, most-recent data movement + total rows applied, and (live) the backlog rollup.

Live by default. Unless you pass --no-live, status also connects to the sync's source and targets to add signals that live only in the databases — the visibility you'd otherwise get from Grafana:

  • SRC_ROWS / TGT_ROWS — live source and target row counts (Redis: key counts). During initial copy TGT_ROWS climbs toward SRC_ROWS; while streaming they track each other.
  • BACKLOG — the per-target unconsumed delta backlog as rows (oldest-age). It distinguishes three states: 0 = caught up (healthy — the common steady state); N (age) = that many deltas waiting; - = unknown, not zero — the signal couldn't be read (a table not captured, Redis's capture-less engine, or a transient error), and a live: partial (…) note says why.

Live collection is best-effort and read-only (it installs nothing): if the source or a target is unreachable, the row still renders from the state store and a live: partial (…) note explains what was missing. --no-live skips all source/target connections (useful if you only have state-store access, or want the cheapest possible check).

--watch <dur> re-renders on the given interval (e.g. 10s, 1m) until SIGINT; combine with --json to stream snapshots.

verify <config> [--json] [--sync <name>] [--watch <dur>]

replicare verify config.yml                    # all syncs
replicare verify config.yml --sync pg-pipeline # one sync
replicare verify config.yml --json             # machine-readable
replicare verify config.yml --watch 30s        # re-check every 30s until Ctrl-C

A read-only source↔target convergence spot-check. For every replicated unit it counts and content-fingerprints the source and each target and compares. It writes nothing and installs nothing, so it is safe to run against a source you may not own and against live targets. Per table it reports one of:

  • ok — row counts and content checksums match.
  • drift-count — row counts differ (the target is missing/extra rows).
  • drift-checksum — counts match but content differs (a value diverged).
  • count-only-ok — counts match; the engine compares membership only (see below).
  • error — a table could not be compared (e.g. missing on the target).

The content fingerprint is an order-independent hash over a name-matched column projection, so it is robust to physical column-order differences and streams in constant server memory. Relational engines (Postgres, MySQL) hash row content; Redis compares the key set (count + membership), which catches missing/extra keys — the dominant Redis divergence — since transport is value-faithful DUMP→RESTORE (per-value Redis content diffing is a documented follow-up).

Exit codes: 0 = every selected sync converged; 1 = drift or an endpoint could not be reached; 2 = usage/config. So replicare verify config.yml is scriptable as a convergence gate. In --watch mode the exit code reflects the last completed pass.

Note: replication has lag — a drift-count/drift-checksum immediately after a burst of source writes usually just means the target hasn't caught up yet. Re-run (or use --watch); persistent drift is the real signal.

capture install|remove <config> [--sync <name>]

replicare capture install config.yml            # all syncs
replicare capture remove config.yml --sync app-to-warehouse

Installs or removes the trigger-based CDC machinery on a sync's source, over its selected tables. run installs capture automatically, so this is for pre-provisioning or manual teardown.

Note: install needs only TRIGGER (plus SELECT/USAGE and schema creation). remove additionally needs table ownership to drop the triggers — if the replicare role doesn't own the tables, removal of the trigger fails (reported), though replicare's own delta/track tables are still dropped.

reseed <config> --sync <name> --target <name>

replicare reseed config.yml --sync app-to-warehouse --target warehouse

Flags a target for a full re-copy. It signals the running daemon (via the state store) to re-copy that target from current source state on its next pass, then resume streaming — honoring single-active ownership rather than acting directly. Use it after out-of-band divergence, or to recover a target that fell behind.

If the target has no cursors yet (you ran reseed before the sync's first run), there is nothing to flag: replicare reports that the target will full-copy on its first run anyway and exits 0. That is expected, not a failure.

reseed-sequences <config> --sync <name> [--target <name>] [--dry-run]

replicare reseed-sequences config.yml --sync app-to-warehouse            # all targets of the sync
replicare reseed-sequences config.yml --sync app-to-warehouse --target warehouse
replicare reseed-sequences config.yml --sync app-to-warehouse --dry-run  # report, write nothing

Advances each replicated table's owned identity/serial/AUTO_INCREMENT counter on a promoted one-way (DR) target to max(id)+1, so after a failover cutover the node never re-issues an id already present in the replicated data. replicare replicates row values faithfully but not the sequence object's counter (data-only); for a passive replica that is harmless, but a promoted node's nextval() would collide with the copied rows. Postgres uses setval, MySQL ALTER TABLE … AUTO_INCREMENT. Tables with no owned counter (UUID / natural / composite PK) are skipped; --dry-run reports the target values and writes nothing.

  • Not the same as reseed. reseed re-copies a target's data; reseed-sequences only bumps its sequence counters.
  • Run order: stop replication into the node, run this, then open it to writes — a write in the gap could still grab a colliding id.
  • Passive / one-way / DR only. It refuses a target that is an active-active cluster member: there the same max(id)+1 collapses every node onto one counter and silently drops rows via last-write-wins. Active-active id allocation is a schema concern — use globally-unique keys (UUID v7 / ULID); see multi-master.
  • Privilege: a DR-only grant beyond the DML set — Postgres UPDATE on each sequence, MySQL ALTER on each table (reads only the target's own max(id); no source privilege). See operations → Sequences & identity counters.
  • Exit codes: 0 done; 1 runtime error (unreachable / missing grant / engine without the capability); 2 usage/config, or a refused mesh target.

Signals

run responds to:

  • SIGTERM / SIGINT — graceful shutdown (drain in-flight pass, checkpoint, exit 0).

An ungraceful kill is also safe: state is checkpointed and applies are idempotent, so a fresh run resumes and converges without duplicates.

Operating replicare

Practical guidance for running replicare in production: the operator's view. How it works underneath is in Architecture.

Lifecycle of a sync

  1. Pre-flight. replicare validate <config> (and the daemon at startup) introspects the source and target and classifies every column pair: identical/widening → ok, risky/lossy → warn, incompatible or missing type → block (refuses to start). It also reports FK components, a giant-component warning, dangling FK edges, and skipped no-key tables. Fix blocks before running.
  2. Capture-first. The daemon installs trigger-based CDC on the source's selected tables, so changes begin queuing into per-table delta tables immediately.
  3. Initial copy. A chunked, parallel copy of each FK component (parents first) runs while capture is active. It is resumable — a restart continues from the last checkpoint, not the top.
  4. Cutover → streaming. Once a component is copied, it flips to continuous streaming: each pass re-reads the current source values of dirty keys and applies them idempotently, FK-ordered.
  5. Purge / retention / reseed. Consumed deltas are purged; a target that falls too far behind is reseeded (see below).

Before a migration: checklist

Work through this before pointing replicare at a production source. Most items are information to gather (sizing, traffic, quiet windows); a few must be true before starting. Run the queries on the source unless stated otherwise. replicare validate <config> then checks the rest (types, foreign-key loops, keys) and reports anything that would block.

Postgres

Sizing and traffic

  1. Total database size.
    SELECT pg_size_pretty(pg_database_size(current_database()));
    
  2. Row counts and sizes per table. row_estimate comes from planner statistics; -1 (or 0 on older versions) means the table has never been analyzed — run ANALYZE first for real numbers.
    SELECT n.nspname AS schema, c.relname AS table_name,
           c.reltuples::bigint                            AS row_estimate,
           pg_size_pretty(pg_total_relation_size(c.oid)) AS total,
           pg_size_pretty(pg_indexes_size(c.oid))        AS indexes
    FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
    WHERE c.relkind IN ('r', 'p')
      AND n.nspname NOT IN ('pg_catalog', 'information_schema', 'replicare')
      AND n.nspname NOT LIKE 'pg_toast%'
    ORDER BY pg_total_relation_size(c.oid) DESC;
    
  3. Write volume over 24 hours. The counters only ever go up, so run this twice, 24 hours apart, and subtract. Every row inserted, updated or deleted in a replicated table adds one row to replicare's change tracking, so inserts + updates + deletes per day is roughly the daily change-tracking volume (and the work streaming has to keep up with).
    SELECT now() AS sampled_at,
           sum(n_tup_ins) AS inserts, sum(n_tup_upd) AS updates, sum(n_tup_del) AS deletes,
           (SELECT xact_commit + xact_rollback FROM pg_stat_database
             WHERE datname = current_database()) AS transactions
    FROM pg_stat_user_tables;
    
    For the busiest tables, SELECT relname, n_tup_ins + n_tup_upd + n_tup_del AS writes FROM pg_stat_user_tables ORDER BY writes DESC LIMIT 20;.
  4. Low-traffic windows (daily and weekly). Busy periods followed by lulls let streaming catch up, and a quiet window is the best time for the final cutover.

Must be true before starting

  1. Every table has a primary key or a UNIQUE constraint. replicare tracks changes by key. It uses the primary key, or else the first UNIQUE constraint; a bare CREATE UNIQUE INDEX without a constraint does not count. Tables listed here are skipped (with a warning), not replicated:
    SELECT n.nspname AS schema, c.relname AS table_name
    FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
    WHERE c.relkind IN ('r', 'p')
      AND n.nspname NOT IN ('pg_catalog', 'information_schema', 'replicare')
      AND n.nspname NOT LIKE 'pg_toast%'
      AND NOT EXISTS (SELECT 1 FROM pg_constraint k
                      WHERE k.conrelid = c.oid AND k.contype IN ('p', 'u'))
    ORDER BY 1, 2;
    
  2. Extensions. List them on the source and confirm the target has every one the replicated tables use (column types, defaults). replicare never installs extensions; a missing one fails pre-flight.
    SELECT extname, extversion FROM pg_extension ORDER BY 1;
    
  3. Materialized views (and plain views) are not replicated — only table data is. Create them on the target from the schema dump, then REFRESH MATERIALIZED VIEW there after the initial copy and again just before cutover.
    SELECT schemaname, matviewname FROM pg_matviews ORDER BY 1, 2;
    
  4. Large objects are not replicated. Their contents live in pg_largeobject, not in a replicated table; a table's oid column is copied but the object it points to is not. Anything above 0 here needs a separate plan:
    SELECT count(*) FROM pg_largeobject_metadata;
    
  5. Schema changes are frozen for the whole replication. DDL is not replicated, so a schema change on the source has to be applied to the target by hand and can break the pipeline. During the initial copy there is an extra reason: an ALTER TABLE waits for replicare's copy read of that table to finish, and the application's queries on that table queue behind the ALTER.
  6. Regular vacuum and maintenance jobs. Find out what runs and when — autovacuum settings, scheduled VACUUM, and especially heavy jobs that lock whole tables (VACUUM FULL, CLUSTER, pg_repack). Plain VACUUM / autovacuum is fine at any time; the heavy jobs should be paused during the initial copy and kept out of peak hours afterwards. See Vacuum and other maintenance.
    SHOW autovacuum;
    -- scheduled jobs, if pg_cron is installed:
    SELECT jobid, schedule, command FROM cron.job;
    -- recent vacuum activity and the tables with the most dead rows:
    SELECT relname, last_vacuum, last_autovacuum, n_dead_tup
    FROM pg_stat_user_tables ORDER BY n_dead_tup DESC LIMIT 20;
    
    Also ask about jobs outside the database (cron, CI pipelines, the hosting provider's maintenance windows).
  7. No long-open transactions. Sessions left open — especially idle in transaction — stop vacuum from cleaning up (replicare's change-tracking tables bloat) and make trigger setup wait. This should normally return nothing:
    SELECT pid, usename, state, now() - xact_start AS open_for, left(query, 80) AS last_query
    FROM pg_stat_activity
    WHERE xact_start < now() - interval '5 minutes' AND pid <> pg_backend_pid()
    ORDER BY xact_start;
    
  8. Versions. SELECT version(); on both sides. The source can be old; the target should be the same major version or newer.
  9. Privileges and disk. The source role needs the grants in Least-privilege grants, and the source needs room for the change-tracking tables to grow with the write volume from item 3 (more if a target falls behind, up to the retention cap).
  10. Sequences at cutover. Row values are copied, sequence counters are not. Plan replicare reseed-sequences (or sync_sequences: true) before the target takes writes — see Sequences & identity counters.

MySQL

Numbers match the Postgres list. Postgres items 4 (quiet windows), 9 (schema freeze), 13 (privileges and disk) and 14 (sequences) apply as written; large objects (8) don't exist in MySQL, and the version check (12) is part of (6). MySQL versions of the rest, plus two MySQL-only items:

  • (1) Total size.
    SELECT ROUND(SUM(data_length + index_length) / 1024 / 1024) AS size_mb
    FROM information_schema.TABLES WHERE table_schema = '<DB_NAME>';
    
  • (2) Row counts, sizes and storage engine per table. table_rows is InnoDB's estimate.
    SELECT table_name, engine, table_rows AS row_estimate,
           ROUND(data_length / 1024 / 1024)  AS data_mb,
           ROUND(index_length / 1024 / 1024) AS index_mb
    FROM information_schema.TABLES
    WHERE table_schema = '<DB_NAME>' AND table_type = 'BASE TABLE'
    ORDER BY data_length + index_length DESC;
    
    Every table must be InnoDB — pre-flight blocks MyISAM/MEMORY tables.
  • (3) Write volume over 24 hours. Server-wide counters (all databases on the server); run twice, 24 hours apart, and subtract.
    SHOW GLOBAL STATUS WHERE Variable_name IN
      ('Innodb_rows_inserted', 'Innodb_rows_updated', 'Innodb_rows_deleted', 'Com_commit', 'Com_rollback');
    
  • (5) Every table has a primary key or a UNIQUE key (tables without one are skipped):
    SELECT t.table_name FROM information_schema.TABLES t
    WHERE t.table_schema = '<DB_NAME>' AND t.table_type = 'BASE TABLE'
      AND NOT EXISTS (SELECT 1 FROM information_schema.TABLE_CONSTRAINTS c
                      WHERE c.table_schema = t.table_schema AND c.table_name = t.table_name
                        AND c.constraint_type IN ('PRIMARY KEY', 'UNIQUE'));
    
  • (6) Extensions don't exist in MySQL. Instead check: MySQL 5.7 or newer (SELECT VERSION();, MariaDB is not supported); on the target, local_infile is ON (SHOW GLOBAL VARIABLES LIKE 'local_infile';) and the target tables have no extra UNIQUE keys beyond the one replicare uses (pre-flight blocks them — see the MySQL page).
  • (7) Materialized views don't exist in MySQL; plain views are not replicated (they come over with the schema dump).
  • (MySQL only) Existing triggers. Tables can keep their own triggers: from MySQL 5.7.2 a table may have several triggers per timing and event, so replicare's triggers sit alongside the application's and both fire. Only a source older than 5.7.2 (in practice a best-effort 5.6) allows a single trigger per timing and event, and there a table with its own AFTER INSERT/UPDATE/DELETE trigger blocks capture. Worth listing either way, to know what else runs on each write:
    SELECT event_object_table, trigger_name, action_timing, event_manipulation
    FROM information_schema.TRIGGERS WHERE trigger_schema = '<DB_NAME>';
    
  • (9) Schema freeze also covers online schema-change tools (pt-online-schema-change, gh-ost). They rebuild a table and swap it in, and the swap leaves replicare's triggers behind on the old table; pt-online-schema-change also adds triggers of its own.
  • (10) Maintenance jobs (the MySQL counterpart of vacuum). MySQL has no VACUUM; the jobs that matter are table rebuilds — OPTIMIZE TABLE, ALTER TABLE … FORCE / ENGINE=InnoDB — run from cron or the event scheduler. A rebuild locks the table briefly at the start and end; during the initial copy it waits for replicare's copy read and the application queues behind it, so pause rebuilds during the copy.
    SELECT event_schema, event_name, interval_value, interval_field, event_definition
    FROM information_schema.EVENTS;
    
  • (11) No long-open transactions — they hold back InnoDB's purge (the history list grows and replicare's change-tracking tables bloat) and make trigger setup wait:
    SELECT trx_mysql_thread_id, trx_started, trx_state, LEFT(trx_query, 80) AS query
    FROM information_schema.INNODB_TRX ORDER BY trx_started;
    
  • (MySQL only) A Postgres database for replicare's state is still required for a MySQL→MySQL sync (see the MySQL page).

Tuning knobs (per sync, under tuning:)

KnobMeaningDefault
drain_intervalTime between streaming drain passes. Longer = more per-PK coalescing (less source load) but higher lag.1s
drain_batchMax dirty deltas applied per table per pass. With drain_interval this is the per-table streaming ceiling.1000
apply_concurrencyHow many of a component's tables apply concurrently per pass. 1 is sequential; higher parallelizes across the copy-worker pool. Raise when several tables backlog at once.1
retention.max_ageCap on the oldest unconsumed delta before a laggard target is reseeded.24h
retention.max_bytesCap on the delta table's on-disk size before reseed.off
pool.max_source_connections / pool.max_target_connectionsConnection caps; the copy worker pool is sized from these.conservative

Defaults favor low source pressure. Raise the pool caps and shorten drain_interval for throughput; lengthen drain_interval to reduce re-read load under heavy churn.

A single high-volume table that lags

Streaming applies at most drain_batch deltas per table per pass, so one table's ceiling is roughly drain_batch / drain_interval rows/s (e.g. 1000 / 1s = ~1000 rows/s). When one big table (say a high-churn events) falls behind while the rest stay at zero backlog, you've hit that ceiling. Two levers, in order:

  1. Shorten drain_interval (1s → 200ms → 100ms): more passes/sec, linearly more throughput, at the cost of more frequent re-read SELECTs on the source. Best when the "Apply batch latency (p95)" is well under the interval (interval-bound).
  2. Raise drain_batch (1000 → 5000+): more rows per pass. More efficient than a tiny interval for a genuinely large table — one COPY-to-staging + upsert of 5000 rows amortizes round-trips far better than five of 1000 — and the right lever when apply latency is near/above the interval (apply-bound). Costs a larger re-read and apply per pass (more source and target work in one shot).

Watch - rate(replicare_delta_backlog_rows[5m]) per table: positive means the backlog is shrinking (catching up).

Several tables backlogged at once

The drain_batch/drain_interval levers above are per-table ceilings on a single-threaded drain: by default a pass applies each component's tables one at a time. When several tables are backlogged together, raise apply_concurrency — it applies a component's tables concurrently, across a pool built from the copy workers (idle during streaming), so they no longer queue behind each other. It's bounded by the connection caps: the pool has one primary pair plus up to apply_concurrency-1 worker pairs, and worker pairs come from pool.max_source_connections/max_target_connections (so raise those alongside it). FK order within a component is still honored per pass (a child that lands before its parent hits a transient FK and retries), and cyclic components on MySQL — which apply as one whole-component transaction — stay sequential. This does not speed up a single hot table (that's still drain_batch/drain_interval + target capacity); it removes the cross-table serialization when many tables are hot. As always, it helps only until the target's write capacity saturates.

Source footprint (the thing to watch)

replicare writes delta and track tables to the source — a database you may not own — so bounding that footprint is a first-class concern. The headline signals (per target/table):

  • replicare_delta_backlog_rows / replicare_delta_backlog_bytes — unconsumed work.
  • replicare_delta_oldest_unconsumed_age_seconds — how stale the queue is.
  • replicare_target_up — reachability (1/0).
  • replicare_reseed_total — forced reseeds.

Retention & forced reseed. A slow or unreachable target pins the delta queue (nothing purges until every target has consumed it), so replicare caps retention by age and/or size. A target over the cap is marked needs-reseed: its consumption record is reset, the unpinned deltas are purged, and it is brought back by a fresh full copy — trading a re-copy for a protected source. This is on by default. Watch retention.cap_approaching logs (they escalate INFO→WARN→ERROR toward the cap).

The xmin-horizon caveat. A long idle-in-transaction session (or other held-back xmin) on the source prevents autovacuum from reclaiming purged delta rows, so the delta table can bloat even though replicare purges it. replicare bounds the logical backlog regardless, but physical bloat under a pinned xmin is surfaced, not fixed — hunt down the idle transaction on the source.

MySQL (InnoDB) has the direct analog. InnoDB does not shrink the tablespace on DELETE, and a long-running source transaction inflates the history list, blocking purge of rows it can still see — the same shape as the xmin horizon. The same headline metrics apply (the delta/track tables live in a dedicated replicare database on the MySQL source), retention/reseed protects the source identically, and the fix is the same: end the long-running source transaction. See the MySQL engine page.

Redis has no source footprint at all. Its CDC is capture-less — replicare writes nothing to the source (no delta/track tables, no replicare schema). So the delta-backlog metrics, retention, and xmin-style bloat simply don't apply. What you watch instead is delete-propagation latency: deletes are found by the periodic target-vs-source sweep, so a source DEL converges only after the next sweep. replicare_delete_reconciliation_lag_seconds (sweep duration) and replicare_deletes_reconciled_total are the Redis-specific signals; tune cadence with delete_sweep_interval. Upserts converge faster via the rolling reconciliation scan. See the Redis engine page.

Vacuum and other maintenance

Plain VACUUM and autovacuum never conflict with streaming. While streaming, replicare runs only ordinary statements on the source — the trigger's INSERT into its change-tracking table on every application write, SELECTs that read those tables and re-read changed rows, and batched DELETEs that clear changes every target has applied. None of them takes a lock that a plain vacuum blocks, or that blocks a plain vacuum. The same holds on the target, where replicare only inserts, updates and deletes rows.

Trigger setup is the one place a plain vacuum can get in the way — only at startup, and only when a trigger actually has to be created or replaced (a new table, or a trigger someone removed). Adding a trigger needs a lock that a running vacuum holds. replicare waits at most 2 seconds for it, backs off and retries, and after repeated failures stops with an error naming the table, rather than holding the application up. A normal restart, with every trigger already in place, takes no lock on application tables at all.

Heavy maintenance that locks a whole table — VACUUM FULL, CLUSTER, pg_repack, ALTER TABLE:

  • On an application table, it blocks replicare's reads of that table just like the application's queries; replicare adds no extra harm. During streaming replicare's reads are short, so the job waits only a moment for one to finish.
  • On one of replicare's change-tracking tables (for example a database-wide VACUUM FULL), it briefly blocks the application's writes to the matching table, because the trigger cannot record the change until the job finishes. These tables are small, so the pause is short; keep it out of peak hours.

During the initial copy two more things apply:

  • Copy reads are long: each chunk, and a table loaded in one statement (a self-referencing table whose link column cannot be emptied), keeps a read open for as long as it runs. While it runs, vacuum cannot remove rows that read might still need to see, anywhere in the database.
  • An ALTER TABLE (or VACUUM FULL / CLUSTER) on a table being copied waits for that read to finish, and the application's queries on the table queue behind it. Pause heavy maintenance and schema migrations during the initial copy, or run them with a lock_timeout.

Long-open transactions are the real vacuum problem. replicare's change-tracking tables get constant inserts and deletes and rely on autovacuum to reclaim the space. A transaction left open on the source — typically an application session idle in transaction — stops vacuum from cleaning up after it anywhere in the database, so those tables grow even though replicare keeps clearing them (see the xmin-horizon caveat). Streaming keeps working; the cost is disk. Ending the stuck session fixes it.

MySQL has no VACUUM; the equivalents are table rebuilds (OPTIMIZE TABLE, ALTER TABLE … FORCE), which behave like the heavy jobs above, and long transactions holding back InnoDB's purge, which behave like the open-transaction problem.

When a target goes down

A drain against an unreachable target fires all three channels at once: an errored trace span with backlog attributes, an ERROR target.unreachable log, and replicare_target_up=0 plus a climbing backlog series. The daemon keeps retrying each pass; nothing is lost. If the target stays down long enough to exceed the retention cap, it is reseeded on recovery.

Whole-DB vs replicated size

replicare emits two size figures for each endpoint, and they answer different questions:

  • Whole-database size — replicare_source_db_bytes / replicare_target_db_bytes (Postgres pg_database_size, MySQL sum over information_schema.tables). This is the entire database the connection points at, including everything that is not the replicated data.
  • Replicated-data size — replicare_source_replicated_bytes / replicare_target_replicated_bytes (SUM(pg_total_relation_size) — heap + indexes + TOAST — or MySQL data_length + index_length, over just the sync's selected tables). Tables absent at an endpoint contribute nothing; a sync with no relational selection reports 0.

Why the whole-DB gauges usually differ while the replicated-data gauges converge. On a healthy, caught-up single→single Postgres sync you will typically see source_db_bytes larger than target_db_bytes, even though the data is identical. That is expected, not drift:

  1. Trigger CDC writes to the source. replicare's own replicare schema lives on the source: per-table delta and track tables, plus trigger functions. Under churn those delta tables are a high-write queue that bloats before autovacuum/purge reclaims it. None of this exists on the target.
  2. Unreplicated tables count too. pg_database_size covers the whole database, so any table on the source you did not select (audit logs, staging tables, keyless tables skipped for lacking a PK) inflates the source figure but never the target's.
  3. The state store is a separate database. When the Postgres StateStore lives on the target host it is its own database (e.g. replicare_state), so pg_database_size(current_database()) for the target's data database does not count it.

The replicated-data gauges strip all of that away — same tables, same size function on both sides — so they are the right series for an apples-to-apples "has the target caught up to the source?" comparison. Watch source_replicated_bytes vs target_replicated_bytes for convergence; watch source_db_bytes (against source_replicated_bytes) for capture/bloat footprint on a source you may not own (see Source footprint).

Metrics reference

Every metric replicare exposes on /metrics (Prometheus text format; port 9090 in the demo, configurable under observability:). These names, types, and labels are the single source of truth locked in the observability contract, so a scrape of a running daemon matches this table exactly.

MetricTypeLabelsMeaning
replicare_rows_copied_totalcountersync, tableRows copied during the initial copy, counted once they are committed on the target (a retried chunk counts once). Tables linked in a loop count too; rows re-copied only to catch up with a busy source do not. Restarts from 0 with the daemon.
replicare_initial_copy_rows_targetgaugesync, tableEstimated total rows to copy (the progress denominator), from the source's own statistics (Postgres pg_class.reltuples, MySQL TABLE_ROWS; no scan). Set at bring-up for tables this run still has to copy; left unset for a table never analyzed.
replicare_throughput_rows_per_secondgaugesyncCurrent apply/copy throughput. 0 on an idle, caught-up pass.
replicare_replication_lag_secondsgaugesync, target, table, componentReplication lag — age of the oldest unconsumed delta (0 when caught up).
replicare_delta_backlog_rowsgaugesync, target, table, componentUnconsumed delta rows (the queue depth).
replicare_delta_backlog_bytesgaugesync, target, table, componentEstimated unconsumed delta bytes.
replicare_delta_oldest_unconsumed_age_secondsgaugesync, target, table, componentAge of the oldest unconsumed delta (how stale the queue is).
replicare_delta_purged_totalcountersync, tableDelta rows purged after consumption.
replicare_reseed_totalcountersync, targetForced reseeds triggered by the retention cap.
replicare_target_upgaugesync, targetTarget reachability (1=up, 0=down).
replicare_source_upgaugesyncSource reachability (1=up, 0=down).
replicare_source_db_bytesgaugesyncSource whole-database size in bytes (Postgres pg_database_size, MySQL information_schema; unset for engines that don't report it). Includes replicare's own capture schema + delta bloat and any unreplicated tables — see Whole-DB vs replicated size.
replicare_target_db_bytesgaugesync, targetTarget whole-database size in bytes (same engine support).
replicare_source_replicated_bytesgaugesyncSize in bytes of just the replicated tables on the source (SUM(pg_total_relation_size) / MySQL data_length+index_length over the selection). The apples-to-apples source↔target figure — see Whole-DB vs replicated size.
replicare_target_replicated_bytesgaugesync, targetSize in bytes of just the replicated tables on the target (same engine support).
replicare_apply_batch_secondshistogramsync, targetApply-batch duration per drain pass.
replicare_errors_totalcountersync, categoryErrors by category (e.g. drain, retention).
replicare_table_phase_infogaugesync, table, phaseTable lifecycle phase as an info gauge — value 1 on the active phase label (initial_copy/streaming); exactly one series per table. Present from bring-up, so it is the series to key dashboards on.
replicare_delete_reconciliation_lag_secondsgaugesync, target, tableRedis only — duration of the last completed delete sweep (staleness of capture-less deletes). Unset for Postgres/MySQL.
replicare_deletes_reconciled_totalcountersync, target, tableRedis only — keys DELed by the target-vs-source delete sweep. Unset for Postgres/MySQL.

The per-table backlog/lag gauges also carry a component label — the table's FK connected component (its first member) — so you can roll lag and backlog up per component (max by (component) (replicare_replication_lag_seconds)).

The headline signals to watch are replicare_source_up, replicare_target_up, replicare_delta_backlog_rows, replicare_delta_oldest_unconsumed_age_seconds, and replicare_replication_lag_seconds — together they answer "is the target healthy, and how far behind is it?" (see Source footprint and When a target goes down).

A ready-to-import Grafana dashboard over these metrics — per-table backlog/lag, catch-up, initial-copy progress, throughput, and errors, with a configurable Prometheus data source — lives at Grafana dashboard.

Traces (OpenTelemetry)

Traces are exported via OTLP/gRPC when observability.otlp_endpoint is set. The span names are a fixed registry:

SpanCovers
replicare.initial_copy.chunkone chunk of the initial copy
replicare.capture.installinstalling trigger capture on the source
replicare.delta.drainone streaming drain pass for a table/target
replicare.apply.componentapplying one FK component's pass on the target
replicare.reseeda forced/retention reseed
replicare.preflightthe start-up compatibility check

A drain/apply span against an unreachable target is recorded with error status plus the backlog/age attributes, so the stall and its blast radius are visible in the trace timeline.

Structured logs (slog)

Logs are JSON or text (logging.format), leveled (logging.level), with a stable event field you can grep/alert on:

EventLevelMeaning
capture.installed / capture.removedINFOcapture lifecycle
table.cutover_to_streamingINFOa table finished initial copy
retention.cap_approachingINFO→WARNdelta backlog nearing the retention cap (escalates with proximity)
target.unreachable / target.recoveredERROR / INFOtarget connectivity, cross-channel with the target_up gauge and the errored span
target.needs_reseedERRORa target was sacrificed to the retention cap and will re-copy
preflight.blockedERRORa sync refused to start on a blocking incompatibility

A degrading target surfaces in metrics, traces, AND logs at once — never just one channel.

The status HTTP API

The daemon serves an operator HTTP surface (bound to observability.status_addr, e.g. :8080):

  • GET /status — JSON snapshot per sync/target/table: phase (initial_copy/streaming), rows copied vs total, replication lag, and last error, read from the state store.
  • GET /healthz — a liveness probe (200 when the daemon is up), for Kubernetes/load balancers.
  • GET /metrics — the Prometheus scrape. It is served here too, and additionally on observability.metrics_addr when that is set to a different address (the conventional split).

The CLI replicare status reads the same state-store snapshot and adds live signals on top — see the next section.

Checking progress from the CLI (no Grafana required)

When you can't deploy Grafana next to replicare — e.g. the daemon runs inside the source cluster because that's the only network path to the source databases — the CLI is your dashboard. Two read-only commands answer "what is it doing, and has it caught up?" directly against the config, with no metrics stack:

replicare status config.yml            # phase, lag, LIVE row counts + delta backlog
replicare status config.yml --watch 5s # refresh in place until Ctrl-C
replicare verify config.yml            # source<->target convergence spot-check (exit 0 = converged)
  • status is live by default: each sync prints a one-line health headline (phase · last healthy pass · last data applied + rows · backlog) over a per-(table, target) grid. Two distinct time signals keep a caught-up sync legible: SEEN (time since the last healthy pass — a liveness heartbeat, small when running) and LAST_SYNC / ROWS (when data was last applied to that table and how many rows). A healthy idle sync shows a recent SEEN with a growing LAST_SYNC — normal, not lag. Live mode adds SRC_ROWS / TGT_ROWS and the per-target delta BACKLOG — 0 = caught up, N (age) = waiting, - = unknown (couldn't read it; a live: partial note says why), never conflated with 0. --no-live drops the source/target connections but keeps the state-store signals (phase, SEEN, LAST_SYNC, reseed).
  • verify is a read-only convergence check: it counts and content-fingerprints every replicated unit on both ends and reports ok / drift-count / drift-checksum per table, exiting non-zero on drift — the same count+checksum convergence logic the load-test harnesses use, folded into the shipped binary. The fingerprint is by value content, including for Redis: it hashes each key's type + canonical logical value (not just which keys exist), so a changed value on an existing key is caught, not only missing/extra keys. (Redis TTL is excluded — replicated relative, so exact remaining differs by design; stream consumer-groups/PEL aren't deep-verified yet.) A paused sync (enabled: false) is tagged [PAUSED] in the output, since convergence of a static source and target says nothing about whether the pipeline is running.

Both take --json (for scripting) and --watch <dur> (repeat on an interval). Both are read-only and install nothing, so they're safe to run against a source you may not own.

Running the CLI in the pod (no shell needed). The container image is minimal, so don't rely on a shell being present — invoke the binary directly, which kubectl exec does without any shell:

kubectl exec <pod> -- /usr/local/bin/replicare status /etc/replicare/config.yml
kubectl exec <pod> -- /usr/local/bin/replicare verify /etc/replicare/config.yml

The runtime image is Chainguard wolfi-base, which does include /bin/sh if you want an interactive session (kubectl exec -it <pod> -- /bin/sh), but the direct-exec form above is all you need for a progress check.

Running multiple pipelines in one instance

One replicare config — and one daemon process — can run several syncs across different engines at once. The single-engine rule is per sync, not per config: each sync's source and targets must share an engine, but a config may hold a Postgres sync and a Redis sync side by side, and the daemon runs them concurrently (each under its own single-active ownership lock).

A complete, annotated version of the example below is at ../examples/postgres-and-redis.yml.

state_store:                     # ALWAYS Postgres — even for a Redis-only deployment
  engine: postgres
  postgres: { host: pg-state, port: 5432, database: replicare_state, user: replicare, password: ${STATE_PW} }
sources:
  pg-src:    { engine: postgres, postgres: { host: pg-source,  port: 5432, database: app, user: replicare, password: ${PG_PW} } }
  redis-src: { engine: redis,    redis:    { mode: standalone, nodes: [redis-source:6379] } }
targets:
  pg-dst:    { engine: postgres, postgres: { host: pg-target,  port: 5432, database: app, user: replicare, password: ${PG_TGT_PW} } }
  redis-dst: { engine: redis,    redis:    { mode: standalone, nodes: [redis-target:6379] } }
syncs:
  - name: pg-pipeline
    source: pg-src
    targets: [pg-dst]
    include: ["public.*"]
  - name: redis-pipeline
    source: redis-src
    targets: [redis-dst]
    include: ["*"]

replicare run config.yml streams both. replicare status config.yml shows both (the Postgres sync with row counts + delta backlog, the Redis sync with key counts and - backlog, since Redis has no source-side queue). replicare verify config.yml checks both. Scope any command to one with --sync pg-pipeline.

State store is always Postgres (v1's only backend), even for a Redis→Redis pipeline where no data touches Postgres — a documented requirement, not a bug. See redis-statestore.md. If you have no Postgres to spare, that is the one piece of infrastructure a Redis-only deployment still needs.

Pausing one pipeline

Set enabled: false on a sync to pause just that pipeline while the others keep running — e.g. to work on the Postgres pipeline above while Redis keeps streaming:

syncs:
  - name: pg-pipeline
    source: pg-src
    targets: [pg-dst]
    enabled: false            # paused; skipped at startup, no ownership lock taken
    include: ["public.*"]
  - name: redis-pipeline      # still runs
    source: redis-src
    targets: [redis-dst]
    include: ["*"]

The pause is data-loss-free: the paused sync's source capture stays installed, so deltas keep queuing and setting enabled: true again (then restarting) drains the backlog. It is start-time only — a config change + daemon restart, not a live command. Because it's not a live toggle, a long pause lets the source-side delta tables grow and suspends the source-footprint protection (retention runs in the streaming loop, which is skipped) — so pause for maintenance/cutover, not indefinitely. Full semantics: configuration.md → Pausing a sync. On Kubernetes, flip it with helm upgrade (see kubernetes.md).

Forcing a reseed

To re-copy a target from scratch (e.g. after out-of-band divergence):

replicare reseed <config> --sync <name> --target <name>

This flags the target in the state store; the running daemon re-copies it on its next pass and resumes streaming. It signals the owning daemon rather than acting directly, honoring single-active ownership.

Sequences & identity counters (DR / promotion)

replicare replicates row values faithfully — the integer id a sequence produced lands on the target verbatim (OVERRIDING SYSTEM VALUE on Postgres, the explicit value on MySQL). What it does not replicate is the sequence object's counter — Postgres last_value, MySQL's per-table AUTO_INCREMENT — because that is schema-object state and replicare is data-only.

For a passive replica this is harmless: the target never calls nextval() for a replicated table (the applier always supplies explicit ids), so the stale counter just sits unused. It bites only when the target starts generating ids — i.e. a DR promotion / failover: the app inserts, nextval() returns a value at or near 1, and it collides with the replicated rows at id 1..N.

Before you open a promoted node to application writes, advance its counters past the data:

replicare reseed-sequences <config> --sync <name> [--target <name>] [--dry-run]

For every replicated table it sets the owned identity/serial/AUTO_INCREMENT counter to max(id)+1 on that target (--dry-run reports what it would set, writing nothing). Tables with no locally-allocated counter (UUID / natural / composite PK) are skipped.

  • Order matters: run it after replication into the node has stopped and before the app can write — a write in that gap could still grab a colliding id.
  • Why max(id)+1, not the source's last_value: the promoted node's data is the replicated rows, so max(id)+1 can't collide; any ids the source allocated beyond its max (gaps, or rows lost in the DR window) belong to rows that aren't present, so re-issuing them is harmless.
  • Privilege: this needs a grant beyond the DML set — Postgres UPDATE on the sequence, MySQL ALTER on the table (see Least-privilege grants). A missing grant fails loudly; the command reads only the target's own max(id) and never touches the source.
  • PASSIVE / one-way / DR ONLY. reseed-sequences refuses a target that is an active-active cluster member: there the same max(id)+1 would collapse every node onto the same counter and silently drop rows via last-write-wins. Active-active id allocation is a schema concern — use globally-unique keys (UUID v7 / ULID); see multi-master → active-active id allocation. (Not the same as reseed, which re-copies a target's data.)

Keeping counters warm continuously (sync_sequences)

The promotion-time reseed-sequences run above is the guarantee. If you'd rather have the counters kept advanced during streaming — so a failover has nothing sequence-related to do — opt the sync into continuous sequence syncing on the neutral config layer:

syncs:
  - name: app-to-dr
    source: primary
    targets: [dr]
    sync_sequences: true      # optional; default false

While streaming, the daemon advances each replicated table's owned counter on the target to max(id)+1 on a lazy interval (the same mechanism as reseed-sequences). Because a passive replica's applier always supplies explicit ids (OVERRIDING SYSTEM VALUE), which never advance the sequence, repeatedly setting it is idempotent and simply holds the counter ahead of the data.

  • Opt-in and off by default — unset or false is exactly the pre-existing behavior.
  • Steady-state privilege. Enabling it means the daemon's target connection needs the same DR-only grant the command needs, now continuously: Postgres UPDATE on each owned sequence, MySQL ALTER on each table. Without it the advance fails — but best-effort: the failure is logged (stream.pass_error with a sequence sync message) and retried next interval; it never aborts the streaming pass or stops data replication.
  • Still PASSIVE / one-way / DR ONLY. Same mesh refusal as the command, enforced at config load: sync_sequences: true on a sync whose target is also an active-active cluster member is rejected by validate/run up front (advancing a mesh counter is silent data loss).
  • Not a substitute for the runbook. It narrows the promotion gap but the cutover order (stop replication → open to writes) is unchanged; run reseed-sequences as the final belt-and- braces step if you want certainty at the instant of promotion.

Runtime type errors

If the target rejects a value (type/constraint), replicare halts the affected component loudly (logs + telemetry) rather than skipping or guessing. The offending row stays dirty and re-applies automatically once you fix the cause (e.g. a missing target type/extension, or a schema mismatch).

Restarts & ownership

Each sync runs under a single-active pg_advisory_lock; a second daemon pointed at the same sync stands by. State (copy progress, cursors) is checkpointed in the Postgres state store — which is always Postgres regardless of the data engine, so MySQL syncs restart and resume through the same mechanism — so a restarted or rescheduled process resumes cleanly from the last checkpoint; an ungraceful kill replays at-least-once and converges idempotently. One daemon can run Postgres, MySQL, and Redis syncs side by side (each sync stays single-engine). A Redis sync checkpoints only coarse phase (snapshot-complete-per-unit; no resumable SCAN cursor), so a crash mid-copy re-runs the whole SCAN idempotently rather than resuming a chunk — see the Redis engine page.

Connection resilience & the liveness probe

A dropped source or target connection — RDS/managed failover, an idle NAT/LB reap, a brief network blip — recovers on its own, no restart needed. All connections set a bounded dial timeout and TCP keepalive so a dead peer surfaces as an error quickly instead of hanging, and after any failed drain pass the daemon health-checks both endpoints and reconnects whichever is down (Postgres re-dials its single connection and re-applies session GUCs; MySQL and Redis rebuild their pooled clients). A reconnect that can't succeed yet (the endpoint is genuinely down) is retried each pass, so the daemon never crash-loops on an outage — it heals when the endpoint returns. Recovery is visible as a stream.reconnected event in /status and the logs.

/healthz reports streaming liveness as a backstop for a true wedge (e.g. a hung query no timeout caught). It is deliberately not target reachability: a loop that keeps cycling — including one correctly retrying against a down target — stays healthy (that state shows up in replicare_target_up and the backlog metrics instead). The probe fails only when a streaming loop completes no pass within observability.stall_timeout (default 2m), so Kubernetes restarts a pod whose loop has stopped iterating. Set stall_timeout above your slowest expected drain pass (a large drain_batch on a slow target lengthens a pass); a negative value disables the check. Liveness tracking starts only once a sync reaches streaming, so a long initial copy is never mistaken for a wedge.

Least-privilege grants

See ../deploy/grants-source.sql and ../deploy/grants-target.sql. Note the documented asymmetry: installing capture needs only the TRIGGER privilege, but capture remove (dropping the trigger) requires table ownership. For the true zero-database-grant path, pre-create the replicare capture schema (and, if the state store lives there, the replicare_state schema) owned by the daemon role — the migration runner then issues no CREATE SCHEMA and needs no CREATE-on-database privilege. See the Postgres engine page.

For MySQL, use ../deploy/grants-source-mysql.sql and ../deploy/grants-target-mysql.sql. MySQL has no such asymmetry — the TRIGGER privilege covers both CREATE TRIGGER and DROP TRIGGER, so the daemon role can fully uninstall capture on its own.

For Redis, use the ACL presets ../deploy/acl-source-redis.txt and ../deploy/acl-target-redis.txt. The source role is read-only (+ping +info +scan +type +dump +pttl +exists +memory|usage); the target role needs +ping +info +restore +del +scan +exists +module|list. Both roles need +info (pre-flight reads INFO server), and in cluster mode both also need +cluster|shards +cluster|slots and the same user granted on every master. Watch the foot-gun: RESTORE is @dangerous and must be granted explicitly with +restore, or every apply fails loud. See the Redis engine page.

reseed-sequences needs one extra target grant beyond the DML set, and only on the node you run it against (a DR promotion step): Postgres UPDATE on each replicated table's owned sequence (or ownership of it); MySQL ALTER on each replicated table. It reads only the target's own max(id) — no source privilege. Grant it as part of the failover runbook, not the steady-state role.

Load & convergence testing

Three manual load-and-verify harnesses drive replicare against broad, high-volume, realistic data and assert the target converges with the source under continuous churn. They are developer/test tools, run from a checkout (go run ./test/…), not part of the shipped binary:

  • Postgres — test/loadgen. Seeds a rich 10-table schema chosen to exercise the hard paths (FK components, composite/UUID/text PKs, PK-changing SKU renames, GENERATED columns, a keyless table it must skip; optional FK cycles under --cyclic), churns random inserts/updates/deletes, and verifys per-table row-count + ordered content checksum.
  • MySQL — test/loadgen-mysql. The MySQL sibling of the Postgres harness: the same 10-table schema in MySQL types (AUTO_INCREMENT, JSON, composite VARCHAR PK, GENERATED … STORED, a keyless table), generated with MySQL-5.7-safe SQL (no generate_series/CTE/window functions), churned the same way, and verifyd with a cross-version, order-independent content checksum (BIT_XOR-folded per-row hashes).
  • Redis — test/loadgen-redis. Seeds every value type (string/hash/list/set/zset/stream + big/TTL'd keys) plus a source-only lgskip:* cohort the sync must exclude, churns value mutations, RENAMEs, and heavy DELs (the delete-reconciliation stress), and verifys a version-independent, type-aware content hash + TTL presence. Standalone, sentinel, and cluster.

All three support active-active load testing via --node-id (each writer node claims a disjoint key slice so a mesh converges to the clean union); --node-id 0 is the default single-writer (active-passive) behaviour. Each README documents the full end-to-end rig (two databases, a Postgres state store, a config, the daemon, and a churn-and-verify loop) and the active-active workflow. task loadgen:*, task loadgen-mysql:*, and task loadgen-redis:* are the shortcuts. These are distinct from the automated Go suite (task test:integration).

Deploying on Kubernetes (Helm)

replicare ships a Helm chart — replicare-controller — that runs the daemon (replicare run) on Kubernetes. This page is the deployment guide; the chart's own README is the exhaustive values reference.

Two GHCR packages

Each vX.Y.Z release publishes two independent OCI artifacts to GHCR, stamped with the same version:

ArtifactPackagePull
Daemon imageghcr.io/rudimk/replicaredocker pull ghcr.io/rudimk/replicare:0.1.1
Helm chartghcr.io/rudimk/replicare-controllerhelm pull oci://ghcr.io/rudimk/replicare-controller --version 0.1.1

The chart's appVersion defaults image.tag to the matching daemon image, so a versioned chart install pulls a matched chart + image pair unless you override image.tag.

Install

# From the published OCI chart (recommended) — pin the version:
helm install my-replicare oci://ghcr.io/rudimk/replicare-controller \
  --version 0.1.1 -f my-values.yaml

# ...or from a checkout of the repo:
helm install my-replicare deploy/helm/replicare-controller -f my-values.yaml

my-values.yaml is where you put your config and point at your image and secrets. There is no separate config file to manage — the chart renders your config value into a ConfigMap at /etc/replicare/config.yml and runs the daemon against it.

Configuring the daemon

config is a structured YAML object (the ergonomic default): Helm merges and overrides it across -f files and --set, so --set config.logging.level=debug changes just that field and leaves the rest intact. It is the exact replicare config — any engines, any number of sources, targets, and syncs.

image:
  repository: <your-registry>/replicare
  tag: <version>

config:
  logging: { level: info, format: json }
  observability: { status_addr: ":8080", metrics_addr: ":9090" }

  # replicare's own state — Postgres, always OUTSIDE this chart (see below).
  state_store:
    engine: postgres
    postgres: { host: state.rds.amazonaws.com, port: 5432, database: replicare_state,
                user: replicare, password: ${STATE_PW}, sslmode: require }

  sources:
    cache:
      engine: redis
      redis: { host: my-redis.default.svc.cluster.local, port: 6379, password: ${SRC_PW:-} }
  targets:
    replica:
      engine: redis
      redis: { host: master.my-ec.cache.amazonaws.com, port: 6379, tls: require,
               user: ${EC_USER}, password: ${EC_PW} }
  syncs:
    - { name: cache-to-replica, source: cache, targets: [replica], include: ["*"],
        tuning: { drain_interval: 1s, drain_batch: 1000 } }

secret:
  existingSecret: my-replicare-secrets   # provides STATE_PW / SRC_PW / EC_USER / EC_PW

Two caveats for the structured form:

  • Quote any all-digit scalar you inline (e.g. a numeric-only password) so it decodes as a string, not an int. ${VAR} placeholders need no quoting here — they render in block style.
  • Prefer a verbatim string for byte-exact control: set config to a block-scalar string and the chart emits it unchanged (Helm can't merge a string, but nothing is re-serialized either).

Secrets

Keep passwords and tokens out of config with ${VAR} placeholders — replicare expands env vars in any field. Supply them via a Secret, injected into the pod with envFrom:

  • secret.existingSecret — reference a Secret you manage (preferred for production; the chart creates none, so secrets never live in Helm state).
  • secret.env — a map the chart renders into a Secret (fine for dev).

Endpoints & monitoring

The pod serves the operator HTTP surface (see operations):

  • /healthz — liveness/readiness (both probes use it), on the status port.
  • /status — per-sync phase, progress, lag (JSON); what replicare status renders.
  • /metrics — Prometheus scrape.

The Service exposes the status and metrics ports. Set serviceMonitor.enabled: true to scrape /metrics via the Prometheus Operator.

Port alignment: service.ports.status / .metrics must match the addresses your config binds (observability.status_addr, metrics_addr). The defaults line up at :8080 / :9090; if you change one, change both.

Checking progress without Prometheus/Grafana

When you can't run a metrics stack next to the pod — e.g. the daemon runs inside the source cluster for network reasons — exec the CLI directly against the mounted config. The image is minimal, so target the binary rather than a shell (kubectl exec runs a command without needing /bin/sh):

kubectl exec deploy/replicare -- /usr/local/bin/replicare status /etc/replicare/config.yml   # live counts + delta backlog
kubectl exec deploy/replicare -- /usr/local/bin/replicare verify /etc/replicare/config.yml   # source<->target convergence (exit 0 = converged)

status is live by default (source/target row counts + per-target delta backlog); verify is a read-only convergence spot-check. Both accept --json, --sync <name>, and --watch <dur>. See the CLI reference and Checking progress from the CLI.

The runtime image is Chainguard wolfi-base, which also ships /bin/sh if you want an interactive session for ad-hoc debugging (kubectl exec -it deploy/replicare -- /bin/sh) — but the direct-exec form above needs no shell.

Single-active, rollouts & state

  • Keep replicaCount: 1. replicare is single-active per sync — a second daemon just stands by on the state store's ownership lock, so extra replicas are wasted and never run two writers. The chart uses a Recreate rollout so the old daemon fully exits (releasing its lock) before the new one starts.
  • Config changes roll the pod automatically: a checksum annotation over the rendered ConfigMap means helm upgrade after editing config restarts the daemon cleanly.
  • Pause one pipeline without touching the others by setting enabled: false on that sync (a per-pipeline switch; omit it or true to run). Because it's a config change, the upgrade's pod rollout applies it — it is start-time, not a live command. Prefer editing the sync's block in your values file by name, so you pause the pipeline you mean to:
    config:
      syncs:
        - name: pg-pipeline
          # ...
          enabled: false        # pauses THIS pipeline only
        - name: redis-pipeline
          # ...                  # (no enabled / true) keeps running
    
    then helm upgrade -f values.yaml. You can use --set, but it addresses syncs by array index, not name — config.syncs[0] is the first sync in your list, which may not be the one you want:
    # WRONG target if redis-pipeline is the 2nd sync — this pauses syncs[0] (e.g. pg-pipeline):
    helm upgrade ... --set-string 'config.syncs[0].enabled=false'
    # Pause the second sync instead:
    helm upgrade ... --set-string 'config.syncs[1].enabled=false'
    
    Verify which sync actually paused in the pod logs: sync disabled (paused); skipping sync=<name> for the paused one, and sync streaming sync=<name> for each still running. replicare status also shows [PAUSED] per sync. The paused sync's source capture stays installed, so re-enabling it drains the queued backlog with no data loss. Full semantics and the long-pause caveat (source-side delta growth): configuration.md → Pausing a sync.
  • The state store lives outside the chart, always. Every sync needs a Postgres state store (v1's only backend) — including Redis→Redis and MySQL→MySQL. Point state_store at your own managed Postgres (e.g. RDS). The chart never bundles it; it is control-plane bookkeeping, separate from your data.

Security

The container runs non-root (uid 65534), with a read-only root filesystem and all capabilities dropped (a writable emptyDir is mounted at /tmp). Grant the daemon's database/Redis roles the least privilege documented in deploy/ — grants-*.sql for Postgres/MySQL and acl-*-redis.txt for Redis (note the explicit +restore on Redis targets). Mount a CA bundle via extraVolumes/extraVolumeMounts if you use tls: verify-full.

Worked example: in-cluster Redis → ElastiCache

A common shape is replicating a self-managed Redis running in EKS to ElastiCache. Run replicare as a pod in the same cluster — with the AWS VPC CNI the pod gets a VPC IP, so it can reach both the in-cluster Redis (via its Service DNS) and ElastiCache (only reachable inside the VPC; open its security group to the node/pod SG). Watch-outs specific to that move:

  • Version direction: RESTORE rejects a newer-than-target payload, so the ElastiCache engine version must be ≥ the source version, or pre-flight blocks.
  • RESTORE must be permitted on the target (it is @dangerous; grant +restore via RBAC). Run replicare validate against the real endpoint first — some managed tiers restrict commands.
  • TLS + AUTH: set tls: require (or verify-full) and the RBAC user/password on the target block. Redis tls defaults to disable, so set it explicitly.
  • Cluster-mode ElastiCache: use mode: cluster seeded with the configuration endpoint (DB 0 only); cluster-mode-disabled connects as standalone to the primary endpoint.
  • Deletes lag the sweep interval, and big keys briefly block the source on DUMP — see the Redis engine page and troubleshooting.

Validate before applying

helm lint deploy/helm/replicare-controller
helm template my-replicare deploy/helm/replicare-controller -f my-values.yaml \
  | kubectl apply --dry-run=client -f -

replicare-controller Helm chart

Deploys the replicare daemon (replicare run) on Kubernetes: one replica that brings up and streams every sync in your config, exposing the /status, /healthz, and /metrics endpoints.

The chart is published as an OCI artifact at ghcr.io/rudimk/replicare-controller — a package distinct from the daemon image (ghcr.io/rudimk/replicare). Both are stamped with the same version on each vX.Y.Z release.

Install

# From the published OCI chart (recommended) — pin the version:
helm install my-replicare oci://ghcr.io/rudimk/replicare-controller \
  --version 0.1.1 \
  -f my-values.yaml

# ...or from a checkout of the repo:
helm install my-replicare deploy/helm/replicare-controller \
  -f my-values.yaml

The chart's appVersion defaults image.tag to the matching daemon image (ghcr.io/rudimk/replicare:<version>), so a versioned OCI install pulls a matched chart + image pair. Override image.tag to decouple them.

How it's configured

replicare is driven by one config file. This chart renders the config value into a ConfigMap mounted at /etc/replicare/config.yml — you get replicare's full expressiveness (any engines, any number of sources/targets/syncs). See the configuration reference.

config is a structured YAML object by default, so Helm merges and overrides it the normal way — layer -f files or --set config.logging.level=debug and only that field changes:

config:
  logging: { level: info, format: json }
  observability: { status_addr: ":8080", metrics_addr: ":9090" }
  state_store:
    engine: postgres           # always OUTSIDE this chart — point at your own Postgres
    postgres: { host: state.rds, port: 5432, database: replicare_state, user: replicare, password: ${STATE_PW}, sslmode: require }
  sources:
    cache: { engine: redis, redis: { host: my-redis, port: 6379, password: ${SRC_PW:-} } }
  targets:
    ec: { engine: redis, redis: { host: my-elasticache, port: 6379, tls: require, user: ${EC_USER}, password: ${EC_PW} } }
  syncs:
    - { name: cache-to-ec, source: cache, targets: [ec], include: ["*"], tuning: { drain_interval: 1s } }

Two small caveats for the structured form: quote any all-digit scalar you inline (e.g. a numeric-only password) so it decodes as a string, not an int; ${VAR} placeholders need no quoting (they render in block style). If you'd rather hand the daemon a byte-exact config, set config to a block-scalar string instead and it is emitted verbatim.

Pausing a pipeline. Each sync (and cluster) takes an optional enabled: true|false — set it false to pause just that pipeline; the others keep running. Because it's a config change, the helm upgrade pod rollout applies it (it's start-time, not live). Prefer editing the sync's block by name in your values so you pause the right one:

config:
  syncs:
    - name: pg-pipeline
      enabled: false      # pauses THIS pipeline only
    - name: redis-pipeline
      # ...               # keeps running

--set works too but addresses syncs by array index, not name — config.syncs[0] is the first sync in your list, which may not be the one you intend (--set-string 'config.syncs[1].enabled=false' for the second, etc.). After the rollout, confirm which sync paused in the pod logs (sync disabled (paused); skipping sync=<name> vs sync streaming sync=<name>) or via replicare status ([PAUSED]). The paused sync's source capture stays installed, so re-enabling it drains the queued backlog with no data loss (a long pause grows the source-side delta tables). Full semantics: configuration.md → Pausing a sync.

Secrets stay out of config using ${VAR} placeholders (replicare expands env vars in any field), supplied via a Secret:

secret:
  # Option A — let the chart create a Secret (fine for dev):
  env: { EC_USER: replicare, EC_PW: the-token }
  # Option B — reference a Secret you manage (preferred for prod; `env` is ignored):
  existingSecret: my-replicare-secrets

The pod loads the Secret via envFrom, so every key becomes an env var the config can reference.

Key values

ValueDefaultNotes
image.repository / image.tagghcr.io/rudimk/replicare / chart appVersionthe daemon image (a separate GHCR package from this chart)
replicaCount1keep at 1 — single-active per sync; more just stand by
configa Redis→Redis sampleyour full replicare config as a structured map (mergeable), or a string for a verbatim config
secret.existingSecret""reference an existing Secret for ${VAR} values
secret.env{}chart-managed Secret (dev)
service.ports.status / .metrics8080 / 9090must match observability.status_addr/metrics_addr in config
serviceMonitor.enabledfalsescrape /metrics via the Prometheus Operator
resources, nodeSelector, tolerations, affinity—standard scheduling knobs
extraVolumes / extraVolumeMounts[]e.g. a CA bundle for tls: verify-full

Notes & gotchas

  • Single active. The rollout is Recreate so the old daemon exits (releasing its ownership lock) before the new one starts — never two writers. Scaling above 1 replica is unsupported.
  • Port alignment. The chart's Service/probe ports must match the addresses your config binds (observability.status_addr, metrics_addr). Defaults line up at :8080/:9090.
  • Scraping metrics. With the Prometheus Operator, set serviceMonitor.enabled=true. Without it (e.g. an OpenTelemetry Collector or plain Prometheus), use the scrape job in the metrics-scraping guide.
  • Grafana dashboard. Once /metrics is scraped (via serviceMonitor.enabled or your own scrape job), import deploy/grafana/replicare-dashboard.json — it has a configurable Prometheus data source and covers per-table backlog/lag, catch-up, initial-copy progress, throughput, and errors.
  • State store. Every sync needs a Postgres state store (v1's only backend), including Redis→Redis and MySQL→MySQL. Point state_store at a reachable Postgres.
  • Config changes roll the pod automatically (a checksum annotation), so helm upgrade after editing config restarts the daemon cleanly.
  • Least privilege. The container runs non-root (uid 65534), read-only root FS, all capabilities dropped. Grant the daemon roles per deploy/ (grants-*.sql, acl-*-redis.txt).

Validate before applying

helm lint deploy/helm/replicare-controller
helm template my-replicare deploy/helm/replicare-controller -f my-values.yaml | kubectl apply --dry-run=client -f -

Grafana dashboard for replicare

replicare-dashboard.json is a ready-to-import Grafana dashboard for replicare's Prometheus metrics. It has a configurable data source — you pick which Prometheus data source it queries at import time (and can switch it later from the dashboard's Data source dropdown), so it is not tied to any one Prometheus.

It assumes replicare's /metrics endpoint is being scraped into Prometheus. See deploy/helm/replicare-controller (the serviceMonitor option, or a raw scrape job) for how to wire that up.

Import

Grafana UI

  1. Dashboards → New → Import.
  2. Upload replicare-dashboard.json (or paste its contents).
  3. When prompted, select your Prometheus data source for the Data source variable, then Import.

Provisioning (file-based)

Drop the JSON into a provisioned dashboards folder and point a provider at it, e.g.:

# /etc/grafana/provisioning/dashboards/replicare.yaml
apiVersion: 1
providers:
  - name: replicare
    type: file
    options:
      path: /var/lib/grafana/dashboards/replicare

The dashboard's data source is a template variable (${datasource}), so provisioning does not hard-code one; Grafana resolves it from the current value (default: your default Prometheus). To pin it, set the datasource variable's current value or pass DS_PROMETHEUS via the import API.

Variables

VariableMeaning
Data source (datasource)Which Prometheus data source to query. Choose at import; change any time.
Sync (sync)Filter to one or more syncs (label_values(replicare_table_phase_info, sync) — a metric present from the moment a sync starts, so the filter works during the initial copy too). Defaults to All.
Target (target)Filter to one or more targets within the selected syncs. Defaults to All.
Table (table)Filter the per-table panels to one or more tables (from replicare_table_phase_info, so every replicated table is listed from the start). Defaults to All.
Component (component)Filter the per-component panels to one or more FK components. Defaults to All.

Panels

Organized into sections (Grafana rows):

Initial copy (top, since it is the first thing a new sync does):

PanelQueryReading
Initial copy (overall)sum(replicare_rows_copied_total) / sum(replicare_initial_copy_rows_target)Percent of the rows this run has to copy. The target is the source database's own row estimate (statistics, no scan), so the end can read a little over or under 100%
Tables still copying / Tables streamingcount(replicare_table_phase_info{phase=…})How many tables are in each phase; copying reaches 0 when the whole sync is streaming
Copy rate (rows/s)rate(replicare_rows_copied_total[2m])Current copy speed
Progress % / Rows copied vs target (per table)the same, by (table)Which tables are done and which are still moving

Rows copied count only rows that have been committed on the target, so a retried chunk is never counted twice. A table that loads in one statement (a self-referencing table whose link column cannot be emptied) jumps from 0 to done in one step. replicare_rows_copied_total restarts from 0 when the daemon restarts, and a restart re-copies whatever was not finished, so the percentage restarts with it.

Source / Target — the controller checks both endpoints and emits their reachability and two size figures — the whole database and just the replicated tables:

PanelQuery (source metric)Reading
Source up / Targets upmin(replicare_source_up) / min(replicare_target_up)UP (green) / DOWN (red) — the controller's health-check of that endpoint
Source / Target DB sizereplicare_source_db_bytes / replicare_target_db_bytesWhole-database on-disk size (Postgres/MySQL). The source usually runs larger — it carries replicare's capture schema + delta bloat and any unreplicated tables
Source / Target replicated sizereplicare_source_replicated_bytes / replicare_target_replicated_bytesSize of just the replicated tables — the apples-to-apples comparison; on a caught-up sync the two converge even when the whole-DB sizes differ

Why two size figures? The whole-DB gauge counts everything in the database — including replicare's own replicare capture schema on the source and any tables you didn't select — so source > target is normal, not drift. The replicated-size gauge sums only the selected tables with the same function on both sides, so it's the series to watch for convergence. See Whole-DB vs replicated size.

Replication lag — the headline lag section:

PanelQueryReading
Max replication lag (SLO)max(replicare_replication_lag_seconds)Worst lag; green < 30s, amber < 5m, red beyond
Catch-up ETAsum(delta_backlog_rows) / clamp_min(sum(throughput),1)Estimated seconds to converge at the current apply rate
Inflow vs drainthroughput vs throughput + rate(sum(delta_backlog_rows)[5m])Inflow above drain ⇒ the backlog is growing
Replication lag / Total backlogreplicare_replication_lag_seconds, sum(replicare_delta_backlog_rows)Per-table lag; total rows still to apply

Per FK component — rolled up by the component label:

PanelQueryReading
Replication lag (per component)max by (component) (replicare_replication_lag_seconds)Which FK component is behind
Delta backlog (per component)sum by (component) (replicare_delta_backlog_rows)Queue depth per component

Backlog detail (per table), Throughput & apply latency, Errors & maintenance — per-table backlog rows/bytes/oldest-age; throughput and apply p95; error rate, purge rate, and reseeds. (Metrics as named.)

The Source / Target health and size panels fill in as soon as a sync starts. The lag, backlog and throughput panels fill in once tables reach streaming — during an initial copy they are expected to read No data.

Notes

  • Metrics refresh once per drain pass (your drain_interval, default 1s), so series update at that cadence — the dashboard auto-refreshes every 30s by default.
  • The backlog and lag series are the catch-up view: after a churn burst they rise, then fall back to baseline as streaming drains the queue. A backlog that climbs while Targets up reads DOWN is the intended "target is unreachable and falling behind" signal.
  • The dashboard uid is replicare-overview; re-importing the same file updates the existing dashboard in place rather than creating a copy.

Scraping replicare metrics (OpenTelemetry Collector / Prometheus)

replicare exposes Prometheus metrics on an HTTP endpoint:

DefaultSet by
Path/metricsfixed
Port9090, named metricschart service.ports.metrics / config observability.metrics_addr
Metric namesreplicare_*the observability contract (see docs/operations.md)

The chart's Service/Pod also exposes /status and /healthz on the status port (8080), so a scrape must target the metrics port specifically — not just "the Service".

There are two ways to get these metrics into your monitoring stack.

Option A — Prometheus Operator (simplest, if you run it)

The Helm chart ships a ServiceMonitor. Just enable it:

helm upgrade --install replicare oci://ghcr.io/rudimk/replicare-controller \
  -f your-values.yaml \
  --set serviceMonitor.enabled=true

That wires the operator to scrape the metrics port for you; nothing below is needed.

Option B — Prometheus scrape job (OTel Collector prometheus receiver, or plain Prometheus)

If you scrape with an OpenTelemetry Collector (its prometheus receiver embeds a Prometheus scrape config) or a plain Prometheus, add the scrape job below. It uses Kubernetes endpointslice service discovery and keeps only replicare's metrics endpoint, selected by the chart's stable label app.kubernetes.io/name=replicare-controller (release-independent — works regardless of your Helm release name).

- job_name: replicare
  honor_labels: true
  scrape_interval: 30s
  scrape_timeout: 10s
  kubernetes_sd_configs:
    - role: endpointslice
      # Optional: scope discovery to replicare's namespace to cut SD churn.
      # namespaces:
      #   names: [replicare]
  relabel_configs:
    # Keep only endpoints of the replicare Service (stable chart label, any release name).
    - action: keep
      regex: replicare-controller
      source_labels:
        - __meta_kubernetes_endpointslice_label_app_kubernetes_io_name
    # Keep only the metrics port (the Service also exposes status:8080).
    - action: keep
      regex: metrics
      source_labels:
        - __meta_kubernetes_endpointslice_port_name
    # Nice-to-have target labels.
    - action: replace
      source_labels: [__meta_kubernetes_namespace]
      target_label: namespace
    - action: replace
      source_labels: [__meta_kubernetes_pod_name]
      target_label: pod
  metric_relabel_configs:
    # Optional: drop anything that isn't a replicare metric (e.g. go_/process_ noise).
    - action: keep
      regex: replicare_.*
      source_labels:
        - __name__

Where it goes in an OpenTelemetry Collector

The job slots into the prometheus receiver's embedded config, then feeds your normal pipeline:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: replicare
          # ... the job from above ...

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [prometheusremotewrite]   # or otlp, etc. — whatever you already use

For a plain Prometheus, the same block goes under scrape_configs: in prometheus.yml unchanged.

Notes

  • Port name over number. The job keys on the port name (metrics), not 9090, so it keeps working if you change service.ports.metrics. If you match on the number instead, use __meta_kubernetes_endpointslice_port with your configured value.
  • No scrape annotations. The chart sets no prometheus.io/scrape annotations by default, so annotation-based discovery won't find replicare — use the label-based relabeling above (or add your own annotations via podAnnotations if your collector discovers by annotation).
  • RBAC. The collector's ServiceAccount needs list/watch on endpointslices (and pods if you set the pod target label) in the target namespace(s). Most Kubernetes Collector deployments already grant this.
  • Scrape interval. replicare refreshes its gauges once per drain pass (tuning.drain_interval, default 1s), so a 15–30s scrape interval captures the movement without over-sampling.
  • What to build on top. Once these metrics land in Prometheus, import the Grafana dashboard — it has a configurable data source and covers per-table backlog/lag, catch-up, initial-copy progress, throughput, and errors.

Troubleshooting

The config won't load (exit code 2)

Before any pre-flight, the config is parsed and structurally validated. These are exit-2 failures with a message naming the problem — the earliest thing a new user hits:

  • ${VAR} is unset and has no default → set the env var, or give a fallback (${VAR:-default}). Env expansion applies to any field, not just secrets.
  • unknown YAML field → strict decoding rejects typos/unknown keys; fix the field name against the configuration reference.
  • unknown engine, or a missing engine block → engine: postgres|mysql|redis must match a present typed block (postgres: / mysql: / redis:).
  • a sync references an undefined source/target, or crosses engines → a sync is single-engine; its source and all targets must share one engine.

replicare validate <config> reports these without connecting. Exit 2 is "config/usage"; exit 1 is a clean pre-flight BLOCK or a connection failure (see CLI reference).

validate reports BLOCK findings / the daemon refuses to start

Pre-flight blocks on an incompatible or missing target type — e.g. a source column whose type doesn't exist on the target, a missing enum/extension, or a narrowing that would lose data. replicare never coerces data to fit, so it stops before starting.

Fix: create the missing type/extension on the target, or widen the target column, so the pair is compatible. Re-run validate. Risky-but-allowed narrowings (e.g. int8→int4) are warnings, not blocks — heed them.

A table isn't being replicated

  • No primary key / usable unique key → the table is skipped with a warning (it can't be captured). Add a PK/unique key, or exclude it deliberately.
  • Not matched by the selection → check the sync's include/exclude globs (schema.table). replicare validate lists the replicable and skipped tables.

capture remove fails to drop a trigger

Dropping a trigger requires table ownership (Postgres allows CREATE TRIGGER with just the TRIGGER privilege, but not DROP TRIGGER). Either run remove as the table owner, or have the owner drop the replicare_trg_* trigger. replicare's own delta/track tables and functions are dropped fine by the daemon role.

The source's disk is filling up (delta/replicare schema growth)

The source-side delta tables are a high-churn queue. If a target is down or slow, deltas can't purge and the queue grows. replicare bounds this with retention: past retention.max_age/max_bytes, the laggard target is reseeded and the over-cap deltas are purged. Check:

replicare status config.yml            # look for needs-reseed / stale cursors
curl :9090/metrics | grep -E 'delta_backlog|oldest_unconsumed|target_up'

If the backlog is bounded (retention firing) but the physical delta table still won't shrink, an idle transaction is pinning the source's xmin horizon so autovacuum can't reclaim. Find and end it:

SELECT pid, state, xact_start, query FROM pg_stat_activity
WHERE state = 'idle in transaction' ORDER BY xact_start;

replicare bounds logical growth regardless, but only ending the idle transaction lets the source reclaim the space.

A target is unreachable

A drain against a down target fails and surfaces across all channels (replicare_target_up=0, an ERROR target.unreachable log, an errored trace span). Nothing is lost — the daemon retries each pass and converges when the target returns. If it stays down past the retention cap, it's reseeded on recovery. The source is still protected meanwhile (retention runs even while the target is down).

A sync halted with a type/constraint error

If the target rejects a value at apply time, replicare halts that component loudly rather than skipping or guessing (a hard product promise — it never mangles data). The offending row stays dirty and re-applies automatically once you fix the cause: usually a missing target type/extension or a schema mismatch. Check the ERROR logs / replicare_errors_total for the specific failure.

"sync owned by another daemon; skipping"

Each sync runs under a single-active lock (v1). Another process already owns it. This is expected if you run two daemons over the same config — only one drives each sync; the other stands by. If it's stale, ensure the previous daemon actually exited (the advisory lock releases on disconnect).

Traces aren't showing up

Traces export only when observability.otlp_endpoint is set, and to an OTLP/gRPC collector (port 4317 by convention) over plaintext — point it at a local collector or sidecar. Metrics (/metrics) and the status API work independently of tracing.

MySQL-specific issues

  • The daemon refuses to connect to a MariaDB server. MariaDB is not supported in v1; it is detected and refused at connect. Use a real MySQL (5.7+) source and target.
  • validate blocks on a non-InnoDB table. MyISAM/MEMORY tables make BEGIN/COMMIT a no-op, breaking per-component atomicity (target) and the atomic delta consume (source). Convert the table to InnoDB.
  • validate blocks on a target table with a secondary UNIQUE key. MySQL INSERT ... ON DUPLICATE KEY UPDATE has no conflict target, so a secondary unique could silently rewrite the wrong row — pre-flight blocks it rather than risk corruption. Remove the extra unique key or replicate a table without one.
  • A copy halts with "target requires local_infile=ON". The copy/apply transport is LOAD DATA LOCAL INFILE, which needs local_infile=ON on the target server (a system variable, not a grant). replicare probes it at connect and halts loud when it is off (v1 has no INSERT fallback). Enable it — SET GLOBAL local_infile=1 or local-infile=ON in my.cnf — and set local_infile: true in the target block.
  • The source's replicare database keeps growing under a long transaction. See Source footprint — InnoDB's history list is the xmin-horizon analog; end the long-running source transaction.

See the MySQL engine page for the full requirements and the two operational wrinkles.

Redis-specific issues

  • validate blocks: newer source → older target (RDB version). RESTORE rejects a payload whose RDB version exceeds the target's max, so replicare blocks a newer→older pair up front (the error mentions the version gap). Upgrade the target to at least the source's version, or replicate the other direction.
  • validate blocks: a source module is missing on the target. Module-defined types (RedisJSON, RedisBloom, …) produce opaque DUMP payloads only the identical module can RESTORE. Load the module on the target, or exclude those keys.
  • Every apply fails loud with a RESTORE permission error. RESTORE is an @dangerous ACL command and is not in the usual read-write presets — grant it explicitly with +restore on the target role (see ../deploy/acl-target-redis.txt).
  • A copy refuses on a big key. Above big_key_refuse_bytes replicare blocks loud rather than doing torn incremental transport (which would be unfaithful). Raise the cap if the key genuinely must replicate, or exclude it. Below the cap but above big_key_warn_bytes you'll see a big-key WARN but the copy proceeds.
  • A deleted key still exists on the target. Deletes are not captured — they're found by the periodic target-vs-source sweep, so a source DEL converges only after the next sweep completes (bounded by delete_sweep_interval). Watch replicare_delete_reconciliation_lag_seconds; shorten delete_sweep_interval for faster delete propagation (at more source/target scan load). Upserts converge faster via the rolling reconciliation scan.
  • Keyspace notifications missed a change. Notifications are a lossy accelerator only (fire-and-forget, lost on disconnect, node-local in cluster). A missed event is always caught by the next reconciliation/sweep; on a subscription gap replicare forces a full pass. Never rely on them for correctness — they only cut latency.
  • "Redis→Redis but it wants a Postgres?" Yes — replicare's own state store is Postgres regardless of the data engine (v1's only StateStore backend). Point it at any small Postgres; it's just control-plane bookkeeping, and Redis writes nothing to its own source. See the Redis engine page.
  • The daemon refuses to connect to Dragonfly. Dragonfly is blocked in v1 (its RDB DUMP/RESTORE compatibility is unverified). Valkey and (best-effort) KeyDB are accepted; a non-Redis-family server is refused at connect.

See the Redis engine page for the full requirements, the two pre-flight gates, and the operational wrinkles.

Getting more detail

Set logging.level: debug and logging.format: text for verbose, readable output while diagnosing. replicare status --json gives the full machine-readable state.

Verifying data integrity — Postgres

Standalone psql queries to confirm a Postgres source and target are converged, for when you can't reach the replicare pod to run replicare status / replicare verify. Run each block on both databases and compare the output.

These mirror what verify does internally: columns are matched by name, GENERATED … STORED columns are excluded (the target regenerates them), and the content hash is order-independent (physical row order, index order, and text collation don't affect it).

The authoritative check is still replicare verify <config>. This is the no-pod fallback, and its results should agree with verify.


0. Pin session GUCs (run on both, every session)

Essential — replicare canonicalizes these on every connection. Row text must render identically on both sides (an old source and a newer target) or you get false mismatches on dates, floats, and bytea. Run this first in every session before hashing.

SET DateStyle = 'ISO, YMD';
SET TimeZone = 'UTC';
SET extra_float_digits = 3;
SET IntervalStyle = 'postgres';
SET bytea_output = 'hex';
SET client_encoding = 'UTF8';

1. Row counts per table (run on both)

Exact counts for every base table in a schema, via a session-local helper (pg_temp.* is dropped automatically at disconnect). A hot table reads a few rows ahead on the source by whatever is in flight — a gap that shrinks toward 0 is normal streaming, not drift.

CREATE OR REPLACE FUNCTION pg_temp.rc_counts(sch text)
RETURNS TABLE(table_name text, rows bigint) AS $$
DECLARE r record;
BEGIN
  FOR r IN SELECT tablename FROM pg_tables WHERE schemaname = sch ORDER BY tablename LOOP
    RETURN QUERY EXECUTE format('SELECT %L::text, count(*)::bigint FROM %I.%I',
                                r.tablename, sch, r.tablename);
  END LOOP;
END $$ LANGUAGE plpgsql;

SELECT * FROM pg_temp.rc_counts('public') ORDER BY table_name;

2. Content digest per table (run on both — digests must match)

The real integrity check: order-independent, column-name-matched, excludes GENERATED … STORED. Identical digest per table = logically converged. Best for static / lookup tables; for actively-written tables see step 3.

CREATE OR REPLACE FUNCTION pg_temp.rc_verify(sch text)
RETURNS TABLE(table_name text, rows bigint, digest text) AS $$
DECLARE r record; cols text;
BEGIN
  FOR r IN SELECT tablename FROM pg_tables WHERE schemaname = sch ORDER BY tablename LOOP
    -- name-sorted, generated-excluded column list: robust to column-order differences
    SELECT string_agg(quote_ident(column_name), ',' ORDER BY column_name)
      INTO cols
      FROM information_schema.columns
     WHERE table_schema = sch AND table_name = r.tablename AND is_generated = 'NEVER';
    IF cols IS NULL THEN CONTINUE; END IF;
    RETURN QUERY EXECUTE format(
      'SELECT %L::text, count(*)::bigint,
              md5(coalesce(string_agg(md5(ROW(%s)::text), '''' ORDER BY md5(ROW(%s)::text)), ''empty''))
         FROM %I.%I',
      r.tablename, cols, cols, sch, r.tablename);
  END LOOP;
END $$ LANGUAGE plpgsql;

SELECT * FROM pg_temp.rc_verify('public') ORDER BY table_name;

Diff the two sides cleanly inside psql, then compare the files in a shell:

-- in psql, on EACH side:
\pset format unaligned
\o /tmp/rc_src.txt          -- use /tmp/rc_tgt.txt on the target
SELECT * FROM pg_temp.rc_verify('public') ORDER BY table_name;
\o
# then in a shell:
diff /tmp/rc_src.txt /tmp/rc_tgt.txt   # no output = fully converged

3. Hot-table digest over a stable key range (run on both)

Actively-written tables won't match whole-table (the source runs ahead). Hash an immutable historical window instead — a key range below the active tail — which must match exactly. Adjust the table, key column, and range to a span you know isn't being updated.

SELECT count(*),
       md5(string_agg(md5(t::text), '' ORDER BY id)) AS digest
FROM public.user_answer t
WHERE id BETWEEN 1000000 AND 1010000;

4. Referential integrity (run on target)

Catches any FK breakage from the copy. The query below generates one anti-join per foreign key; run it, then paste and run the emitted SELECT … UNION ALL …. Every orphans count should be 0.

SELECT string_agg(
  format('SELECT %L AS fk, count(*) AS orphans FROM %I.%I c LEFT JOIN %I.%I p ON %s WHERE %s AND %s',
    conname,
    cn.nspname, cc.relname, pn.nspname, pc.relname,
    (SELECT string_agg(format('c.%I = p.%I', a.attname, af.attname), ' AND ')
       FROM unnest(con.conkey)  WITH ORDINALITY k(attnum,ord)
       JOIN unnest(con.confkey) WITH ORDINALITY fk(attnum,ord) ON k.ord = fk.ord
       JOIN pg_attribute a  ON a.attrelid  = con.conrelid  AND a.attnum  = k.attnum
       JOIN pg_attribute af ON af.attrelid = con.confrelid AND af.attnum = fk.attnum),
    (SELECT string_agg(format('c.%I IS NOT NULL', a.attname), ' AND ')
       FROM unnest(con.conkey) k(attnum)
       JOIN pg_attribute a ON a.attrelid = con.conrelid AND a.attnum = k.attnum),
    format('p.%I IS NULL',
      (SELECT af.attname FROM pg_attribute af
        WHERE af.attrelid = con.confrelid AND af.attnum = con.confkey[1]))
  ), E'\nUNION ALL\n' ORDER BY conname)
FROM pg_constraint con
JOIN pg_class cc ON cc.oid = con.conrelid  JOIN pg_namespace cn ON cn.oid = cc.relnamespace
JOIN pg_class pc ON pc.oid = con.confrelid JOIN pg_namespace pn ON pn.oid = pc.relnamespace
WHERE con.contype = 'f' AND cn.nspname = 'public';

Caveats

  • Live drift. On hot tables use the step 3 stable-range check, or run during a quiet window. The whole-table digest (step 2) is for static / lookup tables, which must match exactly.
  • Schema must match. If the target is missing or has an extra column, step 2's digest differs — that's correct, it's flagging a real schema mismatch, not spurious drift.
  • Order & collation don't matter. Step 2 orders rows by their hash, so different heap / index order or text collation never cause a false diff.
  • Version gap is handled. With the step 0 GUCs pinned, values render identically across a source/target major-version gap (floats, timestamps, bytea, intervals).
  • Nothing is left behind. The pg_temp.* helpers are session-local and vanish at disconnect — safe to run against a source you don't own.

See also: MySQL integrity checks · Redis integrity checks.

Verifying data integrity — MySQL

Standalone SQL to confirm a MySQL source and target are converged, for when you can't reach the replicare pod to run replicare status / replicare verify. Run each block on both databases and compare.

Like the Postgres guide, these match columns by name, exclude generated columns, and use an order-independent content hash — here BIT_XOR over a 64-bit slice of each row's MD5, which folds regardless of row order and has no GROUP_CONCAT length ceiling.

The authoritative check is still replicare verify <config>. This is the no-pod fallback.


0. Pin the session (run on both, every session)

replicare moves MySQL values byte-faithfully (LOAD DATA … CHARACTER SET binary), so the main cross-version rendering risk in a manual hash is the time zone. Pin it to UTC on both sides before hashing.

SET SESSION time_zone = '+00:00';
SET SESSION group_concat_max_len = 1048576;  -- so the generator below isn't truncated

1. Row counts per table (run on both)

information_schema.TABLES.TABLE_ROWS is only an estimate for InnoDB, so generate exact COUNT(*)s. Run this to emit one SELECT … UNION ALL …, then run the emitted statement:

SELECT GROUP_CONCAT(
         CONCAT('SELECT ''', TABLE_NAME, ''' AS table_name, COUNT(*) AS `rows` ',
                'FROM `', TABLE_SCHEMA, '`.`', TABLE_NAME, '`')
         ORDER BY TABLE_NAME SEPARATOR '\nUNION ALL\n') AS sql_to_run
FROM information_schema.TABLES
WHERE TABLE_SCHEMA = DATABASE() AND TABLE_TYPE = 'BASE TABLE';

A hot table reads a few rows ahead on the source by whatever is in flight — a gap that shrinks toward 0 is normal streaming, not drift.

2. Content digest per table (run on both — digests must match)

Generate a single statement that hashes every table: name-matched columns, generated columns excluded, BIT_XOR-folded so row order doesn't matter. Run the generator, then run the emitted SELECT … UNION ALL … on each side and compare table_name, rows, digest.

SELECT GROUP_CONCAT(
  CONCAT(
    'SELECT ''', tab.TABLE_NAME, ''' AS table_name, COUNT(*) AS `rows`, ',
    'LPAD(CONV(BIT_XOR(CAST(CONV(SUBSTRING(MD5(CONCAT_WS(''#'', ',
        tab.collist,
    ')), 1, 16), 16, 10) AS UNSIGNED)), 16, 16), 16, ''0'') AS digest ',
    'FROM `', DATABASE(), '`.`', tab.TABLE_NAME, '`'
  )
  ORDER BY tab.TABLE_NAME SEPARATOR '\nUNION ALL\n'
) AS sql_to_run
FROM (
  SELECT c.TABLE_NAME,
         GROUP_CONCAT(CONCAT('COALESCE(CAST(`', c.COLUMN_NAME, '` AS CHAR), ''\\0N'')')
                      ORDER BY c.COLUMN_NAME SEPARATOR ', ') AS collist
  FROM information_schema.COLUMNS c
  JOIN information_schema.TABLES t
    ON t.TABLE_SCHEMA = c.TABLE_SCHEMA AND t.TABLE_NAME = c.TABLE_NAME
   AND t.TABLE_TYPE = 'BASE TABLE'
  WHERE c.TABLE_SCHEMA = DATABASE()
    AND c.EXTRA NOT LIKE '%GENERATED%'   -- exclude generated columns
  GROUP BY c.TABLE_NAME
) tab;

Each column is COALESCEd to a sentinel (\0N) so a real NULL never collides with an empty string, and CONCAT_WS('#', …) separates columns so ('a','bc') and ('ab','c') don't hash alike.

3. Hot-table digest over a stable key range (run on both)

Actively-written tables won't match whole-table (the source runs ahead). Hash an immutable historical window — a key range below the active tail — which must match exactly. Adjust table, key column, and range to a span you know isn't being updated.

SELECT COUNT(*),
       LPAD(CONV(BIT_XOR(CAST(CONV(SUBSTRING(
              MD5(CONCAT_WS('#', COALESCE(CAST(id AS CHAR),'\0N'),
                                 COALESCE(CAST(note AS CHAR),'\0N'))),
            1,16),16,10) AS UNSIGNED)),16,16),16,'0') AS digest
FROM `rc_it`.`orders`
WHERE id BETWEEN 1000000 AND 1010000;

4. Referential integrity (run on target)

Generate one anti-join per foreign key, then run the emitted SELECT … UNION ALL …. Every orphans count should be 0.

SELECT GROUP_CONCAT(q SEPARATOR '\nUNION ALL\n') AS sql_to_run FROM (
  SELECT CONCAT(
    'SELECT ''', k.CONSTRAINT_NAME, ''' AS fk, COUNT(*) AS orphans ',
    'FROM `', k.TABLE_SCHEMA, '`.`', k.TABLE_NAME, '` c ',
    'LEFT JOIN `', k.REFERENCED_TABLE_SCHEMA, '`.`', k.REFERENCED_TABLE_NAME, '` p ON ',
    GROUP_CONCAT(CONCAT('c.`', k.COLUMN_NAME, '` = p.`', k.REFERENCED_COLUMN_NAME, '`')
                 ORDER BY k.ORDINAL_POSITION SEPARATOR ' AND '),
    ' WHERE ',
    GROUP_CONCAT(CONCAT('c.`', k.COLUMN_NAME, '` IS NOT NULL')
                 ORDER BY k.ORDINAL_POSITION SEPARATOR ' AND '),
    ' AND p.`', MIN(k.REFERENCED_COLUMN_NAME), '` IS NULL'
  ) AS q
  FROM information_schema.KEY_COLUMN_USAGE k
  WHERE k.CONSTRAINT_SCHEMA = DATABASE()
    AND k.REFERENCED_TABLE_NAME IS NOT NULL
  GROUP BY k.CONSTRAINT_NAME, k.TABLE_SCHEMA, k.TABLE_NAME,
           k.REFERENCED_TABLE_SCHEMA, k.REFERENCED_TABLE_NAME
) fks;

Caveats

  • Live drift. On hot tables use the step 3 stable-range check, or run during a quiet window. The whole-table digest (step 2) is for static / lookup tables, which must match exactly.
  • BIT_XOR folding. Two identical rows XOR to zero and cancel. Every replicated MySQL table has a primary/unique key (replicare requires one), so rows are distinct and this is safe — but it's why the digest is a drift indicator, with verify the authority. This is the same fold replicare's own Redis verify uses.
  • Schema must match. A missing or extra column on the target changes step 2's digest — correctly flagging a real schema mismatch.
  • Float rendering. MySQL has no extra_float_digits equivalent; FLOAT/DOUBLE text can differ across major versions. If a float-heavy table shows a digest diff with matching counts, spot-check a few rows before treating it as real drift.
  • Charsets. CAST(... AS CHAR) renders under the connection charset; use the same client charset on both sides (replicare reads under character_set_results = binary).

See also: Postgres integrity checks · Redis integrity checks.

Verifying data integrity — Redis

Standalone redis-cli checks to confirm a Redis source and target are converged, for when you can't reach the replicare pod to run replicare status / replicare verify.

Redis has no query language, so these are shell one-liners and a small script. They mirror what verify does: hash key + type + canonical logical value, using type-aware logical reads (GET / LRANGE / SMEMBERS / ZRANGE WITHSCORES / HGETALL / XRANGE) — never raw DUMP bytes, which are not byte-identical across Redis versions for equal values. TTL is excluded (replicated relative → skew-safe, so the exact remaining time differs by design).

The authoritative check is still replicare verify <config>. This is the no-pod fallback.


1. Key count (run on both)

redis-cli -h "$SRC_HOST" -p "$SRC_PORT" -n 0 DBSIZE
redis-cli -h "$TGT_HOST" -p "$TGT_PORT" -n 0 DBSIZE

The source may read a few keys ahead by whatever the current reconciliation sweep hasn't applied yet — a gap that shrinks toward 0 is normal, not drift. (Deletes on the target lag by the sweep interval by design.)

2. Keyspace content digest (run on both — digests must match)

Save this as rc-redis-digest.sh. It SCANs the keyspace, computes a type-aware canonical value per key, and XOR-folds each key's MD5 so iteration order doesn't matter. Run it against source and target; keys and digest must match.

#!/usr/bin/env bash
# rc-redis-digest.sh <host> <port> [db]
# Order-independent content digest of a Redis DB, mirroring `replicare verify`:
# key + type + canonical logical value (NOT DUMP bytes); TTL excluded.
set -euo pipefail
host=${1:?host}; port=${2:?port}; db=${3:-0}
r(){ redis-cli -h "$host" -p "$port" -n "$db" "$@"; }

acc=0 count=0
while IFS= read -r key; do
  [ -z "$key" ] && continue
  t=$(r type "$key")
  case "$t" in
    string) v=$(r get "$key") ;;
    list)   v=$(r lrange "$key" 0 -1) ;;                 # order is significant, preserved
    set)    v=$(r smembers "$key" | LC_ALL=C sort) ;;    # unordered → sort for canonical form
    zset)   v=$(r zrange "$key" 0 -1 withscores) ;;      # ordered by score, deterministic
    hash)   v=$(r hgetall "$key" | paste - - | LC_ALL=C sort) ;;  # field/value pairs, sorted
    stream) v=$(r xrange "$key" - +) ;;                  # ordered by entry id
    *)      v="" ;;
  esac
  h=$(printf '%s\x00%s\x00%s' "$key" "$t" "$v" | md5sum | cut -c1-16)
  acc=$(( acc ^ 0x$h ))
  count=$(( count + 1 ))
done < <(r --scan)
printf 'keys=%d digest=%016x\n' "$count" "$acc"
chmod +x rc-redis-digest.sh
./rc-redis-digest.sh "$SRC_HOST" "$SRC_PORT" 0
./rc-redis-digest.sh "$TGT_HOST" "$TGT_PORT" 0

Use the same redis-cli binary for both runs so value formatting is identical. Point the source read at the master, not a replica (a lagging replica reads stale).

3. Per-key spot checks

To eyeball a specific key on both sides, read it with its type's logical command and diff:

# string
diff <(redis-cli -h "$SRC_HOST" -p "$SRC_PORT" GET  mykey) \
     <(redis-cli -h "$TGT_HOST" -p "$TGT_PORT" GET  mykey)
# hash (sort so field order doesn't matter)
diff <(redis-cli -h "$SRC_HOST" -p "$SRC_PORT" HGETALL myhash | paste - - | sort) \
     <(redis-cli -h "$TGT_HOST" -p "$TGT_PORT" HGETALL myhash | paste - - | sort)
# zset (with scores)
diff <(redis-cli -h "$SRC_HOST" -p "$SRC_PORT" ZRANGE myzset 0 -1 WITHSCORES) \
     <(redis-cli -h "$TGT_HOST" -p "$TGT_PORT" ZRANGE myzset 0 -1 WITHSCORES)

4. Cluster

--scan and DBSIZE are per-node. On a cluster, run steps 1–2 against each master and sum/compare per shard (replicare scans, subscribes, and delete-sweeps per master; delete detection is master-pinned). Only DB 0 is replicated.


Caveats

  • Never hash DUMP bytes. They differ across Redis versions for equal values, so a byte hash would false-positive across a version gap. The script above uses logical reads, which match because transport is value-faithful.
  • TTL is excluded by design (relative replication → skew-safe; exact remaining time differs). Don't compare TTL/PTTL.
  • XOR folding. Distinct keys fold safely (key is part of each hash); the fold is a drift indicator, with verify the authority. It's the same fold replicare's own Redis verify uses.
  • Big keys. A multi-million-element value serialized through redis-cli is slow; spot-check those individually rather than in the full-keyspace script, or accept the runtime.
  • Streams. Entries are compared; consumer groups / PEL are not deep-verified here (nor by verify in v1).
  • Value reads may use a replica; delete/keyspace comparisons must use the master — a lagging replica would show phantom missing/extra keys.

See also: Postgres integrity checks · MySQL integrity checks.

Postgres engine

Postgres is replicare's first and reference engine — the engine the engine-neutral copy / apply / delta / reseed / pipeline / observability layers were designed around. It replicates Postgres → Postgres with trigger-based, no-WAL CDC. MySQL and Redis reuse those same neutral layers; this page documents the Postgres specifics. The shared docs (getting-started, configuration, operations, troubleshooting) describe the baseline behavior using Postgres, so treat them as this engine's extended manual.

Status: shipped. Trigger capture, chunked parallel initial copy (text-faithful COPY), FK-ordered streaming apply (including cyclic components), delta bloat/retention/reseed, full observability, and the daemon lifecycle all run end to end, and Postgres syncs coexist with MySQL and Redis syncs in one daemon. Try it with the Postgres demo.

Supported versions

  • Source floor: PostgreSQL 9.6 (best-effort). Target: a modern major (12+). The CI matrix runs source 9.6 → target 17. See postgres-version-support.md for the rationale (float fidelity on pre-12, no-native-partition path, old PL/pgSQL portability).
  • Source SQL — trigger functions, capture DDL, introspection — stays conservative and version-tolerant so it runs against genuinely old servers. The target is assumed modern, so the apply path uses INSERT … ON CONFLICT.

How CDC works (the relational baseline)

Postgres has a durable change stream (the WAL), but replicare deliberately does not use it — logical replication needs a REPLICATION role and a wal_level change that managed and locked-down servers won't grant. Instead:

  1. Capture — in a dedicated replicare schema on the source, replicare installs per-table delta and track tables and AFTER INSERT/UPDATE/DELETE triggers. The trigger records only the changed row's primary key (PK-only capture), not the full row image. A key-changing UPDATE enqueues both the old and new PK (= delete-old + upsert-new).
  2. Reconcile — at sync time replicare reads the unconsumed delta rows, re-reads the current values of those PKs from the live source table, and applies them to the target. This is self-healing: it always copies the latest value, so overlapping copy/stream windows converge naturally. Consumption is a delta MINUS track set-difference (a dirty-key set, not an ordered log), deleting by delta_id — which sidesteps the commit-order-≠-assignment-order hazard that a scalar high-water-mark cursor would fall into.
  3. Apply — idempotent upserts keyed on the PK (INSERT … ON CONFLICT DO UPDATE) and deletes by PK, FK-dependency-ordered within each FK component (upserts parent→child, deletes child→parent), with a retry fallback for cycles.
  4. Checkpoint — progress lives in the Postgres StateStore; a crash safely replays from the last confirmed cursor (applies are idempotent).

Capture is enabled first, then a chunked parallel initial copy runs while changes queue — so there is no single frozen snapshot and "too big to snapshot" is not a problem.

Faithful transport (never transform)

Values move verbatim as text COPY and are validated by the target's own input functions — replicare never coerces, truncates, casts, or "repairs" data; on a type/constraint error it halts loud rather than corrupting silently. To keep text round-trippable it pins formatting-only session GUCs on every connection (no privilege needed): DateStyle=ISO,YMD, TimeZone=UTC, extra_float_digits=3 (critical for float fidelity on pre-12 sources), IntervalStyle=postgres, bytea_output=hex, client_encoding=UTF8. A pre-flight compatibility check classifies every source↔target column pair: block on incompatible/missing types, warn on risky-lossy (e.g. int8→int4), ok on identical/widening.

Requirements

  • Every replicated table needs a primary key or usable unique key. Tables without one are skipped with a loud warning surfaced in logs and telemetry.
  • The target schema must pre-exist. replicare replicates data only — no live DDL. You create/migrate target tables with your own tooling; it introspects them to build correct, name-matched apply statements and never installs or alters target types/extensions/enum labels.
  • Cyclic / self-referential FKs are handled by case: a nullable cycle uses a NULL-then-fill two-pass; a DEFERRABLE cycle loads under SET CONSTRAINTS … DEFERRED; a NOT NULL non-deferrable cycle is genuinely unloadable without disabling constraints (which replicare won't do), so pre-flight fails loud and asks you to make the FK DEFERRABLE or break the cycle.
  • A link column inside a unique index where empty values count as equal (for example UNIQUE NULLS NOT DISTINCT (email, parent), or a unique index on an expression such as COALESCE(parent, 0), or one with a WHERE filter) cannot be emptied during the two-pass load — rows that differ only by that column would clash and the copy would stop with a duplicate-key error. replicare spots these indexes on both the source and the target (bare CREATE UNIQUE INDEX included) and handles them without any schema change:
    • Table that points at itself (an account tree, a category tree): the column is kept at its real value and the whole table is copied in a single statement, which Postgres accepts because it checks this kind of FK only when the statement ends. Live changes keep the real value too. The trade-off: that one table is copied in one piece rather than in parallel chunks, and a restart re-copies it from the start.
    • Loop across two or more tables: there is no safe load order, so pre-flight stops the sync before anything is copied and asks for the FK to be made DEFERRABLE.

Installing capture on a busy source

Adding a trigger locks the table for an instant, but if something else already holds a conflicting lock — a long VACUUM, a migration, or an application session left idle in transaction — the trigger change has to wait, and while it waits Postgres queues every new query on that table behind it. replicare therefore:

  • leaves an already-correct trigger alone on restart, so a restart takes no lock on any application table;
  • waits at most 2 seconds for a lock when it does need to add or replace a trigger, then steps back, retries with a growing pause, and after repeated failures stops with an error that names the table, rather than holding the application up.

If that error appears, look for the session holding the table (pg_stat_activity / pg_locks), let it finish, and restart replicare.

The Postgres advantage: effectively-exactly-once

replicare's delivery is at-least-once with idempotent apply. For a Postgres target, there is a bonus: when the StateStore is the target database, a cursor advance can ride the same transaction as its apply batch — yielding effectively-exactly-once apply per batch. This is unique to Postgres-target syncs: MySQL and Redis targets are strictly at-least-once because their StateStore is a different engine/server, so no single transaction spans both. (At-least-once is still correct and convergent everywhere — a crash may re-apply the last batch, which is harmless because applies are idempotent upserts.)

Delta bloat, retention & reseed

Because capture writes delta tables to a source you may not own, bounding the footprint is first-class: per-table delta tables (isolated bloat/contention), a batched, consumption-gated DELETE purge plus aggressive per-table autovacuum reloptions set at install time (no superuser, no global change), and bounded retention (age/size) with forced reseed of an over-cap target to protect the source. The reseed handoff — marking a target needs-reseed, resetting its track, re-copying, and resuming without gaps — is specified in reseed-state-machine.md. Per-target delta backlog, oldest-unconsumed age, purge rate, and reseed events are all surfaced as telemetry (see operations).

Least privilege

replicare needs no superuser, no REPLICATION attribute, and no wal_level change. Ready-to-edit grant SQL ships in ../deploy/grants-source.sql and ../deploy/grants-target.sql.

  • Source: USAGE on each replicated schema; SELECT + TRIGGER on each replicated table; and, to hold the capture machinery, either CREATE on the database or a pre-created replicare schema owned by the daemon role (the no-database-grant option). Teardown asymmetry: CREATE TRIGGER needs only the TRIGGER privilege, but DROP TRIGGER (part of replicare capture remove) requires table ownership — a least-privilege role can install and run capture but needs ownership (or the owner's help) to fully uninstall the trigger. Dropping replicare's own delta/track tables and functions works with the daemon role since it owns them.
  • Target: just SELECT, INSERT, UPDATE, DELETE on the pre-existing target tables. If the StateStore lives on the target database, also grant it CREATE on that database (or pre-create the replicare_state schema owned by the role).

Configuration

A Postgres sync is the engine-neutral envelope with a typed postgres: block per endpoint. The state store is Postgres too (v1's only StateStore backend) and may be the target, the source, or a separate database:

state_store:                       # replicare's own bookkeeping (Postgres)
  engine: postgres
  postgres: { host: state-db, port: 5432, database: replicare_state, user: replicare, password: "${PW}", sslmode: require }

sources:
  app:
    engine: postgres
    postgres: { host: app-db, port: 5432, database: app, user: replicare, password: "${PW}", sslmode: verify-full }

targets:
  warehouse:
    engine: postgres
    postgres: { host: wh-db, port: 5432, database: warehouse, user: replicare, password: "${PW}", sslmode: require }

syncs:
  - name: app-to-warehouse
    source: app
    targets: [warehouse]
    include: ["public.*"]          # schema.table globs
    exclude: ["*_audit"]
  • The Postgres block uses sslmode (libpq's disable | allow | prefer | require | verify-ca | verify-full), matching Postgres's own vocabulary — MySQL and Redis use a tls field with the same spectrum.
  • database is the connection default; selection globs are schema.table and may span other schemas. FK components are computed over the selected tables and copied/streamed with intra-component ordering and cross-component parallelism; a dominating "giant component" is warned about with a config override available.
  • Extra libpq parameters can be passed via params.
  • The single-engine rule is enforced: a Postgres source's targets must all be Postgres — never cross-engine.
  • All neutral knobs (worker/connection caps, drain interval, batch sizes, retention, observability endpoints, logging, env-var overrides) are documented in configuration.

Validating

replicare validate config.yml

connects to both sides, classifies every column pair (block on incompatible, warn on risky-lossy), reports the FK components with giant-component and dangling-FK (FK→excluded table) warnings, flags tables without a usable key, and detects unloadable NOT NULL cyclic FKs — all before any sync starts. See cli.md for the full command surface (run, validate, status, capture, reseed).

Postgres version support

This records replicare's committed Postgres version support floor and the rationale. It is load-bearing: several design decisions hinge on genuinely old source behavior, so the floor is a decision, not an afterthought.

Decision

RoleVersion in the test harnessSupport statement
SourcePostgreSQL 9.6Best-effort support back to 9.6. replicare reads from old sources using conservative, version-tolerant SQL.
TargetPostgreSQL 17Target should be a modern major (12+); apply uses modern features such as INSERT ... ON CONFLICT.

The CI integration matrix runs source 9.6 → target 17.

Why 9.6 as the source floor

  • Float fidelity. extra_float_digits = 3 is critical for round-trippable float4/float8 text on servers older than 12. A floor of 9.6 actually exercises that regime; a floor of 12+ would leave the most important float-fidelity claim untested.
  • No declarative partitioning. Native partitioning arrived in PG 10, so 9.6 exercises the "no native partitions" path and the ctid/keyset chunking fallbacks.
  • Old PL/pgSQL. Trigger functions and capture DDL must run on old servers; 9.6 is a realistic old target for that portability.
  • Availability. postgres:9.6 is still pullable from Docker Hub for CI.

Caveats

  • Apple Silicon (arm64): the postgres:9.6 image is amd64-only, so it runs under emulation locally (platform: linux/amd64 in the harness). CI runners are amd64, so this only affects local dev speed.
  • Even older sources (9.4 / 9.5 and earlier): not covered by the CI matrix. If we later need them, add matrix entries; until then, behavior on pre-9.6 servers is unverified. (Notably, pre-9.0 bytea defaults to escape output, which the modern target still parses — but this is not exercised.)

What is verified vs unverified

  • Verified by CI: everything against source 9.6 → target 17.
  • Unverified (flagged): sources older than 9.6, and target majors other than the one(s) in the matrix. Revisit if a real deployment needs them.

MySQL engine

replicare's second engine replicates MySQL → MySQL using the same trigger-based, no-binlog CDC model as Postgres and reuses the engine-neutral copy/apply/reseed/pipeline/observability layers unchanged.

Status: shipped. The MySQL engine runs end to end — trigger capture, chunked initial copy (byte-faithful LOAD DATA), FK-ordered streaming apply (including cyclic NOT NULL components), bloat control (purge/retention/reseed), observability, and the full daemon lifecycle — and coexists with Postgres syncs in one daemon. This page documents the engine's shape and its two operational wrinkles. Try it with the MySQL demo.

Supported versions

  • Source floor: MySQL 5.7. Target: 8.4 (the committed CI pair). 5.6 is best-effort only. See mysql-version-support.md.
  • MariaDB is not supported in v1: its dialect has forked from MySQL's, so the engine detects it at connect and refuses it loudly rather than mis-driving it.

Requirements

  • InnoDB tables only. Non-transactional engines (MyISAM/MEMORY) break the per-component atomic apply and the source consume/track atomicity, so pre-flight blocks them.
  • local_infile=ON is required on the target — it is the copy/apply transport (LOAD DATA LOCAL INFILE). replicare probes it at connect and, if it is off, halts loud with an actionable error rather than failing cryptically mid-copy. Enable it (SET GLOBAL local_infile=1, or local-infile=ON in my.cnf) and set the local_infile: true hint in the target's mysql: block. (A batched-INSERT fallback for local_infile-off targets is deferred to a future release — doing it byte-faithfully across column charsets needs careful design, and a hardening release favors a loud requirement over a risky fallback.)
  • Every replicated table needs a primary key or a usable unique key. Tables without one are skipped with a loud warning.
  • No secondary UNIQUE keys on target tables (beyond the replication key): MySQL's INSERT … ON DUPLICATE KEY UPDATE fires on any unique key and would silently mutate the wrong row, so pre-flight blocks such target tables. This is a hard faithful-transport guarantee.
  • Least privilege (source): SELECT + TRIGGER on the replicated tables, CREATE/owner on the replicare capture database — no SUPER, no REPLICATION SLAVE, no binlog access. (Exact grant SQL ships with MM8.)

The two operational wrinkles

Both follow from the v1 architecture and are worth knowing up front:

1. A MySQL sync still needs a Postgres for state

replicare's own operational state — which syncs exist, per-table copy progress, per-target cursors, and the single-active ownership lock — lives in a Postgres state store (state_store.engine: postgres), regardless of the data engine. This state is small control-plane bookkeeping, entirely separate from your MySQL data: the correctness-bearing replication state (the delta/track tables) lives on the MySQL source, exactly as Postgres puts them on a Postgres source.

So to replicate MySQL→MySQL you also point replicare at a small Postgres for its state. A MySQL-backed state store is a future item behind the pluggable StateStore interface; it is not in v1.

2. MySQL syncs are strictly at-least-once

replicare's delivery is at-least-once with idempotent apply. For a Postgres target, when the state store is that target database, a cursor advance can ride the same transaction as its apply batch — yielding effectively-exactly-once apply per batch.

That optimization is not available for MySQL syncs: the state store is a different engine/server (Postgres) than the MySQL target, so no single transaction can span both. MySQL syncs are therefore strictly at-least-once — correct and convergent (applies are idempotent upserts keyed on the PK), but a crash may re-apply the last batch. This is harmless by design; it's noted only so you don't expect exactly-once parity with a Postgres-target sync.

Installing capture on a busy source

Adding or replacing a trigger needs an exclusive metadata lock on the table. If an application transaction that touched the table is still open, the trigger change has to wait, and while it waits MySQL queues every new query on that table behind it (lock_wait_timeout defaults to a year). replicare therefore:

  • leaves a trigger alone on restart when its stored body already matches, so a restart takes no lock on any application table;
  • creates its replicare database only when it is missing (even CREATE DATABASE IF NOT EXISTS waits behind open application transactions once capture is installed);
  • waits at most 2 seconds for a lock when it does need to change a trigger, then retries with a growing pause, and after repeated failures stops with an error naming the table.

MySQL cannot replace a trigger in one step (drop and create are separate), so changes to that table are not captured during the instant between them. That only happens when a trigger actually changes (an upgrade that alters its body), never on a plain restart.

Configuration

A MySQL sync is an engine-neutral envelope with a typed mysql: block per endpoint, plus the Postgres state store:

state_store:                       # replicare's own bookkeeping (always Postgres)
  engine: postgres
  postgres: { host: state-db, database: replicare_state, user: replicare, password: "${PW}", sslmode: require }

sources:
  app:
    engine: mysql
    mysql: { host: app-db, port: 3306, database: app, user: replicare, password: "${PW}", tls: verify-full }

targets:
  warehouse:
    engine: mysql
    mysql: { host: wh-db, port: 3306, database: warehouse, user: replicare, password: "${PW}", tls: require, local_infile: true }

syncs:
  - name: app-to-warehouse
    source: app
    targets: [warehouse]
    include: ["app.*"]          # db.table globs (a MySQL schema IS a database)
    exclude: ["*_audit"]
  • tls accepts the full spectrum disable | allow | prefer | require | verify-ca | verify-full (mapped onto the driver, including a custom verify-ca).
  • database is the DSN default; selection globs may span other databases, and cross-database FK components are handled.
  • The single-engine rule is enforced: a MySQL source's targets must all be MySQL — never cross-engine.

Validating

replicare validate config.yml

introspects both sides and classifies every column pair (block on incompatible, warn on risky), reports FK components and giant-component/dangling-FK warnings, and enforces the InnoDB / secondary-unique / no-key rules above — all before any sync starts.

MySQL version support (the MF3 floor decision)

This records replicare's committed MySQL version floor and the MariaDB scope decision for the v1 MySQL engine. It is the MySQL counterpart to Postgres version support, and it fixes the CI test pair every MySQL acceptance test runs against.

The committed pair: MySQL 5.7 source → MySQL 8.4 target

CI runs a single concrete pair — mysql:5.7 as the old source, mysql:8.4 as the modern target — not a broad matrix. This mirrors the Postgres engine's single 9.6 → 17 pair: one integration-mysql job, amd64 (5.7 has no arm64 image, so it runs under emulation locally on Apple Silicon; CI is amd64 natively).

Why 5.7 is the source floor

5.7 is the oldest source we support, chosen because it is the oldest release that still has the features replicare's introspection and capture rely on, while being old enough to genuinely exercise the "old source → modern target" path:

CapabilityIntroducedConsequence if the floor were older (5.6)
Generated columns (VIRTUAL/STORED in information_schema.COLUMNS.EXTRA)5.75.6 has none; the generated-column exclusion path (MM4) would be untestable there.
JSON type5.75.6 lacks it; a headline fidelity-corpus type would be missing.
Usable information_schema5.7 (materially faster than 5.6)5.6's information_schema is notoriously slow, degrading introspection.
sys schema, spatial indexes, etc.5.7Not load-bearing, but part of why 5.7 is the practical floor.

5.6 is best-effort only: the engine may work against it, but no acceptance test is committed to it, and the features above are unverified there. If a future need arises, 5.6 support is additive.

Why 8.4 is the target

8.4 is the current LTS. It exercises the modern-target concerns the plan calls out: utf8mb4 defaults, ROW_NUMBER() window functions for boundary discovery (MM4a), and the strict-sql_mode/zero-date interaction the faithful-transport path depends on. The 5.7→8.4 gap is where the old-source fidelity work earns its keep.

MariaDB: out of scope for v1

replicare v1 does not support MariaDB. Although MariaDB began as a MySQL fork, its dialect, system catalogs, sql_mode semantics, and JSON/sequence behavior have diverged enough that silently driving it as MySQL would risk exactly the value corruption faithful transport forbids. The engine therefore detects MariaDB at connect time (the server advertises MariaDB in VERSION()) and refuses it with a loud error rather than mis-driving it. MariaDB is a candidate for a future engine variant behind the same interface, not a v1 goal.

What this pins

  • The CI job builds and tests only against mysql:5.7 and mysql:8.4.
  • Every MySQL milestone's acceptance is verified on this pair.
  • If CI cannot run 5.7 (e.g. image unavailability), the plan requires listing which fidelity claims become unverified rather than silently raising the floor.

Redis engine

replicare's third engine replicates Redis → Redis with the same no-privileged-change-stream philosophy as Postgres and MySQL, but a very different mechanism: Redis has no WAL/binlog, no triggers, and no way for us to write capture state onto the source, so CDC is full-keyspace SCAN reconciliation rather than trigger deltas. It reuses the engine-neutral copy/apply/state/config/observability layers, plus one additive delete- reconciliation seam that the relational engines don't need.

Status: shipped. The Redis engine runs end to end — value-faithful DUMP/RESTORE copy, reconciliation-driven streaming upserts, a durable target-vs-source delete sweep, the optional keyspace-notification accelerator, standalone and Cluster topologies, TTL fidelity, least-privilege ACLs, and the full daemon lifecycle — and coexists with Postgres and MySQL syncs in one daemon.

How Redis CDC works (it's different)

There is no change stream to tail, so replicare re-reads the source keyspace:

  • Upserts — rolling reconciliation SCAN. Every streaming pass SCANs the source, DUMPs the present keys, and RESTORE … REPLACEs them on the target. This is idempotent and convergent: a rapid rewrite simply converges to the last value on the next pass. Nothing is written to the source — no delta tables, no replicare schema, no source bloat.
  • Deletes — a durable target-vs-source keyspace diff. A deleted key just stops appearing in the source scan; there is nothing to "capture." So replicare runs a separate delete sweep: enumerate the target keyspace, EXISTS-check each key against the source master, and DEL the ones the source no longer has. This is the correctness mechanism for deletes — durable and self-healing, but its latency is bounded by the sweep interval, not instant (see Delete-propagation latency).
  • Keyspace notifications — an optional accelerator only. When enabled they shorten latency by flagging changed keys ahead of the rolling scan, but they are fire-and-forget and lossy, so correctness never depends on them (see Keyspace notifications).

Transport — value-faithful DUMP → RESTORE

replicare moves each value as its RDB-serialized DUMP payload and applies it with RESTORE … REPLACE. It never interprets, coerces, or reconstructs a value — the same faithful-transport promise as the relational text COPY path. All six native types (string, list, set, zset, hash, stream) plus TTLs move this way; a stream's consumer groups and PEL are preserved in the payload.

Transport is value-faithful, not byte-faithful: RESTORE re-encodes on the target and picks its own optimal internal representation, so a target DUMP need not byte-match the source's across versions. Fidelity is verified by type-appropriate logical comparison, not DUMP-byte equality across a version gap.

There is no reconstruction fallback. When a payload can't be RESTOREd (version gate, missing module, oversize), replicare fails loud — it never falls back to GET/HGETALL/… rebuilding, which would be the forbidden value transform (and non-idempotent besides). See the two pre-flight gates.

Supported versions & forks

  • Source floor: Redis 3.0 (RESTORE … REPLACE). Target floor: 5.0. Recommended 6.2+ both ends. CI-tested pair: 6.2 source → 7.4 target plus a 3-node 7.4 cluster.
  • Valkey is supported (shares Redis's RDB format + protocol); KeyDB is best-effort; Dragonfly is blocked in v1 (its RDB DUMP/RESTORE compat is unverified — refused loudly rather than risk corruption). A non-Redis-family server is refused at connect, the same way MariaDB is for MySQL.

Full detail — including the RDB-version directional gate and the module gate — is in redis-version-support.md.

The two pre-flight BLOCKS

Redis has no type-coercion axis like the relational engines, but it has two value-transport gates that pre-flight (validate and daemon startup) blocks on:

  1. RDB-version directional gate. RESTORE rejects a payload whose embedded RDB version exceeds the target's maximum. So older/equal source → newer/equal target works; newer source → older target fails loud. Pre-flight reads INFO server on both ends and blocks a newer→older pair up front rather than failing mid-copy. Fix: raise the target to at least the source's version.

  2. Module gate. A module-defined type (RedisJSON, RedisBloom, …) produces an opaque DUMP payload only the identical module can RESTORE. Pre-flight compares MODULE LIST on both ends and blocks when a source module type is absent on the target — there is no faithful native reconstruction.

    Managed Redis that disables MODULE (ElastiCache returns ERR unknown command 'MODULE'; an ACL may deny it with NOPERM) is handled gracefully: replicare treats a blocked MODULE LIST as "no modules" and proceeds, rather than failing to start. Such servers can't load modules through MODULE anyway, so this only softens the pre-flight gate — a module-typed value, if somehow present, still fails loudly at RESTORE, never silently. +module|list in the target ACL preset is therefore best-effort: grant it where the provider supports it; where it's blocked, replicare degrades on its own.

The operational wrinkles

Both follow from the v1 architecture and are worth knowing up front.

1. A Redis sync still needs a Postgres for state

replicare's own operational state — which syncs exist, per-unit copy progress, per-target cursors, and the single-active ownership lock — lives in a Postgres state store (state_store.engine: postgres), regardless of the data engine. This is a real wart for an all-Redis shop: to replicate Redis→Redis you also point replicare at a small Postgres for its bookkeeping.

Unlike the relational engines, though, Redis writes nothing to the source — its CDC is capture-less, so there are no source-side delta/track tables. All durable state a Redis sync keeps is the coarse control-plane state in that Postgres. A Redis-backed state store is a future item behind the pluggable StateStore interface; it is not in v1. See redis-statestore.md for exactly how a Redis sync's rows are represented.

2. Copy progress is coarse (no resumable SCAN cursor)

A SCAN cursor is an opaque reverse-binary index, not durably resumable across rehashes or restarts. So a Redis unit's copy progress is deliberately coarse: snapshot-complete-per-unit, with no watermark. A crash mid-snapshot re-runs the whole SCAN, which is idempotent (RESTORE REPLACE), so nothing is lost — but a half-finished large copy restarts from the top rather than resuming a chunk.

Configuration

A Redis sync is an engine-neutral envelope with a typed redis: block per endpoint, plus the Postgres state store:

state_store:                       # replicare's own bookkeeping (always Postgres)
  engine: postgres
  postgres: { host: state-db, database: replicare_state, user: replicare, password: "${PW}", sslmode: require }

sources:
  cache:
    engine: redis
    redis: { mode: standalone, host: src-redis, port: 6379, db: 0, user: replicare, password: "${PW}", tls: require }

targets:
  replica:
    engine: redis
    redis: { mode: standalone, host: dst-redis, port: 6379, db: 0, user: replicare, password: "${PW}", tls: require }

syncs:
  - name: cache-to-replica
    source: cache
    targets: [replica]           # v1: exactly one Redis target (fan-out is a non-goal)
    include: ["app:*"]           # key-pattern globs (Redis SCAN MATCH semantics)
    exclude: ["app:tmp:*"]       # exclude-wins

The full redis: block reference is in configuration.md. The most-used fields:

FieldNotes
modestandalone (default) | cluster | sentinel (parsed but experimental)
host / portstandalone seed; nodes (a host:port list) for cluster/sentinel
dblogical DB index (standalone only; forced 0 in cluster)
tlsfull disable|allow|prefer|require|verify-ca|verify-full spectrum
read_from_replicaoffload value reads to replicas; delete detection still hits the master
typesoptional type filter (string/list/set/zset/hash/stream; needs 6.0+)
scan_countSCAN COUNT hint (per-call; distinct from the neutral drain_interval/batch)
reconcile_interval / delete_sweep_intervalcadence of the upsert scan and the delete sweep
big_key_warn_bytes / big_key_refuse_bytesDUMP-and-warn / block-loud thresholds (via MEMORY USAGE)
notificationsenable the lossy keyspace-notification accelerator (default off)
ttl_moderelative (default, skew-safe) | absttl (opt-in, trusted clocks)

The single-engine rule is enforced: a Redis source's targets must all be Redis — never cross-engine.

Selection — key-pattern globs

include/exclude are key-pattern globs with Redis SCAN … MATCH semantics (*, ?, […], hash-tags), applied client-side, exclude-wins. There is one full SCAN per pass regardless (matching is post-scan), except when a types filter lets replicare prune server-side with SCAN … TYPE (6.0+). A key that matches no include glob is simply not replicated.

Topology & Cluster

  • Standalone picks one logical DB (db: 0–15).
  • Cluster runs everything per master node: the reconciliation SCAN, the delete sweep, and (optionally) the notification subscription each fan out across masters; topology is discovered via CLUSTER SHARDS/SLOTS and refreshed on MOVED/ASK. The parallelism unit is the shard. Only DB 0 exists in cluster mode (validated). Per-key DUMP/RESTORE/DEL/EXISTS sidestep cross-slot limits.
  • Sentinel is parsed and the client is constructed, but v1 does not harden or test failover — treat it as experimental.

Delete detection always reads the master. read_from_replica may offload value reads to replicas for politeness, but EXISTS/MissingAtSource are pinned to the master: a lagging replica could report a just-written key absent and cause a false DEL. This is enforced.

TTL fidelity

TTL is captured with PTTL and, by default, applied relative (RESTORE key <pttl> …) so the target evaluates expiry from its own clock — immune to daemon/target clock skew. absttl mode is opt-in for trusted-clock topologies and is computed from the server's own TIME, not the daemon clock. PTTL -1 (no expiry) maps to no expiry; PTTL -2/expired keys are handled by the delete sweep.

Big keys

replicare never does incremental/torn transport of a large value (that would be unfaithful). Instead, above big_key_warn_bytes (measured with MEMORY USAGE) it still DUMPs the whole key and logs a loud big-key warning (accepting the transient source blocking); above big_key_refuse_bytes it blocks loud. This applies to both initial copy and streaming re-reads.

Keyspace notifications (optional accelerator)

Setting notifications: true turns on a latency accelerator: one PSUBSCRIBE per master to __keyevent@<db>__:*, feeding flagged keys into a priority set drained ahead of the rolling scan. It is never trusted for correctness — events are fire-and-forget, value-less, lost on disconnect, and node-local in cluster. A value-less event only marks a key dirty; the current value is always re-read, so a set and a del both flow through the same re-read → RESTORE-or-DEL path. On any subscription gap/reconnect, replicare forces a full reconciliation + delete sweep for that shard. With notifications off, the engine falls back cleanly to pure reconciliation. Enabling notifications on the server needs CONFIG SET (notify-keyspace-events) or, on a managed tier like ElastiCache, a parameter group set out-of-band.

Delete-propagation latency (the thing to know)

Because deletes are not captured — they're found by the periodic target-vs- source sweep — a source DEL converges on the target only after the next sweep completes, bounded by delete_sweep_interval (and the time to SCAN the target). Upserts converge faster (the rolling reconciliation, accelerated by notifications when on). Two Redis-specific metrics make this observable:

MetricTypeLabelsMeaning
replicare_delete_reconciliation_lag_secondsgaugesync, target, tableDuration of the last completed delete sweep — staleness of capture-less deletes.
replicare_deletes_reconciled_totalcountersync, target, tableKeys DELed by the target-vs-source sweep.

These are unset for Postgres/MySQL (whose deletes flow through the delta path). The reused F2 signals apply too, with table carrying the Redis unit (redis.db0); note that replicare_delta_backlog_* stay ~empty for Redis (there is no source-side queue to bound).

Least-privilege ACLs (and the +restore foot-gun)

Redis 6+ ACLs; no @admin/@all/DEBUG/CONFIG/KEYS/FLUSH* in the baseline. Ready-to-edit presets ship in the repo:

Foot-gun: RESTORE is an @dangerous command, so it is excluded from the usual read-write presets and must be granted explicitly (+restore). Without it, every apply fails loud. Granting it lets the role overwrite any key in the ACL's key pattern, so scope ~… tightly.

The optional notification accelerator additionally needs +psubscribe/+subscribe (and CONFIG SET where the server isn't preconfigured) on the source. In cluster mode, grant the same user on every master and add +cluster|shards/|slots.

Active-active mesh (multi-master)

Redis participates in an active-active clusters: mesh (writes accepted on any node, converging on all under HLC last-write-wins) — same config surface as the relational engines, detailed in multi-master.md. Because Redis is capture-less, the (hlc, node_id) version register lives in a parallel metadata keyspace: for each data key K, a sibling hash \x00rc:m:{…}K (co-located in K's cluster slot) holding its version + a logical-value fingerprint. The value key is never wrapped — faithful transport holds. Everything below is cluster-mode-only; a one-way Redis sync is byte-for-byte unchanged:

  • Convergence & loop suppression fall out of one comparison. A re-read RESTOREs/DELs on a peer only when the incoming (hlc, node) strictly beats the local register — so replicare's own applied writes lose (no echo storm) and concurrent same-key writes resolve to one value everywhere. Value+register are written in one atomic Lua.
  • Deletes are version-guarded tombstones, not the stateless sweep (which is disabled in a mesh — a key absent at one node but present here is a peer's legitimate write, never an orphan). Tombstones GC by age (rc_tombstone_retention, default 24h).
  • Extra ACLs: a mesh member is both source and target and additionally needs the metadata keyspace RW + logical readers + +eval/+evalsha — see the mesh sections in deploy/acl-*-redis.txt.
  • Cost: the metadata keyspace ~doubles key count; correctness runs on the local gate (REPLICARE_REDIS=1), not the Postgres-only CI.

Limitations / non-goals (v1)

  • Redis fan-out (one source → multiple Redis targets) — a single source SCAN cursor can't serve two targets at different rates without per-target scan state. Single source → single target only (a mesh is different: each edge is its own job).
  • Sentinel-hardened failover — wired but not hardened/tested.
  • Dragonfly — blocked (RDB compat unverified).
  • Command-reconstruction / big-key incremental transport — refused (it would transform values; transport is faithful only).
  • Cross-key/group atomicity — apply is per-key idempotent, not wrapped in MULTI/EXEC (a deliberate departure from the relational per-component atomicity). This holds in mesh mode too: each key's value+register pair is atomic, but there is no cross-key transaction.
  • Cross-engine and point-in-time consistency — never / deferred, as for the other engines. Redis converges eventually and self-heals. Multi-master IS supported (see above); pre-mesh divergent data / pre-mesh deletes resolve by LWW at join.

Validating

replicare validate config.yml

connects to both ends, runs the RDB-version and module gates, reports each replication unit as a single-member component, and (for cluster) validates per-shard — all before any sync starts.

Try it end-to-end

examples/demo-redis/ is a runnable demo: an old 6.2 source, a modern 7.4 target, a Postgres state-store sidecar, and replicare — replicating app:* keys. Build the image (task docker), then cd examples/demo-redis && docker compose up and watch the seeded keys, a live SET, and a DEL propagate. The config is examples/demo-redis/config.yml.

Redis version & fork support

replicare's Redis engine replicates Redis → Redis using ordinary, grantable commands (SCAN, DUMP, PTTL, RESTORE, DEL) — no replication link, no RDB/AOF access, no PSYNC. Transport is DUMP → RESTORE (value-faithful; the value's RDB-serialized bytes are moved and never transformed). This page records the supported version range and fork stance.

Version range

RoleBest-effort floorRecommendedCI-tested
Source3.0 (RESTORE … REPLACE)6.2+6.2
Target5.0 (RESTORE … ABSTTL opt-in)7.x7.4

The engine keeps source-side commands conservative (SCAN 2.8+, DUMP/RESTORE 2.6+, PTTL 2.6+); SCAN … TYPE (6.0+), ACLs (6.0+), and RESTORE … ABSTTL (5.0+) are modern-only niceties with older fallbacks.

The RDB-version directional gate (important)

RESTORE rejects a payload whose embedded RDB version exceeds the target's maximum. Therefore:

  • older/equal source → newer/equal target: works (backward compatible).
  • newer source → older target: fails loud (DUMP payload version or checksum are wrong).

replicare's pre-flight (RM2) reads INFO server on both ends and blocks a newer-source → older-target pair with an actionable error, rather than failing mid-copy. Upgrade the target to at least the source's version, or replicate the other direction. This is the Redis analog of the relational "block on incompatible type."

Forks

Detected from INFO server:

ServerStatusRationale
RedisSupportedThe reference implementation.
ValkeySupportedShares Redis's RDB format and RESP protocol, so DUMP/RESTORE transport is compatible.
KeyDBBest-effortRedis-protocol compatible; RDB compatibility generally holds but is not CI-verified.
DragonflyBlocked (v1)Its DUMP/RESTORE RDB-format compatibility with Redis is unverified; since transport is DUMP/RESTORE, replicare refuses it loudly rather than risk silent corruption.

A non-Redis-family server is refused at connect, the same way MariaDB is refused for the MySQL engine.

Modules

Module-defined types (RedisJSON ReJSON-RL, RedisBloom, RedisTimeSeries, …) produce module-specific opaque DUMP payloads that only RESTORE on a target loading the identical module. Pre-flight (RM2) compares MODULE LIST on both ends and blocks when a source module type is absent on the target — there is no faithful native reconstruction of an opaque module value.

Redis on the Postgres StateStore

replicare's operational state — which syncs exist, per-unit initial-copy progress, per-target streaming cursors, ownership, and reseed flags — lives in a dedicated schema on a Postgres database (the target, the source, or a separate Postgres; see Architecture). This backend is engine-neutral: the Redis engine ships no new StateStore code. It reuses the identical Postgres backend by overloading the neutral vocabulary. This document records those overloads so an operator reading the replicare_state schema knows how to interpret a Redis sync's rows.

The StateStore is the daemon's own operational state. It is not where Redis change-capture lives — Redis has no source-side delta/track tables; its CDC is SCAN reconciliation (see the Redis engine page).

The overloads

Neutral conceptRelational meaningRedis meaning
TableRef (a copy_progress / cursors row's table)schema.tablea replication unit — the connection's logical DB, rendered redis.db<N> (cluster = db0). One unit per sync.
SyncDef.Selectionschema.table include/exclude globskey-pattern include/exclude globs (Redis SCAN … MATCH semantics).
CopyProgress.Watermark / .Completedkeyset high-water mark + sparse completed PK-rangesalways nil — see coarse checkpointing below.
CopyProgress.Doneall chunks of a table copiedthe unit's SCAN snapshot completed.
Cursor.LastDelta (engine.DeltaID)last confirmed source delta-table row ida per-unit change-id (a reconciliation-pass counter used only for lag/observability).
Cursor.NeedsReseedretention-cap forced reseedidentical — retention/reseed is engine-neutral.
Acquire (advisory lock)single-active per syncidentical — a Redis sync is single-active too.

One unit per sync (not one per shard)

A Redis sync replicates a single logical keyspace. Even against a cluster, the persisted identity is one TableRef (redis.db0); the per-master SCAN fan-out happens inside the Source (CopyChunk / ReadDirtyKeys), never as N-per-shard rows. This keeps a single, stable unit ref that round-trips cleanly through the neutral copy/apply/state layers regardless of topology.

Coarse checkpointing (nil watermark)

A relational table's initial copy is chunked by PK range, so its progress is a keyset watermark plus a sparse set of out-of-order completed ranges, letting a restart skip finished chunks. A Redis unit has no ordered key space to chunk against — its snapshot is a full SCAN pass that is either complete or not. So its CopyProgress is deliberately coarse: Watermark and Completed stay nil, and Done flips false → true when the unit's snapshot finishes. A restart mid-snapshot re-runs the whole SCAN, which is idempotent (RESTORE REPLACE), so no watermark is needed.

Why no new code

Because the store never interprets these fields — it persists TableRef as two opaque strings, KeyValues/DeltaID as JSON/bigint, and the advisory lock by sync name — a Redis unit flows through the exact same read/write path as a Postgres table. The RM3 tests (internal/state/postgres/redis_unit_test.go) assert this by round-tripping Redis-shaped SyncDef, CopyProgress, and Cursor values, and by proving the ownership lock excludes a second daemon for a Redis sync.

Running active-active replication — the rules

Active-active means several databases (called nodes) that an application can write to on any of them, with replicare keeping them all carrying the same data. This is more demanding than one-way replication (one source, one or more read-only copies), and two things decide whether it works:

  1. IDs must never collide across nodes.
  2. Schema changes must reach every node before anything uses them.

Following the two rules below keeps active-active safe. Ignoring them risks silently losing data. replicare is built to stop loudly rather than corrupt, but it cannot fix either problem on its own — both are decisions that live in the database and in the deployment process, before replicare ever sees a row.

For how any of this works under the hood, see the design note, multi-master.md. This page is just the operating rules.


Rule 1 — Use globally-unique IDs (UUID or ULID), always

The problem. Many tables give each row an id that the database counts up automatically — 1, 2, 3, 4… (Postgres serial / IDENTITY, MySQL AUTO_INCREMENT). In active-active this breaks, because each node counts on its own. Two nodes can, at the same moment, create two different rows that both get id 100. When those rows sync to each other, the system sees two rows claiming the same id, keeps one, and the other is silently lost.

The same trap applies to any other column that is forced to be unique and filled in by the database — for example an auto-generated order_number. It is not only about the primary key.

The rule. Every id (and every other auto-filled unique value) in an active-active replicated table must be globally unique by design, so any node can create one with no coordination and they will never clash. In practice that means UUID v7 or ULID — large time-based ids that are effectively impossible to collide. Plain auto-counting integers are not allowed in an active-active cluster.

A few clarifications:

  • Natural keys are fine. A key that is already unique in the real world (an email address, an ISO country code, an id assigned by the application itself) has no collision problem — the rule is specifically about letting the database hand out sequential numbers.
  • It is a schema decision, not a setting. replicare does not and will not invent ids or "sync the counters" between nodes — that approach cannot be made safe. The fix lives in the table design.
  • One-way replication is unaffected. This rule is only for active-active. A normal one-way copy can keep using auto-counting integer ids. (See operations → Sequences & identity counters for the separate, one-way concern of keeping a failover copy's counter in step.)

How replicare helps.

  • At startup (built today): before it will run an active-active cluster, replicare inspects every table it is about to replicate. If it finds an auto-counting integer used as a primary key or a unique key, it refuses to start, names the offending table and column, and points at this rule. This stops the collision setup from ever going live.
  • Continuously (planned, not built yet): the startup check only runs at startup. A schema change made while the cluster is already running — a new table, or a new auto-counting column — could introduce the problem with no restart to catch it. The plan is to run the same check on a schedule (and on demand) so a risky change is flagged the day it ships, not the next time something restarts. Until that exists, it remains the operator's responsibility: do not add auto-counting integer keys to replicated tables in a running cluster.

Rule 2 — Push schema changes to every node first, then deploy the app

replicare never replicates schema changes. It copies data (the rows) between nodes. It does not copy changes to the structure of a table — adding a column, dropping one, renaming one, changing a column's type, or adding a new table. (These structure changes are often called DDL.) This is deliberate and permanent: replicare is a data replicator, not a schema manager.

So a structure change made on one node does not reach the other nodes automatically. It must be applied to each node separately.

What happens if the nodes disagree. Suppose nodes A and B both have a users table, and a phone column is added to A only. Then a new user is written on A with a phone number. replicare tries to copy that row to B, B has no phone column, and the copy fails. replicare does not guess, does not drop the phone, and does not corrupt anything — it stops copying in that direction and says so loudly in its logs and status, holding the row. Once the phone column is added to B as well, replicare resumes on its own and delivers everything it was holding. It heals once the structures match.

Two cautions about that stall:

  • While a node is behind, data for it is piling up, not flowing. replicare waits only so long (about a day by default) before giving up and re-copying that node from scratch. So every node should get the change promptly — nodes must not stay mismatched for long.
  • A stall is safe but not invisible — it is a real interruption to that node staying current. The rule below avoids it entirely.

The rule. For every schema change:

  1. Apply the change to every node first, while nothing is using it yet. Adding an empty new column (or a new table) to all nodes breaks nothing on its own.
  2. Then deploy the app code that reads or writes the new structure.

Because step 1 finishes everywhere before step 2 begins, no node is ever asked to store something its structure cannot hold. Nothing stalls.

The mirror image and the special cases:

  • Dropping a column: reverse the order. Stop using it in the app everywhere first, then drop it from every node last.
  • Renaming a column: never rename in place in a running cluster. Instead: add the new column, copy the values across, switch the app to the new column, then drop the old one. (A bare rename looks like "drop one column, add another" and is the easiest way to lose data.)
  • Changing a column's type: only safe if the old values still fit the new type (for example widening a number). An incompatible change stalls that node loudly — do it the add-new-column-and-migrate way instead.
  • Adding a new table: besides creating it on every node (with globally-unique ids, per Rule 1), replicare must be told to start replicating it — a config change plus a restart on each node. It is a coordinated step, not a one-line migration.

A note on online-schema-change tools (e.g. gh-ost, pt-online-schema-change): these rebuild a table behind the scenes and swap it in, which does not play well with how replicare captures changes. Prefer the database's native, in-place ALTER for simple additive changes, and test carefully before using an online-schema-change tool against a replicated cluster.


The short version

  • IDs: UUID or ULID on every replicated table. No auto-counting integer keys in a cluster. replicare refuses to start if it finds one.
  • Schema changes: replicare will not carry them. Apply every change to all nodes first, then deploy the app that uses it. Drops last; never rename in place.
  • If structures ever disagree, replicare stalls loudly and heals once the structures match — but that is a safety net, not a plan.

Multi-master replication — design note

Status: IN PROGRESS. This document is the design; milestones are landing against it. Shipped for Postgres, MySQL, AND Redis: the nodes:/clusters: config surface (MM0–MM1), loop suppression (MM3), and HLC last-write-wins conflict resolution + tombstones + GC (MM4 Postgres, MM5 MySQL, MM6 Redis) — a full active-active mesh runs on all three engines today: writes accepted on any node converge on all, replicare's own applies are not re-captured and echoed, and concurrent writes to the same key converge to the same value on every node, resolved by a hidden (hlc, node_id) version (no user schema change), with deletes handled by GC'd tombstones (§5.2, §5.3, §5.4, §6.2, §8 status). On Redis the register is a parallel metadata keyspace rather than a side table, since Redis is capture-less (§5.4). Not yet shipped: HA leader election (MM8). The hard constraint on all of this work is that the existing one-way path (source → target(s), changes never flow back) keeps behaving exactly as it does today — see §7 Backward compatibility.

This note covers Postgres, MySQL, and Redis. It records what exists now, why naively wiring a bidirectional topology breaks today, the mechanisms a real multi-master mode would need per engine, the config-schema changes, and a phased plan. It is written to sit alongside the per-engine pages.

Direction set by the project owner (v3): full active-active is required on all three engines, Redis included (not deferred); N-node full mesh (no fixed node count); conflict resolution must need no user schema change and no user-specified column (replicare manages a hidden version itself — §5.3); no priority-fencing; tombstones live in the replicare schema and are GC'd; it must work with no superuser anywhere; the config surface is clusters:.

Operator-facing rules. The plain-language rules for safely running an active-active cluster — the globally-unique-ID requirement and the schema-change procedure — live in active-active.md. This note is the engineering design behind them; if the two ever disagree, the mechanisms described here are the source of truth and active-active.md should be corrected to match.


1. Goal

Keep N database instances (across different clouds) converged, where writes may land on any node — i.e. active-active / multi-master, not just one writer fanned out to read replicas. The target topology is a full mesh of any N (every node ↔ every node — no fixed node count; 3 is only a common example). Ring / partial-mesh forwarding is a deferred efficiency variant (§5.2); the version register (§5.3) already makes such topologies safe.

This is distinct from what replicare does today: one authoritative source fanned out to one or more read-only targets (§3) — which becomes the per-node building block of the mesh rather than the end state.


2. Current behaviour (grounded in the code)

replicare is single-master by construction on every engine. The relevant facts, with file references, because the design has to work with them:

Topology / config (internal/config/config.go, internal/daemon/daemon.go)

  • A Sync is one Source → N Targets (config.go:65-72); at runtime the daemon spawns one Syncer per (sync, target) (daemon.go:129-158), and a Syncer holds a single source, single sink, single target (internal/pipeline/syncer.go) — deliberately one-directional.
  • Config.Validate() (config.go:221-275) validates each sync in isolation. There is no cross-sync graph analysis and no cycle detection: two syncs A→B and B→A, or a ring, pass validation silently.
  • Parsing is strict — unknown YAML keys are rejected (config.go Load), which is important for §6: new keys are additive and old configs are unaffected.
  • Per-sync single-engine is enforced (config.go:254-258).
  • Fan-out (one source → many targets) exists but is explicitly "present, not hardened" (syncer.go docstring) and has no end-to-end test; initial-copy progress is keyed (sync, table) without target (internal/state/postgres/progress.go), so two targets on one sync currently share copy progress.

Postgres capture/apply (internal/engine/postgres/)

  • Capture triggers are AFTER INSERT/UPDATE/DELETE ... FOR EACH ROW with no WHEN guard; the trigger function unconditionally enqueues the changed PK (capture_ddl.go). No current_setting/session check.
  • The apply connection sets only text-formatting GUCs (conn.go sessionGUCs) — not session_replication_role='replica' or anything that suppresses triggers.
  • Apply is blind source-wins: INSERT … ON CONFLICT (pk) DO UPDATE SET col = EXCLUDED.col with no WHERE (sink.go), plus delete-to-match (apply.go).
  • Delta rows are PK-only + delta_id + rc_op + seq/txid/rc_at — no origin column.

MySQL capture/apply (internal/engine/mysql/)

  • Triggers are three separate AFTER I/U/D triggers (MySQL has no WHEN clause); the bodies unconditionally insert the PK (capture_ddl.go). The UPDATE body's IF NOT (NEW.pk <=> OLD.pk) is only a PK-change detector, not an origin guard.
  • MySQL has no session_replication_role and no per-session trigger-skip. The apply path sets only FOREIGN_KEY_CHECKS (for cyclic components); no user variable, no origin marker.
  • Apply is blind source-wins: INSERT … ON DUPLICATE KEY UPDATE col = VALUES(col)
    • delete-to-match (apply.go, apply_tx.go). ON UPDATE CURRENT_TIMESTAMP columns are carried verbatim from the source (faithfulness) — which removes the one timestamp signal a naive LWW might have used.
  • Delta rows are PK-only + delta_id + rc_op + k1..kn + rc_at — no origin, no txid.
  • Constraints that matter later: before MySQL 5.7.2 a table allows only one trigger per (timing,event), so install blocks there if a non-replicare AFTER trigger exists (from 5.7.2 they coexist); InnoDB has no deferrable FKs; FOREIGN_KEY_CHECKS is session-level and non-transactional.

Redis capture/apply (internal/engine/redis/, internal/pipeline/delete.go)

  • Capture-less. Upserts = rolling SCAN + RESTORE … REPLACE (idempotent, value-faithful). Deletes = a target-vs-source keyspace diff: SCAN the target, EXISTS-check the source master, DEL every target key missing at the source (delete.go, stream.go MissingAtSource).
  • The transport record is {key, ttl, flags, dump} (copy.go, framing.go) — no origin, no version. The per-unit changeID is a local lag hint only; it never crosses nodes.
  • No conflict resolution, no origin, no loop suppression anywhere.

State / ownership (internal/state/)

  • Ownership is a per-sync Postgres advisory lock keyed by the sync name; cursors are namespaced (sync, target, table). Running A→B and B→A means two separate sync names → two independent locks → nothing coordinates the two directions.
  • HA / leader election is stubbed (a plain pg_try_advisory_lock; a second daemon is skipped with a warning), not implemented.

3. The fan-out building block (a prerequisite, not an alternative)

A single-writer fan-out — one authoritative primary → N read-only replicas (one source → N targets) — keeps N instances converged when all writes go to the primary. The owner requires symmetric active-active (writes on any node), so fan-out is not the endpoint here — but it is the building block the mesh is made of: each cluster node fans its local changes out to its N−1 peers. So hardening fan-out is the first substantive milestone (§8 phase 2), on the critical path.

Caveats it must close (they matter for the mesh too):

  • Fan-out is present, not hardened and untested end-to-end; initial-copy progress must be re-keyed to include the target before it is trustworthy for 2+ targets.
  • Nothing enforces "read-only" on a plain fan-out target — but in a cluster every node is intentionally writable, and correctness there comes from the version register + HLC-LWW (§5.3), not from read-only discipline.

4. Why naive bidirectional wiring breaks today

Wiring A→B and B→A (or a ring) with today's code does not "sort of work with lag" — it fails destructively:

  • Postgres / MySQL — infinite replication loop. A node in a bidirectional topology is both a source (has triggers) and a target (receives applied writes). Because no trigger distinguishes replicare's own apply writes, an applied write is re-captured and shipped back, forever. (PG has session_replication_role as a potential fix but it needs superuser, which the project forbids; MySQL has no equivalent at all.)
  • Redis — data loss + ping-pong. A key freshly written to B and not yet on A is, to the A→B delete sweep, "present on target, missing at source" → it is DEL'd on B before B→A can propagate it. New writes are indistinguishable from orphans. Upserts also ping-pong via repeated RESTORE … REPLACE.
  • All three — no conflict resolution. Concurrent writes to the same key/row on two nodes clobber each other non-deterministically (whichever apply pass runs last wins).

Because Config.Validate() does no cycle detection, a user can enter this configuration today and get silent corruption. The first concrete change this note recommends (independent of full multi-master) is to make the config layer detect and refuse an un-declared cycle — see §6 and §8.


5. Proposed design — the mechanisms multi-master needs

Three capabilities are missing everywhere and must be built. They are engine-neutral in concept, engine-specific in mechanism.

5.1 Node identity & replication origin

Every participating instance gets a stable node_id (configured, persisted with the daemon's state so it survives restarts). Every replicated change is associated with the node_id where it originated. This is the foundation for both loop suppression (5.2) and conflict attribution (5.3).

5.2 Loop suppression (origin filtering)

Shipped for Postgres (MM3). Implemented exactly as described below for the apply-suppression sub-problem (1). Concretely:

  • Origin-aware capture. Source.InstallOriginCapture installs the capture trigger with the WHEN (current_setting('replicare.apply', true) IS NULL) guard (internal/engine/postgres/capture_ddl.go). One-way InstallCapture installs the byte-identical unguarded trigger — the only difference is that one WHEN clause, asserted by a test — so the one-way path is untouched (§7).
  • Marked apply/copy. A cluster member's Sink is switched to origin marking (EnableOriginMarking); every apply and copy transaction then runs SET LOCAL replicare.apply = '1' (internal/engine/postgres/{apply,sink}.go). SET LOCAL is transaction-scoped, so the marker can never leak onto an ordinary connection. (The value is '1' in v1 — presence is all a full mesh needs; it becomes the origin id only when ring/forwarding lands, see below.)
  • Wiring. A clusters: block expands into one directed edge per ordered pair of members (full mesh); each edge runs as an independent single-active job in cluster mode — origin-aware capture on its source, marked sink — under its own ownership lock (internal/daemon/cluster.go). A config with no clusters runs exactly the one-way daemon.
  • Mesh initial copy. Because every member is simultaneously copied-from and written-to by its reciprocal edges, a member's live table transiently carries rows the other edge just applied; a direct COPY of those bounced rows would collide on the target PK. Cluster edges therefore copy with the idempotent merge path (INSERT … ON CONFLICT DO UPDATE, copy.WithLoadMode(LoadMerge)) and skip the DELETE-range resume. This makes a mesh converge on non-conflicting data (e.g. node-partitioned keys); resolving concurrent same-key writes is MM4 (§5.3).

Not yet shipped: the version register + HLC (§5.3), so cross-node writes are applied last-writer-by-arrival, not by (hlc, node). Same-key conflict resolution is MM4.

Two sub-problems:

  1. Don't re-capture replicare's own apply writes. When the daemon applies a change to a node, that node's capture must not treat it as a fresh local write.
  2. Don't forward a change back toward where it came from. In a ring/partial mesh a change is relayed hop-to-hop and must stop after one lap. In a full mesh with direct edges, (1) alone prevents loops (every node receives each change directly and never re-captures it); origin tagging is still needed for (3) and for rings.

Per-engine mechanism for (1):

  • Postgres. Add a WHEN (current_setting('replicare.apply', true) IS NULL) clause to the capture trigger. The apply connection does SET LOCAL replicare.apply = '<origin_node_id>' inside the apply transaction. Application writes (no GUC set) still capture; replicare's apply writes are skipped. This is a custom GUC (a namespaced replicare.* setting) — it needs no superuser and no privilege beyond what the daemon already has, unlike session_replication_role. (When origin forwarding is needed, the trigger instead records the GUC's value as the origin rather than skipping unconditionally — see below.)
  • MySQL. MySQL has no WHEN and no session_replication_role. Use a user session variable: the apply connection issues SET @replicare_apply = '<origin_node_id>' at BeginApply, and each of the three trigger bodies wraps its delta-insert in IF @replicare_apply IS NULL THEN … END IF. User variables are strictly connection-scoped; the apply pool is already MaxOpenConns=1 with a pinned connection, so the marker is isolated — but it must be reset on connection release, exactly as FOREIGN_KEY_CHECKS already is.
  • Redis. Capture-less, so there is nothing to suppress at "capture" time — the problem moves to the reconciliation passes (5.4). Loop suppression requires per-key origin+version metadata that a reconcile pass consults to decide "I already have the newest version; don't re-RESTORE, don't delete." Redis has no room in the value (DUMP/RESTORE is verbatim, and faithful transport forbids wrapping it), so this metadata lives in a parallel metadata keyspace (see 5.4).

v1 is full-mesh only, so sub-problem (2) does not arise — every node ships directly to every other, so skip-suppression alone prevents loops, and the version register (§5.3) is the correctness backstop: any change that is re-delivered but not newer is a no-op, so a stray loop is self-limiting (this is also what makes ring/partial topologies safe to add later). The source-side origin column and hop-to-hop forward-suppression (skip deltas whose origin is the destination; preserve origin across hops) are only needed to make ring/partial meshes efficient and are deferred post-v1 — which keeps the one-way delta table DDL literally unchanged in v1.

Bootstrapping a new member uses the same marker: the initial copy / reseed into a member runs with replicare.apply / @replicare_apply set (so its cold-copy writes are not self-captured into an echo storm) and seeds the version register from the source's register, so copied rows carry their true (hlc, node) rather than a fresh local stamp.

5.3 Conflict resolution — zero-config, replicare-managed

Shipped for Postgres (MM4), exactly as described below. Implementation:

  • Version register — a per-table reg_<hash> table in the source replicare schema keyed by the primary key, holding (rc_hlc_phys, rc_hlc_log, rc_node, rc_deleted); named by a stable hash of schema.table so every member computes the same name (internal/engine/postgres/mesh.go). Mesh-only — a one-way source has no register.
  • HLC — a single-row replicare.hlc_state (physical, logical, node_id) with replicare.hlc_tick() (advance for a local write) and replicare.hlc_observe() (advance past an incoming version on apply). The cluster capture trigger stamps the register with a fresh tick on every local change; a delete (and the old key of a PK-change) writes a tombstone (rc_deleted=true).
  • Version-guarded apply — the re-read carries each row's version (rereadVersioned), and apply upserts the value + register, or deletes + tombstones, only when the incoming (hlc, node) strictly beats the target register (internal/engine/postgres/apply_mesh.go); otherwise a no-op. The target HLC is then advanced past the max applied version.
  • GC — a tombstone is reclaimed once its delete's delta has been consumed by every peer (the delta is purged), reusing the consume+purge machinery as the distributed watermark (GCTombstones, run each streaming pass).

Not yet: seeding the target register from the source's during the initial COPY (the bootstrap-vs-concurrent caveat in §8); MySQL/Redis (MM5–MM6).

replicare deliberately has no commit-order signal (the entire trigger-CDC premise), so it cannot do true causal ordering. And a hard product requirement: conflict resolution must need no user schema change and no user-specified column. The primary key cannot be the tiebreaker — it is the conflict key (two colliding writes share it), so it says which rows conflict, never which value wins. Resolution needs something ordered, and we will not ask the user to add it.

So replicare maintains the version itself, invisibly: for every replicated row/key it keeps a hidden record PK → (hlc, origin_node) in its own replicare schema — a side "version register" table (Postgres/MySQL) or the metadata keyspace (Redis), never a column on the user's tables. hlc is a hybrid logical clock (max(physical_now, last_seen_hlc) + logical_tick): monotonic, wall-clock-tracking, causality-respecting. Every change is stamped (hlc, node_id); the node_id breaks equal-HLC ties, so the order is total, with no ties.

  • The one policy is HLC last-write-wins, zero-config. On apply of an incoming change for PK, compare (hlc_in, node_in) against the register's stored (hlc, node): apply the value and update the register iff strictly greater, else no-op; then advance the local HLC past hlc_in. This is a LWW-register CRDT (a join-semilattice max) → it converges for any number of nodes regardless of message order.
  • No priority-fencing, no user version column, no custom resolver in v1. The policy interface stays open for a future custom resolver, but v1 ships exactly this default and there is nothing to configure beyond declaring cluster membership.

Cost, stated honestly: the register is ~one small row per replicated row (PK + ~16 bytes) — the size of an extra index. That is the price of touching none of the user's schema (the rejected alternative was a version column on user tables).

Deletes are tombstones carrying (hlc, node_id) in the same register/keyspace, so a delete competes with a concurrent update under the same total order. Tombstones and version records for deleted keys are GC'd once every peer has observed a version ≥ them (a per-cluster min-observed watermark across peers' cursors), keeping the source footprint bounded.

5.4 Redis specifics (the hardest engine) — SHIPPED (MM6)

Redis needed the most new machinery because it is capture-less and value-opaque. As implemented (internal/engine/redis/mesh*.go), gated entirely on cluster mode so one-way Redis is byte-for-byte unchanged:

  • Metadata keyspace (the version register). For each replicated data key K a sibling metadata hash \x00rc:m:{<K's slot tag>}K holds {p, l, n, d, v} = (hlc phys, hlc log, origin node, deleted flag, vhash). The reserved control-char prefix keeps it out of the user keyspace and it is excluded from every data operation (copy, verify, reconcile) at one choke point (selection.match). The value key is never wrapped (faithful transport holds); the metadata co-locates in K's cluster slot (a CRC16-derived hash tag) so value + register are written together in one atomic Lua.
  • Capture-less "capture" (loop suppression + LWW in one comparison). There is no trigger. The version-aware re-read (ClusterReadSource) detects a local write by comparing each key's current logical-value fingerprint (vhash, type-aware reads — never DUMP bytes, so it is version-gap-stable) against the stored register; a change stamps a fresh (hlc, node) off the key's own stored version (per-key monotonic, so no shared clock is needed). Apply (OriginMarkingSink) RESTOREs/DELs only when the incoming (hlc, node) strictly beats the local register — the same comparison is both loop suppression (an echoed write already has a version the peer holds → loses) and HLC-LWW.
  • Tombstone-based deletion replaces the stateless diff. The one-way "missing at source ⇒ DEL" sweep is disabled in cluster mode (a key absent at one source but present here is a legitimate peer write, not an orphan). A local delete is detected by scanning the metadata keyspace for a live entry whose data key is gone; it stamps a tombstone that propagates and version-guard-deletes on peers. Tombstones are GC'd by age (TombstoneGC, default 24h — a peer lagged beyond that is a reseed case, MM7).
  • Conflict resolution on opaque bytes. LWW uses the register (hlc, node); the value is never merged. Sub-key merges (hash-field CRDTs, à la Redis Enterprise Active-Active) are out of scope.

Bootstrap. No separate protocol: capture-first + idempotent versioned apply reconciles the mesh, so the first streaming pass lazily seeds the register (a key with no metadata is treated as a fresh local write) without spurious tombstones (no pre-existing register to misread), mirroring Postgres's deferred copy-time seeding. A consequence, documented: mesh formation over two nodes that already hold divergent data converges each key to one value under LWW, and a delete made before a node joined the mesh is not preserved (there is no pre-mesh history to tombstone from).

Cost, honestly: the metadata keyspace ~doubles key count, and the change-detector adds a logical value read per key per reconcile pass (big keys are already DUMP-and-warn). A same-version DUMP-hash fast path is a documented future optimization.

5.5 Ownership, cursors, topology expansion

  • A mesh expands to a set of directed edges; each edge keeps its own cursor namespace (extend the existing (sync, target, table) keying with the mesh/edge and origin). The per-sync advisory lock generalizes to a per-edge (or per-mesh-member) lock.
  • HA/leader-election is still deferred, but multi-master raises the stakes (a split-brain daemon in a mesh is worse than in one-way), so the ownership interface must be ready for pg_advisory_lock-based leader election before mesh is declared production-ready.

5.6 Active-active id allocation — globally-unique keys

Conflict resolution (§5.3) decides which value wins when two nodes write the same key. It says nothing about how two nodes avoid minting the same key for different rows in the first place — that is a separate, upstream problem, and in a mesh it is the user's schema's job, not replicare's. This section is the decided guidance; a pre-flight guardrail enforces it (§6.3 item 3 — replicare refuses to bring up a mesh member whose replicated tables use a locally-allocated integer key).

The blessed strategy: globally-unique keys — UUID v7 / ULID. Every node mints its own keys with zero coordination — no sequence to sync, no node windows, no interleave, no engine asymmetry — and replicare moves them verbatim (faithful transport already handles a UUID). Id allocation belongs upstream of replication, and UUIDs put the fix exactly there, keeping replicare a pure replicator with nothing to build on the write path.

Do NOT try to sync integer counters across a mesh. This is the single most dangerous anti-pattern here, and it is why reseed-sequences refuses a cluster member and why sync_sequences is one-way only. In a converged mesh every node holds the union of all rows, so max(id) is the same global maximum everywhere; setting every node's counter to max(id)+1 collapses them onto one shared counter. Two nodes then both allocate global_max+1, insert different rows under the same id, replicate, and HLC-LWW keeps one and silently discards the other — the cardinal sin (silent data loss), delivered by an operation that exits 0. On a correctly partitioned mesh it is worse: it yanks a node out of its disjoint window into another's space, destroying the very invariant that was keeping the cluster collision-free. replicare cannot de-collide two independently minted ids after the fact — allocation happened before capture ever saw the row — so the only safe fix is to not collide in the first place.

Fine print — carry these into any "we'll use UUIDs" decision:

  • It solves allocation, not conflict resolution. UUIDs guarantee two nodes never mint the same key for different rows; two concurrent updates to the same row are still HLC-LWW's job (§5.3). UUIDs retire one of the two mesh hazards, not both.
  • It is a schema migration. Conceptually "one move", but on a live system it means every surrogate-int PK → UUID, every referencing FK, a reindex, and app code that assumes integer ids. Greenfield: trivial. An established production schema: a real project — plan it as one.
  • Use time-ordered UUID v7 / ULID, stored binary — uuid on Postgres; BINARY(16) via UUID_TO_BIN(…, 1) on MySQL. Random v4 as a PK scatters inserts (on MySQL/InnoDB the PK is the clustered index, so v4 causes page splits and write amplification); v7/ULID restore insert locality, and binary storage halves the width of the 36-char text form.
  • It must cover every locally-allocated unique value, not just the PK. A secondary UNIQUE column fed by a sequence (a human-facing order_number, say) still collides across nodes even with a UUID PK. "We use UUIDs" has to mean every locally-allocated unique value — which is exactly what the §6.3 guardrail checks.

Why not any of the counter schemes real databases use? Production active-active systems never sync counters — they make allocations disjoint or globally-unique: interleaved sequences (offset + increment = N; MySQL Group Replication, Galera), coordinated block allocation (Postgres BDR galloc, CockroachDB per-node blocks — but the app must allocate through the coordinator), or globally-unique-by-construction (snowflake ids, unique_rowid(), UUID/ULID). The first two either need an engine feature replicare does not control or push allocation into the app's write path; the third is the one that needs nothing from replicare and no coordination at all. That is why UUID/ULID is the blessed answer and not merely an option.

A transparent integer-PK auto-config (replicare assigning node-disjoint sequence windows on Postgres) is a deferred, lower-priority fallback for schemas that genuinely cannot move to UUIDs; it is Postgres-only in practice (MySQL's AUTO_INCREMENT floats to the global max and cannot be pinned to a low window) and is out of scope here. For now, replicare's only mesh-side obligation is to police the chosen strategy (§6.3), never to allocate.


6. Config-schema changes

Design principle: additive and opt-in. Existing configs (no new keys) must be byte-for-byte valid and behave identically. Multi-master is expressed by new, optional constructs; the existing sources / targets / syncs schema is untouched, and a config with no clusters: block is exactly today's one-way daemon.

Naming note. A replicare clusters: entry is a set of active-active peer nodes — do not confuse it with a Redis endpoint's mode: cluster (one sharded Redis). Different scopes; the docs disambiguate.

6.1 New: a nodes: map + node_id (optional)

Cluster members are endpoints that are both read from and written to, so they live in their own top-level nodes: map (separate from sources:/targets:, which stay one-way-only). A node has the same shape as any endpoint (an engine + that engine's connection block) plus an optional node_id — its stable replication-origin identity, stamped into every change's (hlc, node_id) version. node_id defaults to the node's map key, so it is rarely written explicitly.

nodes:
  us:
    engine: postgres
    # node_id defaults to "us"; override only if you need a different origin id
    postgres: { host: pg-us, port: 5432, database: app, user: replicare, password: ${PW}, sslmode: require }
  eu:
    engine: postgres
    postgres: { host: pg-eu, port: 5432, database: app, user: replicare, password: ${PW}, sslmode: require }

nodes: is optional and ignored on the one-way path.

6.2 New: a clusters: block (optional, top-level)

A cluster names its member nodes (by key into nodes:), the topology, and the selection/tuning — mirroring a sync but multi-directional. There is no conflict policy to configure: resolution is the zero-config, replicare-managed HLC last-write-wins of §5.3 (no version column, no priority). The daemon expands the cluster into the full set of directed edges internally.

# Existing one-way syncs keep working, unchanged, alongside clusters.
syncs:
  - name: analytics-fanout
    source: pgprod              # a `sources:` key, as always
    targets: [warehouse]
    include: ["public.*"]

clusters:                        # NEW, entirely optional
  - name: global-app
    engine: postgres             # single-engine, like a sync
    members: [us, eu, ap]        # keys into nodes:; each is source AND target; any N >= 2
    topology: mesh               # v1: full mesh (ring/partial deferred)
    include: ["public.*"]
    exclude: ["*_audit"]
    tuning: { drain_interval: 1s }
    # No conflict block: HLC-LWW is automatic (§5.3). Nothing to declare.

Notes:

  • members reference endpoint definitions; each must carry a node_id and be reachable as both read (capture+snapshot) and write (apply). Any N ≥ 2 — no 3-node limit.
  • topology: mesh is the only v1 value (full mesh, N·(N−1) edges); ring/explicit (partial) are deferred with the origin-column forwarding work (§5.2).
  • The engine registry still owns per-engine connection parsing; clusters adds only neutral wiring, consistent with the sync model.

6.3 New validation (and a safety fix for the one-way path)

Config.Validate() gains:

  1. Cluster validation — single-engine members; every member has a node_id; topology: mesh only (v1); no member endpoint reused in a conflicting plain sync. (No conflict-policy validation — there is no policy to configure.) Engine-capability guard: validateCluster (internal/config/config.go) rejects a cluster on any engine that can't do active-active, at config-load time with an actionable error, rather than letting it fail opaquely later at capture install. All three shipped engines (Postgres, MySQL, Redis) now qualify, so the blocklist is empty; the guard is retained for a future engine that cannot do active-active.
  2. Cycle detection for plain syncs — refuse an un-declared cycle among one-way syncs (A→B + B→A, or a ring) not part of a clusters: block. Closes the silent-corruption footgun in §4, worth doing independently. It only adds a rejection for a config already broken today, so no valid one-way config is affected.
  3. Mesh id-allocation policing (SHIPPED — introspection-based pre-flight). At cluster bring-up, replicare introspects every replicated table's primary key and every unique key and classifies each locally-allocated integer key (serial / identity / AUTO_INCREMENT). A key that is not locally DB-allocated — a UUID/text/natural key, or an app-assigned globally-unique integer where the DB does not generate the value — is allowed (those are the collision-free schemes). A locally-allocated integer PK or secondary unique key → refuse to start, loudly, naming the offending table/column and pointing at the UUID/ULID guidance (§5.6), with narrow (32-bit-or-smaller) keys flagged especially. This converts the silent data-loss setup (§5.6) into a loud startup refusal — the same "block on incompatible" philosophy as the type pre-flight. The classifier is pure over the introspected schema (engine.MeshIDAllocationFindings, internal/engine/preflight_mesh.go) → engine-neutral and CI-tested; the daemon runs it only on a cluster edge (internal/daemon/build.go), so one-way syncs are unaffected. Postgres serial (a plain int with a nextval(…) default, not reported as identity) is caught via a dedicated Column.DefaultSequence introspection flag, kept distinct from Identity so the apply path's OVERRIDING SYSTEM VALUE is unchanged. This is the whole active-active id-allocation code deliverable; replicare polices the strategy, it never allocates or syncs counters. (replicare validate does not yet walk clusters:, so the refusal currently surfaces at run/bring-up, not in validate — a documented follow-up.) Planned — continuous re-check: the classifier runs only at bring-up today, so a mesh-unsafe key introduced by a mid-stream migration (a new table, or a new auto-counting column added to a running cluster) is not caught until the next restart. The plan is to run the same pure classifier on a schedule and on demand, so a risky change is flagged when it ships rather than at the next restart. This is the continuous check referenced in active-active.md Rule 1.

6.4 State-store & source-schema changes

  • The one-way delta table is unchanged — v1 adds no column to it (a full mesh needs no origin column; §5.2). The version register (PK → (hlc, node_id, deleted?)) and tombstones are new, mesh-only tables in the replicare schema, created only for cluster tables. Applied via the existing in-place, idempotent migration runner with a schema_version bump that preserves in-flight deltas/cursors — never drop-recreate.
  • Cursors extend their key with cluster/edge for cluster members; the existing (sync, target, table) shape is unchanged for one-way.
  • Redis adds the metadata keyspace (the register + tombstones, §5.4); nothing changes for one-way Redis.

7. Backward compatibility — the non-negotiable

Every change above is designed so the one-way path is provably unchanged. The invariants:

  1. Config. No clusters: and no node_id: ⇒ identical parse and identical behaviour. New keys are optional; strict parsing still rejects genuine typos. The only new rejection is an un-declared cycle among one-way syncs — a config that is already corrupt today, never a working one.
  2. Triggers (PG/MySQL). The origin guard is a no-op for one-way:
    • A one-way source is never applied to by replicare, so the apply marker (replicare.apply GUC / @replicare_apply) is never set on it → local writes are captured exactly as today.
    • A one-way target has no capture triggers (targets aren't sources), so the guard is irrelevant there.
    • The guard defaults to "capture" when the marker is unset, so any non-replicare write is still captured. Origin-aware triggers are only installed for tables that participate in a cluster; one-way tables keep today's trigger DDL.
  3. Apply path. The apply marker + HLC-LWW version comparison are only engaged for cluster edges. One-way apply remains blind source-wins overwrite + delete-to-match, unchanged, and writes no version register.
  4. Delta schema. v1 adds no column to the delta table — the version register and tombstones are separate, mesh-only tables. One-way delta DDL is literally unchanged; migrations preserve in-flight state.
  5. Redis. The metadata keyspace, tombstones, and delete-diff redesign are gated to cluster mode. One-way Redis keeps the stateless SCAN-reconcile + target-vs-source delete sweep verbatim.
  6. Tests. The entire existing one-way integration suite (PG/MySQL/Redis, plus the test/loadgen and test/loadgen-redis convergence harnesses) must pass unchanged. Multi-master gets its own harness (an N-node cluster convergence + conflict test) rather than modifying the one-way tests.

Rollout is feature-flagged by the presence of a clusters: block: a daemon with none compiles and runs exactly as before.


8. Implementation status

MilestoneScopeStatus
MM0–MM1nodes:/clusters: config surface; one-way cycle guardrail; fan-out per-target progressShipped
MM2Per-target initial-copy progress (fan-out hardening)Shipped
MM3Postgres loop suppression — origin-aware capture, marked apply/copy, cluster→edge wiring, idempotent mesh copyShipped
MM4Postgres HLC-LWW — version register + HLC, version-guarded apply, tombstones, GCShipped
MM5MySQL mesh — mirror of MM3+MM4 for MySQL (@replicare_apply guard, inline HLC, version-guarded apply, GC)Shipped
MM6Redis mesh — metadata-keyspace version register, version-guarded Lua apply, tombstone deletes, age-based GC, slot co-location, config guard lifted (§5.4)Shipped
Id-alloc guardrailPre-flight refusal of a mesh member whose replicated tables use a locally-allocated integer PK/secondary-unique key — the §5.6 policing (§6.3 item 3), pure+CI-tested classifier run at cluster bring-upShipped
MM7–MM11Cluster retention/reseed, HA, observability, E2E gate, releaseNot started

What works today (Postgres, MySQL, AND Redis): define members under nodes:, group them in a clusters: block, and the daemon runs a full-mesh active-active cluster — writes accepted on any node converge on all, initial copy is bidirectional and idempotent, replicare's own applies are not re-captured and echoed (loop suppression), and concurrent writes to the same key converge to the same value on every node under HLC last-write-wins, with deletes resolved by GC'd tombstones. Proven end-to-end for Postgres by internal/daemon.TestDaemonTwoNodeMeshConverges + TestDaemonMeshSameKeyConflictConverges, for MySQL by internal/daemon.TestDaemonMySQLMeshSameKeyConflictConverges, and for Redis by the engine-level mesh suite internal/engine/redis.TestMesh* (disjoint-key + same-key conflict convergence, delete propagation, and the peer-write-not-destroyed safety property), which drive the exact neutral apply path (DrainComponent); Redis correctness runs on the local gate (REPLICARE_REDIS=1), not the Postgres-only CI. Pure mesh logic (slot hashing, version codec, LWW order, HLC) is unit-tested in CI (internal/engine/redis.TestSlotTag, TestCRC16KeySlot, TestVersionBeats, TestHLC*, TestFramingMesh*).

MySQL specifics (MM5). The design mirrors Postgres; three things differ because the engine forces them:

  • Loop suppression uses a @replicare_apply session user variable (MySQL has no trigger WHEN clause): each of the three trigger bodies is wrapped in IF @replicare_apply IS NULL THEN … END IF. The variable is CONNECTION-scoped (not transaction-scoped like Postgres SET LOCAL), so it is reset on connection release alongside FOREIGN_KEY_CHECKS (the apply pins one connection, MaxOpenConns=1), and set + reset around the pinned bootstrap-copy connection.
  • HLC is a table (replicare.hlc_state) + inline SQL (a tick in the trigger body, an observe in the apply), NOT a stored function — a data-modifying MySQL function needs SUPER/CREATE ROUTINE under the managed-provider default, which would break the no-superuser mandate.
  • Version-guarded apply cannot read the register inside an INSERT … SELECT that also targets it (MySQL error 1093), so the winner set is computed once into a staging column and every value/register statement reads that flag.

How MM4 resolves conflicts. For every replicated row the source's capture trigger stamps a hidden version register PK → (hlc, node_id[, tombstone]) in the replicare schema — no user column. hlc is a hybrid logical clock (replicare.hlc_tick()), advanced past every incoming version on apply (replicare.hlc_observe()), so it tracks wall-clock and tolerates skew. The re-read carries each row's version; apply upserts (or, for a tombstone, deletes) only when the incoming (hlc, node) strictly beats the target register, so a stale write arriving out of order loses and a delete competes with a concurrent update under one total order (node_id breaks equal-HLC ties → no ties). This is a last-write-wins register CRDT: it converges for any N regardless of message order. Tombstones are reclaimed once their delete has been consumed by every peer (the delete's delta is purged), so the register stays bounded.

Bootstrap-vs-conflict caveat. The register is stamped on local writes and updated on apply, but the initial COPY does not yet seed the target register from the source's (a small follow-up). So a value placed purely by initial copy, never re-written and never applied-over, has no local register entry until it is next written or received — a remote write for such a key always wins even if the copied value was newer. This only affects a key written concurrently during a member's initial copy; steady-state and post-bootstrap same-key conflicts resolve correctly (each side stamps its own write).

8.1 Usage — running a Postgres active-active cluster

Define each member under nodes: and group them in a clusters: block. The target schema must pre-exist on every member (data-only), and the daemon needs a Postgres state_store (it may live on any member or a separate DB):

state_store:
  engine: postgres
  postgres: { host: pg-us, port: 5432, database: replicare_state, user: replicare, password: ${PW}, sslmode: require }

nodes:
  us:
    engine: postgres
    postgres: { host: pg-us, port: 5432, database: app, user: replicare, password: ${PW}, sslmode: require }
  eu:
    engine: postgres
    postgres: { host: pg-eu, port: 5432, database: app, user: replicare, password: ${PW}, sslmode: require }

clusters:
  - name: global-app
    engine: postgres
    members: [us, eu]        # any N >= 2 (full mesh)
    include: ["public.*"]
    tuning: { drain_interval: 1s }
  • Grants (no superuser). Each member needs the combined cluster-member grant set: the source-side grants (TRIGGER + SELECT on replicated tables, CREATE/USAGE on the replicare schema — or pre-create it owned by the daemon role) and the target-side grants (INSERT/UPDATE/DELETE on the replicated tables), because every member is both. No superuser, no REPLICATION, no wal_level change.
  • What the daemon does. It expands the cluster into one directed edge per ordered pair of members and runs each edge as an independent single-active job: origin-aware capture on the edge's source, a marked (loop-suppressing) sink on its target, and HLC-LWW version-guarded apply. Each edge takes its own ownership lock, so several daemon replicas can share a cluster's edges.
  • Conflict resolution is automatic. Writes may land on any node, including the same key concurrently — HLC last-write-wins converges every node to the same value, with no config and no user schema change (see §5.3). The only remaining caveat is the bootstrap-vs-concurrent edge documented above (initial-copy register seeding is a follow-up).

9. Phased plan

Ordered so each phase is independently shippable and low-risk-first:

  1. Cycle-detection guardrail (small, do first). Make Config.Validate() refuse an un-declared one-way cycle. Pure safety, no behaviour change for valid configs.
  2. Harden fan-out (small–medium; a mesh prerequisite). Re-key initial-copy progress by target; add a 2+-target test. Each cluster node fans out to its N−1 peers, so this is on the critical path (and independently delivers single-writer fan-out).
  3. Postgres loop suppression + version register + bootstrap (medium). node_id; the HLC + version register in the replicare schema; replicare.apply GUC + trigger WHEN guard; apply and cluster-bootstrap copy set the marker and seed the register. Prove no-loop + no-echo on a 2-node PG mesh, no superuser.
  4. Postgres HLC-LWW + tombstones + GC (medium). Version-guarded upsert over (hlc, node_id); tombstones; coordinated GC. N-node PG convergence + conflict harness.
  5. MySQL mesh (medium). Mirror 3–4 with the @replicare_apply user-variable guard in all three trigger bodies; reset-on-release alongside FOREIGN_KEY_CHECKS.
  6. Redis mesh (large / highest-risk, but co-equal — NOT deferred). Metadata keyspace as the version register + tombstones; tombstone/version-aware delete reconciliation; HLC-LWW via the metadata (hlc, node). Sequenced after 4 (reuses its abstractions).
  7. Cluster lifecycle (retention/reseed). Reseed runs marked, seeds the register, and never resurrects a tombstoned key.
  8. HA / leader election (before "production-ready"). Per-edge pg_advisory_lock leader election + cursor fencing, so a cluster can't split-brain.

10. Open questions / risks

  • Version-register cost. ~1 row per replicated row (PK + (hlc, node_id)) — the accepted price of zero user-schema change (the rejected alternative was a version column on user tables). Index-sized but non-trivial for very large tables; measured as milestone acceptance.
  • HLC clock skew. The node_id tie in (hlc, node_id) guarantees a total order, so nodes always converge to the same winner regardless of skew; skew only affects which of two truly-concurrent writes is deemed "latest" (a bounded wobble, never divergence). Document the assumed max skew and surface an HLC-skew metric.
  • Tombstone/register GC coordination. A per-cluster min-observed-version watermark across peers (no global coordinator) — sub-design before the GC milestone ships. Live keys keep their register row; only tombstones + deleted-key records are collected.
  • Redis metadata atomicity & cost. A metadata entry per key ~doubles key count and needs Lua/MULTI atomicity; big-key and cluster-slot interactions need care (the metadata key must hash-tag to its value key's slot). The single biggest risk item.
  • Least-privilege, everywhere. The PG replicare.apply GUC and MySQL user-variable guards are grantable (no superuser); re-verified against managed offerings (RDS/Cloud SQL/etc.) as milestone acceptance, and the grants docs updated. No superuser is required anywhere.
  • Schema drift across members. One-way assumes the target schema pre-exists; a cluster assumes all members share a compatible schema. Pre-flight must check every member pair, not just source→target.
  • Full-mesh edge scaling. N·(N−1) directed edges — fine for small-to-moderate N; beyond ~tens of nodes a hub/ring (the deferred forwarding work) would be preferable.
  • Faithful transport is preserved — none of this transforms values; the HLC version/origin are metadata about a change, never a mutation of the replicated value.

11. Summary

CapabilityPG todayMySQL todayRedis todayNeeded for multi-master
Loop suppression✗ (no trigger guard)✗ (no guard, no session_replication_role)✗ (re-RESTORE loop)replicare.apply GUC guard / @replicare_apply var guard + version-comparison backstop
Conflict resolution✗ (blind overwrite)✗ (blind overwrite)✗ (blind RESTORE)replicare-managed HLC-LWW over (hlc, node_id) — zero user schema change; tombstones for deletes
Version register / origin✗✗✗node_id + a hidden PK → (hlc, node) register in the replicare schema / Redis metadata keyspace
Topology / cycle safety✗ (silent cycles)✗✗clusters: block (full mesh, any N) + cycle-detection validation
Delete handling in a clusterdelete-to-matchdelete-to-matchdestructive difftombstone-based, (hlc, node)-aware, GC'd
One-way path✓✓✓must remain unchanged (§7)

Multi-master is a substantial, multi-phase project across all three engines (Redis included, not deferred) — heaviest on Redis. Conflict resolution is zero-config (replicare-managed HLC-LWW; no user column, no schema change, clusters: just declares membership). The one-way, source-authoritative path stays the default and is protected by the §7 invariants throughout. If a single writer happens to suffice, hardening fan-out (§3, phase 2) delivers "same data in N clouds" even sooner — but it is a prerequisite of the mesh, not an alternative to it.

Delta purge, retention & the reseed state machine

Status: implemented and shipped. This began as the design artifact for the purge/retention/reseed work and is now the reference for the code that ships it: internal/reseed (the state machine) and internal/pipeline/stream.go (the daemon loop that enforces retention and drives reseeds every pass). It specifies the purge mechanics, the retention caps, and — the load-bearing part — the reseed state machine, with a stated invariant for why no source change is lost across the handoff.

This re-derives the two guarantees M5c has to hold together: capture-first initial copy and crash-safe at-least-once consumption.


1. What M5c protects against

The source-side delta table is a high-churn queue we write to a database we may not own. A slow or down target pins it: a delta row cannot be purged until every target has consumed it, so one laggard makes the source grow without bound. M5c bounds that growth in three layers:

  1. Consumption-gated purge — the baseline: a delta consumed by all targets is deleted. Batched + rate-limited; paired with the aggressive per-table autovacuum already set at capture install.
  2. Bounded retention (age + size) — a cap on how far the queue may grow because of a laggard. When a target pushes the queue past the cap, that target is sacrificed rather than the source.
  3. Forced reseed — the sacrificed target is marked needs_reseed, its pinning deltas are dropped, and it is brought back to convergence by a fresh full copy + streaming resume. This is the state machine specified in §4.

The xmin horizon is the residual risk the batched-DELETE path cannot beat on old sources: autovacuum cannot reclaim dead tuples older than the source's oldest running transaction, so a long idle-in-transaction session makes the delta table bloat regardless of purge. M5c cannot fix that on an old source — it surfaces it loudly (telemetry) and still bounds logical growth via retention/reseed. On PG ≥ 10 the partition+DROP path avoids the trap; see §5.


2. Objects involved (grounding)

Per captured source table (keyed by a stable rel_id, in the source replicare schema):

  • replicare.delta_<rel_id> — the dirty-key set. Columns: delta_id bigserial PK, rc_op, key columns k1..kn, and rc_txid / rc_seq / rc_at (lag/ordering hints only, never a resume cursor).
  • replicare.track_<rel_id> — per (target, delta_id) consumption record. A row means that target has applied that delta. PK (target, delta_id).

Daemon state (the StateStore, a separate Postgres schema, not to be confused with the source-side tables):

  • cursors(sync, target, table) carries phase (initial_copy / streaming), last_delta (lag only), and needs_reseed — the flag M5c drives.
  • copy_progress(sync, table) — the resumable initial-copy watermark.

Separation of concerns: Source.Purge touches only source objects (delta/track). It never writes the StateStore. It returns which targets it sacrificed; the engine-neutral orchestration (internal/reseed) is what writes needs_reseed, records events, and runs the re-copy. This keeps the delta lifecycle self-contained on the source and the daemon state on the StateStore.


3. Purge mechanics

Source.Purge(ctx, table, targets, retention) → (PurgeStats, error)

targets is the full configured target set for the sync — required because "consumed by all targets" is only decidable against the complete set (a target that has consumed nothing has no track rows, yet still pins the queue).

3.1 Consumption-gated batched purge (baseline, all versions)

A delta is purgeable once every configured target has a track row for it. Batched by delta_id to bound transaction churn, looped until a pass deletes < batch:

WITH purgeable AS (
    SELECT d.delta_id
    FROM   replicare.delta_<rel> d
    WHERE  (SELECT count(*) FROM replicare.track_<rel> tr
            WHERE tr.delta_id = d.delta_id AND tr.target = ANY($targets)) = $ntargets
    ORDER  BY d.delta_id
    LIMIT  $batch
)
DELETE FROM replicare.delta_<rel>
WHERE  delta_id IN (SELECT delta_id FROM purgeable);

count(*) is exact because track PK is (target, delta_id) — each target tracks a delta_id at most once, so the count is the number of distinct targets that consumed it. When it equals $ntargets, all have. Deletes are always by delta_id (never by PK), preserving the read-and-clear discipline.

3.2 Retention evaluation (per target)

For each target T, its pinned backlog is delta MINUS track_T. From it:

  • oldest-unconsumed age = now() - min(rc_at) over the backlog. (rc_at is a hint, never a cursor — it is fine as a retention signal because we only ever compare it to a wall-clock cap, never resume from it.)
  • size signal = pg_total_relation_size('replicare.delta_<rel>') — the actual on-disk footprint of the queue, which is what "protect the source" means. Attributed to the laggard: the target with the oldest unconsumed delta.

T is over-cap when MaxAge > 0 && age > MaxAge, or MaxBytes > 0 && size > MaxBytes && T pins the oldest delta. Both caps are opt-outable (0 = unbounded); age defaults on (24h), size defaults off.

3.3 Retention-forced purge (sacrifice the laggard)

When a set R of targets is over-cap, Purge, in one pass:

  1. Records R in PurgeStats.TargetsReseeded (the orchestration will mark them needs_reseed).
  2. Resets each R's track: DELETE FROM track_<rel> WHERE target = ANY($R). A reseeded target is about to be re-derived from scratch, so its partial consumption record is void. (We reset to empty, never to a high-water mark — a fast-forward would re-introduce the commit-order hazard.)
  3. Purges the now-unpinned deltas — those consumed by all remaining (non-reseeding) targets, treating R as already satisfied:
WITH purgeable AS (
    SELECT d.delta_id
    FROM   replicare.delta_<rel> d
    WHERE  (SELECT count(*) FROM replicare.track_<rel> tr
            WHERE tr.delta_id = d.delta_id AND tr.target = ANY($remaining)) = $nremaining
    ORDER  BY d.delta_id
    LIMIT  $batch
)
DELETE FROM replicare.delta_<rel>
WHERE  delta_id IN (SELECT delta_id FROM purgeable);

For the single-target case (the v1 common path) remaining is empty and $nremaining = 0, so count(*) = 0 matches every row → the queue drains fully. That is exactly the intended behaviour: the one laggard is being reseeded, so all its deltas are discardable and the source is fully unpinned.

Why dropping R's unconsumed deltas is safe: because R gets a full re-copy of current source state. A delta is a pointer to "PK X changed"; re-copy re-reads X's current value regardless. Discarding the pointer loses nothing the re-copy does not re-derive. See the invariant in §4.3.

PurgeStats{ DeltasPurged, TargetsReseeded } is returned for telemetry (replicare_delta_purged_total, replicare_reseed_total) and orchestration.


4. The reseed state machine

Driven by internal/reseed over the Source / Sink / StateStore interfaces, reusing the M4 copy driver and the M5a/M5b drain. Per reseeding target T, over the tables of its FK component:

        ┌─────────────┐
        │  STREAMING  │  (normal drain: delta MINUS track_T)
        └──────┬──────┘
               │ Purge reports T over-cap  (source unpinned, track_T reset)
               ▼
        ┌─────────────┐
        │  MARKED     │  StateStore: cursor.needs_reseed=true, phase=initial_copy
        │             │  event target.needs_reseed (ERROR); metric reseed_total++
        └──────┬──────┘
               │ reset copy_progress(T-tables) to fresh
               ▼
        ┌─────────────┐
        │  RECOPYING  │  per table: DeleteRange(nil,nil) → copy.Component (LoadDirect)
        │             │  checkpointed by copy_progress watermark
        └──────┬──────┘
               │ copy done for the component
               ▼
        ┌─────────────┐
        │  RESUMING   │  drain delta MINUS track_T with track_T EMPTY:
        │             │  re-consume every retained delta (idempotent re-read+upsert)
        └──────┬──────┘
               │ backlog drained; phase=streaming; needs_reseed=false
               ▼
        ┌─────────────┐
        │  STREAMING  │
        └─────────────┘

4.1 Why DeleteRange(nil,nil) before re-copy

The reseed target is non-empty (stale data). A fresh copy_progress (Watermark=nil) makes the copy driver plan every chunk and LoadDirect (COPY FROM) — which would collide with existing rows and, worse, leave orphans for source rows deleted since the last copy. So reseed first clears the table with DeleteRange(ctx, ref, nil, nil) (predicate TRUE, i.e. delete-all — within the DML-only target grant, no TRUNCATE privilege needed), then copies into the now- empty table. This re-derives target = current source exactly, deletes included.

4.2 Why track stays empty through RESUMING

After re-copy, track_T is empty, so the resume drain re-consumes all currently-retained deltas. Each is an idempotent re-read of the key's current value + upsert (self-healing) — re-applying a delta the copy already covered is a no-op-shaped upsert, never corruption. We deliberately do not fast- forward the track: emptiness is both correct (idempotent) and hazard-free (no commit-order skew). It re-consumes redundant work once; convergence is exact.

4.3 Invariant — no source change is lost across a reseed

**Capture is never uninstalled during reseed, and initial copy is capture-first

  • idempotent; therefore for every key the target ends at the key's current source value, regardless of when changes landed relative to the handoff.**

Proof sketch. Triggers stay installed for the whole machine, so every source change during reseed enqueues a fresh delta_id. Take any key X and any change C to it during the reseed window. C reaches the target by at least one path:

  • (a) copy path — if C committed before RECOPYING read X's chunk, the re-copy loaded X's current value directly.
  • (b) delta path — C left a delta row (retained; not purged, since the §3.3 purge only removes rows consumed by remaining targets, and track_T is empty during resume). RESUMING re-reads X's current value and upserts.

Because re-read always fetches the current value (self-healing), intermediate states are irrelevant and the two paths cannot disagree on the final value. A change after copy but before resume falls in (b). A delete of X at the source is path (a) (absent from the re-copy → not loaded) or path (b) (re-read finds X absent → delete-absent). Thus X converges to its current source state — which is the one-way guarantee (source authoritative, target overwritten to match). The only thing discarded is R's historical intermediate deltas, which reseed intentionally supersedes with a current-state copy. ∎

4.4 Crash safety across the handoff (at-least-once, no 2PC)

Every transition is idempotent and independently checkpointed, so a crash at any point resumes correctly — the same delete-by-id discipline as consumption, applied to the machine:

Crash pointOn restart
after track reset, before MARKED persistedtrack_T empty; Purge is idempotent; next purge pass re-detects over-cap → re-marks. No loss (deltas already unpinned).
after MARKED, before/into RECOPYINGneeds_reseed=true + phase=initial_copy in StateStore → orchestration re-invokes Run, which is idempotent from the top: empty-target + fresh-progress reset re-run harmlessly and the re-copy restarts (its per-chunk watermark resumes within a single Run; a Run-crash restarts the copy).
mid RESUMINGtrack_T still (partially) empty; drain re-reads current values; idempotent upserts replay harmlessly.
after resume drained, before clearing needs_reseedflag still set → one more (no-op) resume pass, then cleared.

needs_reseed is cleared only after the resume backlog is drained and phase set to streaming, so the flag conservatively over-approximates "reseed in flight." At-least-once throughout; no distributed transaction.

4.5 Ordering guard (the trap we must not fall into)

The reseed must reset track_T to empty, never to max(delta_id) or any high-water mark. A high-water reset would declare late-committing low-delta_id rows "already consumed" and skip them forever — the exact commit-order hazard the dirty-key consumption model was built to defeat. Emptiness re-consumes them idempotently instead. This is the single most important correctness rule of the machine.


5. Partition+DROP decision (Momus M-4)

Decision: the batched-DELETE + aggressive-autovacuum path is the universal v1 default, on every supported source version. Time-partitioned delta tables purged by DROP/TRUNCATE of fully-consumed partitions remain the documented, recommended optimization for high-churn deployments on PG ≥ 10 (it is bloat-proof and dodges the xmin horizon), but it is not auto-enabled in v1 and is not the default even on modern sources.

Rationale: partition routing changes the delta-table DDL and the trigger's insert target and requires a partition-maintenance loop (create-ahead, drop-behind) — a larger surface that would delay a correct, universal M5c. Keeping one code path (batched DELETE) correct on the 9.6 floor and PG 17 is worth more in v1 than the bloat win, especially since retention/reseed already bounds logical growth and telemetry makes the xmin-horizon physical-bloat case loud. Partition+DROP is revisited as a tuning pass in M9 (partition granularity/cadence numbers are still open). Until then it is an opt-in extension point, not a promise.

This settles whether partition+DROP should be the default on PG≥10: no, not default in v1; opt-in optimization, tune in M9.


6. Telemetry (F2 contract, emitted by M5c, verified by M6)

Purge/retention/reseed emit against the frozen F2 registry (internal/observability) — no new names:

  • replicare_delta_backlog_rows / _bytes, replicare_delta_oldest_unconsumed_age_seconds — per (sync, target, table); the "is the source footprint healthy?" headline.
  • replicare_delta_purged_total — per (sync, table); purge progress.
  • replicare_reseed_total — per (sync, target); forced reseeds.
  • span replicare.reseed — wraps a reseed; error status on failure, backlog/age attributes.
  • events retention.cap_approaching (WARN, escalating with retention_cap_proximity toward 1.0) and target.needs_reseed (ERROR).

The cross-channel rule holds: a target pushing the queue toward the cap is visible in metrics (climbing backlog/age + proximity), logs (escalating WARN→ERROR), and traces (reseed span) simultaneously.


7. Scope / non-goals for M5c

  • Fan-out purge is implemented correctly (consumed-by-all across the full target set) but only lightly exercised — v1 hardens single-target (plan goals).
  • No partition+DROP implementation (see §5).
  • Retention-cap default numbers are set conservatively here (age 24h on, size off) and tuned in M9 against the at-scale fault matrix.
  • The daemon loop that schedules purge/reseed passes is M7; M5c delivers the reusable mechanics + primitives and proves them under integration tests.

loadgen — a Postgres load + verification harness for replicare

loadgen is a manual, idempotent tool for exercising replicare against broad, high-volume, realistic Postgres data. You run it whenever you like against a source database; it seeds a rich schema the first time and applies random inserts/updates/deletes on every subsequent run. Point replicare at the source and a target, and use loadgen verify to assert the target converged.

It is a developer/test tool — not part of the shipped binary. Run it from a checkout with go run ./test/loadgen (or task loadgen:*).

What it generates

One schema (loadgen) of 10 tables, chosen to exercise replicare's hard paths, not just to hold rows:

TableWhat it stresses
tenantshub table many others FK to (giant-component pressure); identity PK, jsonb, text[], timestamptz, bool
categoriesself-referential hierarchy column (self-ref FK under --cyclic)
usersbytea, inet, numeric(20,4), date, jsonb, many nullable columns; unique (tenant_id,email)
productscomposite text PK (tenant_id, sku); float8[]; timestamp (no tz); target of PK-changing updates
ordersUUID PK; char(3); nullable FK to users
order_itemscomposite PK; composite FK to products ON UPDATE CASCADE (a SKU rename is a PK change that cascades)
product_categoriesM:N join, composite PK, two FKs
eventsthe high-volume table (~1M rows at default scale); wide type coverage
generated_metricsGENERATED … STORED column + identity PK
audit_logno primary/unique key → replicare skips it with a loud warning (verify excludes it too)

Data is generated server-side (INSERT … SELECT generate_series, random() made reproducible with setseed, md5(…)::uuid) — fast to a million rows, robust across old Postgres, no client-side type encoding. The churn mix includes PK-changing SKU renames and mutation of the nullable FK-cycle edge.

At --scale 1.0 (default) the seed writes ~1.2M rows (events dominate). Scale it down for quick loops: --scale 0.01 ≈ 12k rows in ~150ms.

Usage

# 1. Create the schema on the TARGET (replicare replicates data only — target
#    tables must pre-exist).
go run ./test/loadgen ddl --dsn "$TARGET"

# 2. Seed the SOURCE (first run seeds; later runs churn).
go run ./test/loadgen run --dsn "$SOURCE"                 # ~1M+ rows
go run ./test/loadgen run --dsn "$SOURCE" --scale 0.05    # smaller, faster

# 3. Start replicare source -> target (your own config), then generate change:
go run ./test/loadgen run --dsn "$SOURCE" --ops 500       # a burst of I/U/D

# 4. Assert the target converged (retries for replication lag):
go run ./test/loadgen verify --source "$SOURCE" --target "$TARGET" --wait 60s

--dsn/--source/--target take a libpq/pgx URL (postgres://user:pw@host:port/db) or keyword string; empty uses the standard PG* environment variables.

Commands & flags

  • run --dsn <src> [--scale F] [--ops N] [--seed S] [--cyclic] [--duration D] [--node-id K] — ensure schema, then seed if empty or churn N statements. --seed makes a run reproducible. --duration D churns continuously for D (repeated --ops bursts) — run it while replicare does its initial copy to exercise the live-source cyclic-copy skew: parent rows (tenants/users) inserted mid-copy are referenced by children copied from a later snapshot, a transient FK the cyclic copy must recover from rather than crash-loop on. Pair with --cyclic. --node-id K (default 0) selects this writer's key slice for active-active testing — see Active-active; 0 is the default single-writer (active-passive) behaviour, unchanged.
  • ddl --dsn <db> [--cyclic] — apply the schema only (prep the target). One shared schema serves every node; --node-id is a run-only concept (it partitions the data, not the schema).
  • verify --source <src> --target <tgt> [--wait D] [--interval D] — per-table row-count + ordered content-checksum comparison. Exits non-zero on drift. --wait polls until converged or the timeout. GUCs are pinned to match replicare's transport so row::text renders identically on both ends. It is a pure equality check, so it doubles as the active-active convergence oracle — run it between any two mesh nodes (they must hold the identical union).
  • reset --dsn <db> --yes — DROP SCHEMA loadgen CASCADE.

verify compares only the 9 keyed tables (it skips audit_log, which replicare skips). The GENERATED … STORED column is included and matches because both servers recompute it.

Active-active (multi-master) load testing

The same harness drives a clusters: mesh. The model is disjoint partitioning: each writer node gets --node-id K, which shifts every generated key and every FK reference into a private slice [K·1e9+1, K·1e9+1e9), so nodes never collide on a key and the mesh converges to the clean union of all slices (predictable counts, matching checksums). --node-id 0 is the default and is byte-for-byte the single-writer behaviour above, so active-passive is unaffected.

What --node-id K changes on run:

  • Seed writes ids base+g, order UUIDs md5('order:'||(base+g)), SKUs SKU-(base+g), and offsets every FK reference by base, so a node's children point only at that node's own parents. Identity sequences are then restarted past the node's seeded rows so churn inserts stay in the slice.
  • Churn inserts into the node's slice and scopes every UPDATE/DELETE to node-owned rows, so a node only ever mutates what it wrote — no accidental cross-node conflict (resolving concurrent same-key writes is the separate, planned conflict-storm mode; see below).

--node-id 0 and any --node-id K seeded/churned against the same node's DB must be consistent — always pass the same --node-id for a given node. Keep --scale well under ~500M rows per node (the harness refuses a scale that approaches the slice).

Three-node mesh example

Assume a running replicare clusters: mesh over three Postgres nodes A, B, C (config per multi-master.md), with $A/$B/$C their DSNs.

# 1. Schema on every node (data-only replication; targets must pre-exist).
for n in "$A" "$B" "$C"; do go run ./test/loadgen ddl --dsn "$n"; done

# 2. Seed each node into its own disjoint slice.
go run ./test/loadgen run --dsn "$A" --node-id 0
go run ./test/loadgen run --dsn "$B" --node-id 1
go run ./test/loadgen run --dsn "$C" --node-id 2

# 3. Churn every node concurrently (each in its own slice).
go run ./test/loadgen run --dsn "$A" --node-id 0 --ops 800 --seed 1 &
go run ./test/loadgen run --dsn "$B" --node-id 1 --ops 800 --seed 2 &
go run ./test/loadgen run --dsn "$C" --node-id 2 --ops 800 --seed 3 &
wait

# 4. Convergence oracle: every pair must be identical (all hold the union).
go run ./test/loadgen verify --source "$A" --target "$B" --wait 300s
go run ./test/loadgen verify --source "$A" --target "$C" --wait 300s

Convergence is transitive, so A==B and A==C imply all three agree. A pass ending CONVERGED: all 9 replicated tables match on every pair — with the daemon logs clean of HALTED / stream pass error — is a passing active-active load test.

Not yet: conflict-storm mode. The disjoint model above never has two nodes write the same key, so it tests convergence of the union. A follow-up --overlap mode will deliberately overlap node slices to exercise HLC last-write-wins under concurrent same-key writes; there verify still asserts convergence (all nodes equal), but not which value wins (the LWW winner is nondeterministic).

Reset the state store per node when re-seeding (see Gotchas) — this applies per node in a mesh: clear each node's replicare schema and its state-store entries alongside loadgen reset.

Running a full load test end-to-end

The commands above assume replicare is already running with "your own config". Here is the complete rig — two Postgres instances, a state store, a config, the daemon, and a heavy-churn convergence loop — that this harness is meant to drive. It is the exact flow used to validate replicare's copy + streaming under load.

1. Two Postgres instances + a state-store database. Any two reachable Postgres servers work; the quickest is two containers. replicare's state store is a third logical database (here, one on the target server):

docker run -d --name lg-src -e POSTGRES_PASSWORD=pw -p 55432:5432 postgres:16
docker run -d --name lg-tgt -e POSTGRES_PASSWORD=pw -p 55433:5432 postgres:16
# state store (replicare's own progress/cursors — NOT your data):
docker exec lg-tgt psql -U postgres -c 'CREATE DATABASE replicare_state'

export SOURCE="postgres://postgres:pw@localhost:55432/postgres"
export TARGET="postgres://postgres:pw@localhost:55433/postgres"

2. A minimal replicare config (loadtest.yaml). drain_interval is short so streaming keeps up under churn:

logging: { level: info, format: text }
observability: { metrics_addr: ":19090", status_addr: ":18080" }
state_store:
  engine: postgres
  postgres: { host: localhost, port: 55433, database: replicare_state, user: postgres, password: pw, sslmode: disable }
sources:
  src: { engine: postgres, postgres: { host: localhost, port: 55432, database: postgres, user: postgres, password: pw, sslmode: disable } }
targets:
  tgt: { engine: postgres, postgres: { host: localhost, port: 55433, database: postgres, user: postgres, password: pw, sslmode: disable } }
syncs:
  - name: loadtest
    source: src
    targets: [tgt]
    include: ["loadgen.*"]
    tuning: { drain_interval: 1s }

3. Seed the source, create the target schema, start replicare. Capture is enabled first, then the chunked copy runs — so seeding before the daemon starts is fine (the initial copy reproduces it):

go run ./test/loadgen run --dsn "$SOURCE"      # seed ~1M+ rows on the source
go run ./test/loadgen ddl --dsn "$TARGET"      # create the (empty) target schema
go run ./cmd/replicare run loadtest.yaml &     # start the daemon

4. Verify the initial copy converged:

go run ./test/loadgen verify --source "$SOURCE" --target "$TARGET" --wait 120s
# => CONVERGED: all 9 replicated tables match

5. Churn-and-verify loop — this is the actual load test. Each round applies a random burst of inserts/updates/deletes (including PK-changing SKU renames) and asserts the target re-converges while the daemon streams deltas:

for r in 1 2 3 4; do
  go run ./test/loadgen run --dsn "$SOURCE" --ops 800 --seed $r
  go run ./test/loadgen verify --source "$SOURCE" --target "$TARGET" --wait 120s
done

To stress cross-batch dependencies and delta backlog (what shook out the FK livelocks), churn several times back-to-back before verifying, so a large backlog builds while streaming lags, then converges:

for r in $(seq 1 8); do go run ./test/loadgen run --dsn "$SOURCE" --ops 800 --seed $r; done
go run ./test/loadgen verify --source "$SOURCE" --target "$TARGET" --wait 300s

A run finishing with CONVERGED: all 9 replicated tables match — and the daemon log showing no HALTED / stream pass error — is a passing load test. For the cyclic variant, add --cyclic to every ddl and run (see below).

This is a manual test, distinct from task test:integration (the automated Go suite). Those integration tests must run serially (go test -p 1 ...) because they share schemas; the load test above is driven by hand and has no such constraint.

The --cyclic flag

By default the schema is acyclic. --cyclic adds two nullable FK cycles — the categories self-reference and a users ↔ orders 2-table cycle — to exercise replicare's cyclic-copy (null_then_fill) path. Apply it consistently to source and target (ddl --cyclic, run --cyclic).

Both initial copy and streaming of a cyclic component converge under --cyclic, with standard non-DEFERRABLE target FKs (the common real-world case). Initial copy loads every table with its cyclic FK columns NULL, in an order that respects the non-cyclic edges, then fills those columns. Streaming uses the same NULL-then-fill idea per pass, but per table: each table's upsert, delete, and cyclic-column fill commit independently, so a parent advances through its own delta queue even when a cross-batch child is transiently blocked (the same progress guarantee the acyclic drain gives), and the cycle itself is closed by a final fill phase. This harness's --cyclic schema drains cleanly under heavy churn with no DEFERRABLE requirement.

Gotchas

  • Reset the state store when re-seeding from scratch. replicare records per-table copy-progress in its state store; if you wipe and re-seed the source without clearing that state, replicare treats already-Done tables as copied and skips them (their parents go missing → FK errors on children). Clear the replicare schema in your state-store database (or use a fresh one) alongside loadgen reset.
  • The target schema must exist first — run ddl against the target before starting replicare.
  • Deletes cascade (orders → order_items, tenants → everything), so a delete_orders/delete_users op can remove more rows than it names.

Task shortcuts

task loadgen:ddl    DSN="$TARGET"                    # schema on the target
task loadgen:seed   DSN="$SOURCE" SCALE=0.1          # seed / scale
task loadgen:churn  DSN="$SOURCE" OPS=500            # a churn burst
task loadgen:verify SRC="$SOURCE" TGT="$TARGET"      # convergence check

loadgen-mysql — a MySQL load + verification harness for replicare

loadgen-mysql is the MySQL sibling of test/loadgen. It is a manual, idempotent tool for exercising replicare's MySQL engine against broad, high-volume, realistic data. You run it whenever you like against a source MySQL; it seeds a rich schema the first time and applies random inserts/updates/deletes on every subsequent run. Point replicare at the source and a target, and use loadgen-mysql verify to assert the target converged.

It is a developer/test tool — not part of the shipped binary. Run it from a checkout with go run ./test/loadgen-mysql (or task loadgen-mysql:*).

Everything lives in a dedicated loadgen database (MySQL's analog of the Postgres loadgen schema), created by ddl/run, so replicare can select it wholesale (include: ["loadgen.*"]) and the harness never collides with anything else on the server. Every statement is schema-qualified, so the DSN's default database is irrelevant and need not exist.

What it generates

One database (loadgen) of 10 tables, chosen to exercise replicare's hard MySQL paths, not just to hold rows (the same 10 as the Postgres harness, in MySQL types):

TableWhat it stresses
tenantshub table many others FK to (giant-component pressure); AUTO_INCREMENT PK, JSON, DATETIME, TINYINT(1) bool
categoriesself-referential hierarchy column (self-ref FK under --cyclic)
usersVARBINARY (avatar), IP text, DECIMAL(20,4), DATE, JSON, many nullable columns; unique (tenant_id,email)
productscomposite VARCHAR PK (tenant_id, sku); JSON dims; target of PK-changing updates
ordersCHAR(32) id PK (MD5 hex, the UUID analog); CHAR(3); nullable FK to users
order_itemscomposite PK; composite FK to products ON UPDATE CASCADE (a SKU rename is a PK change that cascades)
product_categoriesM:N join, composite PK, two FKs
eventsthe high-volume table (~1M rows at default scale); wide type coverage incl. DOUBLE
generated_metricsGENERATED … STORED column + AUTO_INCREMENT PK
audit_logno primary/unique key → replicare skips it with a loud warning (verify excludes it too)

Data is generated server-side (INSERT … SELECT over a cross-joined digit-table row generator, RAND() seeded with RAND(seed), MD5(…) ids) — fast to a million rows and, crucially, MySQL-5.7 safe: 5.7 is the oldest supported source and has no generate_series, no CTEs, and no window functions, so the harness uses none of them. The churn mix includes PK-changing SKU renames (cascaded) and mutation of the nullable FK-cycle edge.

At --scale 1.0 (default) the seed writes ~1.2M rows (events dominate). Scale it down for quick loops: --scale 0.01 ≈ 12k rows.

Usage

# 1. Create the schema on the TARGET (replicare replicates data only — target tables
#    must pre-exist).
go run ./test/loadgen-mysql ddl --dsn "$TARGET"

# 2. Seed the SOURCE (first run seeds; later runs churn).
go run ./test/loadgen-mysql run --dsn "$SOURCE"                 # ~1M+ rows
go run ./test/loadgen-mysql run --dsn "$SOURCE" --scale 0.05    # smaller, faster

# 3. Start replicare source -> target (your own config), then generate change:
go run ./test/loadgen-mysql run --dsn "$SOURCE" --ops 500       # a burst of I/U/D

# 4. Assert the target converged (retries for replication lag):
go run ./test/loadgen-mysql verify --source "$SOURCE" --target "$TARGET" --wait 60s

--dsn/--source/--target take a go-sql-driver DSN, e.g. root:replicare@tcp(127.0.0.1:3340)/ (source) and …@tcp(127.0.0.1:3341)/ (target) for the bundled harness. The harness pins time_zone='+00:00' on every connection so NOW()/DATETIME values render deterministically in UTC, matching replicare's own MySQL session canonicalization.

Commands & flags

  • run --dsn <src> [--scale F] [--ops N] [--seed S] [--cyclic] [--duration D] [--node-id K] — ensure schema, then seed if empty or churn N statements. --seed seeds both client op-selection and server RAND() (server-side value reproducibility is best-effort on MySQL; op selection is exact). --duration D churns continuously for D (repeated --ops bursts) — run it while replicare does its initial copy to exercise the live-source skew path: parent rows inserted mid-copy referenced by children copied from a later snapshot (a transient FK the copy must recover from). Pair with --cyclic. --node-id K (default 0) selects this writer's key slice for active-active testing — see Active-active; 0 is the default single-writer (active-passive) behaviour, unchanged.
  • ddl --dsn <db> [--cyclic] — apply the schema only (prep the target). One shared schema serves every node; --node-id is a run-only concept (it partitions the data, not the schema).
  • verify --source <src> --target <tgt> [--wait D] [--interval D] — per-table row-count + order-independent content-checksum comparison (per row, MD5 over a NULL-safe, type-aware rendering of every column; folded with BIT_XOR). Exits non-zero on drift. --wait polls until converged or the timeout. It is a pure equality check, so it doubles as the active-active convergence oracle — run it between any two mesh nodes (they must hold the identical union).
  • reset --dsn <db> --yes — DROP DATABASE loadgen.

verify compares only the 9 keyed tables (it skips audit_log, which replicare skips). The GENERATED … STORED column is included and matches because both servers recompute it.

Why not compare a table checksum directly? The harness runs an old source (5.7) against a modern target (8.4). verify renders each value server-side into a version-stable form before hashing: binary/blob via HEX, FLOAT/DOUBLE normalized via CAST … AS DECIMAL(65,15) (replicare moves the value bit-faithfully, so both ends hold the identical double; normalizing removes any raw double-to-string formatting difference between server versions), everything else via CAST … AS CHAR under a pinned time_zone. It compares that canonical rendering, not raw storage bytes (which differ across versions for identical data).

Active-active (multi-master) load testing

The same harness drives a MySQL clusters: mesh. The model is disjoint partitioning: each writer node gets --node-id K, which shifts every generated key and every FK reference into a private slice [K·1e9+1, K·1e9+1e9), so nodes never collide on a key and the mesh converges to the clean union of all slices (predictable counts, matching checksums). --node-id 0 is the default and is byte-for-byte the single-writer behaviour above, so active-passive is unaffected.

What --node-id K changes on run:

  • Seed writes ids base+g, order ids MD5(CONCAT('order:', base+g)), SKUs SKU-(base+g), and offsets every FK reference by base, so a node's children point only at that node's own parents. AUTO_INCREMENT counters are then restarted past the node's seeded rows so churn inserts stay in the slice.
  • Churn inserts into the node's slice and scopes every UPDATE/DELETE to node-owned rows, so a node only ever mutates what it wrote — no accidental cross-node conflict (resolving concurrent same-key writes is the separate, planned conflict-storm mode; see below).

--node-id 0 and any --node-id K seeded/churned against the same node's DB must be consistent — always pass the same --node-id for a given node. Keep --scale well under ~500M rows per node (the harness refuses a scale that approaches the slice).

Three-node mesh example

Assume a running replicare clusters: mesh over three MySQL nodes A, B, C (config per multi-master.md), with $A/$B/$C their DSNs.

# 1. Schema on every node (data-only replication; targets must pre-exist).
for n in "$A" "$B" "$C"; do go run ./test/loadgen-mysql ddl --dsn "$n"; done

# 2. Seed each node into its own disjoint slice.
go run ./test/loadgen-mysql run --dsn "$A" --node-id 0
go run ./test/loadgen-mysql run --dsn "$B" --node-id 1
go run ./test/loadgen-mysql run --dsn "$C" --node-id 2

# 3. Churn every node concurrently (each in its own slice).
go run ./test/loadgen-mysql run --dsn "$A" --node-id 0 --ops 800 --seed 1 &
go run ./test/loadgen-mysql run --dsn "$B" --node-id 1 --ops 800 --seed 2 &
go run ./test/loadgen-mysql run --dsn "$C" --node-id 2 --ops 800 --seed 3 &
wait

# 4. Convergence oracle: every pair must be identical (all hold the union).
go run ./test/loadgen-mysql verify --source "$A" --target "$B" --wait 300s
go run ./test/loadgen-mysql verify --source "$A" --target "$C" --wait 300s

Convergence is transitive, so A==B and A==C imply all three agree. A pass ending CONVERGED: all 9 replicated tables match on every pair — with the daemon logs clean of HALTED / stream pass error — is a passing active-active load test.

Not yet: conflict-storm mode. The disjoint model above never has two nodes write the same key, so it tests convergence of the union. A follow-up --overlap mode will deliberately overlap node slices to exercise HLC last-write-wins under concurrent same-key writes; there verify still asserts convergence (all nodes equal), but not which value wins (the LWW winner is nondeterministic).

Reset the state store per node when re-seeding (see Gotchas) — this applies per node in a mesh: clear each node's source-side replicare schema and its state-store entries alongside loadgen-mysql reset.

Running a full load test end-to-end

The commands above assume replicare is already running with "your own config". Here is the complete rig — an old 5.7 source, a modern 8.4 target, a Postgres state store, a config, the daemon, and a heavy-churn convergence loop — that this harness is meant to drive. It is the exact flow used to validate replicare's MySQL copy + streaming under load.

1. Two MySQL instances + a Postgres state store. The bundled harness is the quickest source/target pair (an old 5.7 source on :3340 and a modern 8.4 target on :3341); the state store is Postgres (v1's only StateStore backend — the replicated data is MySQL, the daemon's own progress/cursors live in Postgres):

task harness:mysql:up          # source on :3340, target on :3341
docker run -d --name lg-state -e POSTGRES_PASSWORD=pw -p 55450:5432 postgres:16
docker exec lg-state psql -U postgres -c 'CREATE DATABASE replicare_state'

export SOURCE="root:replicare@tcp(127.0.0.1:3340)/"
export TARGET="root:replicare@tcp(127.0.0.1:3341)/"

2. A minimal replicare config (loadtest-mysql.yaml). drain_interval is short so streaming keeps up under churn:

logging: { level: info, format: text }
observability: { metrics_addr: ":19092", status_addr: ":18082" }
state_store:
  engine: postgres
  postgres: { host: localhost, port: 55450, database: replicare_state, user: postgres, password: pw, sslmode: disable }
sources:
  src: { engine: mysql, mysql: { host: 127.0.0.1, port: 3340, database: loadgen, user: root, password: replicare, tls: disable } }
targets:
  # local_infile: true opts into the LOAD DATA fast path (the target server must also
  # permit it — the bundled harness sets --local-infile=ON).
  tgt: { engine: mysql, mysql: { host: 127.0.0.1, port: 3341, database: loadgen, user: root, password: replicare, tls: disable, local_infile: true } }
syncs:
  - name: loadtest-mysql
    source: src
    targets: [tgt]
    include: ["loadgen.*"]
    tuning: { drain_interval: 1s }

3. Seed the source, create the target schema, start replicare. Capture (triggers) is installed first, then the chunked copy runs — so seeding before the daemon starts is fine (the initial copy reproduces it):

go run ./test/loadgen-mysql run --dsn "$SOURCE"      # seed ~1M+ rows on the source
go run ./test/loadgen-mysql ddl --dsn "$TARGET"      # create the (empty) target schema
go run ./cmd/replicare run loadtest-mysql.yaml &     # start the daemon

4. Verify the initial copy converged:

go run ./test/loadgen-mysql verify --source "$SOURCE" --target "$TARGET" --wait 120s
# => CONVERGED: all 9 replicated tables match

5. Churn-and-verify loop — this is the actual load test. Each round applies a random burst of inserts/updates/deletes (including PK-changing SKU renames) and asserts the target re-converges while the daemon streams deltas:

for r in 1 2 3 4; do
  go run ./test/loadgen-mysql run --dsn "$SOURCE" --ops 800 --seed $r
  go run ./test/loadgen-mysql verify --source "$SOURCE" --target "$TARGET" --wait 120s
done

To stress cross-batch dependencies and delta backlog, churn several times back-to-back before verifying, so a large backlog builds while streaming lags, then converges:

for r in $(seq 1 8); do go run ./test/loadgen-mysql run --dsn "$SOURCE" --ops 800 --seed $r; done
go run ./test/loadgen-mysql verify --source "$SOURCE" --target "$TARGET" --wait 300s

A run finishing with CONVERGED: all 9 replicated tables match — and the daemon log showing no HALTED / stream pass error — is a passing load test. For the cyclic variant, add --cyclic to every ddl and run (see below).

This is a manual test, distinct from the automated Go integration suite. MySQL correctness is a local gate (REPLICARE_INTEGRATION=1 with the MySQL harness up); CI is Postgres-only.

The --cyclic flag

By default the schema is acyclic. --cyclic adds two nullable FK cycles — the categories self-reference and a users ↔ orders 2-table cycle — to exercise replicare's cyclic-copy (null_then_fill) path. Apply it consistently to source and target (ddl --cyclic, run --cyclic). MySQL has no ADD CONSTRAINT IF NOT EXISTS, so the harness adds each cyclic FK only when it is absent, keeping --cyclic idempotent.

Gotchas

  • Reset the state store when re-seeding from scratch. replicare records per-table copy-progress in its state store; if you wipe and re-seed the source without clearing that state, replicare treats already-Done tables as copied and skips them. Clear the replicare schema in your state-store database (or use a fresh one) alongside loadgen-mysql reset.
  • The target schema must exist first — run ddl against the target before starting replicare.
  • local_infile on the target. replicare's MySQL initial copy uses LOAD DATA LOCAL INFILE, so the target server must permit it (the bundled harness-mysql sets --local-infile=ON). The harness itself does not need it (it seeds via ordinary INSERT … SELECT).
  • Deletes cascade (orders → order_items, tenants → everything), so a delete_orders/delete_users op can remove more rows than it names.

Task shortcuts

task loadgen-mysql:ddl    DSN="$TARGET"                    # schema on the target
task loadgen-mysql:seed   DSN="$SOURCE" SCALE=0.1          # seed / scale
task loadgen-mysql:churn  DSN="$SOURCE" OPS=500            # a churn burst
task loadgen-mysql:verify SRC="$SOURCE" TGT="$TARGET" WAIT=60s
task loadgen-mysql:reset  DSN="$SOURCE"                    # DROP DATABASE loadgen

loadgen-redis — a Redis load + verification harness for replicare

loadgen-redis is the Redis sibling of test/loadgen. It is a manual, idempotent tool for exercising replicare's Redis engine against broad, multi-type, high-volume data. You run it against a source Redis; it seeds a rich keyspace the first time and applies random mutations on every subsequent run. Point replicare at the source and a target, and use loadgen-redis verify to assert the target converged.

It is a developer/test tool — not part of the shipped binary. Run it from a checkout with go run ./test/loadgen-redis (or task loadgen-redis:*).

Unlike the Postgres harness there is no ddl step: Redis has no schema, and replicare RESTOREs keys straight into the target keyspace, which need not pre-exist.

What it generates

Keys are laid out as lg:{<bucket>}:<type>:<n>. Everything is chosen to exercise replicare's Redis paths, not just to hold data:

CohortWhat it stresses
strplain strings; ~20% carry a TTL (volatile keys → TTL replication)
hash / list / set / zsetevery collection type, through DUMP→RESTORE
streamstreams with consumer groups — the trickiest DUMP/RESTORE payload
bigdeliberately large keys (~100 KB hashes) → the big-key path / MEMORY USAGE gate
lgskip:*written to the source only; replicare's include: ["lg:*"] must exclude it — the Redis analog of loadgen's keyless audit_log. verify asserts the target holds none of these.

The {bucket} hash tag is the cluster story: on a standalone/sentinel server the braces are inert literal bytes, but on a Redis Cluster they spread the keyspace across slots/shards (exercising replicare's per-master SCAN fan-out) and keep a RENAME's old+new key in one slot (a cross-slot RENAME is an error).

The churn mix includes the two cases replicare's Redis engine handles specially:

  • RENAME = delete-old + create-new — the analog of a PK change, and a stress on the delete-reconciliation sweep.
  • heavy DEL — Redis has no delete capture, so deletes are only caught by replicare's target-vs-source diff; this is the genuinely hard path.

At --scale 1.0 (default) the seed writes ~430k keys. Scale down for quick loops: --scale 0.02 ≈ 8.6k keys in well under a second.

Usage

# 1. Seed the SOURCE (first run seeds; later runs churn).
go run ./test/loadgen-redis run --dsn "$SOURCE"                 # ~430k keys
go run ./test/loadgen-redis run --dsn "$SOURCE" --scale 0.05    # smaller, faster

# 2. Start replicare source -> target (your own config), then generate change:
go run ./test/loadgen-redis run --dsn "$SOURCE" --ops 500       # a burst of mutations

# 3. Assert the target converged (retries for replication lag):
go run ./test/loadgen-redis verify --source "$SOURCE" --target "$TARGET" --wait 60s

Connection specs

  • standalone / sentinel: a redis:// URL, e.g. redis://:pw@localhost:6390/0 (empty = redis://localhost:6379/0).
  • cluster: pass --cluster and a comma-separated seed list, e.g. --dsn "n1:6379,n2:6379,n3:6379" (use --password for AUTH). One tool, both topologies — see the hash-tag note above.

Commands & flags

  • run --dsn <src> [--scale F] [--ops N] [--seed S] [--cluster] [--node-id K] — seed if empty or churn N ops. --seed makes a run reproducible (op selection + generated values). --node-id K (default 0) selects this writer's key namespace for active-active testing — see Active-active; 0 is the default single-writer (active-passive) behaviour, unchanged.
  • verify --source <src> --target <tgt> [--wait D] [--interval D] [--cluster] [--node-id K] — compares every lg:* key version-independently: a type-aware canonical content hash (GET / sorted HGETALL / LRANGE / sorted SMEMBERS / ZRANGE WITHSCORES / XRANGE + group last-ids) plus TTL presence. Also asserts no lgskip:* leaked to the target. It is a pure equality check over the whole lg:* keyspace, so it doubles as the active-active convergence oracle (run it between any two mesh nodes — both hold the union). --node-id K is the source's node index; it scopes the skip-leak check to that source's own lgskip:* namespace (0 = the classic single-writer check). --wait polls until converged or the timeout. Exits non-zero on drift.
  • reset --dsn <db> --yes [--cluster] — DEL every lg:*, lgskip:*, and the lgmeta:* marker key (all nodes' namespaces). Scoped by prefix — it never runs FLUSHDB.

Why not compare DUMP bytes directly? The harness runs an old source (6.2) against a modern target (7.4), whose RDB serializations differ even for identical data (and replicare deliberately preserves each side's own encoding). So verify compares a canonical rendering of each value, not its serialized bytes. TTL is compared as presence only (has-TTL vs no-TTL) — the exact remaining seconds legitimately differ by the replication delay.

Active-active (multi-master) load testing

The same harness drives a Redis clusters: mesh. The model is disjoint key namespaces: each writer node gets --node-id K, which moves every key it writes into a private segment right after the prefix — lg:n<K>:{<bucket>}:<type>:<n> (and lgskip:n<K>:… for the source-only cohort). Node segments never overlap, so nodes never collide on a key and the mesh converges to the clean union of all segments. --node-id 0 is the default and is byte-for-byte the single-writer layout above (lg:{<bucket>}:…, no segment), so active-passive is unaffected.

The segment sits after the lg:/lgskip: prefix, so a replicare include: ["lg:*"] selection still matches every node's data keys and still excludes every node's lgskip:* keys — no per-node config change is needed.

What --node-id K changes on run:

  • Seed writes only this node's namespace (lg:n<K>:…), and the seeded/empty check scans that namespace alone, so a node re-seeds only when its own slice is empty — peers' replicated-in keys don't count as "already seeded".
  • Churn sets/updates/deletes only keys in this node's namespace, and a RENAME keeps the new key in the same segment (and hash-tag bucket), so a node only ever mutates what it wrote — no accidental cross-node conflict (resolving concurrent same-key writes is the separate, planned conflict-storm mode; see below).

On verify, --node-id K is the source's node index: it scopes the skip-leak check to that source's own lgskip:n<K>:* namespace, so a mesh target — which is itself a source with its own lgskip:* — never false-positives on the peer's excluded cohort. The content comparison itself is a pure equality check over the whole lg:* keyspace, so it doubles as the convergence oracle: run it between any two mesh nodes and both must hold the identical union.

Always pass the same --node-id for a given node's DB. Keep --scale modest per node (Redis holds everything in RAM, and the mesh target holds every node's union).

Three-node mesh example

Assume a running replicare clusters: mesh over three Redis nodes A, B, C (config per multi-master.md), with $A/$B/$C their connection specs. There is no ddl step — Redis has no schema.

# 1. Seed each node into its own disjoint namespace.
go run ./test/loadgen-redis run --dsn "$A" --node-id 0 --scale 0.1
go run ./test/loadgen-redis run --dsn "$B" --node-id 1 --scale 0.1
go run ./test/loadgen-redis run --dsn "$C" --node-id 2 --scale 0.1

# 2. Churn every node concurrently (each in its own namespace).
go run ./test/loadgen-redis run --dsn "$A" --node-id 0 --ops 800 --seed 1 &
go run ./test/loadgen-redis run --dsn "$B" --node-id 1 --ops 800 --seed 2 &
go run ./test/loadgen-redis run --dsn "$C" --node-id 2 --ops 800 --seed 3 &
wait

# 3. Convergence oracle: every pair must be identical (all hold the union).
#    --node-id names the SOURCE so the skip-leak check scopes to its lgskip namespace.
go run ./test/loadgen-redis verify --source "$A" --target "$B" --node-id 0 --wait 300s
go run ./test/loadgen-redis verify --source "$A" --target "$C" --node-id 0 --wait 300s

Convergence is transitive, so A==B and A==C imply all three agree. A pass ending CONVERGED: … no skip leak on every pair — with the daemon logs clean of HALTED / stream pass error — is a passing active-active load test.

Not yet: conflict-storm mode. The disjoint model above never has two nodes write the same key, so it tests convergence of the union. A follow-up --overlap mode will deliberately overlap node namespaces to exercise HLC last-write-wins under concurrent same-key writes; there verify still asserts convergence (all nodes equal), but not which value wins (the LWW winner is nondeterministic).

Reset the state store per node when re-seeding (see Gotchas) — in a mesh this applies per node: clear each node's state-store entries alongside loadgen-redis reset (which already removes every node's lg:/lgskip: namespace).

Running a full load test end-to-end

The commands above assume replicare is already running with "your own config". Here is the complete rig — two Redis instances, a Postgres state store, a config, the daemon, and a churn-and-verify loop — that this harness is meant to drive. This is the exact flow used to validate replicare's Redis copy + streaming under load.

1. Two Redis instances + a Postgres state store. The bundled harness is the quickest source/target pair (an old 6.2 source and a modern 7.4 target); the state store is Postgres (v1's only StateStore backend — the replicated data is Redis, the daemon's own progress/cursors live in Postgres):

task harness:redis:up          # source on :6390, target on :6391
docker run -d --name lg-state -e POSTGRES_PASSWORD=pw -p 55440:5432 postgres:16
docker exec lg-state psql -U postgres -c 'CREATE DATABASE replicare_state'

export SOURCE="redis://localhost:6390/0"
export TARGET="redis://localhost:6391/0"

2. A minimal replicare config (loadtest-redis.yaml). drain_interval is short so the reconciliation SCAN + delete sweep keep up under churn:

logging: { level: info, format: text }
observability: { metrics_addr: ":19091", status_addr: ":18081" }
state_store:
  engine: postgres
  postgres: { host: localhost, port: 55440, database: replicare_state, user: postgres, password: pw, sslmode: disable }
sources:
  src: { engine: redis, redis: { host: localhost, port: 6390, db: 0 } }
targets:
  tgt: { engine: redis, redis: { host: localhost, port: 6391, db: 0 } }
syncs:
  - name: redisload
    source: src
    targets: [tgt]
    include: ["lg:*"]          # excludes lgskip:* by construction
    tuning: { drain_interval: 1s }

3. Seed the source, start replicare. Redis CDC is capture-less; the initial copy SCANs the source, so seeding before the daemon starts is fine:

go run ./test/loadgen-redis run --dsn "$SOURCE"          # seed the source
go run ./cmd/replicare run loadtest-redis.yaml &         # start the daemon

4. Verify the initial copy converged:

go run ./test/loadgen-redis verify --source "$SOURCE" --target "$TARGET" --wait 120s
# => CONVERGED: N replicated keys match, no skip leak

5. Churn-and-verify loop — this is the actual load test. Each round applies a random burst of mutations (including RENAMEs and heavy DELs) and asserts the target re-converges while the daemon streams:

for r in 1 2 3 4; do
  go run ./test/loadgen-redis run --dsn "$SOURCE" --ops 800 --seed $r
  go run ./test/loadgen-redis verify --source "$SOURCE" --target "$TARGET" --wait 120s
done

To stress the delete-reconciliation sweep and streaming backlog, churn several times back-to-back before verifying, so deletes and renames pile up while the sweep runs, then converge:

for r in $(seq 1 8); do go run ./test/loadgen-redis run --dsn "$SOURCE" --ops 800 --seed $r; done
go run ./test/loadgen-redis verify --source "$SOURCE" --target "$TARGET" --wait 300s

A run finishing with CONVERGED: … no skip leak — and the daemon log showing no HALTED / stream pass error — is a passing load test. Watch replicare_deletes_reconciled_total on :19091/metrics climb as DEL/RENAME churn is reconciled.

Cluster mode

The same tool drives a Redis Cluster target. Bring up the cluster harness and pass --cluster plus a seed list:

task harness:redis:cluster:up
go run ./test/loadgen-redis run    --dsn "127.0.0.1:7000,127.0.0.1:7001,127.0.0.1:7002" --cluster --scale 0.1
go run ./test/loadgen-redis verify --source "…standalone source…" --target "127.0.0.1:7000,…" --cluster --wait 120s

The {bucket} hash tags spread keys across the cluster's slots (so the per-master SCAN fan-out is genuinely exercised) while keeping each RENAME intra-slot.

Gotchas

  • Reset the state store when re-seeding from scratch. replicare records per-unit copy-progress in its state store; if you wipe and re-seed the source without clearing that state, it may treat the unit as already copied. Use a fresh replicare_state database (or clear the replicare schema in it) alongside loadgen-redis reset.
  • RENAME needs an existing source key. The tool renames within the seeded string range and silently skips a key that a prior op already renamed/deleted — benign for the harness.
  • lgmeta:counts is a small bookkeeping key (per-type seeded counts, so churn can address existing keys without scanning). It is neither lg:* nor lgskip:*, so replicare and verify both ignore it; reset removes it.

Task shortcuts

task loadgen-redis:seed   DSN="$SOURCE" SCALE=0.1          # seed / scale
task loadgen-redis:churn  DSN="$SOURCE" OPS=500            # a churn burst
task loadgen-redis:verify SRC="$SOURCE" TGT="$TARGET" WAIT=60s
task loadgen-redis:reset  DSN="$SOURCE"                    # add CLUSTER=1 for a cluster

Changelog

All notable changes to replicare are documented here. The format follows Keep a Changelog, and the project aims to follow Semantic Versioning.

0.1.0 — first release

The initial release: a complete, single-engine Postgres → Postgres data replicator. Feature-complete and verified end to end on a Postgres 9.6 → 17 integration matrix with fault-injection, perf, and fidelity gates.

Replication engine

  • Trigger-based CDC (Bucardo-style) — no WAL access, no REPLICATION role, no wal_level change. Works on managed and old Postgres.
  • PK-only capture with re-read at sync time — self-healing dirty-key-set model; PK changes enqueue both old and new keys.
  • Resumable, parallel initial copy — chunked keyset/ctid ranges, text COPY streamed source→target, parents-first within FK components, components in parallel; resumes from checkpoints after a crash.
  • Continuous streaming apply — FK-ordered per-component transactions (parent→child upserts, child→parent deletes) with a bounded retry fallback for cycles/self-references; idempotent, at-least-once.
  • Cyclic / self-referential FK initial copy — NULL-then-fill, deferred constraints, or a loud pre-flight block for the genuinely-unloadable case.

Correctness & safety

  • Faithful transport, never transform — values move verbatim (text COPY, both copy and delta paths) with session-GUC canonicalization; a type/constraint error halts loudly instead of corrupting. Verified across the 9.6→17 gap, including enum and range types, on both copy and delta paths.
  • Bounded source footprint — batched consumption-gated purge, aggressive per-table autovacuum, and bounded retention (age/size) with forced reseed of a laggard target. Protects the source even while a target is down, and under a stalled xmin horizon.
  • Crash-safe checkpointing — Postgres state store; a restarted or rescheduled process resumes cleanly; single-active ownership via pg_advisory_lock.

Operations

  • Daemon managing multiple named syncs concurrently, with graceful (SIGTERM → drain + checkpoint) shutdown.
  • CLI — run, validate, status, capture install|remove, reseed, version.
  • Pre-flight — classifies every source↔target column pair; blocks on incompatible types, warns on risky ones, skips no-key tables.
  • Least-privilege — documented, verified source and target grants; no superuser.

Observability

  • Prometheus /metrics, a /status + /healthz HTTP API, structured slog logs, and OpenTelemetry traces (OTLP/gRPC export when configured).
  • A degrading target (unreachable, or backlog nearing the retention cap) is surfaced across metrics, logs, and traces simultaneously.

Packaging

  • Single static, CGO-free binary; ~5 MB scratch-based Docker image; sample systemd unit; grant SQL; a runnable docker-compose demo; and full docs.

Not in this release (roadmap)

  • MySQL and Redis engines (the core is engine-agnostic and designed for them).
  • HA / leader election, multi-master conflict resolution, live DDL replication, Helm chart, hardened fan-out.