Operating replicare
Practical guidance for running replicare in production: the operator's view. How it works underneath is in Architecture.
Lifecycle of a sync
- Pre-flight.
replicare validate <config>(and the daemon at startup) introspects the source and target and classifies every column pair: identical/widening → ok, risky/lossy → warn, incompatible or missing type → block (refuses to start). It also reports FK components, a giant-component warning, dangling FK edges, and skipped no-key tables. Fix blocks before running. - Capture-first. The daemon installs trigger-based CDC on the source's selected tables, so changes begin queuing into per-table delta tables immediately.
- Initial copy. A chunked, parallel copy of each FK component (parents first) runs while capture is active. It is resumable — a restart continues from the last checkpoint, not the top.
- Cutover → streaming. Once a component is copied, it flips to continuous streaming: each pass re-reads the current source values of dirty keys and applies them idempotently, FK-ordered.
- Purge / retention / reseed. Consumed deltas are purged; a target that falls too far behind is reseeded (see below).
Before a migration: checklist
Work through this before pointing replicare at a production source. Most items are information to
gather (sizing, traffic, quiet windows); a few must be true before starting. Run the queries on the
source unless stated otherwise. replicare validate <config> then checks the rest (types,
foreign-key loops, keys) and reports anything that would block.
Postgres
Sizing and traffic
- Total database size.
SELECT pg_size_pretty(pg_database_size(current_database())); - Row counts and sizes per table.
row_estimatecomes from planner statistics;-1(or0on older versions) means the table has never been analyzed — runANALYZEfirst for real numbers.SELECT n.nspname AS schema, c.relname AS table_name, c.reltuples::bigint AS row_estimate, pg_size_pretty(pg_total_relation_size(c.oid)) AS total, pg_size_pretty(pg_indexes_size(c.oid)) AS indexes FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE c.relkind IN ('r', 'p') AND n.nspname NOT IN ('pg_catalog', 'information_schema', 'replicare') AND n.nspname NOT LIKE 'pg_toast%' ORDER BY pg_total_relation_size(c.oid) DESC; - Write volume over 24 hours. The counters only ever go up, so run this twice, 24 hours apart,
and subtract. Every row inserted, updated or deleted in a replicated table adds one row to
replicare's change tracking, so
inserts + updates + deletesper day is roughly the daily change-tracking volume (and the work streaming has to keep up with).
For the busiest tables,SELECT now() AS sampled_at, sum(n_tup_ins) AS inserts, sum(n_tup_upd) AS updates, sum(n_tup_del) AS deletes, (SELECT xact_commit + xact_rollback FROM pg_stat_database WHERE datname = current_database()) AS transactions FROM pg_stat_user_tables;SELECT relname, n_tup_ins + n_tup_upd + n_tup_del AS writes FROM pg_stat_user_tables ORDER BY writes DESC LIMIT 20;. - Low-traffic windows (daily and weekly). Busy periods followed by lulls let streaming catch up, and a quiet window is the best time for the final cutover.
Must be true before starting
- Every table has a primary key or a
UNIQUEconstraint. replicare tracks changes by key. It uses the primary key, or else the firstUNIQUEconstraint; a bareCREATE UNIQUE INDEXwithout a constraint does not count. Tables listed here are skipped (with a warning), not replicated:SELECT n.nspname AS schema, c.relname AS table_name FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE c.relkind IN ('r', 'p') AND n.nspname NOT IN ('pg_catalog', 'information_schema', 'replicare') AND n.nspname NOT LIKE 'pg_toast%' AND NOT EXISTS (SELECT 1 FROM pg_constraint k WHERE k.conrelid = c.oid AND k.contype IN ('p', 'u')) ORDER BY 1, 2; - Extensions. List them on the source and confirm the target has every one the replicated
tables use (column types, defaults). replicare never installs extensions; a missing one fails
pre-flight.
SELECT extname, extversion FROM pg_extension ORDER BY 1; - Materialized views (and plain views) are not replicated — only table data is. Create them on
the target from the schema dump, then
REFRESH MATERIALIZED VIEWthere after the initial copy and again just before cutover.SELECT schemaname, matviewname FROM pg_matviews ORDER BY 1, 2; - Large objects are not replicated. Their contents live in
pg_largeobject, not in a replicated table; a table'soidcolumn is copied but the object it points to is not. Anything above 0 here needs a separate plan:SELECT count(*) FROM pg_largeobject_metadata; - Schema changes are frozen for the whole replication. DDL is not replicated, so a schema change
on the source has to be applied to the target by hand and can break the pipeline. During the
initial copy there is an extra reason: an
ALTER TABLEwaits for replicare's copy read of that table to finish, and the application's queries on that table queue behind theALTER. - Regular vacuum and maintenance jobs. Find out what runs and when — autovacuum settings,
scheduled
VACUUM, and especially heavy jobs that lock whole tables (VACUUM FULL,CLUSTER,pg_repack). PlainVACUUM/ autovacuum is fine at any time; the heavy jobs should be paused during the initial copy and kept out of peak hours afterwards. See Vacuum and other maintenance.
Also ask about jobs outside the database (cron, CI pipelines, the hosting provider's maintenance windows).SHOW autovacuum; -- scheduled jobs, if pg_cron is installed: SELECT jobid, schedule, command FROM cron.job; -- recent vacuum activity and the tables with the most dead rows: SELECT relname, last_vacuum, last_autovacuum, n_dead_tup FROM pg_stat_user_tables ORDER BY n_dead_tup DESC LIMIT 20; - No long-open transactions. Sessions left open — especially
idle in transaction— stop vacuum from cleaning up (replicare's change-tracking tables bloat) and make trigger setup wait. This should normally return nothing:SELECT pid, usename, state, now() - xact_start AS open_for, left(query, 80) AS last_query FROM pg_stat_activity WHERE xact_start < now() - interval '5 minutes' AND pid <> pg_backend_pid() ORDER BY xact_start; - Versions.
SELECT version();on both sides. The source can be old; the target should be the same major version or newer. - Privileges and disk. The source role needs the grants in Least-privilege grants, and the source needs room for the change-tracking tables to grow with the write volume from item 3 (more if a target falls behind, up to the retention cap).
- Sequences at cutover. Row values are copied, sequence counters are not. Plan
replicare reseed-sequences(orsync_sequences: true) before the target takes writes — see Sequences & identity counters.
MySQL
Numbers match the Postgres list. Postgres items 4 (quiet windows), 9 (schema freeze), 13 (privileges and disk) and 14 (sequences) apply as written; large objects (8) don't exist in MySQL, and the version check (12) is part of (6). MySQL versions of the rest, plus two MySQL-only items:
- (1) Total size.
SELECT ROUND(SUM(data_length + index_length) / 1024 / 1024) AS size_mb FROM information_schema.TABLES WHERE table_schema = '<DB_NAME>'; - (2) Row counts, sizes and storage engine per table.
table_rowsis InnoDB's estimate.
Every table must be InnoDB — pre-flight blocks MyISAM/MEMORY tables.SELECT table_name, engine, table_rows AS row_estimate, ROUND(data_length / 1024 / 1024) AS data_mb, ROUND(index_length / 1024 / 1024) AS index_mb FROM information_schema.TABLES WHERE table_schema = '<DB_NAME>' AND table_type = 'BASE TABLE' ORDER BY data_length + index_length DESC; - (3) Write volume over 24 hours. Server-wide counters (all databases on the server); run twice,
24 hours apart, and subtract.
SHOW GLOBAL STATUS WHERE Variable_name IN ('Innodb_rows_inserted', 'Innodb_rows_updated', 'Innodb_rows_deleted', 'Com_commit', 'Com_rollback'); - (5) Every table has a primary key or a
UNIQUEkey (tables without one are skipped):SELECT t.table_name FROM information_schema.TABLES t WHERE t.table_schema = '<DB_NAME>' AND t.table_type = 'BASE TABLE' AND NOT EXISTS (SELECT 1 FROM information_schema.TABLE_CONSTRAINTS c WHERE c.table_schema = t.table_schema AND c.table_name = t.table_name AND c.constraint_type IN ('PRIMARY KEY', 'UNIQUE')); - (6) Extensions don't exist in MySQL. Instead check: MySQL 5.7 or newer (
SELECT VERSION();, MariaDB is not supported); on the target,local_infileisON(SHOW GLOBAL VARIABLES LIKE 'local_infile';) and the target tables have no extraUNIQUEkeys beyond the one replicare uses (pre-flight blocks them — see the MySQL page). - (7) Materialized views don't exist in MySQL; plain views are not replicated (they come over with the schema dump).
- (MySQL only) Existing triggers. Tables can keep their own triggers: from MySQL 5.7.2 a table may
have several triggers per timing and event, so replicare's triggers sit alongside the
application's and both fire. Only a source older than 5.7.2 (in practice a best-effort 5.6) allows
a single trigger per timing and event, and there a table with its own
AFTER INSERT/UPDATE/DELETEtrigger blocks capture. Worth listing either way, to know what else runs on each write:SELECT event_object_table, trigger_name, action_timing, event_manipulation FROM information_schema.TRIGGERS WHERE trigger_schema = '<DB_NAME>'; - (9) Schema freeze also covers online schema-change tools (
pt-online-schema-change,gh-ost). They rebuild a table and swap it in, and the swap leaves replicare's triggers behind on the old table;pt-online-schema-changealso adds triggers of its own. - (10) Maintenance jobs (the MySQL counterpart of vacuum). MySQL has no
VACUUM; the jobs that matter are table rebuilds —OPTIMIZE TABLE,ALTER TABLE … FORCE/ENGINE=InnoDB— run from cron or the event scheduler. A rebuild locks the table briefly at the start and end; during the initial copy it waits for replicare's copy read and the application queues behind it, so pause rebuilds during the copy.SELECT event_schema, event_name, interval_value, interval_field, event_definition FROM information_schema.EVENTS; - (11) No long-open transactions — they hold back InnoDB's purge (the history list grows and
replicare's change-tracking tables bloat) and make trigger setup wait:
SELECT trx_mysql_thread_id, trx_started, trx_state, LEFT(trx_query, 80) AS query FROM information_schema.INNODB_TRX ORDER BY trx_started; - (MySQL only) A Postgres database for replicare's state is still required for a MySQL→MySQL sync (see the MySQL page).
Tuning knobs (per sync, under tuning:)
| Knob | Meaning | Default |
|---|---|---|
drain_interval | Time between streaming drain passes. Longer = more per-PK coalescing (less source load) but higher lag. | 1s |
drain_batch | Max dirty deltas applied per table per pass. With drain_interval this is the per-table streaming ceiling. | 1000 |
apply_concurrency | How many of a component's tables apply concurrently per pass. 1 is sequential; higher parallelizes across the copy-worker pool. Raise when several tables backlog at once. | 1 |
retention.max_age | Cap on the oldest unconsumed delta before a laggard target is reseeded. | 24h |
retention.max_bytes | Cap on the delta table's on-disk size before reseed. | off |
pool.max_source_connections / pool.max_target_connections | Connection caps; the copy worker pool is sized from these. | conservative |
Defaults favor low source pressure. Raise the pool caps and shorten drain_interval for throughput;
lengthen drain_interval to reduce re-read load under heavy churn.
A single high-volume table that lags
Streaming applies at most drain_batch deltas per table per pass, so one table's ceiling is
roughly drain_batch / drain_interval rows/s (e.g. 1000 / 1s = ~1000 rows/s). When one big table
(say a high-churn events) falls behind while the rest stay at zero backlog, you've hit that ceiling.
Two levers, in order:
- Shorten
drain_interval(1s→200ms→100ms): more passes/sec, linearly more throughput, at the cost of more frequent re-readSELECTs on the source. Best when the "Apply batch latency (p95)" is well under the interval (interval-bound). - Raise
drain_batch(1000→5000+): more rows per pass. More efficient than a tiny interval for a genuinely large table — one COPY-to-staging + upsert of 5000 rows amortizes round-trips far better than five of 1000 — and the right lever when apply latency is near/above the interval (apply-bound). Costs a larger re-read and apply per pass (more source and target work in one shot).
Watch - rate(replicare_delta_backlog_rows[5m]) per table: positive means the backlog is shrinking
(catching up).
Several tables backlogged at once
The drain_batch/drain_interval levers above are per-table ceilings on a single-threaded drain:
by default a pass applies each component's tables one at a time. When several tables are backlogged
together, raise apply_concurrency — it applies a component's tables concurrently, across a pool
built from the copy workers (idle during streaming), so they no longer queue behind each other. It's
bounded by the connection caps: the pool has one primary pair plus up to apply_concurrency-1 worker
pairs, and worker pairs come from pool.max_source_connections/max_target_connections (so raise
those alongside it). FK order within a component is still honored per pass (a child that lands before
its parent hits a transient FK and retries), and cyclic components on MySQL — which apply as one
whole-component transaction — stay sequential. This does not speed up a single hot table (that's
still drain_batch/drain_interval + target capacity); it removes the cross-table serialization when
many tables are hot. As always, it helps only until the target's write capacity saturates.
Source footprint (the thing to watch)
replicare writes delta and track tables to the source — a database you may not own — so bounding that footprint is a first-class concern. The headline signals (per target/table):
replicare_delta_backlog_rows/replicare_delta_backlog_bytes— unconsumed work.replicare_delta_oldest_unconsumed_age_seconds— how stale the queue is.replicare_target_up— reachability (1/0).replicare_reseed_total— forced reseeds.
Retention & forced reseed. A slow or unreachable target pins the delta queue (nothing purges
until every target has consumed it), so replicare caps retention by age and/or size. A target over
the cap is marked needs-reseed: its consumption record is reset, the unpinned deltas are purged,
and it is brought back by a fresh full copy — trading a re-copy for a protected source. This is on
by default. Watch retention.cap_approaching logs (they escalate INFO→WARN→ERROR toward the cap).
The xmin-horizon caveat. A long idle-in-transaction session (or other held-back xmin) on the
source prevents autovacuum from reclaiming purged delta rows, so the delta table can bloat even
though replicare purges it. replicare bounds the logical backlog regardless, but physical bloat
under a pinned xmin is surfaced, not fixed — hunt down the idle transaction on the source.
MySQL (InnoDB) has the direct analog. InnoDB does not shrink the tablespace on DELETE, and a
long-running source transaction inflates the history list, blocking purge of rows it can still
see — the same shape as the xmin horizon. The same headline metrics apply (the delta/track tables
live in a dedicated replicare database on the MySQL source), retention/reseed protects the source
identically, and the fix is the same: end the long-running source transaction. See
the MySQL engine page.
Redis has no source footprint at all. Its CDC is capture-less — replicare writes nothing to the
source (no delta/track tables, no replicare schema). So the delta-backlog metrics, retention, and
xmin-style bloat simply don't apply. What you watch instead is delete-propagation latency:
deletes are found by the periodic target-vs-source sweep, so a source DEL converges only after the
next sweep. replicare_delete_reconciliation_lag_seconds (sweep duration) and
replicare_deletes_reconciled_total are the Redis-specific signals; tune cadence with
delete_sweep_interval. Upserts converge faster via the rolling reconciliation scan. See
the Redis engine page.
Vacuum and other maintenance
Plain VACUUM and autovacuum never conflict with streaming. While streaming, replicare runs
only ordinary statements on the source — the trigger's INSERT into its change-tracking table on
every application write, SELECTs that read those tables and re-read changed rows, and batched
DELETEs that clear changes every target has applied. None of them takes a lock that a plain
vacuum blocks, or that blocks a plain vacuum. The same holds on the target, where replicare only
inserts, updates and deletes rows.
Trigger setup is the one place a plain vacuum can get in the way — only at startup, and only when a trigger actually has to be created or replaced (a new table, or a trigger someone removed). Adding a trigger needs a lock that a running vacuum holds. replicare waits at most 2 seconds for it, backs off and retries, and after repeated failures stops with an error naming the table, rather than holding the application up. A normal restart, with every trigger already in place, takes no lock on application tables at all.
Heavy maintenance that locks a whole table — VACUUM FULL, CLUSTER, pg_repack,
ALTER TABLE:
- On an application table, it blocks replicare's reads of that table just like the application's queries; replicare adds no extra harm. During streaming replicare's reads are short, so the job waits only a moment for one to finish.
- On one of replicare's change-tracking tables (for example a database-wide
VACUUM FULL), it briefly blocks the application's writes to the matching table, because the trigger cannot record the change until the job finishes. These tables are small, so the pause is short; keep it out of peak hours.
During the initial copy two more things apply:
- Copy reads are long: each chunk, and a table loaded in one statement (a self-referencing table whose link column cannot be emptied), keeps a read open for as long as it runs. While it runs, vacuum cannot remove rows that read might still need to see, anywhere in the database.
- An
ALTER TABLE(orVACUUM FULL/CLUSTER) on a table being copied waits for that read to finish, and the application's queries on the table queue behind it. Pause heavy maintenance and schema migrations during the initial copy, or run them with alock_timeout.
Long-open transactions are the real vacuum problem. replicare's change-tracking tables get
constant inserts and deletes and rely on autovacuum to reclaim the space. A transaction left open on
the source — typically an application session idle in transaction — stops vacuum from cleaning up
after it anywhere in the database, so those tables grow even though replicare keeps clearing them
(see the xmin-horizon caveat). Streaming keeps working; the
cost is disk. Ending the stuck session fixes it.
MySQL has no VACUUM; the equivalents are table rebuilds (OPTIMIZE TABLE,
ALTER TABLE … FORCE), which behave like the heavy jobs above, and long transactions holding back
InnoDB's purge, which behave like the open-transaction problem.
When a target goes down
A drain against an unreachable target fires all three channels at once: an errored trace span
with backlog attributes, an ERROR target.unreachable log, and replicare_target_up=0 plus a
climbing backlog series. The daemon keeps retrying each pass; nothing is lost. If the target stays
down long enough to exceed the retention cap, it is reseeded on recovery.
Whole-DB vs replicated size
replicare emits two size figures for each endpoint, and they answer different questions:
- Whole-database size —
replicare_source_db_bytes/replicare_target_db_bytes(Postgrespg_database_size, MySQL sum overinformation_schema.tables). This is the entire database the connection points at, including everything that is not the replicated data. - Replicated-data size —
replicare_source_replicated_bytes/replicare_target_replicated_bytes(SUM(pg_total_relation_size)— heap + indexes + TOAST — or MySQLdata_length + index_length, over just the sync's selected tables). Tables absent at an endpoint contribute nothing; a sync with no relational selection reports0.
Why the whole-DB gauges usually differ while the replicated-data gauges converge. On a healthy,
caught-up single→single Postgres sync you will typically see source_db_bytes larger than
target_db_bytes, even though the data is identical. That is expected, not drift:
- Trigger CDC writes to the source. replicare's own
replicareschema lives on the source: per-table delta and track tables, plus trigger functions. Under churn those delta tables are a high-write queue that bloats before autovacuum/purge reclaims it. None of this exists on the target. - Unreplicated tables count too.
pg_database_sizecovers the whole database, so any table on the source you did not select (audit logs, staging tables, keyless tables skipped for lacking a PK) inflates the source figure but never the target's. - The state store is a separate database. When the Postgres
StateStorelives on the target host it is its own database (e.g.replicare_state), sopg_database_size(current_database())for the target's data database does not count it.
The replicated-data gauges strip all of that away — same tables, same size function on both
sides — so they are the right series for an apples-to-apples "has the target caught up to the
source?" comparison. Watch source_replicated_bytes vs target_replicated_bytes for convergence;
watch source_db_bytes (against source_replicated_bytes) for capture/bloat footprint on a source
you may not own (see Source footprint).
Metrics reference
Every metric replicare exposes on /metrics (Prometheus text format; port 9090 in the demo,
configurable under observability:). These names, types, and labels are the single source of truth
locked in the observability contract, so a scrape of a running daemon matches this table exactly.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
replicare_rows_copied_total | counter | sync, table | Rows copied during the initial copy, counted once they are committed on the target (a retried chunk counts once). Tables linked in a loop count too; rows re-copied only to catch up with a busy source do not. Restarts from 0 with the daemon. |
replicare_initial_copy_rows_target | gauge | sync, table | Estimated total rows to copy (the progress denominator), from the source's own statistics (Postgres pg_class.reltuples, MySQL TABLE_ROWS; no scan). Set at bring-up for tables this run still has to copy; left unset for a table never analyzed. |
replicare_throughput_rows_per_second | gauge | sync | Current apply/copy throughput. 0 on an idle, caught-up pass. |
replicare_replication_lag_seconds | gauge | sync, target, table, component | Replication lag — age of the oldest unconsumed delta (0 when caught up). |
replicare_delta_backlog_rows | gauge | sync, target, table, component | Unconsumed delta rows (the queue depth). |
replicare_delta_backlog_bytes | gauge | sync, target, table, component | Estimated unconsumed delta bytes. |
replicare_delta_oldest_unconsumed_age_seconds | gauge | sync, target, table, component | Age of the oldest unconsumed delta (how stale the queue is). |
replicare_delta_purged_total | counter | sync, table | Delta rows purged after consumption. |
replicare_reseed_total | counter | sync, target | Forced reseeds triggered by the retention cap. |
replicare_target_up | gauge | sync, target | Target reachability (1=up, 0=down). |
replicare_source_up | gauge | sync | Source reachability (1=up, 0=down). |
replicare_source_db_bytes | gauge | sync | Source whole-database size in bytes (Postgres pg_database_size, MySQL information_schema; unset for engines that don't report it). Includes replicare's own capture schema + delta bloat and any unreplicated tables — see Whole-DB vs replicated size. |
replicare_target_db_bytes | gauge | sync, target | Target whole-database size in bytes (same engine support). |
replicare_source_replicated_bytes | gauge | sync | Size in bytes of just the replicated tables on the source (SUM(pg_total_relation_size) / MySQL data_length+index_length over the selection). The apples-to-apples source↔target figure — see Whole-DB vs replicated size. |
replicare_target_replicated_bytes | gauge | sync, target | Size in bytes of just the replicated tables on the target (same engine support). |
replicare_apply_batch_seconds | histogram | sync, target | Apply-batch duration per drain pass. |
replicare_errors_total | counter | sync, category | Errors by category (e.g. drain, retention). |
replicare_table_phase_info | gauge | sync, table, phase | Table lifecycle phase as an info gauge — value 1 on the active phase label (initial_copy/streaming); exactly one series per table. Present from bring-up, so it is the series to key dashboards on. |
replicare_delete_reconciliation_lag_seconds | gauge | sync, target, table | Redis only — duration of the last completed delete sweep (staleness of capture-less deletes). Unset for Postgres/MySQL. |
replicare_deletes_reconciled_total | counter | sync, target, table | Redis only — keys DELed by the target-vs-source delete sweep. Unset for Postgres/MySQL. |
The per-table backlog/lag gauges also carry a component label — the table's FK connected
component (its first member) — so you can roll lag and backlog up per component
(max by (component) (replicare_replication_lag_seconds)).
The headline signals to watch are replicare_source_up, replicare_target_up, replicare_delta_backlog_rows,
replicare_delta_oldest_unconsumed_age_seconds, and replicare_replication_lag_seconds — together
they answer "is the target healthy, and how far behind is it?" (see Source footprint
and When a target goes down).
A ready-to-import Grafana dashboard over these metrics — per-table backlog/lag, catch-up, initial-copy progress, throughput, and errors, with a configurable Prometheus data source — lives at Grafana dashboard.
Traces (OpenTelemetry)
Traces are exported via OTLP/gRPC when observability.otlp_endpoint is set. The span names are a
fixed registry:
| Span | Covers |
|---|---|
replicare.initial_copy.chunk | one chunk of the initial copy |
replicare.capture.install | installing trigger capture on the source |
replicare.delta.drain | one streaming drain pass for a table/target |
replicare.apply.component | applying one FK component's pass on the target |
replicare.reseed | a forced/retention reseed |
replicare.preflight | the start-up compatibility check |
A drain/apply span against an unreachable target is recorded with error status plus the backlog/age attributes, so the stall and its blast radius are visible in the trace timeline.
Structured logs (slog)
Logs are JSON or text (logging.format), leveled (logging.level), with a stable event field you
can grep/alert on:
| Event | Level | Meaning |
|---|---|---|
capture.installed / capture.removed | INFO | capture lifecycle |
table.cutover_to_streaming | INFO | a table finished initial copy |
retention.cap_approaching | INFO→WARN | delta backlog nearing the retention cap (escalates with proximity) |
target.unreachable / target.recovered | ERROR / INFO | target connectivity, cross-channel with the target_up gauge and the errored span |
target.needs_reseed | ERROR | a target was sacrificed to the retention cap and will re-copy |
preflight.blocked | ERROR | a sync refused to start on a blocking incompatibility |
A degrading target surfaces in metrics, traces, AND logs at once — never just one channel.
The status HTTP API
The daemon serves an operator HTTP surface (bound to observability.status_addr, e.g. :8080):
GET /status— JSON snapshot per sync/target/table: phase (initial_copy/streaming), rows copied vs total, replication lag, and last error, read from the state store.GET /healthz— a liveness probe (200 when the daemon is up), for Kubernetes/load balancers.GET /metrics— the Prometheus scrape. It is served here too, and additionally onobservability.metrics_addrwhen that is set to a different address (the conventional split).
The CLI replicare status reads the same state-store snapshot and adds live signals on top —
see the next section.
Checking progress from the CLI (no Grafana required)
When you can't deploy Grafana next to replicare — e.g. the daemon runs inside the source cluster because that's the only network path to the source databases — the CLI is your dashboard. Two read-only commands answer "what is it doing, and has it caught up?" directly against the config, with no metrics stack:
replicare status config.yml # phase, lag, LIVE row counts + delta backlog
replicare status config.yml --watch 5s # refresh in place until Ctrl-C
replicare verify config.yml # source<->target convergence spot-check (exit 0 = converged)
statusis live by default: each sync prints a one-line health headline (phase · last healthy pass · last data applied + rows · backlog) over a per-(table, target) grid. Two distinct time signals keep a caught-up sync legible:SEEN(time since the last healthy pass — a liveness heartbeat, small when running) andLAST_SYNC/ROWS(when data was last applied to that table and how many rows). A healthy idle sync shows a recentSEENwith a growingLAST_SYNC— normal, not lag. Live mode addsSRC_ROWS/TGT_ROWSand the per-target deltaBACKLOG—0= caught up,N (age)= waiting,-= unknown (couldn't read it; alive: partialnote says why), never conflated with0.--no-livedrops the source/target connections but keeps the state-store signals (phase,SEEN,LAST_SYNC, reseed).verifyis a read-only convergence check: it counts and content-fingerprints every replicated unit on both ends and reportsok/drift-count/drift-checksumper table, exiting non-zero on drift — the same count+checksum convergence logic the load-test harnesses use, folded into the shipped binary. The fingerprint is by value content, including for Redis: it hashes each key's type + canonical logical value (not just which keys exist), so a changed value on an existing key is caught, not only missing/extra keys. (Redis TTL is excluded — replicated relative, so exact remaining differs by design; stream consumer-groups/PEL aren't deep-verified yet.) A paused sync (enabled: false) is tagged[PAUSED]in the output, since convergence of a static source and target says nothing about whether the pipeline is running.
Both take --json (for scripting) and --watch <dur> (repeat on an interval). Both are read-only and
install nothing, so they're safe to run against a source you may not own.
Running the CLI in the pod (no shell needed). The container image is minimal, so don't rely on a
shell being present — invoke the binary directly, which kubectl exec does without any shell:
kubectl exec <pod> -- /usr/local/bin/replicare status /etc/replicare/config.yml
kubectl exec <pod> -- /usr/local/bin/replicare verify /etc/replicare/config.yml
The runtime image is Chainguard wolfi-base, which does include /bin/sh if you
want an interactive session (kubectl exec -it <pod> -- /bin/sh), but the direct-exec form above is
all you need for a progress check.
Running multiple pipelines in one instance
One replicare config — and one daemon process — can run several syncs across different engines at once. The single-engine rule is per sync, not per config: each sync's source and targets must share an engine, but a config may hold a Postgres sync and a Redis sync side by side, and the daemon runs them concurrently (each under its own single-active ownership lock).
A complete, annotated version of the example below is at
../examples/postgres-and-redis.yml.
state_store: # ALWAYS Postgres — even for a Redis-only deployment
engine: postgres
postgres: { host: pg-state, port: 5432, database: replicare_state, user: replicare, password: ${STATE_PW} }
sources:
pg-src: { engine: postgres, postgres: { host: pg-source, port: 5432, database: app, user: replicare, password: ${PG_PW} } }
redis-src: { engine: redis, redis: { mode: standalone, nodes: [redis-source:6379] } }
targets:
pg-dst: { engine: postgres, postgres: { host: pg-target, port: 5432, database: app, user: replicare, password: ${PG_TGT_PW} } }
redis-dst: { engine: redis, redis: { mode: standalone, nodes: [redis-target:6379] } }
syncs:
- name: pg-pipeline
source: pg-src
targets: [pg-dst]
include: ["public.*"]
- name: redis-pipeline
source: redis-src
targets: [redis-dst]
include: ["*"]
replicare run config.yml streams both. replicare status config.yml shows both (the Postgres
sync with row counts + delta backlog, the Redis sync with key counts and - backlog, since Redis has
no source-side queue). replicare verify config.yml checks both. Scope any command to one with
--sync pg-pipeline.
State store is always Postgres (v1's only backend), even for a Redis→Redis pipeline where no data touches Postgres — a documented requirement, not a bug. See redis-statestore.md. If you have no Postgres to spare, that is the one piece of infrastructure a Redis-only deployment still needs.
Pausing one pipeline
Set enabled: false on a sync to pause just that pipeline while the others keep running — e.g.
to work on the Postgres pipeline above while Redis keeps streaming:
syncs:
- name: pg-pipeline
source: pg-src
targets: [pg-dst]
enabled: false # paused; skipped at startup, no ownership lock taken
include: ["public.*"]
- name: redis-pipeline # still runs
source: redis-src
targets: [redis-dst]
include: ["*"]
The pause is data-loss-free: the paused sync's source capture stays installed, so deltas keep
queuing and setting enabled: true again (then restarting) drains the backlog. It is start-time
only — a config change + daemon restart, not a live command. Because it's not a live toggle, a long
pause lets the source-side delta tables grow and suspends the source-footprint
protection (retention runs in the streaming loop, which is
skipped) — so pause for maintenance/cutover, not indefinitely. Full semantics:
configuration.md → Pausing a sync. On Kubernetes, flip it
with helm upgrade (see kubernetes.md).
Forcing a reseed
To re-copy a target from scratch (e.g. after out-of-band divergence):
replicare reseed <config> --sync <name> --target <name>
This flags the target in the state store; the running daemon re-copies it on its next pass and resumes streaming. It signals the owning daemon rather than acting directly, honoring single-active ownership.
Sequences & identity counters (DR / promotion)
replicare replicates row values faithfully — the integer id a sequence produced lands on the
target verbatim (OVERRIDING SYSTEM VALUE on Postgres, the explicit value on MySQL). What it does
not replicate is the sequence object's counter — Postgres last_value, MySQL's per-table
AUTO_INCREMENT — because that is schema-object state and replicare is data-only.
For a passive replica this is harmless: the target never calls nextval() for a replicated
table (the applier always supplies explicit ids), so the stale counter just sits unused. It bites
only when the target starts generating ids — i.e. a DR promotion / failover: the app inserts,
nextval() returns a value at or near 1, and it collides with the replicated rows at id 1..N.
Before you open a promoted node to application writes, advance its counters past the data:
replicare reseed-sequences <config> --sync <name> [--target <name>] [--dry-run]
For every replicated table it sets the owned identity/serial/AUTO_INCREMENT counter to max(id)+1
on that target (--dry-run reports what it would set, writing nothing). Tables with no
locally-allocated counter (UUID / natural / composite PK) are skipped.
- Order matters: run it after replication into the node has stopped and before the app can write — a write in that gap could still grab a colliding id.
- Why
max(id)+1, not the source'slast_value: the promoted node's data is the replicated rows, somax(id)+1can't collide; any ids the source allocated beyond its max (gaps, or rows lost in the DR window) belong to rows that aren't present, so re-issuing them is harmless. - Privilege: this needs a grant beyond the DML set — Postgres
UPDATEon the sequence, MySQLALTERon the table (see Least-privilege grants). A missing grant fails loudly; the command reads only the target's ownmax(id)and never touches the source. - PASSIVE / one-way / DR ONLY.
reseed-sequencesrefuses a target that is an active-active cluster member: there the samemax(id)+1would collapse every node onto the same counter and silently drop rows via last-write-wins. Active-active id allocation is a schema concern — use globally-unique keys (UUID v7 / ULID); see multi-master → active-active id allocation. (Not the same asreseed, which re-copies a target's data.)
Keeping counters warm continuously (sync_sequences)
The promotion-time reseed-sequences run above is the guarantee. If you'd rather have the counters
kept advanced during streaming — so a failover has nothing sequence-related to do — opt the sync
into continuous sequence syncing on the neutral config layer:
syncs:
- name: app-to-dr
source: primary
targets: [dr]
sync_sequences: true # optional; default false
While streaming, the daemon advances each replicated table's owned counter on the target to
max(id)+1 on a lazy interval (the same mechanism as reseed-sequences). Because a passive
replica's applier always supplies explicit ids (OVERRIDING SYSTEM VALUE), which never advance the
sequence, repeatedly setting it is idempotent and simply holds the counter ahead of the data.
- Opt-in and off by default — unset or
falseis exactly the pre-existing behavior. - Steady-state privilege. Enabling it means the daemon's target connection needs the same
DR-only grant the command needs, now continuously: Postgres
UPDATEon each owned sequence, MySQLALTERon each table. Without it the advance fails — but best-effort: the failure is logged (stream.pass_errorwith asequence syncmessage) and retried next interval; it never aborts the streaming pass or stops data replication. - Still PASSIVE / one-way / DR ONLY. Same mesh refusal as the command, enforced at config
load:
sync_sequences: trueon a sync whose target is also an active-active cluster member is rejected byvalidate/runup front (advancing a mesh counter is silent data loss). - Not a substitute for the runbook. It narrows the promotion gap but the cutover order
(stop replication → open to writes) is unchanged; run
reseed-sequencesas the final belt-and- braces step if you want certainty at the instant of promotion.
Runtime type errors
If the target rejects a value (type/constraint), replicare halts the affected component loudly (logs + telemetry) rather than skipping or guessing. The offending row stays dirty and re-applies automatically once you fix the cause (e.g. a missing target type/extension, or a schema mismatch).
Restarts & ownership
Each sync runs under a single-active pg_advisory_lock; a second daemon pointed at the same sync
stands by. State (copy progress, cursors) is checkpointed in the Postgres state store — which is
always Postgres regardless of the data engine, so MySQL syncs restart and resume through the same
mechanism — so a restarted or rescheduled process resumes cleanly from the last checkpoint; an
ungraceful kill replays at-least-once and converges idempotently. One daemon can run Postgres, MySQL,
and Redis syncs side by side (each sync stays single-engine). A Redis sync checkpoints only coarse
phase (snapshot-complete-per-unit; no resumable SCAN cursor), so a crash mid-copy re-runs the whole
SCAN idempotently rather than resuming a chunk — see the Redis engine page.
Connection resilience & the liveness probe
A dropped source or target connection — RDS/managed failover, an idle NAT/LB reap, a brief network
blip — recovers on its own, no restart needed. All connections set a bounded dial timeout and TCP
keepalive so a dead peer surfaces as an error quickly instead of hanging, and after any failed drain
pass the daemon health-checks both endpoints and reconnects whichever is down (Postgres re-dials its
single connection and re-applies session GUCs; MySQL and Redis rebuild their pooled clients). A
reconnect that can't succeed yet (the endpoint is genuinely down) is retried each pass, so the daemon
never crash-loops on an outage — it heals when the endpoint returns. Recovery is visible as a
stream.reconnected event in /status and the logs.
/healthz reports streaming liveness as a backstop for a true wedge (e.g. a hung query no
timeout caught). It is deliberately not target reachability: a loop that keeps cycling — including
one correctly retrying against a down target — stays healthy (that state shows up in
replicare_target_up and the backlog metrics instead). The probe fails only when a streaming loop
completes no pass within observability.stall_timeout (default 2m), so Kubernetes restarts a pod
whose loop has stopped iterating. Set stall_timeout above your slowest expected drain pass (a large
drain_batch on a slow target lengthens a pass); a negative value disables the check. Liveness
tracking starts only once a sync reaches streaming, so a long initial copy is never mistaken for a
wedge.
Least-privilege grants
See ../deploy/grants-source.sql and
../deploy/grants-target.sql. Note the documented asymmetry:
installing capture needs only the TRIGGER privilege, but capture remove (dropping the trigger)
requires table ownership. For the true zero-database-grant path, pre-create the replicare
capture schema (and, if the state store lives there, the replicare_state schema) owned by the
daemon role — the migration runner then issues no CREATE SCHEMA and needs no CREATE-on-database
privilege. See the Postgres engine page.
For MySQL, use ../deploy/grants-source-mysql.sql and
../deploy/grants-target-mysql.sql. MySQL has no such asymmetry
— the TRIGGER privilege covers both CREATE TRIGGER and DROP TRIGGER, so the daemon role can
fully uninstall capture on its own.
For Redis, use the ACL presets ../deploy/acl-source-redis.txt
and ../deploy/acl-target-redis.txt. The source role is read-only
(+ping +info +scan +type +dump +pttl +exists +memory|usage); the target role needs +ping +info +restore +del +scan +exists +module|list. Both roles need +info (pre-flight reads INFO server), and in cluster mode both also need +cluster|shards +cluster|slots and the same user
granted on every master. Watch the foot-gun: RESTORE is @dangerous and must be granted
explicitly with +restore, or every apply fails loud. See the Redis engine page.
reseed-sequences needs one extra target grant beyond the DML set, and only on the node you run
it against (a DR promotion step): Postgres UPDATE on each replicated table's owned sequence (or
ownership of it); MySQL ALTER on each replicated table. It reads only the target's own max(id) —
no source privilege. Grant it as part of the failover runbook, not the steady-state role.
Load & convergence testing
Three manual load-and-verify harnesses drive replicare against broad, high-volume, realistic data and
assert the target converges with the source under continuous churn. They are developer/test
tools, run from a checkout (go run ./test/…), not part of the shipped binary:
- Postgres —
test/loadgen. Seeds a rich 10-table schema chosen to exercise the hard paths (FK components, composite/UUID/text PKs, PK-changingSKUrenames,GENERATEDcolumns, a keyless table it must skip; optional FK cycles under--cyclic), churns random inserts/updates/deletes, andverifys per-table row-count + ordered content checksum. - MySQL —
test/loadgen-mysql. The MySQL sibling of the Postgres harness: the same 10-table schema in MySQL types (AUTO_INCREMENT,JSON, compositeVARCHARPK,GENERATED … STORED, a keyless table), generated with MySQL-5.7-safe SQL (nogenerate_series/CTE/window functions), churned the same way, andverifyd with a cross-version, order-independent content checksum (BIT_XOR-folded per-row hashes). - Redis —
test/loadgen-redis. Seeds every value type (string/hash/list/set/zset/stream + big/TTL'd keys) plus a source-onlylgskip:*cohort the sync must exclude, churns value mutations,RENAMEs, and heavyDELs (the delete-reconciliation stress), andverifys a version-independent, type-aware content hash + TTL presence. Standalone, sentinel, and cluster.
All three support active-active load testing via --node-id (each writer node claims a disjoint
key slice so a mesh converges to the clean union); --node-id 0 is the default single-writer
(active-passive) behaviour. Each README documents the full end-to-end rig (two databases, a Postgres
state store, a config, the daemon, and a churn-and-verify loop) and the active-active workflow.
task loadgen:*, task loadgen-mysql:*, and task loadgen-redis:* are the shortcuts. These are
distinct from the automated Go suite (task test:integration).