Operating replicare

Practical guidance for running replicare in production: the operator's view. How it works underneath is in Architecture.

Lifecycle of a sync

  1. Pre-flight. replicare validate <config> (and the daemon at startup) introspects the source and target and classifies every column pair: identical/widening → ok, risky/lossy → warn, incompatible or missing type → block (refuses to start). It also reports FK components, a giant-component warning, dangling FK edges, and skipped no-key tables. Fix blocks before running.
  2. Capture-first. The daemon installs trigger-based CDC on the source's selected tables, so changes begin queuing into per-table delta tables immediately.
  3. Initial copy. A chunked, parallel copy of each FK component (parents first) runs while capture is active. It is resumable — a restart continues from the last checkpoint, not the top.
  4. Cutover → streaming. Once a component is copied, it flips to continuous streaming: each pass re-reads the current source values of dirty keys and applies them idempotently, FK-ordered.
  5. Purge / retention / reseed. Consumed deltas are purged; a target that falls too far behind is reseeded (see below).

Before a migration: checklist

Work through this before pointing replicare at a production source. Most items are information to gather (sizing, traffic, quiet windows); a few must be true before starting. Run the queries on the source unless stated otherwise. replicare validate <config> then checks the rest (types, foreign-key loops, keys) and reports anything that would block.

Postgres

Sizing and traffic

  1. Total database size.
    SELECT pg_size_pretty(pg_database_size(current_database()));
    
  2. Row counts and sizes per table. row_estimate comes from planner statistics; -1 (or 0 on older versions) means the table has never been analyzed — run ANALYZE first for real numbers.
    SELECT n.nspname AS schema, c.relname AS table_name,
           c.reltuples::bigint                            AS row_estimate,
           pg_size_pretty(pg_total_relation_size(c.oid)) AS total,
           pg_size_pretty(pg_indexes_size(c.oid))        AS indexes
    FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
    WHERE c.relkind IN ('r', 'p')
      AND n.nspname NOT IN ('pg_catalog', 'information_schema', 'replicare')
      AND n.nspname NOT LIKE 'pg_toast%'
    ORDER BY pg_total_relation_size(c.oid) DESC;
    
  3. Write volume over 24 hours. The counters only ever go up, so run this twice, 24 hours apart, and subtract. Every row inserted, updated or deleted in a replicated table adds one row to replicare's change tracking, so inserts + updates + deletes per day is roughly the daily change-tracking volume (and the work streaming has to keep up with).
    SELECT now() AS sampled_at,
           sum(n_tup_ins) AS inserts, sum(n_tup_upd) AS updates, sum(n_tup_del) AS deletes,
           (SELECT xact_commit + xact_rollback FROM pg_stat_database
             WHERE datname = current_database()) AS transactions
    FROM pg_stat_user_tables;
    
    For the busiest tables, SELECT relname, n_tup_ins + n_tup_upd + n_tup_del AS writes FROM pg_stat_user_tables ORDER BY writes DESC LIMIT 20;.
  4. Low-traffic windows (daily and weekly). Busy periods followed by lulls let streaming catch up, and a quiet window is the best time for the final cutover.

Must be true before starting

  1. Every table has a primary key or a UNIQUE constraint. replicare tracks changes by key. It uses the primary key, or else the first UNIQUE constraint; a bare CREATE UNIQUE INDEX without a constraint does not count. Tables listed here are skipped (with a warning), not replicated:
    SELECT n.nspname AS schema, c.relname AS table_name
    FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
    WHERE c.relkind IN ('r', 'p')
      AND n.nspname NOT IN ('pg_catalog', 'information_schema', 'replicare')
      AND n.nspname NOT LIKE 'pg_toast%'
      AND NOT EXISTS (SELECT 1 FROM pg_constraint k
                      WHERE k.conrelid = c.oid AND k.contype IN ('p', 'u'))
    ORDER BY 1, 2;
    
  2. Extensions. List them on the source and confirm the target has every one the replicated tables use (column types, defaults). replicare never installs extensions; a missing one fails pre-flight.
    SELECT extname, extversion FROM pg_extension ORDER BY 1;
    
  3. Materialized views (and plain views) are not replicated — only table data is. Create them on the target from the schema dump, then REFRESH MATERIALIZED VIEW there after the initial copy and again just before cutover.
    SELECT schemaname, matviewname FROM pg_matviews ORDER BY 1, 2;
    
  4. Large objects are not replicated. Their contents live in pg_largeobject, not in a replicated table; a table's oid column is copied but the object it points to is not. Anything above 0 here needs a separate plan:
    SELECT count(*) FROM pg_largeobject_metadata;
    
  5. Schema changes are frozen for the whole replication. DDL is not replicated, so a schema change on the source has to be applied to the target by hand and can break the pipeline. During the initial copy there is an extra reason: an ALTER TABLE waits for replicare's copy read of that table to finish, and the application's queries on that table queue behind the ALTER.
  6. Regular vacuum and maintenance jobs. Find out what runs and when — autovacuum settings, scheduled VACUUM, and especially heavy jobs that lock whole tables (VACUUM FULL, CLUSTER, pg_repack). Plain VACUUM / autovacuum is fine at any time; the heavy jobs should be paused during the initial copy and kept out of peak hours afterwards. See Vacuum and other maintenance.
    SHOW autovacuum;
    -- scheduled jobs, if pg_cron is installed:
    SELECT jobid, schedule, command FROM cron.job;
    -- recent vacuum activity and the tables with the most dead rows:
    SELECT relname, last_vacuum, last_autovacuum, n_dead_tup
    FROM pg_stat_user_tables ORDER BY n_dead_tup DESC LIMIT 20;
    
    Also ask about jobs outside the database (cron, CI pipelines, the hosting provider's maintenance windows).
  7. No long-open transactions. Sessions left open — especially idle in transaction — stop vacuum from cleaning up (replicare's change-tracking tables bloat) and make trigger setup wait. This should normally return nothing:
    SELECT pid, usename, state, now() - xact_start AS open_for, left(query, 80) AS last_query
    FROM pg_stat_activity
    WHERE xact_start < now() - interval '5 minutes' AND pid <> pg_backend_pid()
    ORDER BY xact_start;
    
  8. Versions. SELECT version(); on both sides. The source can be old; the target should be the same major version or newer.
  9. Privileges and disk. The source role needs the grants in Least-privilege grants, and the source needs room for the change-tracking tables to grow with the write volume from item 3 (more if a target falls behind, up to the retention cap).
  10. Sequences at cutover. Row values are copied, sequence counters are not. Plan replicare reseed-sequences (or sync_sequences: true) before the target takes writes — see Sequences & identity counters.

MySQL

Numbers match the Postgres list. Postgres items 4 (quiet windows), 9 (schema freeze), 13 (privileges and disk) and 14 (sequences) apply as written; large objects (8) don't exist in MySQL, and the version check (12) is part of (6). MySQL versions of the rest, plus two MySQL-only items:

  • (1) Total size.
    SELECT ROUND(SUM(data_length + index_length) / 1024 / 1024) AS size_mb
    FROM information_schema.TABLES WHERE table_schema = '<DB_NAME>';
    
  • (2) Row counts, sizes and storage engine per table. table_rows is InnoDB's estimate.
    SELECT table_name, engine, table_rows AS row_estimate,
           ROUND(data_length / 1024 / 1024)  AS data_mb,
           ROUND(index_length / 1024 / 1024) AS index_mb
    FROM information_schema.TABLES
    WHERE table_schema = '<DB_NAME>' AND table_type = 'BASE TABLE'
    ORDER BY data_length + index_length DESC;
    
    Every table must be InnoDB — pre-flight blocks MyISAM/MEMORY tables.
  • (3) Write volume over 24 hours. Server-wide counters (all databases on the server); run twice, 24 hours apart, and subtract.
    SHOW GLOBAL STATUS WHERE Variable_name IN
      ('Innodb_rows_inserted', 'Innodb_rows_updated', 'Innodb_rows_deleted', 'Com_commit', 'Com_rollback');
    
  • (5) Every table has a primary key or a UNIQUE key (tables without one are skipped):
    SELECT t.table_name FROM information_schema.TABLES t
    WHERE t.table_schema = '<DB_NAME>' AND t.table_type = 'BASE TABLE'
      AND NOT EXISTS (SELECT 1 FROM information_schema.TABLE_CONSTRAINTS c
                      WHERE c.table_schema = t.table_schema AND c.table_name = t.table_name
                        AND c.constraint_type IN ('PRIMARY KEY', 'UNIQUE'));
    
  • (6) Extensions don't exist in MySQL. Instead check: MySQL 5.7 or newer (SELECT VERSION();, MariaDB is not supported); on the target, local_infile is ON (SHOW GLOBAL VARIABLES LIKE 'local_infile';) and the target tables have no extra UNIQUE keys beyond the one replicare uses (pre-flight blocks them — see the MySQL page).
  • (7) Materialized views don't exist in MySQL; plain views are not replicated (they come over with the schema dump).
  • (MySQL only) Existing triggers. Tables can keep their own triggers: from MySQL 5.7.2 a table may have several triggers per timing and event, so replicare's triggers sit alongside the application's and both fire. Only a source older than 5.7.2 (in practice a best-effort 5.6) allows a single trigger per timing and event, and there a table with its own AFTER INSERT/UPDATE/DELETE trigger blocks capture. Worth listing either way, to know what else runs on each write:
    SELECT event_object_table, trigger_name, action_timing, event_manipulation
    FROM information_schema.TRIGGERS WHERE trigger_schema = '<DB_NAME>';
    
  • (9) Schema freeze also covers online schema-change tools (pt-online-schema-change, gh-ost). They rebuild a table and swap it in, and the swap leaves replicare's triggers behind on the old table; pt-online-schema-change also adds triggers of its own.
  • (10) Maintenance jobs (the MySQL counterpart of vacuum). MySQL has no VACUUM; the jobs that matter are table rebuilds — OPTIMIZE TABLE, ALTER TABLE … FORCE / ENGINE=InnoDB — run from cron or the event scheduler. A rebuild locks the table briefly at the start and end; during the initial copy it waits for replicare's copy read and the application queues behind it, so pause rebuilds during the copy.
    SELECT event_schema, event_name, interval_value, interval_field, event_definition
    FROM information_schema.EVENTS;
    
  • (11) No long-open transactions — they hold back InnoDB's purge (the history list grows and replicare's change-tracking tables bloat) and make trigger setup wait:
    SELECT trx_mysql_thread_id, trx_started, trx_state, LEFT(trx_query, 80) AS query
    FROM information_schema.INNODB_TRX ORDER BY trx_started;
    
  • (MySQL only) A Postgres database for replicare's state is still required for a MySQL→MySQL sync (see the MySQL page).

Tuning knobs (per sync, under tuning:)

KnobMeaningDefault
drain_intervalTime between streaming drain passes. Longer = more per-PK coalescing (less source load) but higher lag.1s
drain_batchMax dirty deltas applied per table per pass. With drain_interval this is the per-table streaming ceiling.1000
apply_concurrencyHow many of a component's tables apply concurrently per pass. 1 is sequential; higher parallelizes across the copy-worker pool. Raise when several tables backlog at once.1
retention.max_ageCap on the oldest unconsumed delta before a laggard target is reseeded.24h
retention.max_bytesCap on the delta table's on-disk size before reseed.off
pool.max_source_connections / pool.max_target_connectionsConnection caps; the copy worker pool is sized from these.conservative

Defaults favor low source pressure. Raise the pool caps and shorten drain_interval for throughput; lengthen drain_interval to reduce re-read load under heavy churn.

A single high-volume table that lags

Streaming applies at most drain_batch deltas per table per pass, so one table's ceiling is roughly drain_batch / drain_interval rows/s (e.g. 1000 / 1s = ~1000 rows/s). When one big table (say a high-churn events) falls behind while the rest stay at zero backlog, you've hit that ceiling. Two levers, in order:

  1. Shorten drain_interval (1s → 200ms → 100ms): more passes/sec, linearly more throughput, at the cost of more frequent re-read SELECTs on the source. Best when the "Apply batch latency (p95)" is well under the interval (interval-bound).
  2. Raise drain_batch (1000 → 5000+): more rows per pass. More efficient than a tiny interval for a genuinely large table — one COPY-to-staging + upsert of 5000 rows amortizes round-trips far better than five of 1000 — and the right lever when apply latency is near/above the interval (apply-bound). Costs a larger re-read and apply per pass (more source and target work in one shot).

Watch - rate(replicare_delta_backlog_rows[5m]) per table: positive means the backlog is shrinking (catching up).

Several tables backlogged at once

The drain_batch/drain_interval levers above are per-table ceilings on a single-threaded drain: by default a pass applies each component's tables one at a time. When several tables are backlogged together, raise apply_concurrency — it applies a component's tables concurrently, across a pool built from the copy workers (idle during streaming), so they no longer queue behind each other. It's bounded by the connection caps: the pool has one primary pair plus up to apply_concurrency-1 worker pairs, and worker pairs come from pool.max_source_connections/max_target_connections (so raise those alongside it). FK order within a component is still honored per pass (a child that lands before its parent hits a transient FK and retries), and cyclic components on MySQL — which apply as one whole-component transaction — stay sequential. This does not speed up a single hot table (that's still drain_batch/drain_interval + target capacity); it removes the cross-table serialization when many tables are hot. As always, it helps only until the target's write capacity saturates.

Source footprint (the thing to watch)

replicare writes delta and track tables to the source — a database you may not own — so bounding that footprint is a first-class concern. The headline signals (per target/table):

  • replicare_delta_backlog_rows / replicare_delta_backlog_bytes — unconsumed work.
  • replicare_delta_oldest_unconsumed_age_seconds — how stale the queue is.
  • replicare_target_up — reachability (1/0).
  • replicare_reseed_total — forced reseeds.

Retention & forced reseed. A slow or unreachable target pins the delta queue (nothing purges until every target has consumed it), so replicare caps retention by age and/or size. A target over the cap is marked needs-reseed: its consumption record is reset, the unpinned deltas are purged, and it is brought back by a fresh full copy — trading a re-copy for a protected source. This is on by default. Watch retention.cap_approaching logs (they escalate INFO→WARN→ERROR toward the cap).

The xmin-horizon caveat. A long idle-in-transaction session (or other held-back xmin) on the source prevents autovacuum from reclaiming purged delta rows, so the delta table can bloat even though replicare purges it. replicare bounds the logical backlog regardless, but physical bloat under a pinned xmin is surfaced, not fixed — hunt down the idle transaction on the source.

MySQL (InnoDB) has the direct analog. InnoDB does not shrink the tablespace on DELETE, and a long-running source transaction inflates the history list, blocking purge of rows it can still see — the same shape as the xmin horizon. The same headline metrics apply (the delta/track tables live in a dedicated replicare database on the MySQL source), retention/reseed protects the source identically, and the fix is the same: end the long-running source transaction. See the MySQL engine page.

Redis has no source footprint at all. Its CDC is capture-less — replicare writes nothing to the source (no delta/track tables, no replicare schema). So the delta-backlog metrics, retention, and xmin-style bloat simply don't apply. What you watch instead is delete-propagation latency: deletes are found by the periodic target-vs-source sweep, so a source DEL converges only after the next sweep. replicare_delete_reconciliation_lag_seconds (sweep duration) and replicare_deletes_reconciled_total are the Redis-specific signals; tune cadence with delete_sweep_interval. Upserts converge faster via the rolling reconciliation scan. See the Redis engine page.

Vacuum and other maintenance

Plain VACUUM and autovacuum never conflict with streaming. While streaming, replicare runs only ordinary statements on the source — the trigger's INSERT into its change-tracking table on every application write, SELECTs that read those tables and re-read changed rows, and batched DELETEs that clear changes every target has applied. None of them takes a lock that a plain vacuum blocks, or that blocks a plain vacuum. The same holds on the target, where replicare only inserts, updates and deletes rows.

Trigger setup is the one place a plain vacuum can get in the way — only at startup, and only when a trigger actually has to be created or replaced (a new table, or a trigger someone removed). Adding a trigger needs a lock that a running vacuum holds. replicare waits at most 2 seconds for it, backs off and retries, and after repeated failures stops with an error naming the table, rather than holding the application up. A normal restart, with every trigger already in place, takes no lock on application tables at all.

Heavy maintenance that locks a whole table — VACUUM FULL, CLUSTER, pg_repack, ALTER TABLE:

  • On an application table, it blocks replicare's reads of that table just like the application's queries; replicare adds no extra harm. During streaming replicare's reads are short, so the job waits only a moment for one to finish.
  • On one of replicare's change-tracking tables (for example a database-wide VACUUM FULL), it briefly blocks the application's writes to the matching table, because the trigger cannot record the change until the job finishes. These tables are small, so the pause is short; keep it out of peak hours.

During the initial copy two more things apply:

  • Copy reads are long: each chunk, and a table loaded in one statement (a self-referencing table whose link column cannot be emptied), keeps a read open for as long as it runs. While it runs, vacuum cannot remove rows that read might still need to see, anywhere in the database.
  • An ALTER TABLE (or VACUUM FULL / CLUSTER) on a table being copied waits for that read to finish, and the application's queries on the table queue behind it. Pause heavy maintenance and schema migrations during the initial copy, or run them with a lock_timeout.

Long-open transactions are the real vacuum problem. replicare's change-tracking tables get constant inserts and deletes and rely on autovacuum to reclaim the space. A transaction left open on the source — typically an application session idle in transaction — stops vacuum from cleaning up after it anywhere in the database, so those tables grow even though replicare keeps clearing them (see the xmin-horizon caveat). Streaming keeps working; the cost is disk. Ending the stuck session fixes it.

MySQL has no VACUUM; the equivalents are table rebuilds (OPTIMIZE TABLE, ALTER TABLE … FORCE), which behave like the heavy jobs above, and long transactions holding back InnoDB's purge, which behave like the open-transaction problem.

When a target goes down

A drain against an unreachable target fires all three channels at once: an errored trace span with backlog attributes, an ERROR target.unreachable log, and replicare_target_up=0 plus a climbing backlog series. The daemon keeps retrying each pass; nothing is lost. If the target stays down long enough to exceed the retention cap, it is reseeded on recovery.

Whole-DB vs replicated size

replicare emits two size figures for each endpoint, and they answer different questions:

  • Whole-database size — replicare_source_db_bytes / replicare_target_db_bytes (Postgres pg_database_size, MySQL sum over information_schema.tables). This is the entire database the connection points at, including everything that is not the replicated data.
  • Replicated-data size — replicare_source_replicated_bytes / replicare_target_replicated_bytes (SUM(pg_total_relation_size) — heap + indexes + TOAST — or MySQL data_length + index_length, over just the sync's selected tables). Tables absent at an endpoint contribute nothing; a sync with no relational selection reports 0.

Why the whole-DB gauges usually differ while the replicated-data gauges converge. On a healthy, caught-up single→single Postgres sync you will typically see source_db_bytes larger than target_db_bytes, even though the data is identical. That is expected, not drift:

  1. Trigger CDC writes to the source. replicare's own replicare schema lives on the source: per-table delta and track tables, plus trigger functions. Under churn those delta tables are a high-write queue that bloats before autovacuum/purge reclaims it. None of this exists on the target.
  2. Unreplicated tables count too. pg_database_size covers the whole database, so any table on the source you did not select (audit logs, staging tables, keyless tables skipped for lacking a PK) inflates the source figure but never the target's.
  3. The state store is a separate database. When the Postgres StateStore lives on the target host it is its own database (e.g. replicare_state), so pg_database_size(current_database()) for the target's data database does not count it.

The replicated-data gauges strip all of that away — same tables, same size function on both sides — so they are the right series for an apples-to-apples "has the target caught up to the source?" comparison. Watch source_replicated_bytes vs target_replicated_bytes for convergence; watch source_db_bytes (against source_replicated_bytes) for capture/bloat footprint on a source you may not own (see Source footprint).

Metrics reference

Every metric replicare exposes on /metrics (Prometheus text format; port 9090 in the demo, configurable under observability:). These names, types, and labels are the single source of truth locked in the observability contract, so a scrape of a running daemon matches this table exactly.

MetricTypeLabelsMeaning
replicare_rows_copied_totalcountersync, tableRows copied during the initial copy, counted once they are committed on the target (a retried chunk counts once). Tables linked in a loop count too; rows re-copied only to catch up with a busy source do not. Restarts from 0 with the daemon.
replicare_initial_copy_rows_targetgaugesync, tableEstimated total rows to copy (the progress denominator), from the source's own statistics (Postgres pg_class.reltuples, MySQL TABLE_ROWS; no scan). Set at bring-up for tables this run still has to copy; left unset for a table never analyzed.
replicare_throughput_rows_per_secondgaugesyncCurrent apply/copy throughput. 0 on an idle, caught-up pass.
replicare_replication_lag_secondsgaugesync, target, table, componentReplication lag — age of the oldest unconsumed delta (0 when caught up).
replicare_delta_backlog_rowsgaugesync, target, table, componentUnconsumed delta rows (the queue depth).
replicare_delta_backlog_bytesgaugesync, target, table, componentEstimated unconsumed delta bytes.
replicare_delta_oldest_unconsumed_age_secondsgaugesync, target, table, componentAge of the oldest unconsumed delta (how stale the queue is).
replicare_delta_purged_totalcountersync, tableDelta rows purged after consumption.
replicare_reseed_totalcountersync, targetForced reseeds triggered by the retention cap.
replicare_target_upgaugesync, targetTarget reachability (1=up, 0=down).
replicare_source_upgaugesyncSource reachability (1=up, 0=down).
replicare_source_db_bytesgaugesyncSource whole-database size in bytes (Postgres pg_database_size, MySQL information_schema; unset for engines that don't report it). Includes replicare's own capture schema + delta bloat and any unreplicated tables — see Whole-DB vs replicated size.
replicare_target_db_bytesgaugesync, targetTarget whole-database size in bytes (same engine support).
replicare_source_replicated_bytesgaugesyncSize in bytes of just the replicated tables on the source (SUM(pg_total_relation_size) / MySQL data_length+index_length over the selection). The apples-to-apples source↔target figure — see Whole-DB vs replicated size.
replicare_target_replicated_bytesgaugesync, targetSize in bytes of just the replicated tables on the target (same engine support).
replicare_apply_batch_secondshistogramsync, targetApply-batch duration per drain pass.
replicare_errors_totalcountersync, categoryErrors by category (e.g. drain, retention).
replicare_table_phase_infogaugesync, table, phaseTable lifecycle phase as an info gauge — value 1 on the active phase label (initial_copy/streaming); exactly one series per table. Present from bring-up, so it is the series to key dashboards on.
replicare_delete_reconciliation_lag_secondsgaugesync, target, tableRedis only — duration of the last completed delete sweep (staleness of capture-less deletes). Unset for Postgres/MySQL.
replicare_deletes_reconciled_totalcountersync, target, tableRedis only — keys DELed by the target-vs-source delete sweep. Unset for Postgres/MySQL.

The per-table backlog/lag gauges also carry a component label — the table's FK connected component (its first member) — so you can roll lag and backlog up per component (max by (component) (replicare_replication_lag_seconds)).

The headline signals to watch are replicare_source_up, replicare_target_up, replicare_delta_backlog_rows, replicare_delta_oldest_unconsumed_age_seconds, and replicare_replication_lag_seconds — together they answer "is the target healthy, and how far behind is it?" (see Source footprint and When a target goes down).

A ready-to-import Grafana dashboard over these metrics — per-table backlog/lag, catch-up, initial-copy progress, throughput, and errors, with a configurable Prometheus data source — lives at Grafana dashboard.

Traces (OpenTelemetry)

Traces are exported via OTLP/gRPC when observability.otlp_endpoint is set. The span names are a fixed registry:

SpanCovers
replicare.initial_copy.chunkone chunk of the initial copy
replicare.capture.installinstalling trigger capture on the source
replicare.delta.drainone streaming drain pass for a table/target
replicare.apply.componentapplying one FK component's pass on the target
replicare.reseeda forced/retention reseed
replicare.preflightthe start-up compatibility check

A drain/apply span against an unreachable target is recorded with error status plus the backlog/age attributes, so the stall and its blast radius are visible in the trace timeline.

Structured logs (slog)

Logs are JSON or text (logging.format), leveled (logging.level), with a stable event field you can grep/alert on:

EventLevelMeaning
capture.installed / capture.removedINFOcapture lifecycle
table.cutover_to_streamingINFOa table finished initial copy
retention.cap_approachingINFO→WARNdelta backlog nearing the retention cap (escalates with proximity)
target.unreachable / target.recoveredERROR / INFOtarget connectivity, cross-channel with the target_up gauge and the errored span
target.needs_reseedERRORa target was sacrificed to the retention cap and will re-copy
preflight.blockedERRORa sync refused to start on a blocking incompatibility

A degrading target surfaces in metrics, traces, AND logs at once — never just one channel.

The status HTTP API

The daemon serves an operator HTTP surface (bound to observability.status_addr, e.g. :8080):

  • GET /status — JSON snapshot per sync/target/table: phase (initial_copy/streaming), rows copied vs total, replication lag, and last error, read from the state store.
  • GET /healthz — a liveness probe (200 when the daemon is up), for Kubernetes/load balancers.
  • GET /metrics — the Prometheus scrape. It is served here too, and additionally on observability.metrics_addr when that is set to a different address (the conventional split).

The CLI replicare status reads the same state-store snapshot and adds live signals on top — see the next section.

Checking progress from the CLI (no Grafana required)

When you can't deploy Grafana next to replicare — e.g. the daemon runs inside the source cluster because that's the only network path to the source databases — the CLI is your dashboard. Two read-only commands answer "what is it doing, and has it caught up?" directly against the config, with no metrics stack:

replicare status config.yml            # phase, lag, LIVE row counts + delta backlog
replicare status config.yml --watch 5s # refresh in place until Ctrl-C
replicare verify config.yml            # source<->target convergence spot-check (exit 0 = converged)
  • status is live by default: each sync prints a one-line health headline (phase · last healthy pass · last data applied + rows · backlog) over a per-(table, target) grid. Two distinct time signals keep a caught-up sync legible: SEEN (time since the last healthy pass — a liveness heartbeat, small when running) and LAST_SYNC / ROWS (when data was last applied to that table and how many rows). A healthy idle sync shows a recent SEEN with a growing LAST_SYNC — normal, not lag. Live mode adds SRC_ROWS / TGT_ROWS and the per-target delta BACKLOG — 0 = caught up, N (age) = waiting, - = unknown (couldn't read it; a live: partial note says why), never conflated with 0. --no-live drops the source/target connections but keeps the state-store signals (phase, SEEN, LAST_SYNC, reseed).
  • verify is a read-only convergence check: it counts and content-fingerprints every replicated unit on both ends and reports ok / drift-count / drift-checksum per table, exiting non-zero on drift — the same count+checksum convergence logic the load-test harnesses use, folded into the shipped binary. The fingerprint is by value content, including for Redis: it hashes each key's type + canonical logical value (not just which keys exist), so a changed value on an existing key is caught, not only missing/extra keys. (Redis TTL is excluded — replicated relative, so exact remaining differs by design; stream consumer-groups/PEL aren't deep-verified yet.) A paused sync (enabled: false) is tagged [PAUSED] in the output, since convergence of a static source and target says nothing about whether the pipeline is running.

Both take --json (for scripting) and --watch <dur> (repeat on an interval). Both are read-only and install nothing, so they're safe to run against a source you may not own.

Running the CLI in the pod (no shell needed). The container image is minimal, so don't rely on a shell being present — invoke the binary directly, which kubectl exec does without any shell:

kubectl exec <pod> -- /usr/local/bin/replicare status /etc/replicare/config.yml
kubectl exec <pod> -- /usr/local/bin/replicare verify /etc/replicare/config.yml

The runtime image is Chainguard wolfi-base, which does include /bin/sh if you want an interactive session (kubectl exec -it <pod> -- /bin/sh), but the direct-exec form above is all you need for a progress check.

Running multiple pipelines in one instance

One replicare config — and one daemon process — can run several syncs across different engines at once. The single-engine rule is per sync, not per config: each sync's source and targets must share an engine, but a config may hold a Postgres sync and a Redis sync side by side, and the daemon runs them concurrently (each under its own single-active ownership lock).

A complete, annotated version of the example below is at ../examples/postgres-and-redis.yml.

state_store:                     # ALWAYS Postgres — even for a Redis-only deployment
  engine: postgres
  postgres: { host: pg-state, port: 5432, database: replicare_state, user: replicare, password: ${STATE_PW} }
sources:
  pg-src:    { engine: postgres, postgres: { host: pg-source,  port: 5432, database: app, user: replicare, password: ${PG_PW} } }
  redis-src: { engine: redis,    redis:    { mode: standalone, nodes: [redis-source:6379] } }
targets:
  pg-dst:    { engine: postgres, postgres: { host: pg-target,  port: 5432, database: app, user: replicare, password: ${PG_TGT_PW} } }
  redis-dst: { engine: redis,    redis:    { mode: standalone, nodes: [redis-target:6379] } }
syncs:
  - name: pg-pipeline
    source: pg-src
    targets: [pg-dst]
    include: ["public.*"]
  - name: redis-pipeline
    source: redis-src
    targets: [redis-dst]
    include: ["*"]

replicare run config.yml streams both. replicare status config.yml shows both (the Postgres sync with row counts + delta backlog, the Redis sync with key counts and - backlog, since Redis has no source-side queue). replicare verify config.yml checks both. Scope any command to one with --sync pg-pipeline.

State store is always Postgres (v1's only backend), even for a Redis→Redis pipeline where no data touches Postgres — a documented requirement, not a bug. See redis-statestore.md. If you have no Postgres to spare, that is the one piece of infrastructure a Redis-only deployment still needs.

Pausing one pipeline

Set enabled: false on a sync to pause just that pipeline while the others keep running — e.g. to work on the Postgres pipeline above while Redis keeps streaming:

syncs:
  - name: pg-pipeline
    source: pg-src
    targets: [pg-dst]
    enabled: false            # paused; skipped at startup, no ownership lock taken
    include: ["public.*"]
  - name: redis-pipeline      # still runs
    source: redis-src
    targets: [redis-dst]
    include: ["*"]

The pause is data-loss-free: the paused sync's source capture stays installed, so deltas keep queuing and setting enabled: true again (then restarting) drains the backlog. It is start-time only — a config change + daemon restart, not a live command. Because it's not a live toggle, a long pause lets the source-side delta tables grow and suspends the source-footprint protection (retention runs in the streaming loop, which is skipped) — so pause for maintenance/cutover, not indefinitely. Full semantics: configuration.md → Pausing a sync. On Kubernetes, flip it with helm upgrade (see kubernetes.md).

Forcing a reseed

To re-copy a target from scratch (e.g. after out-of-band divergence):

replicare reseed <config> --sync <name> --target <name>

This flags the target in the state store; the running daemon re-copies it on its next pass and resumes streaming. It signals the owning daemon rather than acting directly, honoring single-active ownership.

Sequences & identity counters (DR / promotion)

replicare replicates row values faithfully — the integer id a sequence produced lands on the target verbatim (OVERRIDING SYSTEM VALUE on Postgres, the explicit value on MySQL). What it does not replicate is the sequence object's counter — Postgres last_value, MySQL's per-table AUTO_INCREMENT — because that is schema-object state and replicare is data-only.

For a passive replica this is harmless: the target never calls nextval() for a replicated table (the applier always supplies explicit ids), so the stale counter just sits unused. It bites only when the target starts generating ids — i.e. a DR promotion / failover: the app inserts, nextval() returns a value at or near 1, and it collides with the replicated rows at id 1..N.

Before you open a promoted node to application writes, advance its counters past the data:

replicare reseed-sequences <config> --sync <name> [--target <name>] [--dry-run]

For every replicated table it sets the owned identity/serial/AUTO_INCREMENT counter to max(id)+1 on that target (--dry-run reports what it would set, writing nothing). Tables with no locally-allocated counter (UUID / natural / composite PK) are skipped.

  • Order matters: run it after replication into the node has stopped and before the app can write — a write in that gap could still grab a colliding id.
  • Why max(id)+1, not the source's last_value: the promoted node's data is the replicated rows, so max(id)+1 can't collide; any ids the source allocated beyond its max (gaps, or rows lost in the DR window) belong to rows that aren't present, so re-issuing them is harmless.
  • Privilege: this needs a grant beyond the DML set — Postgres UPDATE on the sequence, MySQL ALTER on the table (see Least-privilege grants). A missing grant fails loudly; the command reads only the target's own max(id) and never touches the source.
  • PASSIVE / one-way / DR ONLY. reseed-sequences refuses a target that is an active-active cluster member: there the same max(id)+1 would collapse every node onto the same counter and silently drop rows via last-write-wins. Active-active id allocation is a schema concern — use globally-unique keys (UUID v7 / ULID); see multi-master → active-active id allocation. (Not the same as reseed, which re-copies a target's data.)

Keeping counters warm continuously (sync_sequences)

The promotion-time reseed-sequences run above is the guarantee. If you'd rather have the counters kept advanced during streaming — so a failover has nothing sequence-related to do — opt the sync into continuous sequence syncing on the neutral config layer:

syncs:
  - name: app-to-dr
    source: primary
    targets: [dr]
    sync_sequences: true      # optional; default false

While streaming, the daemon advances each replicated table's owned counter on the target to max(id)+1 on a lazy interval (the same mechanism as reseed-sequences). Because a passive replica's applier always supplies explicit ids (OVERRIDING SYSTEM VALUE), which never advance the sequence, repeatedly setting it is idempotent and simply holds the counter ahead of the data.

  • Opt-in and off by default — unset or false is exactly the pre-existing behavior.
  • Steady-state privilege. Enabling it means the daemon's target connection needs the same DR-only grant the command needs, now continuously: Postgres UPDATE on each owned sequence, MySQL ALTER on each table. Without it the advance fails — but best-effort: the failure is logged (stream.pass_error with a sequence sync message) and retried next interval; it never aborts the streaming pass or stops data replication.
  • Still PASSIVE / one-way / DR ONLY. Same mesh refusal as the command, enforced at config load: sync_sequences: true on a sync whose target is also an active-active cluster member is rejected by validate/run up front (advancing a mesh counter is silent data loss).
  • Not a substitute for the runbook. It narrows the promotion gap but the cutover order (stop replication → open to writes) is unchanged; run reseed-sequences as the final belt-and- braces step if you want certainty at the instant of promotion.

Runtime type errors

If the target rejects a value (type/constraint), replicare halts the affected component loudly (logs + telemetry) rather than skipping or guessing. The offending row stays dirty and re-applies automatically once you fix the cause (e.g. a missing target type/extension, or a schema mismatch).

Restarts & ownership

Each sync runs under a single-active pg_advisory_lock; a second daemon pointed at the same sync stands by. State (copy progress, cursors) is checkpointed in the Postgres state store — which is always Postgres regardless of the data engine, so MySQL syncs restart and resume through the same mechanism — so a restarted or rescheduled process resumes cleanly from the last checkpoint; an ungraceful kill replays at-least-once and converges idempotently. One daemon can run Postgres, MySQL, and Redis syncs side by side (each sync stays single-engine). A Redis sync checkpoints only coarse phase (snapshot-complete-per-unit; no resumable SCAN cursor), so a crash mid-copy re-runs the whole SCAN idempotently rather than resuming a chunk — see the Redis engine page.

Connection resilience & the liveness probe

A dropped source or target connection — RDS/managed failover, an idle NAT/LB reap, a brief network blip — recovers on its own, no restart needed. All connections set a bounded dial timeout and TCP keepalive so a dead peer surfaces as an error quickly instead of hanging, and after any failed drain pass the daemon health-checks both endpoints and reconnects whichever is down (Postgres re-dials its single connection and re-applies session GUCs; MySQL and Redis rebuild their pooled clients). A reconnect that can't succeed yet (the endpoint is genuinely down) is retried each pass, so the daemon never crash-loops on an outage — it heals when the endpoint returns. Recovery is visible as a stream.reconnected event in /status and the logs.

/healthz reports streaming liveness as a backstop for a true wedge (e.g. a hung query no timeout caught). It is deliberately not target reachability: a loop that keeps cycling — including one correctly retrying against a down target — stays healthy (that state shows up in replicare_target_up and the backlog metrics instead). The probe fails only when a streaming loop completes no pass within observability.stall_timeout (default 2m), so Kubernetes restarts a pod whose loop has stopped iterating. Set stall_timeout above your slowest expected drain pass (a large drain_batch on a slow target lengthens a pass); a negative value disables the check. Liveness tracking starts only once a sync reaches streaming, so a long initial copy is never mistaken for a wedge.

Least-privilege grants

See ../deploy/grants-source.sql and ../deploy/grants-target.sql. Note the documented asymmetry: installing capture needs only the TRIGGER privilege, but capture remove (dropping the trigger) requires table ownership. For the true zero-database-grant path, pre-create the replicare capture schema (and, if the state store lives there, the replicare_state schema) owned by the daemon role — the migration runner then issues no CREATE SCHEMA and needs no CREATE-on-database privilege. See the Postgres engine page.

For MySQL, use ../deploy/grants-source-mysql.sql and ../deploy/grants-target-mysql.sql. MySQL has no such asymmetry — the TRIGGER privilege covers both CREATE TRIGGER and DROP TRIGGER, so the daemon role can fully uninstall capture on its own.

For Redis, use the ACL presets ../deploy/acl-source-redis.txt and ../deploy/acl-target-redis.txt. The source role is read-only (+ping +info +scan +type +dump +pttl +exists +memory|usage); the target role needs +ping +info +restore +del +scan +exists +module|list. Both roles need +info (pre-flight reads INFO server), and in cluster mode both also need +cluster|shards +cluster|slots and the same user granted on every master. Watch the foot-gun: RESTORE is @dangerous and must be granted explicitly with +restore, or every apply fails loud. See the Redis engine page.

reseed-sequences needs one extra target grant beyond the DML set, and only on the node you run it against (a DR promotion step): Postgres UPDATE on each replicated table's owned sequence (or ownership of it); MySQL ALTER on each replicated table. It reads only the target's own max(id) — no source privilege. Grant it as part of the failover runbook, not the steady-state role.

Load & convergence testing

Three manual load-and-verify harnesses drive replicare against broad, high-volume, realistic data and assert the target converges with the source under continuous churn. They are developer/test tools, run from a checkout (go run ./test/…), not part of the shipped binary:

  • Postgres — test/loadgen. Seeds a rich 10-table schema chosen to exercise the hard paths (FK components, composite/UUID/text PKs, PK-changing SKU renames, GENERATED columns, a keyless table it must skip; optional FK cycles under --cyclic), churns random inserts/updates/deletes, and verifys per-table row-count + ordered content checksum.
  • MySQL — test/loadgen-mysql. The MySQL sibling of the Postgres harness: the same 10-table schema in MySQL types (AUTO_INCREMENT, JSON, composite VARCHAR PK, GENERATED … STORED, a keyless table), generated with MySQL-5.7-safe SQL (no generate_series/CTE/window functions), churned the same way, and verifyd with a cross-version, order-independent content checksum (BIT_XOR-folded per-row hashes).
  • Redis — test/loadgen-redis. Seeds every value type (string/hash/list/set/zset/stream + big/TTL'd keys) plus a source-only lgskip:* cohort the sync must exclude, churns value mutations, RENAMEs, and heavy DELs (the delete-reconciliation stress), and verifys a version-independent, type-aware content hash + TTL presence. Standalone, sentinel, and cluster.

All three support active-active load testing via --node-id (each writer node claims a disjoint key slice so a mesh converges to the clean union); --node-id 0 is the default single-writer (active-passive) behaviour. Each README documents the full end-to-end rig (two databases, a Postgres state store, a config, the daemon, and a churn-and-verify loop) and the active-active workflow. task loadgen:*, task loadgen-mysql:*, and task loadgen-redis:* are the shortcuts. These are distinct from the automated Go suite (task test:integration).