Architecture

How replicare moves data, end to end: the overview for newcomers and architects. Everything here is engine-neutral unless noted — Postgres, MySQL, and Redis all ride the same pipeline, differing only behind the Source/Sink driver interfaces.

The one non-negotiable

replicare assumes it will never have a privileged, durable change stream — no Postgres WAL, no MySQL binlog, no durable Redis change feed. Change capture must work with ordinary, grantable privileges. That single constraint shapes everything below.

The pipeline

A sync (one source → one or more targets, with a table/key selection) runs this lifecycle. Copy and streaming are the two phases; capture and checkpointing wrap them.

 install capture ─► chunked parallel COPY ─► cut over ─► stream deltas ─► (repeat)
   (first)             (initial copy)                     (continuous)
        └──────────────── checkpoint to the StateStore throughout ─────────────┘
  1. Capture is installed FIRST (relational engines), so changes begin queuing before and during the copy. For Redis there is nothing to install — capture is replaced by re-reading the live keyspace (see Engines).
  2. Initial copy streams each table/unit in chunks, in parallel, source → target through an io.Pipe that never fully buffers a chunk. Because capture is already running and apply is idempotent, any change during the copy window reconciles automatically — there is no single frozen snapshot, which is how replicare handles sources too large to snapshot.
  3. Cutover flips each table to streaming once its copy completes.
  4. Streaming drains changes continuously and applies them idempotently, forever, until the daemon is stopped.
  5. Checkpointing records progress in the StateStore at every step, so a crash resumes from the last confirmed point (applies are idempotent, so replay is harmless).

The four ideas that make it non-obvious

Most of replicare's design collapses into four decisions. Understand these and the rest follows.

1. Capture is a dirty-key set, not an ordered log

Triggers on the source record only the changed row's primary key (PK-only capture) into a per-table delta table — never the full row image. At sync time replicare reads the unconsumed keys, re-reads the current values from the live source table, and applies those. This is self-healing: it always copies the latest value, so multiple changes to one key coalesce into one re-read and transient intermediate states don't matter.

Crucially, consumption is a set-difference (delta MINUS track, per target), not a WHERE seq > cursor scan. A scalar high-water-mark cursor is broken here: a value a trigger assigns inside a transaction (a sequence, a txid) is fixed when the trigger fires, but the delta row only becomes visible when the transaction commits — and commits happen in a different order than those values were grabbed. A slow transaction holding a low sequence that commits after the daemon's read would be skipped forever. The set-difference model can't skip: an uncommitted or late-committing delta is simply "not yet in track," caught on the next pass. Rows are deleted by unique delta_id (never by PK) to survive the read-and-clear race.

2. FK connected components are the unit of ordering, parallelism & consistency

Within a sync, replicare auto-partitions the selected tables into FK connected components (from the source catalogs, over the selected tables only). No FK edge crosses a component boundary, which gives three properties for free:

  • Ordering is always complete — upserts apply parent→child, deletes child→parent, within a component; no child is ever stranded from its parent. Cycles and self-references can't be topo-sorted, so a retry fallback handles them.
  • Distinct components are referentially independent → safe to copy and stream fully in parallel.
  • A component's changes for one drain pass apply in a single target transaction, so the target is always referentially consistent within the group.

A "giant component" (everything wired through a users/tenant hub table) erases parallelism, so replicare warns and lets you override.

3. Faithful transport — never transform

replicare is a pipe, not an interpreter. Values move verbatim: the source server renders them, the target server's own input function validates them. It never coerces, truncates, casts, or "repairs" data — on a type/constraint error it halts loud rather than corrupting silently. This is a hard product promise: no value-transform capability exists, even as an opt-in.

  • Relational engines move values as text COPY (both initial copy and delta apply), pinning formatting-only session settings (e.g. extra_float_digits=3) so text round-trips losslessly across version gaps.
  • Redis moves values as DUMP → RESTORE — RDB-serialized bytes, value-faithful (the target re-encodes), never reconstructed.
  • A pre-flight check classifies every source↔target type pair up front: block on incompatible/missing, warn on risky-lossy, ok on identical/widening — turning mid-stream failures into an actionable report before any sync starts.

4. The StateStore is separate from source-side capture

Two different kinds of state, often confused:

  • Source-side capture state — the delta and track tables — lives on the source database (that's where triggers write). It is the correctness-bearing replication state.
  • The StateStore — which syncs exist, per-table copy progress, per-target cursors, and the single-active ownership lock — is the daemon's own operational bookkeeping. It lives in a dedicated schema on a Postgres database (v1's only StateStore backend), which may be the target, the source, or a separate server.

They are separate concerns even when both happen to be Postgres. One consequence: every sync needs a Postgres StateStore — including a MySQL→MySQL or Redis→Redis sync (a documented wart for the non-Postgres engines).

How the three engines map on

The pipeline is identical; only the driver behind Source/Sink changes.

Postgres (reference)MySQLRedis
Capturetrigger CDC → delta/track tablestrigger CDC (same model)none — capture-less
Upsert CDCre-read dirty PKsre-read dirty PKsfull-keyspace SCAN reconciliation (re-read every pass)
Delete CDCdelta row for the deleted PKdelta rowdurable target-vs-source keyspace diff
Transporttext COPYLOAD DATA LOCAL INFILEDUMP → RESTORE REPLACE
UnitFK component of tablesFK component of tablesone key-namespace unit (redis.db<N>), single-member
Apply atomicityper-component transactionper-component transactionper-key (no cross-key atomicity)
Deliveryat-least-once; effectively-exactly-once when the StateStore is the targetat-least-onceat-least-once

Redis is the outlier: with no change stream at all, upserts come from re-reading the whole keyspace each pass and deletes from diffing the target against the source (keyspace notifications are an optional, lossy accelerator, never trusted). See the Redis engine page for the full story, and the Postgres / MySQL pages for the relational specifics.

Delivery, topology & bloat

  • Delivery: at-least-once with idempotent apply (upserts keyed on PK/key, deletes by PK/key), checkpointed cursors. A crash replays from the last cursor; replays are harmless.
  • Topology: user choice — default single→single; fan-out (one source → many targets) is supported for the relational engines; multi-master and HA/leader election are on the roadmap. A sync is always single-engine.
  • Source bloat: the delta tables are a high-churn queue on a source you may not own, so replicare bounds its footprint with per-table delta tables, a batched consumption-gated purge + aggressive autovacuum, and bounded retention with forced reseed of an over-cap target. See operations and the reseed state machine.

Process & observability

One daemon runs many named syncs as goroutine worker pools (one process, central observability). Everything is visible across four channels at once: a Prometheus /metrics endpoint, OpenTelemetry traces, structured slog logs, and a /status HTTP API. A degrading target must surface in metrics, logs, and traces simultaneously — see operations for the metric/span/event reference.