System design

This page explains how Synck’s parts work together, and why they are shaped the way they are. It follows one small project through writes, reads, failures and copying, and states the trade-offs as it goes. It assumes you have read Items, events and commits and Commit identity.

This is an explanation, not a specification. The normative contracts for the engine, the local repository formats and the complete-repository format ship with the source, in the docs/ directory of the Synck repository. Where this page and those contracts disagree, the contracts win.

1. Reading this page

Sections 2 and 3 set out what Synck is trying to achieve and the workload it is designed for. Sections 4 to 6 cover the write path: identity, concurrency and ordering. Section 7 covers reads, section 8 follows a write end to end, and sections 9 and 10 cover complete copies and the authority boundary. Section 11 explains why there are two local formats. Trade-offs and alternatives continues where this page ends.

A few terms recur:

  • Accepted: a commit has passed validation and is durably part of the history. Acceptance is the point of no return.
  • Sealed: an accepted commit has been given its repository sequence, and so belongs to an ordered, complete prefix of the history.
  • Closure: the set of content objects that a history refers to.
  • Authority: the single writer that accepts an item’s changes and orders them.
  • Published: a local output, such as a copy or an extracted file, has been installed at its requested path.

2. Goals and non-goals

Synck’s goals, in priority order:

  1. Durable ownership of data. A complete copy of a repository must remain readable, verifiable and usable without the service that produced it.
  2. Exact, append-only history. Accepted changes are immutable and attributable. Undo adds history.
  3. Stable identity. Items keep their identity through renames and moves, and history follows identity rather than paths.
  4. Atomic multi-item changes without making unrelated writers queue behind one another.
  5. Verifiability. Every commit and every content object can be checked against its hash, and every structural rule can be re-checked from history.
  6. A small engine. The core stays policy-free so that applications can be built on it without the core becoming an application.

And its deliberate non-goals:

  • No branching or merging. Each item has one authority and one linear history. Independently modified copies are never reconciled.
  • No access control. The engine records authors but does not authenticate them or enforce permissions. That is the application’s job.
  • No application services. Synck preserves a chat. It does not deliver messages, notify anyone or show presence.
  • No partial copies presented as complete. A copy either carries the entire closure of its history or it is not a complete repository.

3. The workload: a file and a conversation

Imagine a consulting team preparing a client proposal. Its repository contains a folder Launch, a file proposal.txt inside it, and a chat Launch review. Someone saves a revised proposal while someone else posts “Please check the budget.” Later the file is renamed, another message arrives, and an earlier revision of the file gets a named version label. A task to approve the budget, and the email sent to the client, relate to the same proposal.

These changes have different shapes:

  • a file save replaces one file’s content while keeping its earlier bytes;
  • a message append adds a record to a growing conversation;
  • a rename changes where something is, not what it is;
  • a label refers to an existing revision without changing it.

Representing all of that as files would need conventions for message files, IDs, ordering and relationships, and each application would invent its own. A plain application database supplies atomic transactions, but not independently interpretable history, reconstruction of past states, or a way to copy the complete history elsewhere. Synck supplies one shared history mechanism for all of these shapes. Typed readers and applications give it meaning.

The target is therefore broader than a source tree and narrower than an application platform. Synck can preserve a conversation without providing chat delivery, and email without becoming a mail server. Portability of stored data and portability of a running service are different promises, and Synck makes only the first.

4. Identity survives a change of location

Give the example items short labels:

Repository R
├── FOLDER F: Launch
│   └── FILE P: proposal.txt
└── CHAT C: Launch review
    ├── message: Please check the budget.
    └── message: The revised figures are ready.

Rename proposal.txt to approved.txt, or move it to another folder, and it is still item P. Its history does not depend on guessing that two similar paths describe the same thing. Conversely, deleting it and creating a new file at the old path does not give the newcomer P’s identity. A path is a current location. An ID identifies the thing whose changes are recorded. Messages belong to C’s event stream. They are records inside an item, not top-level items of their own.

The engine validates the namespace: parents must be folders in the same repository, an item has at most one parent, and folders cannot form cycles. Live siblings cannot share a canonical name. Names are compared after Unicode normalization and locale-independent lower-casing, with explicit character restrictions. That favors one predictable namespace on every platform over every naming trick a particular file system allows: two Linux files whose names differ only in case cannot both live in one Synck folder.

Lifecycle is part of history too. A trashed item disappears from the live tree but can be recovered. Tombstoning is final for future events on that item. It does not mean the earlier history or its content has been erased, and a complete copy must still account for both.

5. Concurrency is per item

5.1 Commits without a global head

A commit is an ordered group of events accepted atomically. Creating Launch and its first file can be one commit, and a failure rejects the whole thing rather than leaving half a folder behind. The shared FILE/FOLDER profile defines exactly which events each file operation produces, and checks the resulting live tree at the commit boundary, so that every client and every embedding agrees on what an accepted file operation means.

As Commit identity showed, a commit names no parent and no repository head. Ordering is per item: each event takes the next item_seq of the item it changes, and a change declares the base_seq it was based on.

5.2 What conflicts and what does not

Suppose one writer saves P while another appends to C. Their histories concern different items, so on a backend that supports concurrent transactions, neither waits for the other. There is no shared branch tip to contend for.

Two writers saving P, however, must agree on its next position. The proposal carries the head it observed as base_seq. The engine checks it while holding P’s conflict lock, and if another accepted write already moved P forward, the stale proposal fails instead of silently overwriting it. Some trusted append paths, such as adding the next chat message, let the engine fill in the base under the lock, so clients need not predict a busy conversation’s next position. That is still serialized acceptance on that one item, not parallel histories to be merged later.

The set of locks can be larger than the items being edited:

  • creating items under Launch takes Launch’s namespace lock, even with different names, so that duplicate names are impossible;
  • moving a folder must preserve ancestry and naming rules across the subtree;
  • a commit that changes both P and C joins both of their conflict domains.

Large atomic operations therefore trade a larger locked working set for a stronger all-or-nothing result.

5.3 What this is not

These are logical properties, not a performance claim. The local CLI takes an exclusive lock on the repository for each operation, and SQLite has one writer at a time, so independent item histories do not make one local directory a concurrent server. A PostgreSQL-backed embedding such as SynckHub can use the narrower conflict model, but busy items, namespace operations, storage latency and transaction work still bound throughput. One busy chat is one busy, ordered item.

6. Sequencing comes after acceptance

Readers still need one order for the whole repository: to scan changes, to build indexes, and to name checkpoints for --through. That order is the repository sequence, and it is assigned after acceptance:

 acceptance transactions                     after acceptance

 save FILE P ────────┐
                     ├──▶ accepted history ──▶ sequencer ──▶ repo positions
 append CHAT C ──────┘                                           │
                                                                 ├──▶ history index
 per-item order is fixed                                         └──▶ ordered consumers
 here, at acceptance

Why not simply number commits inside each write transaction? Because a transaction can reserve number 41, take longer to commit than the transaction that reserved 42, and become visible after a reader has already scanned past 41. That reader would silently miss a commit. The sequencer only numbers commits that are already accepted, and publishes their positions transactionally, so an ordered scan never skips an accepted commit. The cost is a separate stage with its own backlog and recovery. That is why a local write can report “accepted, but needs recovery”, and why repository finalize exists.

The repository sequence is an activity order for scanning. It does not record which reads a writer depended on and is not a universal causal clock. It never feeds back into commit identity.

7. Three read models

An append-only history does not mean every read replays everything. Synck keeps three kinds of derived state, each answering a different question:

Representation Answers When it fails
Canonical events and commits What happened, exactly? This is the history itself Corruption needs recovery from a good copy
Current item rows What is live under Launch now? Updated in the same transaction as acceptance A mismatch with history is an invariant failure, never ordinary lag
History index What did the repository contain through sealed position N? Rebuild it from intact history. Rebuilding cannot restore missing content
Application consumers Search, previews, activity feeds, notifications Diagnose the consumer. Freshness varies by design

Current rows make listing a folder cheap: no replay at all. To show an item, Synck replays only that item’s own events and refs, with typed readers interpreting them. A long chat is still more work to show than a folder is to list. Current rows make reads cheap, not free.

The history index maps each sealed repository position to an immutable snapshot of item state, namespace and history heads, built as a Merkle tree that shares unchanged structure between positions. It makes reading old states fast. Writers never have to agree on its root before they commit. It is derived after acceptance, so it can lag, and it can be rebuilt with fsck --rebuild-state-index.

Application consumers are where an embedding such as SynckHub does the rest of its work. SynckHub also materializes the fields its applications need inside the acceptance transaction, so an ordinary read that starts after a save sees that save. Search text, thumbnails and activity feeds have their own consumers and freshness, so a file can already be current while its search text is a little older. Moving all of that into the acceptance transaction would make every write pay for unrelated work.

8. Following a write end to end

The complete-repository CLI offers a small set of writes on an activated repository. Following two of them shows the stages above in action.

8.1 Saving the proposal

synck repository save-file P new.txt:

  1. Checks that P is a locally owned FILE, and loads its current head.
  2. Copies the new bytes into object staging and hashes the bytes it actually copied. It does not trust a file name or a checksum supplied by the caller. If the object already exists, it is verified before reuse. New objects are made durable before any history can refer to them.
  3. Builds the canonical content-update events using the observed head as base_seq, and has the engine accept them in one transaction, together with the new current state. P keeps its ID. Its old content stays reachable through its old event.
  4. Sequences the commit and brings the history index forward, then returns the commit ID.

8.2 Appending the review message

synck repository append-chat C message.txt:

  1. Requires an existing, locally owned CHAT, and validates the UTF-8 body.
  2. Encodes a body object with a fresh random salt, and stores it before any event refers to it.
  3. Has the engine append a CHAT::MESSAGE::CREATE event to C, filling in the base under C’s lock.
  4. Sequences and indexes as for a file save.

It does not create chats, deliver notifications or reproduce SynckHub’s message service.

8.3 The boundary that matters after a failure

Storing content and accepting history are separate stages, in that order:

  • a failure before acceptance can leave unreferenced objects, but can never leave accepted history pointing at missing bytes;
  • a failure after acceptance leaves the commit accepted, however the later stages fare.

So when a write reports “accepted, needs recovery”, the right response is to finish it with finalize, never to repeat it. The engine deduplicates retries of the same append intent through operation IDs, but two separate CLI invocations are two intents, and repeating one would record a second file version or a duplicate message. Reads of a complete repository never perform this recovery silently.

9. Portability means carrying the closure

9.1 A current read is not a copy

Right after the proposal save commits, an ordinary current read may already show the new content while its repository position is still unassigned. That is valid current state, but it is not yet part of a sealed prefix that could be exported.

A producer of a complete copy must capture a consistent boundary: every visible accepted commit sealed, the current rows matching exactly that history, and all required content available. Running the sequencer and then taking a database snapshot is not enough, because another write can be accepted in between. The producer must detect that and retry, or refuse. It must never publish an older history beside a newer namespace. “The file opened” and “the repository copy is ready” are different checks.

9.2 The closure

A complete repository has a deliberately plain physical shape (see Inside .synck): one SQLite database for history, current state and the history index, and one directory of objects named by their SHA-256.

The objects it must contain are the closure of the retained history: for the example, every retained version of the proposal, both message bodies, and everything that ref events and trashed or tombstoned items refer to. Renaming the file, editing a message or trashing an item does not remove an older reference. Collecting only today’s tree gives you a useful file export, not a history copy.

Closure discovery walks every retained item and ref event and follows typed references. It does not infer completeness from current heads, or from strings that happen to look like hashes.

9.3 Making a copy

A supported copy locks its source, uses SQLite’s backup mechanism (which includes committed write-ahead-log contents), and copies the required objects into private staging. There it verifies the history, the replayed item and ref heads, the history index and the content closure, and then publishes to an absent destination without replacing anything. A missing object, an unknown codec or malformed history prevents publication.

A copy preserves identities, attribution, timestamps and canonical bytes, including body salts, and adds no commit. It is bounded by its checkpoint. No local tool can recover a message that existed only in later source history, or recreate a missing file from its hash. Once a verified copy is in hand, though, losing the source does not prevent reading any of the retained history.

10. Authority, permissions and services stay explicit

A copied complete repository starts as a follower. Reading needs neither an author nor an account. Writing needs an explicit decision about authority:

  • Checkout creates an independent local line of history. The source may carry on, and the two never rejoin.
  • Activation with --source-retired declares succession. The copy continues the source’s history because the source has stopped. The flag acknowledges an operational fact. Synck cannot contact or fence a machine you do not control, and two activated copies would be two writers with no reconciliation path.

Items sequenced by an external authority, as in federated setups, keep their stream identities and sequencing modes in every copy. Local activation never acquires their authority.

A complete directory also has a different access boundary from a hosted API. Anyone holding it can read everything retained in it. Historical permission metadata and author strings do not authenticate a local reader or hide rows, and hash verification cannot establish who produced a copy or encrypt it.

The open-source engine, CLI, shared definitions and formats provide history and local workflows. SynckHub’s applications and platform add authentication, authorization, tenancy, live services and product workflows. A saved chat is portable data, not a portable chat account.

11. One repository format, with or without a checkout

Every repository uses the same format (format 6). It holds the retained history of every item family, and it does not need any files on disk to be read, verified or copied. A checkout adds a folder of ordinary editable files on top. A scanner detects changes using file-system identity and a stat cache, the CLI turns them into canonical file and folder events, and there is no staging area: a commit takes the whole observed change or the paths named in that invocation. Saved .eml and .msg files become email items rather than files. A checkout can be partial, and items with no place in a folder, such as chats, stay readable through the item commands.

Content is stored in one of two physical layouts. Chunked storage keeps repeated saves of large files affordable. Whole-file storage keeps every object as a plain file named by its SHA-256, checkable with standard tools. The layout is a property of one directory, not of the history: commit IDs, item IDs and content IDs are identical in both, copies can convert between them, and every read reassembles and re-hashes the original bytes. Changing a format number in a file is never a migration.

What all of this costs, and when another system is a better fit, is the subject of Trade-offs and alternatives.

Previous
Inside .synck
Next
Trade-offs and alternatives