Storage space

Keeping every version of everything costs disk space. Synck stores history as compactly as it safely can, but it never throws history away to save space. This page explains where the space goes and the few tools that manage it.

Why history grows

Each new version of a file is kept, and is never removed while history refers to it. That includes versions of files you have since deleted, because a deleted file is in the trash, not gone. A folder whose current files take 1 GB can therefore hold several gigabytes of history after a year of edits to large files.

Synck has no automatic retention policy: it never prunes old versions in the background. If a repository has grown too large, the options are honest ones: more disk, or a new repository for future work, with the old one kept as an archive.

How Synck saves space

Repositories created with synck init use chunked storage. Each stored file is split into variable-size chunks of around 64 KiB, chosen by content, so an insertion near the start of a file does not shift every boundary after it. Each chunk is stored once, by its hash, and compressed with zstd when that actually makes it smaller. When you save a new version of a large file that differs only in places, most of its chunks are already stored, and only the changed ones take new space. Appending a paragraph to a 10 MB file costs about one chunk, not another 10 MB.

Microsoft Office files (.docx, .xlsx, .pptx) are ZIP packages, and a small edit can shift much of a ZIP file. Synck chunks them along the package’s internal entries, without unpacking or altering them, so that unchanged parts of the document stay shared between versions.

None of this affects identity. A file’s hash is always the SHA-256 of its complete, original bytes. Chunking is purely a storage detail, and the original bytes are reassembled and verified on every read.

Two layouts

A repository records its layout as storage_mode in .synck/config.json:

A whole-file repository is the simplest possible layout to inspect by hand, which makes it a natural format for exports. The cost is space: every version of a large file is stored in full. The layout never changes what a repository contains. Commit IDs, item IDs and file hashes are identical in both, and every command works with both. Inside .synck describes the files in detail.

To create a repository with whole-file storage, use synck init --storage-mode WHOLE_FILE.

Seeing what is stored

synck storage reports the content that history uses and anything that nothing refers to:

$ synck storage
referenced_objects 4
referenced_bytes 378
unreachable_objects 0
unreachable_bytes 0

referenced covers every file version and message body that any commit refers to, current or historical. unreachable covers content that no commit refers to. That can only happen when a commit failed after its content had been stored. The figures are logical sizes of file content. They do not include compression, the database or file-system overhead, or your working files themselves.

Removing unreferenced content

synck gc (garbage collection) deletes exactly the unreachable content:

$ synck gc
removed_objects 0
removed_bytes 0

gc never removes anything history refers to: not old versions, not files in the trash, not the content of named versions. It only reclaims what failed operations left behind. It runs only when you ask, so check synck storage first to see whether there is anything to gain. A copy made with synck archive or synck repository import never contains unreachable content in the first place.

Changing the layout

synck storage optimize converts a whole-file repository to chunked storage in place:

$ synck storage optimize
storage optimized

It works with or without a checkout; for a repository without one, run it inside the repository’s directory. Synck verifies the repository, stores every object as chunks, verifies the chunks, and only then switches the layout and removes the whole files. If it is interrupted before the switch, the whole files are still in charge and running it again continues the conversion. If it is interrupted after the switch, running it again removes the leftovers. On a repository that is already chunked, it reports storage already optimized.

To go the other way, or to keep the original untouched, make a converted copy instead. The source is never changed:

$ synck archive ../launch-whole-file --storage-mode WHOLE_FILE
$ synck repository import launch-project launch-chunked --storage-mode CHUNKED