Offline diffing & patching

When you butler push a build, patch generation happens in two phases. The first phase runs on your machine, the second runs on the itch.io backend after the upload is received. Both phases are built from primitives that ship with butler as standalone commands, so the entire pipeline can be reproduced with no network connectivity or in your own infrastructure. This page describes each phase, its intent, and the commands that perform it.

The two patching modes

A build's patch exists in two forms:

• The default patch is what butler push computes on your machine. It uses an rsync-style algorithm that matches fixed-size blocks between the old and new build, and compresses the result with Brotli at a low quality setting (quality 1). Both choices are deliberate: the diff is fast, memory use stays low, and the push finishes quickly. As soon as the upload completes, the build is live and players can update using this patch.

• The optimized patch is regenerated by the backend after the push. It re-diffs files with bsdiff, a byte-level diff that catches changes the block-based pass misses, and recompresses with Brotli at quality 9. This produces a significantly smaller patch, but costs far more CPU and memory, which is why it runs server-side instead of holding up your push. Once ready, it silently replaces the default patch.

See Pushing builds for how this fits into the push workflow. The sections below cover the local commands that implement each step.

Creating a default patch: butler diff

butler diff computes the differences between two builds, given as folders or archives. This is the same operation butler push performs locally. It generates both a patch file (patch.pwr) and a signature file (patch.pwr.sig):

butler diff old-build/ new-build/ patch.pwr

The signature file contains hashes of the new build. It is used later to confirm that applying the patch reproduces the new build exactly.

You can use /dev/null in place of the old build to produce a patch against an empty container. This works everywhere and does not require the special file /dev/null to actually exist or make sense in your operating system.

The old build can also be given as a signature file instead of the actual files. Since the block-based diff only needs the old build's block hashes, a signature is enough to diff against a build you no longer have on disk.

Pass --verify to immediately apply the generated patch in a temporary location and check the result against the signature, at the cost of a slower run.

Rebuilding a new version: butler apply

butler apply uses a patch file to transform an old build into the new one. This is also how the backend works: after a push, it applies the uploaded patch against the parent build's archive to reconstruct the new build's full archive, so players who want a complete download instead of a patch can get one.

butler apply --staging-dir tmp/ --signature patch.pwr.sig patch.pwr old-build/

Patching happens in place by default: old-build/ is transformed into the new version. To leave the old build untouched and write the new version somewhere else, pass a fresh directory with --dir:

butler apply --staging-dir tmp/ --dir new-build/ --signature patch.pwr.sig patch.pwr old-build/

--staging-dir is required. It holds temporary files and checkpoints, which make the operation resumable: if patching is interrupted, running the same command again picks up from the last checkpoint.

A signature can be given via --signature. Patched files are then checked against the new build's hashes, and the command fails if anything does not match. When no signature is given, butler assumes the folder being patched is a non-corrupted instance of the older version.

When patching in place, untouched files are not rewritten or checked. This allows for game updates that do not clobber mods sitting alongside the game's own files.

Creating an optimized patch: butler rediff

butler rediff takes a default patch and regenerates it with the expensive algorithms, exactly like the backend's processing step. It reads the default patch to learn which files changed and how they correspond, then re-diffs those files with bsdiff and recompresses with Brotli:

butler rediff --patch patch.pwr --old old-build/ --new new-build/ \
  --output patch-optimized.pwr --rediff-quality 9

Unlike diff, rediff needs the actual contents of both the old and the new build, because bsdiff works on raw bytes rather than block hashes. The backend uses Brotli quality 9 for optimized patches; the flag defaults to 1, so pass --rediff-quality 9 to match server behavior.

Expect this to take much longer than diff and to use much more memory. That trade-off is the entire reason the two modes exist: the default patch gets the build out the door, the optimized patch makes updates smaller for players.

--partitions and --concurrency control how the bsdiff work is split across CPU cores.

The optimized patch describes the same transformation as the default patch, so it is applied with the same butler apply command and validated against the same signature file. No new signature is needed.

Validating an optimized patch

Before the backend lets an optimized patch replace the default one, it applies the optimized patch and checks the output against the build's signature. To do the same locally, apply the optimized patch to a scratch directory with the signature given:

butler apply --staging-dir tmp/ --dir scratch/ --signature patch.pwr.sig \
  patch-optimized.pwr old-build/

If the command succeeds, the optimized patch rebuilds the new version correctly and can be used in place of the default patch.

The full pipeline, locally

Putting it all together, this reproduces what butler push and the itch.io backend do for a build update, entirely offline:

# phase 1: what `butler push` does on your machine
butler diff v1/ v2/ patch.pwr

# phase 2: what the backend does after receiving the push
butler rediff --patch patch.pwr --old v1/ --new v2/ \
  --output patch-optimized.pwr --rediff-quality 9

# validation: prove the optimized patch rebuilds v2 exactly
butler apply --staging-dir tmp/ --dir v2-rebuilt/ \
  --signature patch.pwr.sig patch-optimized.pwr v1/

Integrity checking: butler verify and butler sign

butler verify reads hashes from a signature file and compares them with the contents of a folder. It exits with a non-zero code if they don't match:

butler verify patch.pwr.sig some-build/

This can be used to check that an installation of a game wasn't corrupted. When given --heal with a path to a pristine copy of the build, it also repairs any files that fail the check.

butler sign generates a signature file for a directory, in the same format as the one butler diff produces, and suitable for use with verify, apply --signature, and as the old side of a diff:

butler sign some-build/ some-build.sig

This is useful when patching is irrelevant but integrity checking is important, or to keep a lightweight record of a build you don't want to store in full.

Inspecting files: butler file and butler ls

butler file displays whether a file is a patch file, a signature file, or another type of file, along with some general information about it.

butler ls displays the list of files contained in a patch file, or the list of files that can be checked via a signature file.

Using butler programmatically

butler's output tries really hard to be readable by humans, but on occasion, will lend itself to being parsed by other tools.

To enable JSON output mode, use the -j (or --json) flag. In JSON mode, each line butler outputs is a valid JSON object of the following form:

• {type: "log", "message": "Doing something", level: "info"}
• {type: "progress", "percentage": "80"}

This is, notably, how the itch app uses butler.