Operations

Data directoryCopy Anchor Link

<dir>/fusee.meta                      format, shard count and log format
<dir>/wal/<segment>.wal                the shared log (log format 3): segments
                                       from 1 MiB doubling to 64 MiB, reused
<dir>/shard-000/log/<segment>.seg      per-shard log segments (log format 2)
<dir>/shard-000/runs-<gen>.dat         run store (parked and finished runs)
<dir>/shard-000/checkpoint-<lsn>.fck   the two newest checkpoints (a second one
                                       of the same position: <lsn>b.fck)
<dir>/shard-000/visibility.redb        rebuildable search-attribute index
<dir>/shard-000/archive/              immutable retained journals, when enabled
<dir>/reshard.new/, reshard.old/       only while `fusee reshard` runs

The shard count is set when the directory is created. An engine opened with another count refuses the directory and names the command that changes it (see Changing the shard count); the TypeScript, Python and Go SDKs pass their shards option to the same check.

ToolsCopy Anchor Link

• fusee disk-check <dir> [seconds]: flush latency of the disk under dir (--json for machine-readable output).
• fusee inspect <dir>: log segments (the chain's batch ranges, records per shard, torn tails, spares), checkpoints and the log position each covers.
• fusee upgrade <dir>: converts a closed log-format-2 directory to the shared log (see Format versions).
• fusee reshard <dir> <shards>: changes a closed local directory's shard count (see Changing the shard count).
• fusee restore <target> <dir> [--at <time>], fusee backups <target> [--verify], fusee backup <dir> <target>: continuous backup (see Backup and restore).
• fusee server ...: server mode. Workers on the server's host can connect to its Unix socket with ?shm=1 to send log records through shared memory (server-mode.md); each such worker takes about 4 MiB of shared memory (unnamed, gone with both processes), and the server needs to read the socket's peer credentials.

RecoveryCopy Anchor Link

On open, each shard loads its newest valid checkpoint, reads the run index it names (32 bytes per parked run, written into the run store with each checkpoint: a base, then what changed), and replays log records after it. Once the deltas outweigh the base (half its bytes, or at least 64 delta frames and eight per 256 KiB of it), a new base is written between checkpoints, a few frames per 100 ms sweep, and the first checkpoint after it is complete names it with the deltas written meanwhile; the shard thread never rewrites the whole index at once, except after a run-store compaction. A checkpoint from before the index (format 2) makes the open scan the whole run store prefix instead; the first checkpoint after it rewrites the store in the current format. The shared log is read once, from its oldest retained segment, and each shard decodes only its records after its checkpoint. A torn tail (a flush that never completed) is discarded and the next write resumes in place after the last valid batch; damage to acknowledged records stops the open with an error naming the batch. Segments every shard's two retained checkpoints cover are retired: their header is zeroed and the file is kept as a spare that a later roll renames and reuses, so a steady workload creates and zero-fills no files. A clean shutdown checkpoints every shard (Config::checkpoint_on_shutdown) and keeps a second copy of that checkpoint; with every shard's records then covered by two files, the log moves to an empty segment and retires the rest, so the next open reads the log's header and no batches, however large the log had grown (a directory with a continuous backup keeps its segments). An open that replays a tail checkpoints soon after, so a crash loop does not grow the log. A failed flush stops the shard (Linux may already have dropped the dirty pages); restart the process.

Replicated groupsCopy Anchor Link

A replica's directory (see replication.md):

<dir>/vsr/log, vsr/superblock, vsr/commit   protocol log, view, commit hint
<dir>/vsr/members                           the membership, once it has changed
<dir>/vsr/admin.sock                        local admin socket (fusee server)
<dir>/engine/shard-NNN/                     each shard's run store and checkpoints

Failure detection. fusee server --heartbeat-ms 5 --election-ms 40 (the defaults). A backup suspects a silent primary after 40 to 80 ms, and a crashed one (its sockets closed) after about 20 ms. Raise the election timeout on hosts whose scheduling or disks stall for tens of milliseconds: an election timeout shorter than such stalls causes needless elections. A primary that hears from no quorum for four election timeouts (at least 200 ms) steps down.

Read leases assume each host's clock rate stays within 1 % of real time; the clock used counts suspended time. Do not restore a replica's VM from a memory snapshot while it may still be primary.

Hot standby. An embedded member that opens its engine with ReplicaNode::standby_config() while it is a backup takes over without opening an engine when it is promoted; open the next one when it loses the role (see replication.md, Failover timing).

vsr/log on Linux is written with O_DIRECT and RWF_DSYNC when direct_io is on (the default); a filesystem that refuses O_DIRECT gets buffered writes and fdatasync, with one line on stderr. Put <dir> on a filesystem that supports it (ext4 or XFS on the device).

Stage timings. ReplicaObserver::enable_timings records where each replicated step's time goes on every member (replication.md, Where a step's time goes); fusee-bench enables them in replicated mode. FUSEE_REPLICA_BUSY_POLL_US (off by default) keeps the I/O thread polling its sockets that long after any activity: measure it before using it, and only on hosts with cores to spare.

Membership. fusee replicas --dir <dir> status prints a member's role, view, epoch, commit and the members; fusee replicas --dir <dir> set id=host:port,... changes the membership from any member (it answers once that member has entered the new epoch, or after 30 seconds). One change adds or removes members or replaces a minority: a quorum of the new list must be members now.

• Add members (3 → 5): start each new member with an empty --dir, --join, its --replica-id and --replicas listing at least the current members (0=a:7400,1=b:7400,2=c:7400,3=d:7400,4=e:7400). Then fusee replicas --dir <dir> set with all five on any member. A joining member's status says joining until it is added.
• Remove members (5 → 3): set with the three that stay. The removed members report retired; stop them once status on the others shows the new epoch with their commit caught up.
• Replace a dead member: start its successor with a new ID and --join, then set the surviving members plus the successor. The surviving members must be a quorum of the new list (two of three).

The primary's engine stops at each change and reopens on the first member of the new list (the current primary if it stays), as after a failover. Under mutual TLS, a new member's certificate must name the host of its address. Once vsr/members exists it overrides --replicas.

Format versionsCopy Anchor Link

Segment headers, the run store and checkpoints carry format versions. A build refuses data newer than it understands with a message saying so. A release that changes a format converts old data with a command or refuses it clearly.

New local directories use log format 3 (log=3 in fusee.meta): one shared log for every shard. Config::shared_log(false) creates a directory with a log per shard (format 2) instead. To convert a closed format-2 directory (a local one, or a server-mode one with its server stopped), run fusee upgrade <dir> (or call fusee::upgrade::upgrade_to_shared_log). It copies every valid record of every shard into a new shared log under wal.tmp, syncs and renames it to wal, switches fusee.meta to log=3, and only then deletes the old segments, so a crash at any point leaves a directory that opens: before the switch in format 2 (the next attempt removes the partial wal), after it in format 3 (the next attempt removes leftover segments). Run stores, checkpoints and archives are kept as they are. The tool refuses a log with damaged acknowledged records instead of copying around them. New server-mode directories use log format 3 too (see server-mode.md); replicated directories keep their own logs and are not converted.

Changing the shard countCopy Anchor Link

fusee reshard <dir> <shards>

(or fusee::reshard::reshard(dir, shards, &ReshardOptions::default())) changes the shard count of a local directory in log format 3 while nothing has it open; the tool takes the directory's lock. It grows and shrinks. Opening the directory afterwards continues every workflow where it was. It writes a copy of every run store and history archive, so it needs that much free space, and takes time in proportion to them. Resharding is offline: stop the engine, run fusee reshard, and open it with the new count; the stop lasts as long as the tool takes to copy the run stores.

What moves. Every run the old shards know goes to the shard its workflow ID hashes to under the new count: running runs, suspended (parked) runs and finished runs still within retention, each with its whole journal. Timers, buffered signals, accepted updates and their results, search attributes and async steps waiting for a completion are journal records, so they move with their run. A run that was in memory at its shard's last checkpoint is in memory in the new shard's checkpoint and its code resumes at the next open (a step that was running when the engine stopped runs again, as after any restart); a parked run stays parked with its timers. Messages between shards that were not acknowledged, including those a log tail regenerates at recovery, move to the new shards' outboxes and are sent again; receivers drop repeats as they always do (child starts attach, signals keep their deduplication keys, child results for a finished wait are acknowledged). History archives are rewritten into the shards their IDs hash to. Schedules, debounce and batch coordinators are workflows and move like any other. The search-attribute index is rebuilt, and the new shards start a new shared log.

Run numbers. Run numbers count per shard, so runs from different old shards can share one. In each new shard a run keeps its number unless another run took it first (runs whose workflow is unfinished are placed first) or an older run of the same workflow ID had to move; the others get numbers above every number any old shard had used, in their old order, and later starts number from there. Growing to a multiple of the old count renumbers nothing, because each new shard draws from one old shard. Every reference to a run follows it: a child's parent, a run's predecessor, the next run a continue-as-new names, the parent in an unacknowledged message. A reference to a run that no longer exists (its retention ended) is rewritten to one that matches no run. Two things outside the directory name a run number and change for a renumbered unfinished run: its Ctx::key values (<workflow ID>/<run>/<position>, so a step that runs again after the change offers a service a different idempotency key) and its completion tokens. The command lists each renumbered unfinished run, and for every async step still waiting, the token handed out before and the one the step answers to now (ReshardReport::renumbered_unfinished); an old token is refused ("the workflow run this token belongs to is gone").

Crash safety. The new shards are built under <dir>/reshard.new and synced; until then the old directory is untouched except by the engine's own recovery, which reads it. The switch is fusee.meta, replaced atomically by one with format=reshard and the new count, which no build opens: an engine refuses it with a message to run the command again. Then the old wal and shard directories move into <dir>/reshard.old, the new ones move in, a final fusee.meta names the new count, and reshard.old is deleted. A crash before the switch leaves the directory as it was (the next run deletes reshard.new); a crash during it leaves a directory the next fusee reshard <dir> <n> finishes, every step checking what is done; a crash after the final fusee.meta leaves leftovers the next run deletes. fusee reshard with the count the directory has only does that cleanup.

Invariants (checked by crates/fusee/tests/reshard.rs):

1. Every run in an old shard's run table is in exactly one new shard, the one its ID hashes to, with the same journal except for run numbers, and in the same state: in memory if it was at the old checkpoint and has not finished, else parked with the same timers (a finished run is parked, as any open parks it).
2. Run numbers are unique within each new shard; among the runs of one workflow ID the newest has the highest number (the one its ID finds); every new shard's next number is above every number an old shard used, so a stale token or key never names a different run.
3. Every reference to a run names the same run after the change, or no run.
4. Every unacknowledged message is in exactly one new outbox: a child start in its parent's shard (which drops it when the child's result arrives), any other message in its target's shard. Deduplication keys of signals are unchanged.
5. Every moved record is at or before the new shards' checkpoint position (the end of the longest old log), so nothing waits for a position that is never written.
6. Before the switch the directory opens with the old count and holds everything it held; during it nothing opens it; after it, it opens with the new count.

Not covered: directories in log format 2 (refused: run fusee upgrade first), replicated directories (refused), and server-mode directories, which have the same layout but are untested with the tool; stop the server first in any case. A continuous backup refuses a target written for another shard count: give a resharded directory a new target. The tool reads the old log without a backup attached, so like any open without one it may retire segments every checkpoint covers; if a reshard is abandoned and the old directory goes back to its backup, the backup continues in a new timeline.

Backup and restoreCopy Anchor Link

A local directory in log format 3 can be backed up continuously to object storage, and restored to a point in time on another machine. Server-mode and replicated directories are not covered (a replicated directory's other replicas are its copies).

use fusee::backup::{BackupOptions, LocalTarget};
let target = std::sync::Arc::new(LocalTarget::new("/mnt/backup/orders")?);
// or, with the `s3` feature:
// let target = Arc::new(fusee::backup::s3::S3Target::from_url("s3://bucket/orders")?);
let engine = Engine::open(Config::local("./data").backup(target, BackupOptions::default())).await?;
let point = engine.backup().unwrap().sync().await?; // everything acknowledged so far

A background thread uploads every sealed segment of the shared log as soon as the chain moves past it, the active segment's acknowledged prefix every upload_interval (1 s by default), and every completed checkpoint with the part of the run store it names that is not uploaded yet and the history archives it covers. The write path does not wait for any of it: the backup reads files that are already durable, and the log keeps a segment the backup has not uploaded from being retired (fusee inspect shows it as part of the chain). While the target is unreachable, sealed segments therefore stay on local disk; the backup retries with backoff, says so on stderr and in Backup::status(), and catches up when the target returns. Engine::shutdown makes a last upload of everything and waits for it.

What can be lost. With upload_interval set, the log acknowledged since the last upload: about one interval plus the time an upload takes. With None, at most the active segment (up to the configured segment size, 64 MiB by default). A clean shutdown loses nothing.

Restore points. Every resolution, the backup reads the log's acknowledged position together with the time; once the log is uploaded through it, that pair is a restore point. Restoring it rebuilds exactly the batches acknowledged at that time: every workflow start, step and signal acknowledged by then is there, nothing acknowledged later is. fusee restore <target> <dir> --at <time> (or fusee::backup::restore) picks the newest restore point at or before the time (RFC 3339, Unix milliseconds, or -15m before now; without --at, the newest) and writes a data directory that opens and recovers like one after a crash at that moment: runs that were suspended resume, timers fire, finished results are readable. The directory is built under a hidden sibling (.<dir>.restoring) and renamed into place when complete, so an interrupted restore leaves no partial directory and a rerun starts over. <dir> must not exist or be empty. Points older than ten minutes are thinned to one per minute; for points older than an hour the backup keeps one checkpoint per shard per hour, so restoring one replays up to an hour of log.

Targets. A directory (LocalTarget, or file:///path and a plain path on the command line), or with the cargo feature s3 (on in the fusee command) an S3-compatible bucket: s3://bucket/prefix?region=eu-west-1, plus endpoint=http://127.0.0.1:9000&path_style=1 for MinIO and other stores and ca=/path/ca.pem for a private CA. Requests are signed with Signature Version 4 from AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN; HTTPS uses TLS 1.3 and the system's CA bundle (or SSL_CERT_FILE). Other stores implement fusee::backup::Target: put, get, list and delete of immutable named objects, and an atomically replaced manifest. One directory writes to a target (prefix) at a time: a backup whose manifest write finds a manifest it did not write stops.

Layout.

manifest.json                                    replaced atomically after each upload
state/<version>.json                             snapshot of the backup's state
<timeline>/log/<segment>/<offset>-<end>.wal      whole batches of one log segment
<timeline>/shard-NNN/checkpoint-<lsn>-<ms>.fck   a checkpoint, as written
<timeline>/shard-NNN/runs-<gen>-e<epoch>/<offset>-<end>.dat   run store bytes
<timeline>/shard-NNN/archive/<name>.arc          history archives

The manifest names a snapshot and the events since it (log chunks, checkpoints, store chunks, archives, restore points, with their CRC-32C); a new snapshot is written every 128 events and after retention. Small log and store chunks are merged as they accumulate (eight of one size into one of the next, up to 16 MiB), so a quiet log does not leave an object per upload. A checkpoint is usable for a restore point when it had completed before the point's batch was acknowledged; a point is restorable when every shard has a usable checkpoint (or the log reaches back to its first record) and the log from the oldest one is uploaded without a gap.

Crashes. Objects are uploaded before a manifest names them and deleted only after a manifest that no longer names them is in place, so the manifest names only complete objects whenever the process dies. The next open deletes objects no manifest names and continues right after the last batch the manifest has, after checking that the local log still holds that batch with the same checksum. A directory restored from a backup and backed up to the same target again starts a new timeline (unless it was restored from the newest point, which its history simply continues); restore considers every timeline's restore points by time, and --timeline <id> selects one. fusee backups <target> --verify lists timelines and restore points and reads every object the manifest names.

MemoryCopy Anchor Link

Parked runs take a 32-byte table entry each plus their share of two hash indexes (about 44 bytes in all); loaded runs are held to Config::memory_budget (128 MiB by default across shards). On an i8g.2xlarge, ten million parked runs over eight shards reopen in 252 ms with 447 MB resident (46.9 bytes per run, everything included).

A process that has just parked millions of runs holds more than that: glibc keeps what the loaded runs used. Two things keep it from staying. The engine fixes glibc's mmap and trim thresholds once per process (256 KiB and 1 MiB) at the first shard open; glibc otherwise raises both each time a large block is freed (a run index rebuilt into a bigger one), after which each shard's arena keeps up to twice that block resident for good, and malloc_trim cannot return it. Setting MALLOC_MMAP_THRESHOLD_ or MALLOC_TRIM_THRESHOLD_ in the environment leaves glibc's own values. And after a mass park, a background thread calls malloc_trim once a second until the memory stops falling. With two million parked runs in a Linux container, the two took the resident size from 203 MB to 147 MB, against 105 MB that malloc counts in use.

Do not set MALLOC_ARENA_MAX=2 for this: it saves memory, but the shards then share an allocator lock, and on an i8g.2xlarge it made creating ten million runs three times slower and waking them 2.7 times slower. jemalloc or mimalloc as the global allocator are fine.

Web UI and HTTP APICopy Anchor Link

The HTTP API (/v1, JSON; GET /v1/openapi.json describes it) serves the web UI at / and Prometheus metrics at GET /metrics, one series per counter per shard (fusee_<counter>{shard="N"}). It reads what the engine's own entry points answer, on a task of its own, so it adds nothing to a step. It is served by:

• the dev profile, at FUSEE_HTTP or else 127.0.0.1:7601 (fusee dev prints the address);
• Config::http(addr) or fusee::http::serve(engine, addr, HttpOptions) in Rust, engine.serve() in TypeScript, Python and Go;
• a server, with fusee server --http host:port.

It listens on loopback unless told otherwise, and off loopback it needs a token (Authorization: Bearer …; the UI asks for it once). A read token (--read-token-file) reaches lists, runs, results, events and metrics and nothing that changes a run; namespace tokens (--namespaces-file) reach the runs under one workflow-ID prefix, and cannot start the engine's own fusee.* types (schedules, delayed starts), which name other runs in their input. Behind a proxy that signs people in, --identity-header names the person in the records of the signals, cancellations, terminations and resets they send (see features.md). A request that changes something must carry a JSON body or X-Fusee-Action: 1, which a page on another site cannot send through a browser, and a browser Origin must match the host. Without a token, a loopback API answers only requests for this machine's names (localhost, 127.0.0.1, [::1]): a page on another site that points its own name at 127.0.0.1 (DNS rebinding) is refused. cargo run -p fusee --example ui serves a populated memory engine on http://127.0.0.1:7601.

Under the dev profile a run's step list is also a timeline: each step record carries when the step began and how long it took (a STEP_TIMED record, which production steps never write), and GET /v1/runs/{id} returns them as stepTimes.

MetricsCopy Anchor Link

Rust exposes Engine::shard_metrics().await and an aggregate Engine::metrics().await. TypeScript exposes engine.shardMetrics() and Python exposes engine.shard_metrics(). The shard number is the position in the returned list. Counters cover records and bytes, flushes, flushed records, flush time, gated waits and wait time, steps executed and replayed, checkpoints, parking, deterministic replay errors and busy rejections. Durations are in microseconds. A delta in flushed_records divided by the corresponding delta in flushes gives the mean batch size.

Applications that install an OpenTelemetry meter provider can enable the Rust crate feature opentelemetry and open Config::local(dir).opentelemetry(false). Instruments include per-shard append and executed-step counters and histograms for batch size, flush duration and gate wait duration. The rate of fusee.steps.executed gives steps per second. Passing true also records flush and gate-wait spans through the application's tracer provider. The application owns the exporter and destination. Fusee’s default configuration opens no telemetry connection.

TracesCopy Anchor Link

A run joins the trace of the request that started it. With telemetry on (telemetry: openTelemetry(api) in TypeScript, telemetry=fusee.open_telemetry() in Python, the opentelemetry feature in Rust), a start records the active span's W3C traceparent in the run's Start record. The run's code then runs under a fusee.workflow span in that trace, with a fusee.step span per attempt below it, in every process that runs it: after a crash the resumed code gets a new fusee.workflow span in the same trace, and replayed steps make none. Children and continue-as-new keep the trace. The HTTP API takes a traceparent header on starts; Rust takes .trace(traceparent) on the start builder, and workflow code reads the recorded one with ctx.traceparent() to parent the spans of the calls its steps make. fusee runs show and the journal show it as the start's trace. A start without one records nothing.

Signed releasesCopy Anchor Link

The release workflow verifies an annotated v* tag against the repository's configured public signers. Set the Actions variable FUSEE_RELEASE_SIGNING_KEYS to ASCII-armored public GPG keys, or set FUSEE_RELEASE_SSH_SIGNERS to an OpenSSH allowed-signers file containing the approved principals and public keys. Run scripts/verify-release-tag.sh v1.0.0 with the same environment locally. Tag verification precedes the jobs that build and upload registry artifacts.