Skip to main content

Module reshard

Module reshard 

Expand description

Changes the shard count of a closed local data directory, offline (fusee reshard <dir> <shards>; design and invariants in docs/operations.md, “Changing the shard count”).

What moves. Every run the old shards know (running, suspended, finished within retention) goes to the shard its workflow ID hashes to under the new count, with its whole journal: its timers, buffered signals, accepted updates and their results, search attributes and pending async completions are all journal records, so they move with it. A run that was in memory at the old shard’s last checkpoint is in memory in the new shard’s checkpoint and its code resumes at the next open; a parked run stays parked, with its timers. Messages between shards that were not acknowledged (child starts and results, signals, cancels) move to the new shards’ outboxes and are sent again; their receivers drop repeats as they always do. History archives are rewritten into the new shards.

Reading. Each old shard is opened with the engine’s own recovery (checkpoint, run store, log tail, outputs the tail regenerates) and no workflow code, and is never ticked: no timer fires, no code runs, no record is appended and nothing expires.

Run numbers. Run numbers count per shard, so runs that come from different old shards may share one. Within each new shard a run keeps its number unless another run took it first (unfinished runs first); the others get numbers above every number any old shard had used, in an order that keeps the newest run of each workflow ID the highest. Growing to a multiple of the old count renumbers nothing (each new shard draws from one old shard). Every reference to a run is rewritten with it: children’s parent references, a run’s predecessor, the next run named by a continue-as-new, and the parent references in messages. A reference to a run that no longer exists (its retention ended) becomes one that matches no run. A renumbered unfinished run’s Ctx::key values and completion tokens change; ReshardReport::renumbered_unfinished lists them with the tokens their pending completions now answer to.

Crash safety. The new shards are built under <dir>/reshard.new and synced. The switch is fusee.meta: it is replaced (atomically) by one with format=reshard and the new count, which every build refuses to open. Then the old wal and shard directories move to <dir>/reshard.old, the new shard directories move in, and a final fusee.meta names the new count; only then is reshard.old deleted. A crash before the switch leaves the old directory, which opens as it was (the next attempt deletes reshard.new); a crash after it leaves a directory that reshard finishes (every step can be repeated); a crash after the final meta file leaves only leftovers the next attempt deletes. The new shards start a new shared log.

Structs§

Renumbered
An unfinished run whose run number changed: its Ctx::key values (<workflow ID>/<run>/<position>) and its completion tokens change.
ReshardOptions
ReshardReport

Enums§

Resharded
What reshard did.

Constants§

MAX_SHARDS
The most shards a directory can have: the language bindings pack a shard number into 10 bits.
SWITCHING
The format= value of a fusee.meta written while the directory’s files are being switched to a new shard count. No build opens it.

Functions§

reshard
Changes the shard count of the local data directory at dir to shards. The directory must be closed: this takes its lock. It must be in log format 3 (fusee upgrade converts an older one). Also finishes a reshard that was interrupted after its switch began.