Skip to Content
Fusee Docs
Internals

Benchmark

Method

The harness is bench/harness (fusee-bench). Every result line is JSON with the machine (CPU, cores, kernel, host load), the engine configuration and the durability boundary.

  • Durable step overhead: the time from one step returning to the next step starting, where the next depends on the first and the first is durable before it starts. Steps are no-ops; timestamps are taken inside the step functions. p50, p99, p99.9.
  • Closed loop (one workflow at a time) for pure latency; open loop (arrivals on a schedule) for throughput, with latency measured from each workflow's scheduled start so queueing is never hidden.
  • Workloads: (1) sequential chains of 1, 10, 100 and 1,000 steps; (2) fan-out of 100 parallel steps and fan-in; (3) steps calling a mock service with 1 ms and 50 ms latency; (4) highest sustained rate with step overhead p99 ≤ 1 ms; (5) start-to-first-step; (6) many sleeping workflows: memory and wake accuracy; (7) signal to resumed code; (8) SIGKILL at full load (fusee-crashtest); (9) CPU per million steps at three load points.
  • Floors: fusee-bench floors <dir> (flush latency and loopback round trip) and fusee-bench io-floors (4 KiB O_DIRECT + RWF_DSYNC writes at depth 1 on the measured volume); fio, io_uring and sockperf on the benchmark machines.
  • Matched durability: see bench/drivers/README.md. A comparison with another engine counts only when both make a step's result durable before the next step starts, release outputs only after durability, and survive the same failures. Margins are claimed only within a deployment class.
  • Procedure: warm-up, then five trials with engine and case order rotated; host load recorded with every line.

Instrumentation

  • Stall trace (any cell): --stall-us N turns on the engine's stall tracer (every tick or flush over N µs, with what else the shard was doing); --stall-trace FILE names the trace (default <dir>/stalls-<pid>.jsonl). Every result line carries stall_us and stalls, and each open-loop point and sub-cell its own stalls: count, max_us, by_kind (tick, inline-flush, flush), by_context (preparing_segment, checkpointing, parking, no_context; each with count, max_us, share), largest (five lines verbatim), and the engine's own engine_counted, engine_max_tick_us, engine_max_flush_us. null when off. The AWS scripts trace at 1 ms and archive <cell>.stalls.jsonl beside <cell>.jsonl.
  • CPU by thread: open-loop points and the extra cells report cpu over the measured window. On Linux, threads groups /proc/self/task by name (shard, flush, segment, checkpoint, remote, ping, replica, tokio, harness) with user, system and schedstat time; engine_us_per_step counts the engine's threads and the tokio workers, engine_dedicated_us_per_step leaves the tokio workers out (they also run the harness's per-workflow tasks), harness_us_per_step is the generator, and exited_threads_us is CPU of threads that ended inside the window. process_us_per_step is the whole process. Other systems report threads: null with a reason. cost shows both next to cpu_seconds_per_million_steps.
  • Interval series: --window-seconds W adds windows, latency and stall counts per W seconds of scheduled arrivals.
  • The log, per open-loop point: log gives the shared log's device writes, writes per second, shards' sections per write, the bytes the writes put on the device against the records' bytes (headers and padding), and segment rolls. The open loop starts each workflow and waits for its result with one engine call (Engine::execute_then), records into per-thread histograms, and above 100,000 starts a second paces arrivals from two threads (arrival_threads); their CPU is the generator's, in harness_us_per_step. Arrivals at least 500 µs apart are slept to within 75 µs and then spun to, so the generator is on time and its lateness is not counted as the engine's; closer arrivals are slept to with the smallest timer slack, at least 50 µs at a time (FUSEE_BENCH_SPIN=1 spins to every arrival).
  • Cluster runs: with chain among the workloads, cluster_run.py also runs five chain trials with adaptive receive moderation off on every member (--no-rx-off skips them) and, with --busy-poll-trial-us N, five with busy-polled receive; every member records ethtool -c and -S. check_cluster.py reports them as closed_loop_rx_off and closed_loop_busy_poll, with each trial's io_thread_tick and io_thread_lock_hold stages.

Extra cells

fanout-1000, fanout-10000; payload (1 KiB, 64 KiB, 1 MiB step results, --steps each); children (10 sequential and 100 parallel child workflows); service-concurrent (1 ms and 50 ms steps with --concurrency workflows in flight, 10,000 by default); saturate (open loop from --rate, times --growth per point (1.5 unless given), until step p99 passes 1 ms or throughput rises less than 2%, then --past points more (2 unless given) to show whether latency recovers as batches grow; an incomplete point ends it; highest_within_1ms is the highest complete point at or under 1 ms wherever it falls, beside highest_within_1ms_before_first_miss); soak (open loop at each of --rates for --seconds, a table row per rate with p50, p99, p99.9 and maximum).

Running on AWS

bench/aws runs every case on a fresh, disposable EC2 instance with no inbound access and verifies the archive it uploads. Set FUSEE_AWS_SPEND_LIMIT (USD) first: each launch reserves its worst-case cost against it and refuses to start past it.

  • host_run.py --case local|floors|gp3|cells|soak|sleepers|sdk|micro|pgo|comparisons for one host (--case cells [--cells …] [--cell-trials N]; --case soak [--soak-rates 1600,12800,51200] [--soak-seconds 600]; --stall-us N, 0 turns tracing off). pgo builds the harness with profile-guided optimization on the host, trained on the open loop and a mass wake.
  • cluster_run.py for three replicas (--workloads chain,open,saturate,failover).
  • check_local.py, check_sdk.py, check_cluster.py and the other check_*.py scripts recompute each summary from the archive.

Results

All on the release code, AWS spot instances, ext4.

One host

AWS i8g.2xlarge: Graviton4, 8 vCPUs, instance-store NVMe. The floor, a 4 KiB O_DIRECT + RWF_DSYNC write at depth 1, is 12 µs at p50 (p99 20 µs).

WorkloadResult
Sequential 1,000-step chain, step to step (five trials)p50 14 µs, p99 22 µs, p99.9 108–112 µs
Start to first step, open loop at 1,000 starts/sp50 24–27 µs, p99 40–44 µs
One-step workflow, start to result, memory modep50 6.2 µs
Fan-out of 100 parallel steps and fan-inp50 211–457 µs, three log flushes
Signal to resumed codep50 55–69 µs
Engine overhead per step calling a service (1 ms and 50 ms)p50 17–19 µs
Rust memory mode, 200,000 steps0.22 µs per step (best chain average over five repetitions), about 1,750–1,950 instructions per step
Step p50 from 3,200 to 51,200 steps/s, 32-step workflows14 µs at every point (1.03 records per flush)
1.64 million steps/sstep p50 51 µs, p99 101 µs; 46 / 97 µs with the profile-guided build
Highest rate with step p99 ≤ 1 ms6.55 million steps/s (p99 495 µs); 8 million with the profile-guided build (p99 708–763 µs, three trials of three)
CPU per step at 100,000 steps/s9.7 µs; 8.5 µs with the profile-guided build
Soak, 32-step workflows, ten minutes per rate51,000 steps/s: workflow p50 549 µs, p99 651 µs, p99.9 911 µs; 410,000: 1.9 / 3.3 / 3.7 ms; 1.64 million: 2.1 / 3.0 / 19.7 ms
Ten million parked workflows461 MiB resident (47 bytes each); reopened in 323 ms
Ten million timers due at once, eight shardsall fired within 4.2 s (p50 2.2 s), 2.4 million wakes/s
A million timers due at once, eight shardsall fired within 405 ms, 2.5 million wakes/s; within 331 ms (3.0 million/s) with the profile-guided build
Twenty SIGKILLs under loadno acknowledged workflow lost (1,521 workflows, 6,254 service calls)

On EBS gp3 (c7i.2xlarge) the same 1,000-step chain runs at p50 1,040 µs, within 21–39 µs of the volume's own 4 KiB durable write (1,001–1,019 µs).

SDKs

Same instance type. The SDK runs use local durability and eight shards.

Result
TypeScript (Node.js), step to step, one chain with seven workflows waitingp50 19.1 µs, p99 63.8 µs
TypeScript, eight chains at once on one threadp50 51.4 µs
TypeScript, fan-out of 100p50 441 µs
TypeScript worker threads, step p99 ≤ 1 ms256,000 steps/s on one thread, 805,000 on eight
Python, step to step, pipelined flushp50 21.8 µs, p99 92.4 µs
Server mode over shared memory, one shardp50 15.9 µs, p99 59.2 µs (local in the same test took 14.9 µs and 55.5 µs, respectively)
Server mode over a Unix socket / TCP loopbacklocal + 12–18 µs / + 20–21 µs

For the 1,000-step chains, each SDK percentile is the median of the reported values across 15 Node repetitions (sdk-live-1-7-default-*.jsonl) or five Python trials (sdk-python-pipeline-*.jsonl). The server benchmark alternates paths for seven rounds of 500 steps each. It takes the median of each percentile across rounds, then across three trials (server-step-cost-*.txt).

Replicated

Three i8g.2xlarge in one placement group; member round trips 30–134 µs (sockperf p50 by pair), backup flush p50 17 µs.

Result
Step to step, closed loop (five trials)p50 64–68 µs, p99 87–137 µs; with adaptive receive moderation off, p50 61–62 µs and p99 72–75 µs
Highest rate with step p99 ≤ 1 ms1.3 million steps/s in both trials (p99 813 µs), peak 2.9 million

Three i4i.large in one placement group (round trips 56–98 µs, flush 31 µs): chain p50 126–146 µs, p99 177–212 µs. Over 30 primary kills, the 10 failovers that completed before the next kill were ready for new work in p50 25.0 ms (at most 30.3 ms) and committed new work in p50 30.3 ms (at most 35.3 ms).

Operations

Previous Page

How Fusee is tested

Next Page

On this page

MethodInstrumentationExtra cellsRunning on AWSResultsOne hostSDKsReplicated