Skip to content

V1 C: fault matrix, exact synthetic tokens, and baseline stubs - #4

Merged
loverustfs merged 1 commit into
mainfrom
feat/v1c-fault-matrix
Oct 15, 2026
Merged

loverustfs merged 1 commit into
mainfrom
feat/v1c-fault-matrix

Conversation

@loverustfs

Copy link
Copy Markdown
Contributor

Summary

V1 C adds the CPU fault matrix on top of the V1 A/B contract. A failed durable write is abandoned, so a short write or a full disk is not a later hit. Synthetic blocks are compared as exact token ids (tolerance=NONE). Wrong-tenant and wrong-model lookups return no payload.

GPU paired baselines are harness stubs that print NOT_RUN_HARDWARE and no timings. ADR 0001 stays Pending. No erasure coding, RDMA, or DPU work.

What is implemented

  • Protocol fuzz (random, truncated, duplicated, and reordered frames) inside cargo test
  • Fault tests: short write, truncated file, corruption, duplicate commit, commit-before-reserve, timeout, cancel, restart, injected disk full, Unix unwritable directory, pinned eviction race, concurrent TCP readers, tenant collision, model/tokenizer/adapter/layout changes, memory quota exhaustion
  • fault-demo: hit, miss, and recompute fallback from a clean temp directory
  • paired-baseline: engine-native prefix cache, local offload, and GPUCache CPU/TCP goodput, plus cold/warm/no-reuse/multi-turn/HBM rows
  • x86_64 and aarch64 CI jobs kept. An aarch64 build is not a BlueField test. TCP remains the correctness backend

What is not implemented

  • Real vLLM KV export/import and engine-output comparison (NOT_RUN_HARDWARE; ADR 0001 Pending)
  • Token goodput, TTFT/TPOT, and the five GPU workload timings (NOT_RUN_HARDWARE; no FAIL_PERFORMANCE figure because nothing was measured)
  • Filling a physical volume (NOT_RUN; the disk-full path is injected, and ENOSPC is classified)
  • Live RustFS (NOT_RUN_HARDWARE)
  • aarch64 execution on this x86_64 build host (NOT_RUN; CI jobs still run it)
  • DOCA/model license review (NOT_RUN_LICENSE) and paid GPU/DPU (NOT_RUN_BUDGET)
  • V2 RDMA and V3 DPU. No erasure coding

Repro

cargo test --workspace --all-targets
cargo run -p gpucache-transport --bin fault-demo
cargo run -p gpucache-bench --bin paired-baseline

Matrix: docs/V1C_TEST_MATRIX.md. Criteria: docs/CORRECTNESS.md.

Open in Web Open in Cursor 

Add protocol fuzz, fault injection, and concurrency tests for short
writes, corruption, duplicates, reordering, timeout, cancel, restart,
disk full, eviction races, tenant collisions, identity changes, and
quota exhaustion. A failed durable write is not a hit. fault-demo
checks hit, miss, and recompute with exact token ids. GPU baselines
stay NOT_RUN_HARDWARE.

Co-authored-by: RustFS <hello@rustfs.com>
@loverustfs
loverustfs marked this pull request as ready for review October 15, 2026 06:19
@loverustfs
loverustfs merged commit cf4910f into main Oct 15, 2026
4 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants