Skip to content

V2: NIC RDMA transport (feature-gated) + shard directory + G4 cold path - #5

Merged
loverustfs merged 1 commit into
mainfrom
cursor/v2-rdma-shard-cold-8b62
Oct 15, 2026
Merged

loverustfs merged 1 commit into
mainfrom
cursor/v2-rdma-shard-cold-8b62

Conversation

@loverustfs

Copy link
Copy Markdown
Contributor

Summary

First V2 slice. V1 A/B/C stay green. TCP is still the correctness path. Live RDMA, live RustFS, GPU export, GDS, and BlueField are NOT_RUN_HARDWARE. No erasure coding and no durability replicas.

  • Transport. KvTransport with TcpTransport as the default. submit is one bounded host copy (dma_posted=false). Feature rdma-nixl compiles and adds no crate. NixlTransport::open returns rdma not linked with or without the feature. NIXL was chosen over Mooncake after a binding/license audit (nixl-sys 1.3.1, Apache-2.0, not pinned). stub-api aborts on call, and this workspace forbids unsafe, so libnixl is not linked. ADR 0004.
  • Registered buffer pool. Bounded Vec<u8> slots with batch checkout, registration reuse, backpressure, quarantine on unknown completion, and no free on lease expiry. Region ids are not recycled. This is not ibv_reg_mr.
  • Data paths. docs/V2_DATA_PATHS.md lists P/D handoff, G3.5 reuse, and G4 refill hop by hop (GPU / NIC / host / NVMe). No full zero-copy claim.
  • Shard directory. One process is one shard (single node = shard 0, same API). Membership epoch fencing, single-flight, and capacity credits. No per-token central directory and no per-block quorum. The directory stores no payload. ADR 0005. Client wire format (GCK1 / GCKR) is unchanged.
  • Cold path. Mock Put/Head/RangeGet/Delete, pack/manifest, bounded async sink, and cost-capped refill. Over budget returns no bytes. Live RustFS still depends on GPUCACHE_RUSTFS_*.
  • Docs. README status table, COMPATIBILITY, REUSE_AUDIT, BENCHMARKS, and the two ADRs. V3 is untouched.
  • Tests / demos. cargo test --workspace --all-targets and cargo test -p gpucache-transport --all-targets --features rdma-nixl do not need a GPU, DPU, or Soft-RoCE. shard-demo and cold-demo print the NOT_RUN_HARDWARE labels. paired-baseline still prints no timings.

Test plan

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace --all-targets
  • cargo test -p gpucache-transport --all-targets --features rdma-nixl
  • cargo run -p gpucache-transport --bin shard-demo
  • cargo run -p gpucache-cold --bin cold-demo (rustfs_live=NOT_RUN_HARDWARE, gds=NOT_RUN_HARDWARE)
  • cargo run -p gpucache-bench --bin paired-baseline
  • Live NIXL / Soft-RoCE — NOT_RUN_HARDWARE
  • Live RustFS — NOT_RUN_HARDWARE (no GPUCACHE_RUSTFS_*)
  • GPU TTFT/TPOT paired baselines — NOT_RUN_HARDWARE
Open in Web Open in Cursor 

TCP stays the correctness path. The rdma-nixl feature does not link
libnixl. A shard-local directory, a simulated registration pool, and
the cold sink/refill contract are covered by CPU tests.

Co-authored-by: RustFS <hello@rustfs.com>
@loverustfs
loverustfs marked this pull request as ready for review October 15, 2026 07:54
@loverustfs
loverustfs merged commit eb7c29a into main Oct 15, 2026
4 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants