Gateway API Inference Extension (v1) • llm-d DisaggregatedSet • vllm-torchtpu • Cloud TPU v6e
Google Developer Expert
Google Developer Expert — AI & ML
ML Scientist — PayPal
Companion Repository & Resources • 01:00 – 01:20
Google Developer Expert
Every GKE manifest, Go ext_proc router package, and automated test gate (make gate-all) is in the repo.
Go 1.23 FleetKV EPP router, TPU v6e DisaggregatedSet YAML & Digital Twin.
Follow along with live speaker notes (press S) and copyable YAML/Go snippets.
Kubernetes SIG Networking Gateway API Inference Extension v1 specification.
Part 1: The Agent Tree Workload • 01:20 – 02:30
Google Developer Expert
Part 1: Fork-Storm Topology • 02:30 – 03:20
Google Developer Expert
Every 256-token block hash in the 80-block parent trunk is identical across all 8 child sub-agents:
🔍 Click diagram to enlarge full-screen
Part 2: Two Routing Failure Modes • 03:20 – 05:00
Google Developer Expert
Part 3: Co-Designing Routing & Disaggregation • 05:00 – 06:15
Google Developer Expert
Solving both 1/N scatter misses and fork-storm herding requires coordinating the GAIE control plane with a disaggregated Prefill/Decode data plane:
Part 3: Control Plane (FleetKV EPP) • 06:15 – 07:30
Google Developer Expert
func ScoreEndpoint(pod *PodState, req *RouteInput) float64 {
// 1. Hard HBM Fit Gate: spill to peer if branch horizon overflows HBM
effectiveFree := pod.FreeHBMBlocks - pod.ReservedBlocks
if effectiveFree < req.EstimatedBranchBlocks + SafetyReserveBlocks {
return RejectOverflowSpillToPeer
}
// 2. Trade off ZMQ-indexed block hits against in-flight prefill queue
savedMs := float64(pod.MatchedHBMBlocks)*HBMBlockMs +
float64(pod.MatchedDRAMBlocks)*DRAMBlockMs
queueWait := float64(pod.InFlightPrefillTokens) / PodTokPerMs
return savedMs - queueWait
}
The three control surfaces:
ZeroMQ KV Indexer (:5557) — Subscribes to vllm-torchtpu BlockStored & BlockRemoved events across HBM and Tier-2 Host DRAM.Hard HBM Fit Gate — Rejects pods where admitting the branch horizon (+24 safety blocks) would evict warm parent trunks.Staged Lease Accounting — AcquireStagedLease releases InFlightPrefillTokens on ResponseHeaders (TTFT) while holding ReservedBlocks until stream EOF.
Part 3: Data Plane (vllm-torchtpu + HMA) • 07:30 – 08:30
Google Developer Expert
Upstream PyTorch models lowered via torch.compile(backend='tpu') onto Pallas paged-attention kernels:
🔍 Click diagram to enlarge full-screen
Part 4: Live 3-Stage Traffic Flip Demo • 08:30 – 11:45
Google Developer Expert
./bench/demo_ctl.sh stage1
./bench/demo_ctl.sh stage2
./bench/demo_ctl.sh stage3
As demo_ctl.sh hot-swaps policies via POST /admin/policy (or patches the GKE HTTPRoute), bench/stage_tui.py freezes the completed stage's 10-second sliding-window telemetry so all 3 stages remain visible side-by-side.
# Launch 3-pane tmux layout (Scorecard TUI + EPP Route Stream + Shell)
./bench/demo_ctl.sh prep
# Flip live traffic across all 3 architectures:
./bench/demo_ctl.sh stage2 # Trigger single-pod herd & HBM thrash
./bench/demo_ctl.sh stage3 # Enable FleetKV Fit Gate + Disagg P/D
Part 5: Empirical Benchmark Scorecard • 11:45 – 12:30
Google Developer Expert
Verified Gate G1 run (`make gate-g1`, 416 streaming requests/stage) & 8-chip GKE TPU v6e (`2x ct6e-standard-4t`) reference. (Click to zoom)
Part 6: Production GKE Invariants • 12:30 – 13:30
Google Developer Expert
Python randomizes process hash seeds by default. Set PYTHONHASHSEED="0" on every vllm-torchtpu worker and fleetkv-epp container so chained 256-token block hashes match across processes.
GPU pools often default to block size 64. On Trillium TPU v6e, 64-token blocks overflow Scalar Memory (SMEM) in Pallas paged-attention kernels. Set --block-size 256 in both vLLM and fleetkv-epp.
Expose ports 8000 and 5557 via Headless Services (clusterIP: None) so fleetkv-epp resolves every pod IP via DNS, filter ZMQ to llm-d.ai/role=prefill, and set VPC mtu: 8896 for TPUConnectorHMA.
Reference: GKE & Local Commands • 13:30 – 14:00
Google Developer Expert
# Run all verification gates (Go race + E2E SLOs + K8s YAML)
make gate-all
# Launch interactive 3-pane conference TUI
./bench/demo_ctl.sh prep
./bench/demo_ctl.sh stage2
./bench/demo_ctl.sh stage3
./deploy/gke/00-bootstrap-gke-tpu.sh
kubectl apply -f deploy/gke/01-baseline-agg-service.yaml
kubectl apply -f deploy/gke/02-disaggregatedset-torchtpu.yaml
kubectl apply -f deploy/gke/03-fleetkv-epp-and-pool.yaml
FLEETKV_GATEWAY_URL="http://${GATEWAY_VIP}" \
./bench/demo_ctl.sh stage3 --target=gke
Google Developer Expert
Defeating the Prefix-Cache Penalty on GKE
• Agent trees fork in bursts: gate KV routing on HBM fit
• Release prefill leases on ResponseHeaders (TTFT)
• Disaggregate Prefill & Decode with Tier-2 Host DRAM
Companion GitHub Repo (Apache-2.0)
Connect on LinkedIn
Interactive Slide Deck & Script