AWS Machine Learning Blog· Sunita Nadampalli·· 12d agoSelected
AWS Introduces Push‑Based KV Transfer for vLLM to Cut TTFT and TPOT
Original source: AWS Machine Learning BlogRead original
AI brief
AWS has added a push‑based NIXL connector to the open‑source vLLM inference engine, moving KV cache transfer off the critical path between prefill and decode phases. In benchmarks on EC2 P5en instances linked with EFA, the push mode achieved up to three‑fold faster time‑to‑first‑token and 30% lower time‑per‑output‑token for the Qwen3‑32B model compared to the default pull mode. The improvement is driven by eliminating sequential coordination steps that previously extended the TTFT path.
Why it matters
This update gives AI practitioners a tangible way to reduce latency in large‑model inference workloads, especially at high request rates, by leveraging AWS’s RDMA‑enabled infrastructure.
Source: AWS Machine Learning Blog · aws.amazon.com