Nexus adds depth-adaptive KV-cache splicing and decoupled tool routing for agentic LLMs on unified memory
agents
| Source: ArXiv | Original article
Researchers introduce Nexus, a depth‑adaptive KV‑cache splicing and retrieval‑decoupled tool routing method that reduces prefill costs for agentic LLMs using unified memory.
A new arXiv pre‑print, *“Nexus: Depth‑Adaptive KV‑Cache Splicing and Retrieval‑Decoupled Tool Routing for Agentic LLMs on Unified Memory”* (arXiv:2608.20397v1), introduces a system designed to cut the latency that plagues agentic large language models (LLMs) when they repeatedly re‑encode extensive tool schemas. The authors note that, under the Model Context Protocol (MCP), each turn forces a full pre‑fill of the tool registry, making the operation quadratic in sequence length and inflating time‑to‑first‑token (TTFT) as the registry expands.
Nexus tackles this bottleneck by decoupling two core stages of inference. First, it applies depth‑adaptive KV‑cache splicing, turning the traditionally request‑local key‑value cache into a shared, reusable resource across requests, workers and storage tiers. Second, it separates tool‑schema retrieval from the main decoding path, allowing the model to route tool calls without re‑encoding the entire schema each turn. The result is a unified memory fabric that can serve multiple inference engines—vLLM V2, SGLang, TensorRT‑LLM, LMDeploy, TGI and custom C++/Rust runtimes—without being tied to a specific hardware or model architecture.
Why it matters: Agentic LLMs are increasingly central to enterprise AI workflows, from autonomous assistants to automated decision‑making pipelines. The quadratic pre‑fill cost has become a performance ceiling, especially as tool libraries grow. By turning the KV cache into a system‑wide asset and streamlining tool routing, Nexus promises faster TTFT, lower compute overhead and more scalable deployments, potentially narrowing the gap between open‑source and proprietary offerings that dominate the market.
What to watch next: The research team plans to release benchmark results comparing Nexus‑enabled inference against baseline pipelines. Industry observers will be looking for early integrations into major platforms and any open‑source contributions to the NexusKV repository. If the performance gains hold up, the approach could become a new standard for building cost‑effective, high‑throughput agentic LLM services.
Sources
Back to AIPULSEN