Together Compiler Enables Continuous Compilation for AI Inference
inference
| Source: ArXiv | Original article
Together AI introduces the Together Compiler, a continual compilation system designed to adapt AI inference to rapidly evolving models, architectures, and traffic patterns.
Together AI unveiled the “Together Compiler” (TKCC), a continual‑compilation framework that automatically rewrites and optimises the forward pass of large language models for production inference. The system ingests a server launch command, a traffic profile and a hardware description, then deploys high‑throughput coding agents to propose alternative implementations. A persistent engine measures each candidate in‑place, hot‑reloading faster passes without restarting the whole service.
The announcement targets a pressing problem: AI workloads evolve rapidly, with new models, diverse architectures and shifting request patterns appearing on a near‑daily basis. Existing compilers rely on offline benchmarks that capture only low‑fidelity timings, especially for data‑dependent primitives such as mixture‑of‑experts layers. TKCC’s agent‑driven hill‑climbing loop closes that gap by coupling rapid measurement with continuous code generation, delivering optimisation cycles in under 24 hours.
Early results show up to a 2.6× increase in throughput (TPS) on frontier open‑source models—including Kimi K3, GLM 5.3, MiniMax M3.1‑Flash and DeepSeek v4.1‑Flash—while also setting a new state‑of‑the‑art on SemiAnalysis’s InferenceX benchmark for low‑latency scenarios. By automating the optimisation pipeline, the compiler promises to reduce the engineering effort required to keep inference services performant as model families proliferate.
What to watch next: Together AI plans to open the TKCC platform to external developers, potentially extending the agent ecosystem beyond internal code generation. Observers will also be keen to see how the system scales across different hardware back‑ends, from GPUs to emerging AI accelerators, and whether the continuous‑compilation model can be integrated with existing orchestration tools used in cloud‑native AI deployments. If the early performance gains hold at scale, TKCC could become a cornerstone technology for keeping inference costs in check as the AI model zoo continues to expand.
Sources
Back to AIPULSEN