# Together AI says its compiler speeds up AI inference

> Together AI says its compiler can optimize inference workloads in under 24 hours, with up to 2.6× higher throughput on tested open models.

Oossa · 2026-11-04 · https://oossa.com/en/together-ai-says-its-compiler-speeds-up-ai-inference-3381353

Together AI says it has built a compiler that continually tunes how AI models run, reporting up to 2.6× higher throughput on several open models after less than 24 hours of optimization. The company says the system, called Together Compiler (TKCC), is already serving its workloads, which handle trillions of tokens.

The compiler is designed to adapt to changes in models, computer chips and user traffic rather than relying on one fixed setup. Together says its tests covered models including Kimi K3, GLM 5.3, MiniMax M3.1-Flash and DeepSeek v4.1-Flash. Throughput means how many tokens a system can process over time; higher throughput can let a provider serve more requests on the same hardware.

## How does the compiler work?

TKCC uses AI agents to propose and test changes to an inference engine, the software that runs a model when someone sends it a request. The company says a key design choice is keeping test engines running instead of repeatedly starting them from scratch. In its comparison, starting an engine took 467 seconds with the usual process, versus nine seconds with TKCC.

Several agents work on possible optimizations in parallel, while persistent engines test their proposals. A fast testing tier supports quick experiments; a slower tier checks whether changes still work with more of the model and realistic input data. Together says this setup ran experiments 252 times faster than baseline agentic performance-optimization systems. That is an internal comparison, and the company has not described independent validation of the results.

## What did Together test?

For the SemiAnalysis InferenceX AgentX benchmark, Together reports up to 2.6× higher throughput than the best available open-source engine configurations it used as baselines. In a separate low-latency, batch-size-one test, it reports 1,111 tokens per second for MiniMax M3.1-Flash and 1,205 for DeepSeek v4.1-Flash. These are Together’s results, not an independent assessment.

The company also reports 1.7× higher throughput than open-source megakernels on the Llama 1B, batch-size-one workload. Beyond language models, it says TKCC outperformed using Claude Code with its “/goal” command on three of four tests across weather modeling, partial differential equations, time-series forecasting and biology. Together says the compiler is in use internally; it does not explain whether or when other developers can access it.

## The facts

- Together AI reports up to 2.6× higher throughput on tested open models after less than 24 hours of optimization.
- Together compares engine startup times of 467 seconds for the usual process and nine seconds for Together Compiler.
- On its batch-size-one InferenceX low-latency tests, Together reports 1,111 tokens per second for MiniMax M3.1-Flash and 1,205 for DeepSeek v4.1-Flash.
- Together reports 1.7× higher throughput than open-source megakernels on Llama 1B at batch size one.

## Why it matters

For AI providers, higher throughput could mean serving more requests on existing hardware, or improving response speed if they choose to use the extra capacity that way. Together’s benchmark results are company-reported, and it has not said whether customers or outside developers can use the compiler.

## Sources & references

1. [ResearchTogether Compiler: Continual Compilation for AI inferenceTogether Compiler uses agents and persistent engines to optimize open models in under 24 hours, delivering up to 3.7x higher throughput](https://www.together.ai/blog/together-compiler-continual-compilation-for-ai-inference) – Together AI
2. [Together AI Releases Together Compiler: Agents Make Inference Up to 2.6x Faster](https://www.marktechpost.com/2026/11/05/together-ai-together-compiler-continual-compilation-ai-inference/) – MarkTechPost, 2026-11-05

Last updated: 2026-11-05