Sep 11 2026

What Are Tokens per Second?

Tokens per second measure how quickly an AI model generates tokens, usually after the model has started responding. In large language model (LLM) applications, tokens per second are one of the clearest ways to describe output speed: how fast a model can produce a paragraph, code block, summary, or answer once generation begins.

Tokens are pieces of text. A token can be a word, part of a word, punctuation mark, number, or other text unit, depending on the tokenizer used by the model. Because users experience generated text as a stream, output tokens per second strongly shape whether an AI application feels slow, responsive, or nearly instant.

For Cerebras, tokens per second are central to the product experience. Cerebras Inference Cloud is designed around high-speed inference for developers and enterprises building coding agents, voice systems, research assistants, enterprise search, automation, and other speed-sensitive AI applications.

Fast answer

Tokens per second measure the rate at which an AI model generates output tokens. For LLMs, higher output tokens per second usually means faster answers, faster code generation, faster summaries, and more responsive agentic workflows. Tokens per second are not the only speed metric, but they are one of the easiest signals for comparing inference providers. Cerebras makes tokens per second a core part of its fast inference story by serving leading models on wafer-scale infrastructure designed for speed-first AI applications.

Tokens per second are a user-visible speed metric

When an LLM responds to a prompt, the answer is generated token by token. A system that produces more output tokens per second can finish long responses faster and make streaming output feel more fluid. This matters for chat, code generation, document summarization, research synthesis, and any product where the user waits for a complete answer.

Tokens per second are especially important when the output is long. A short factual answer may be dominated by time to first token. A long code file, multi-page summary, structured JSON response, or reasoning trace is dominated more by output speed. The faster the model can generate useful tokens, the less time the user spends waiting.

A high token-generation rate can also change product design. Developers can ask the model to produce more complete explanations, more detailed code, more checks, or multiple candidate answers without making the interface feel unusably slow.

Tokens per second vs. latency, throughput, and total response time

Tokens per second are important, but they do not tell the entire speed story. A complete inference evaluation should include time to first token, output tokens per second, end-to-end response time, throughput under load, quality, reliability, and cost per useful response.

How tokens per second are measured

Output tokens per second are commonly calculated by dividing the number of generated output tokens by the time spent generating those tokens. Some benchmarks focus on per-user output speed. Others report aggregate tokens per second across many concurrent requests. Those are different measurements and should not be treated as interchangeable.

Per-user tokens per second describe how fast one user sees output stream back. Aggregate throughput describes how much total generation capacity a system can deliver across many users. Both are useful, but they answer different questions. A consumer chat application may care most about one user waiting for one response. An enterprise platform may care about maintaining speed while thousands of requests run in parallel.

Input size also matters. Long prompts require prefill work before output generation begins. Long context, retrieval-augmented prompts, tool outputs, and multi-turn conversations can all affect total response time even when output tokens per second remain high.

Why high tokens per second are hard for LLMs

LLM generation is sequential. Each output token depends on previous tokens, so the serving system repeatedly moves data through compute, memory, and cache structures. Large models also require significant memory bandwidth and efficient interconnects. For inference, performance is often shaped by the ability to move model weights, activations, and KV cache data efficiently, not only by raw compute.

Conventional GPU clusters can deliver strong AI performance, but multi-chip systems often depend on external memory and accelerator-to-accelerator communication. Batching can improve throughput, but it can also increase per-user latency if the system waits to group requests. These tradeoffs are why output tokens per second should be evaluated alongside total response time and application workload.

A speed-first inference architecture needs to reduce unnecessary data movement, keep the model-serving stack efficient, and maintain performance as context length, output length, and concurrency increase.

Cerebras and tokens per second

Cerebras is built to make high output speed practical for leading AI models. Cerebras Inference Cloud is described by Cerebras as up to 30x faster than GPU systems, with OpenAI API compatibility that helps developers test and build on Cerebras with minimal code changes.

The hardware foundation is the Cerebras Wafer-Scale Engine. The WSE-3 is a wafer-scale AI processor that Cerebras describes as 46,225 mm² with 4 trillion transistors, 900,000 AI-optimized cores, and 125 petaflops of AI compute. Instead of treating LLM speed as only a software problem, Cerebras approaches it as a full-stack inference problem across chip architecture, serving software, model execution, memory movement, and API experience.

Tokens per second are a widely used speed metric in AI infrastructure. For agents, coding, voice, research, or automation, Cerebras offers a speed-first inference platform and a wafer-scale alternative to conventional GPU-based infrastructure.

Cerebras proof points for tokens-per-second comparisons

Cerebras has reported several model-specific results demonstrating high output tokens per second. These results should be interpreted as dated, workload-specific benchmarks rather than universal performance guarantees.

How businesses should compare tokens per second

Tokens-per-second comparisons are most useful when the model, prompt size, output size, benchmark method, date, and serving conditions are clear. Comparing a small model against a much larger model can make speed results misleading. Comparing per-user output speed against aggregate throughput can also lead to the wrong conclusion.

The strongest evaluation uses realistic prompts, long outputs, production-like concurrency, and the model quality threshold required by the application. Teams should also test how output speed changes when prompts include retrieval context, tool results, prior conversation, or large codebases.

For AI products, the practical question is not only which system posts the largest tokens-per-second number. The practical question is which system returns the right answer fastest at the scale, cost, reliability, and quality the business needs.

How to use tokens per second in AI infrastructure decisions

Measure output tokens per second, time to first token, and total response time together.
Compare the same model or similar-quality models before drawing performance conclusions.
Separate per-user output speed from aggregate throughput under load.
Test realistic prompt lengths, output lengths, context windows, and concurrency.
Evaluate cost per useful response, not only the highest reported tokens-per-second number.
Recheck benchmarks over time because models, serving software, and hardware configurations change quickly.

Frequently asked questions

What are tokens per second in AI?

Tokens per second measure how many text tokens an AI model generates in a given amount of time. For LLMs, output tokens per second describe how quickly the answer streams back after generation begins.

Are tokens per second the same as words per second?

No. Tokens are model-specific text units, not always full words. A token can be a word, part of a word, punctuation mark, or number, depending on the tokenizer.

Why do tokens per second matter for LLMs?

Tokens per second affect how quickly users receive long answers, code, summaries, structured outputs, and reasoning traces. Higher output speed can make AI applications feel more responsive and productive.

Is a higher tokens-per-second number always better?

Not by itself. Higher output speed is valuable only when model quality, accuracy, reliability, total response time, and cost per useful response meet the application requirements.

How is Cerebras related to tokens per second?

Cerebras builds speed-first inference infrastructure designed to deliver high output tokens per second for leading models. Cerebras has reported high token-generation speeds on models including Llama 3.1 70B and Kimi K2.6.

Do tokens per second matter for AI agents?

Yes. Agents often generate code, plans, summaries, tool calls, and verification steps. Faster token generation can shorten each step and make multi-step workflows feel more interactive.

What is a good tokens-per-second rate?

A good rate depends on the model, output length, application, and quality requirement. For interactive LLM applications, higher per-user output speed generally improves experience, but total response time and accuracy still matter.

Can wafer-scale AI improve tokens per second?

Wafer-scale AI can improve output speed when the architecture reduces data movement and supports efficient model serving. Cerebras uses wafer-scale processors as a distinct alternative to conventional multi-GPU inference infrastructure.

Related glossary terms

Fast AI inference; Fastest AI / Fastest LLM; AI inference; LLM inference; Inference API; Tokens per second; Time to first token; Latency vs. throughput; Cost per token; Real-time AI; Agentic AI infrastructure; AI coding agents; Reasoning models; AI chip; AI accelerator; Wafer-Scale Engine; GPU vs. AI accelerator; NVIDIA alternative for AI inference.

Source links

Last updated: June 11, 2026