ElevenLabs Wants to Take the Assembly Work Out of Voice AI

Building a voice agent has become easier. Making one behave naturally while completing a customer’s request still takes considerable work.

Developers must coordinate speech recognition, a language model, speech generation, and the logic that decides when an agent should speak, wait, or stop talking. Each component can perform well individually while the conversation still feels awkward.

With the launch of Eleven Realtime, ElevenLabs is offering to take on more of that assembly and tuning. Announced October 20, the offering introduces v1 Mini for fast, natural conversations and v1 Max for complex instructions and tasks involving multiple steps. Both support more than 90 languages and access to over 17,000 voices.

“The hardest part of a voice agent isn’t the voice. It’s orchestrating the conversation: how long to wait after a pause. When to pick back up after an interruption - and when the interruption wasn’t one. When “mm-hm” means keep going. All of that is essential to solve problems seamlessly and make the conversation enjoyable.

Now in addition to our suite of cascaded models already available, we are adding Eleven Realtime which does that tuning for you, speaking in under half a second and answering spoken reasoning questions correctly. It's one of the fastest models of this class. ”

Mati Staniszewski

ElevenLabs, Co-Founder

That framing captures the significance for customer experience. ElevenLabs is taking responsibility for more of the interaction, including the small conversational decisions that determine whether an agent feels responsive or frustrating.

From Choosing Components to Choosing a System

In ElevenAgents’ Custom mode, developers select and configure the underlying models themselves. Realtime mode transfers control over settings including the LLM, speech recognition provider, speech generation model, and reasoning configuration to ElevenLabs. Customers retain their prompts, voices, tools, and knowledge bases.

The development proposition changes accordingly. A business can spend less time choosing and tuning components and more time defining what its agent should accomplish.

There is an important architectural distinction, however.

Although ElevenLabs describes Realtime as a new class of speech-to-speech models, its documentation says Mini and Max use a cascaded architecture. Separate component models remain underneath, optimized to work together. They are not fused models, and the specific underlying components have not been disclosed.

The practical innovation is in the coordinated system and how customers consume it. Whether that delivers better results depends on the quality of the orchestration.

ElevenLabs’ published comparison of Mini and Max across reasoning, conversational dynamics, preference, and task success. Source: ElevenLabs.

A Fast Answer Is Only Part of the Job

The announcement addresses a familiar voice-agent problem: responding quickly without misunderstanding the caller or interrupting at the wrong moment.

A pause may mean the customer has finished speaking, or that they are searching for an account number. An “mm-hmm” may signal encouragement rather than an interruption. Handling those distinctions is part of making automation usable.

ElevenLabs reports that independent Artificial Analysis testing gave Max 98.8% accuracy on Big Bench Audio, a benchmark of 1,000 spoken reasoning questions, with an average 0.57 seconds to first audio. Mini scored 75.9% on reasoning and began speaking in 0.62 seconds on average.

Time to first audio measures how quickly speech begins. It does not measure how long an agent takes to complete a customer’s request. Source: ElevenLabs.

The distinction becomes more important when agents must take action. In the launch’s reported tool-use results, Max achieved 87.1% task success, compared with 60.3% for Mini. That helps explain why ElevenLabs positions Max for customer support, lead qualification, and workflows requiring multiple steps.

For buyers, answering a spoken reasoning question correctly is a different test from changing a reservation, applying a policy, or completing a refund.

Artificial Analysis’s current leaderboard also lists ElevenLabs’ existing default cascaded system at 90.5% task success, compared with Max’s 87.1%. Their confidence intervals overlap, so the comparison does not establish a clear performance difference. It does show why the launch should not be interpreted as proof of universal improvement over existing configurations.

An Easier Route Into Existing Applications

ElevenLabs is also reducing the work required to evaluate its offering.

Its Realtime API uses the same WebSocket wire protocol as OpenAI’s Realtime API, allowing developers to reuse familiar SDKs and event structures. That provides a more accessible route for testing ElevenLabs inside an existing application.

Compatibility comes with limits. WebRTC applications require transport changes, and the standalone API does not automatically include the broader ElevenAgents platform features.

Even with those qualifications, this is a meaningful competitive move. Reducing integration effort makes it easier for developers to compare providers on the behavior of the complete voice system.

What This Changes for CX Buyers

Our assessment is that Eleven Realtime advances a meaningful shift in voice AI procurement. Providers are increasingly asking customers to buy an optimized interaction system rather than assemble individual capabilities.

That could reduce engineering effort and shorten deployment cycles. It also gives buyers a stronger reason to demand accountability for the complete experience.

Enterprise fit still requires scrutiny. Realtime is rolling out gradually, compliance and regional availability differ between Mini and Max, and ElevenAgents billing combines a per-minute platform charge with model token charges.

The next test is whether simpler configuration translates into reliable customer outcomes at an acceptable cost.

For enterprises evaluating voice AI, that means measuring completed tasks, incorrect actions, escalation quality, and cost per successful resolution alongside conversational speed.

ElevenLabs is offering to take more of the assembly work off the customer’s plate. The opportunity is substantial. So is the responsibility that comes with owning more of the conversation.