Sparrow-2: Beyond Turn-Taking to Whole-Scene Conversational Understanding

Written by: Brian Johnson and Eloi Du Bois

[This is a research and technical overview of Sparrow-2. To jump to what the model unlocks, click here]

See the official announcement on X and LinkedIn.

Human conversation relies on a largely subconscious understanding of speaker intent, prosody, and the acoustic environment. We continuously interpret what another person is saying, how they are saying it, and where they are in a thought to determine when to respond. 

Machines have historically struggled to model this. Prior systems reduced turn-taking to endpoint detection (through silence timers or acoustic end-of-turn models) and relied on noise cancellation to isolate the primary speaker, stripping away the broader acoustic environment in order to produce a clean signal for those systems to interpret. 

This is the same underlying challenge humans solve effortlessly in a crowded room: not simply hearing sound, but understanding which signals belong to the conversation and what they mean. Humans do this continuously and naturally. Machines historically could not, so conversational systems made the same fundamental trade in both turn-taking and audio processing: discard information before the system ever had a chance to understand whether it mattered.

Sparrow-2 inverts this approach. Rather than filtering the audio stream down to a clean signal from the primary speaker, it treats the entire incoming audio stream as evidence of the conversational state. Rebuilt from the ground up, Sparrow-2 jointly models semantic content, lexical structure, prosody, speaker identity, backchannels, interruptions, and more, alongside signals from the broader audio environment, including background speech, noise, and unintelligible audio. Together with the PAL’s own speaking state, modeled through a semi-duplex architecture, these signals inform both when the system should listen, wait, speak, or continue speaking, and how it should respond to what is happening around it.

The result is a model that pushes the frontier of turn-taking accuracy, but expands conversational AI beyond the quiet, one-speaker environment it has historically required. Sparrow-2 can distinguish nearby speech from someone actually talking to it, understand when background noise matters, and recognize when audio is too unclear to answer confidently. Together, these capabilities represent a new approach to the cocktail party problem and unlock conversational AI across cafés, retail floors, open offices, cars, healthcare settings, and other noisy or shared environments that were historically impractical.

Human Truth: Listening is Understanding 

Conversation is a dance. Not metaphorically: mechanically. Two partners, in continuous negotiation over a shared floor: when to lead, when to follow, when to hand the moment across. We don’t always know when to speak, but we feel it. And like a dance, a conversation doesn't just start or stop: it flows like a spin, a dip, a twirl. Each partner inherently knows their next step, not by watching for the other to finish moving, but by understanding movement itself. Inertia, timing, collision. The physics of conversation, like the physics of dance, naturally shape a shared subconscious rhythm. This is not heuristic, it’s holistic.

Conversation flow is the choreography of human interplay: the protocol for when to speak, when to listen, when to trade the floor, and what in the world around us actually matters to the interaction - running underneath every conversation you've ever had, so smoothly you've never once thought about it. But it’s a consistent and natural phenomenon that can be learned.

The timing works because each party is continuously reading the whole interaction: the movement, the partner, and the space around them.

Conversation works the same way. You know which voice belongs to the person you are speaking with. You know whether a “woo!” is encouragement, an interruption, or someone speaking across the room. You hear the door close, the next song start, another conversation nearby, a breath, a hesitation- and each signal carries a different weight. Some should change the dance. Some should be ignored. Some tell you to wait, some to respond, and some tell you that you did not hear clearly enough to respond at all.

None of these signals are inherently noise. Some matter, some do not, and their meaning changes with the conversation. The environment is not separate from the interaction; it is part of the context that gives each signal meaning. Humans understand enough of the whole scene to know the difference.

A good dance partner does not stare at your feet and wait for you to stop moving. They understand the movement as it unfolds- and the room around it- well enough to know what comes next. Conversation works the same way. Listening is not waiting for someone to finish. Listening is understanding.

‍

AI Hasn’t Learned How to Dance (yet)

Conversational AI has historically modeled this very differently. Understanding the full state of a live conversation was simply too difficult to model in real time, so the problem was reduced into smaller, tractable objectives.

That reduction happened in two directions. The objective was narrowed from understanding conversational state to detecting when someone had finished speaking. At the same time, the input was narrowed from the full acoustic scene to a clean signal from the presumed primary speaker.

Each made the problem easier to solve. Each also removed information humans naturally rely on to understand what is happening.

‍

The Filtered Objectives

For turn taking, that decomposition centered the objective around detecting an ending. Typical pipelines cascade audio to ASR to transcript, sometimes with prosody features, and then apply either a silence timer or an end-of-turn classifier trained on the final few hundred milliseconds of audio.

Sparrow-1 represented the state of the art in that paradigm: rather than waiting on a fixed silence threshold, it incorporated tone, rhythm, hesitation, and other acoustic cues to model floor transfer directly. But even Sparrow-1 was ultimately optimized around the same question: has the speaker reached the end of their turn?

The ceiling of this approach is empirical, not a matter of tuning. Across more than one million silences in natural conversations, the probability that a pause marked a true turn handoff plateaued around 40%. Most pauses are simply part of speaking- thinking, breathing, searching for a word- not an invitation for someone else to jump in. That makes silence alone a fundamentally weak signal for deciding when an AI should speak.

No silence duration, at any threshold, reliably indicates that the floor has changed hands. More sophisticated end-of-turn models add acoustic cues, but still optimize for detecting an ending rather than understanding conversational state, forcing the same tradeoff: respond aggressively and interrupt, or wait conservatively and introduce unnatural latency. 

The Accidental Sacrifice

The same reduction that narrowed the objective also narrowed the input. To make speech recognition and turn detection reliable, conversational systems came to depend on clean audio. Noise cancellation isolates the presumed primary speaker and suppresses everything else before downstream models ever see it.

That works when the system gets it right, but when it does not, the failure is absolute: the real speaker can be cancelled along with the noise. The audio is gone, the transcript is gone, and the system is no longer mishearing the user- it is not hearing them at all.

More importantly, even successful cancellation comes at a cost. Background speech, breaths, sighs, backchannels, interruptions, and environmental sounds- the very cues humans use to dance- are discarded before the system has a chance to determine whether they matter.

That is the deeper limitation of cancellation: it decides what matters before understanding what it is. A backchannel becomes noise. A second speaker becomes interference. A loud room disappears entirely. In simplifying the audio enough for downstream systems to interpret, we also remove the context they would need to understand the conversation more completely.

Both reductions make the same trade: throw away information to make the problem tractable. 

This is the cocktail party problem hidden inside the conversational AI stack. Humans do not solve a crowded room by deleting every sound except one voice; we continuously decide who is speaking, what is directed at us, and which parts of the acoustic scene matter. Traditional systems had to decide what was noise before they understood what was happening.

Sparrow-2 takes a step back and asks whether we still need to make that trade at all.

Introducing Sparrow-2

Sparrow-2 is an audio-native, streaming-first, multilingual conversational understanding model built to preserve the full audio scene, understand it, and let conversational decisions emerge from that understanding. 

Rather than reducing the interaction to isolated signals, Sparrow-2 takes a rich multi-objective approach to the entire audio space. The question shifts from “has the user finished speaking?” to “what is happening in the conversation, and what should the system do next?”

That paradigm shift expands the model in two directions:

1. The whole turn. Sparrow-2 models semantics, lexical structure, prosody, timing, and speaker identity together, allowing it to understand where a speaker is in a thought and whether that thought is actually complete.
2. The whole room. Background speech, noise, backchannels, interruptions, and unintelligible audio are preserved and classified rather than discarded, allowing those signals to inform both conversational flow and the conversation itself.

This fundamentally changes what a conversational system can understand in practice.

An “mhm” while the PAL is speaking can be understood as a backchannel telling it to continue, while a genuine interruption can tell it to stop. Speech elsewhere in the room can be distinguished from someone actually trying to take the floor. And when audio is too noisy or unclear to understand reliably, the system can recognize that uncertainty and ask the user to repeat themselves rather than confidently responding to something it did not hear.

Together, these changes move conversational flow from endpoint detection toward continuous conversational-state modeling: a single system reasoning over more of the information humans naturally use to decide when to listen, wait, speak, or respond.

The result is not only more natural conversational timing. By understanding rather than cancelling the broader acoustic scene, Sparrow-2 can operate across noisy, shared, and uncontrolled environments that historically broke conversational systems, while navigating pauses, backchannels, interruptions, hesitation, and ambiguity with substantially more human-like conversational behavior.

How Sparrow-2 Works: A Novel, Integrated Approach

Sparrow-2 is built around a richer representation of conversational audio, allowing it to push the boundaries of conversational AI in two directions at once:

1. Where conversational AI can work. by understanding rather than cancelling the broader acoustic scene, Sparrow-2 can operate across noisy, shared, and uncontrolled environments that historically broke conversational systems.
2. How natural the conversation can be: by understanding the full turn rather than detecting its endpoint, Sparrow-2 can navigate pauses, backchannels, interruptions, hesitation, and ambiguity with substantially more human-like conversational timing.

The shift is possible because Sparrow-2 no longer has to choose between rich audio understanding and real-time inference. A new Tavus encoder preserves semantic, lexical, prosodic, speaker, and environmental information in a shared representation, which a causal transformer reasons over continuously in real time.

Core Properties and Capabilities:

• Noise understanding, not cancellation: classifies background speech, environmental noise, and other acoustic events with confidence rather than discarding them before they can inform the interaction.
• Streaming-first at 10 ms: operates on native 10 ms frame rate, updating its estimate of conversational state up to 100 times per second. Four times more granular than Sparrow-1's 40 ms.
• 4x faster inference: processes 80 ms of audio in ~7 ms, compared with ~30 ms for Sparrow-1, reducing inference overhead and system contention while leaving more latency budget for the rest of the conversation stack.
• Native backchannel and interruption awareness: signals such as “mhm” can be distinguished from genuine attempts to take the floor instead of being treated as equivalent speech events.
• Intent-aware patience: holding through genuine mid-thought pauses of up to 6 to 8 seconds, where prior systems were typically limited to 1 or 2.
• Deep semantic and lexical understanding: Tavus audio encoder that maps multiple conversational objectives into a single latent space. Beyond individual words, it captures full sentences, the meaning of the language itself, and speaker identity.
• Nonsense detection: the model recognizes audio that could not be reliably interpreted, allowing the system to ask for clarification rather than confidently acting on a bad transcription.
• Semi-duplex awareness: the model tracks its own speech state internally, allowing its own conversational state to inform interruption and floor-transfer decisions without external heuristics.
• Unified conversational control: Consolidates turn-taking and audio-scene understanding into one model, allowing timing, intent, speaker, and environmental signals to be interpreted jointly rather than passed between independent systems.

Sparrow-2 retains Sparrow-1’s speculative inference, real-time speaker adaptation, and core design for real human speech, while expanding capabilities substantially. 

Together, these capabilities unlock more natural conversations in more real-world conditions- from longer pauses and nuanced interruptions to noisy rooms, shared spaces, and less structured interactions.

[Test Sparrow-2 Demo Environment >] 

Training and Inference

Raw audio passes through a Tavus encoder that projects multiple conversational objectives into a shared latent space, encoding lexical content, full sentences, and linguistic meaning. A six-layer causal transformer runs on the resulting streaming embeddings at a native 10 ms frame rate, producing an updated judgment of conversational state at every frame. 

The model was trained using 16,000 real and 2,000 synthetic conversations with backchannels, interruptions, and other conversational behaviors explicitly represented. These objectives are learned jointly. There is no silence timer anywhere in the system

Outcomes

More Places Conversational AI Can Work

Sparrow-2 breaks through one of conversational AI’s biggest constraints: the cocktail party problem. It expands conversational AI beyond the quiet, one-speaker environment it has traditionally required.

With noise understanding, a kiosk can hold a conversation in a busy lobby without treating every nearby voice as the user. A retail assistant can operate on a shop floor with music, customers, and staff talking around it. A candidate can take an interview from a café or shared room without background speech constantly breaking the interaction. 

And in any conversation, a PAL can recognize that the user is in an environment too loud to understand reliably and ask them to move or change microphones- just like a human would.

The same applies to open offices, healthcare environments, cars, public spaces, and other settings where conversational AI has historically been unreliable because the acoustic scene could not be cleanly isolated.

This materially expands the deployment surface for conversational AI: use cases no longer have to be designed around a quiet room.

More Natural Conversations

Sparrow-2 also changes what is possible inside the more controlled environments conversational AI already serves.

Users can pause for several seconds to think without being interrupted. They can say “mhm” or “yeah” while the system is speaking without accidentally stopping it. They can interrupt naturally when they actually want the floor. They can trail off, restart a sentence, search for a number, or speak unclearly without breaking the flow. 

And when the system genuinely did not understand something, it no longer has to guess. It can admit it didn’t hear you and ask you to repeat yourself.

In an interview, a five-second pause can mean the candidate is thinking, not finished. In intake, interrupting mid-thought can mean losing information entirely. In sales or support, a backchannel should encourage the speaker rather than accidentally take the floor. Across interviews, intake, onboarding, support, sales, coaching, and education, this changes what conversational AI can be trusted to do. In interviews and intake, people can think without losing the floor. 

In sales and support, natural feedback no longer breaks the interaction. Across coaching and education, people can speak the way they naturally would instead of adapting their behavior to the AI.

The result is conversational AI that can absorb the messiness of real human speech without losing the rhythm of the conversation.

Benchmark Results

Sparrow-2 is built to understand what is happening in a conversation as a whole, from turn-taking and interruptions to the broader acoustic scene around it. By reasoning over that full scene, it takes a new approach to the cocktail party problem, allowing conversational AI to work more reliably in noisy, shared, real-world environments. The evaluations below focus on just one part of that broader problem- turn-taking- where we can compare directly against existing systems on real conversational audio.

Industry metrics such as time-to-first-token measure how fast a system speaks, not whether it spoke at the right time. So we scored turn-taking the way you would judge a listener: on real, hand-labeled conversations, against what real people did on those same turns.

We froze two evaluation sets, disjoint from every model's training data: 96 human-to-human recordings (287 finished turns, 281 genuine mid-turn pauses) and 95 production video calls (95 finished turns, 480 mid-turn pauses). Each detector hears the same audio up to a labeled moment and decides when to speak. All models get the same 600 ms end-to-end response floor - a voice agent cannot land a reply faster than that - so nothing wins by firing at physically impossible speeds.

In summary, prior systems trade interruption rate for latency, or the reverse. Sparrow-2 improves both at once because as numbers come from the same underlying state estimate: median reply ties the fastest detectors, its fastest quarter of replies lands sooner than other models, and it makes a fourth of the errors.

Conversation Failure Rates

Across 575 real conversational events from production video calls - every finished turn and every genuine mid-turn pause, we counted how often each detector got the moment wrong: barging into a pause, or never replying to a finished turn. Sparrow-2 fails on 2.1% of these events. The best alternative fails nearly four times as often.

Interruptions by Pause Length

People pause while they think, and the length of the pause is what separates the models. All models can deal with short 250 ms pauses without interrupting.

But when a speaker stops for a second or more to gather a thought, Smart Turn and LiveKit interrupt roughly every other pause - and Sparrow-2 lets 97% of them pass without interruption. This is the difference between an agent you can think in front of and one you have to keep feeding words.

Turns Left Unanswered

There is a second way to fail, and it hides inside good latency averages: the speaker finishes, and the detector never replies at all. When a turn ends ambiguously - a trailing thought, a soft finish - instant-trigger detectors simply miss it.

LiveKit never answered 18% of the human-to-human turns and Smart Turn 12.5%. Sparrow-2 missed none.

Reply latency, honestly measured, by including misses

On the turns each model answers, median reply time is a three-way tie at 680 ms. The fuller picture is the second bar: the average wait for a reply across all finished turns, where a missed turn is charged only the seven seconds we waited for it. By that measure Sparrow-2 answers in 880 ms on average, while the alternatives stretch toward two seconds on the strength of the turns they never answer.

Replies land at human pace

When real people answered these same turns, their replies clustered between 500 and 750 ms after the speaker finished - the black outline in each panel.

The solid bars show each model's raw decision time, which arrives before a person could physically speak; the faint bars shift those same replies to where they land once real speech production is accounted for, squarely on the human cluster. What remains different between the panels is the far-right bar: the turns the fast detectors never answer at all.

What This Means for Human Computing 

For decades, interfaces have required people to learn how machines work. Conversational AI inherited the same pattern: wait for the beep, avoid background noise, do not pause too long, speak one at a time.

These are not natural conversational behaviors, whether in a one-on-one interaction or a crowded social setting. They are accommodations for the system. The more a person has to think about how to speak to the machine, the less the interaction feels like conversation at all.

The real benchmark for conversational AI is not latency alone. It is whether the interaction creates the same sense of being heard that we expect from a good listener- or rather, a shared rhythm we’d expect from a dance partner. 

We trust people who do not interrupt us while we are thinking, who wait when we hesitate, who understand when a quick backchannel means “keep going,” and who admit when they did not hear us clearly enough to respond. These behaviors feel small, but together they are how attentiveness- and ultimately trust- is felt in conversation.

Our goal with human computing is to reverse the relationship between people and machines: instead of having to learn how to interact with machines, teach the machine how people already behave. 

Sparrow-2 moves us closer to that goal, enabling more natural conversational flow while expanding conversational AI into environments and use cases that previous systems could not reliably support.

‍

Sparrow-2 is now available to GA across PALs, the Tavus APIs and platform, already powering conversational experiences in the Tavus PALs and enterprise deployments.

Try the demo at sparrow2.tavuslabs.org or learn more in our docs.