AI microservices architecture for real-time video agents

A face-to-face conversation gives you about the length of a syllable to answer before the other person feels the silence. People hit that window without effort; software has to be engineered for it.

A monolithic build packages the model that hears you, the model that predicts when to answer, the model that decides what to say, and the model that shows a face into one deployable artifact. When one of them stalls, the whole face freezes.

AI microservices architecture reduces the risk of a system-wide stall by running each capability as its own service. Each service can fail and scale independently. For product leaders weighing real-time conversational video, the result determines how much presence it conveys.

What AI microservices architecture means for real-time video agents

AI microservices architecture is an application design pattern that splits a system into small, independently deployable services, each owning one business capability and communicating through lightweight interfaces.

Applied to a real-time video agent that sees, hears, remembers, and responds face-to-face, the pattern separates perception, conversational flow, reasoning, and facial behavior into distinct services that can scale and update independently.

Perception and floor prediction process the speaker's signals against tight deadlines. Reply reasoning and facial behavior generation carry different hardware demands and timing constraints. Each service needs careful coordination to keep the combined response inside the sub-200ms pipeline window associated with natural exchange, and coupling those services in one deployment makes that coordination harder.

Why a monolithic AI stack breaks down for live video conversations

In a monolith, the perception model, the conversational flow model, the large language model (LLM), and the facial behavior generator ship as one artifact. That single deployment introduces failure and scaling problems that a live conversation cannot absorb.

The specific breakdowns show up in four places:

• One artifact, one release cadence. Every change requires building and deploying a new version of the entire application, so a fix in the facial behavior generator waits behind unrelated work in perception or reasoning.  
• Failures propagate across the loop. A retrieval timeout can make the thread driving facial motion wait, and the face freezes while the reasoning layer recovers.  
• Scaling is coupled across workloads. LLM workloads place different demands on hardware than perception or rendering, so binding them together wastes capacity and increases latency variance.  
• Capacity tracks the wrong signal. Perception and rendering scale alongside the LLM regardless of their own capacity needs, driving up cost without protecting the response window.

Splitting those workloads into separate services removes the coupling and gives each phase its own release cadence, failure boundary, and scaling policy. That separation leads to four core services with distinct responsibilities.

The core services inside a real-time video agent stack

Four services own the loop. Each handles one phase under its own contract and latency budget, and each earns its place by what it protects: timing, understanding, response, or presence.

1. Perception service

The perception service fuses the user's audio and visual signals into a running description of emotional state, attention, and intent. It takes raw media from the Web Real-Time Communication (WebRTC) connection.

Its output is rolling context kept no more than 300ms stale. Any older, and the reasoning layer answers a state the user has already left. That sets the floor for everything downstream.

In a candidate interview, the perception layer combines confident wording with the hesitation before it, then gives the LLM a natural language description of the mismatch between the claim and the certainty behind it. In parallel, the flow service tracks when the response should begin.

2. Conversational flow service

The conversational flow service predicts who owns the floor at every moment by working on raw audio. Transcripts discard the prosody and rhythm cues that distinguish a hand-off from a breath.

Silence-based endpointing waits through a fixed timeout, often several times the gap people leave between turns. It should fire when a human listener would respond: firing earlier causes interruptions, firing later creates dead air.

That requires streaming inference with persistent state across the turn, rather than a request-response call on a buffered chunk. Once the floor changes, the response path moves to intelligence and retrieval.

3. Intelligence and retrieval service

The intelligence service is the LLM reasoning layer paired with a retrieval-augmented generation (RAG) component that grounds each reply in domain knowledge. Uncached retrieval and vector database calls can add enough latency to land as an audible pause mid-thought.

The LLM layer reasons about what to say and do next. Its output becomes the input for real-time behavior generation.

4. Real-time behavior generation service

The behavior generation service converts the intelligence layer's output into synchronized expressions, listening signals such as nods and facial micro-expressions, and response timing. It runs full-duplex and generates listening behavior while the user speaks, because a face that goes still the moment it stops talking looks like a recording.

Output has to hold the frame rate current systems target: 40 frames per second (fps) at 1080p. It also has to stay within a narrow sync tolerance. The ITU Radiocommunication Sector (ITU-R) puts detectability at audio leading video by 45ms or trailing by 125ms.

The four services form the loop. Their communication patterns determine whether it stays inside the response window.

Communication patterns that protect the real-time response window

Not every service in the loop should talk to the others the same way. Perception updates and response tokens carry different obligations, and the transport that fits one will stall the other. Two patterns cover the loop:

• Event-driven streaming for perception and floor prediction. These services emit a stream of small updates where only the latest matters, so teams publish on every frame instead of blocking the producer on an acknowledgment.  
• Low-latency RPC for the response path. The LLM and behavior generation services sit on a single response path, so teams keep it off a broker and use remote procedure calls streamed token by token.  
• Pipelining across phases. Streaming lets each phase start before the previous one finishes, so floor prediction, reasoning, and behavior generation overlap instead of running in strict sequence.

Consider Dana, a new account executive running a discovery call against a video agent for sales coaching. The agent plays a procurement lead. While the intelligence service composes the pricing objection and the behavior service renders it, the perception service fuses her quickened pace with her eyes dropping to her notes.

The fused perception update reaches the LLM before the objection ends. The next turn presses the point, and Dana rehearses the objection she loses deals on before it costs her one. That sequence depends on a latency budget that covers the full response loop.

The latency problem that standard microservices guidance skips

Generic microservices guidance omits a deadline. A real-time video agent has one. Presence depends on keeping full-system response latency under a second while the processing pipeline targets sub-200ms. Human turn-transition research places the modal gap around 200ms, and a 2012 conversation delay study rated conversations most unnatural at 600ms and above.

A tuned budget covers every stage that touches the response, so the work has to overlap rather than run in sequence:

• Parallelize the loop. Floor prediction lets the LLM start generating before the user finishes, and the behavior service starts listening for motion before the reply exists.  
• Drop stale work during degradation. Exponential backoff on a failed retrieval belongs to a moment that has passed; hold a listening expression and serve the next frame instead.

Those requirements shape whether teams build each service or integrate an existing stack.

The decision comes down to build vs. integrate

Building the perception, conversational flow, and behavior generation services in-house is a multi-quarter research program. It pulls machine learning (ML) engineers off work that differentiates the product, and it delays the feedback loop that comes from putting a live agent in front of real users.

A Forbes enterprise AI summary covers MIT Project NANDA's 2025 study of enterprise AI initiatives. The study found that pilots built through strategic partnerships reached full deployment at twice the rate of internal builds.

The practical pattern that follows is to integrate the three behavioral services and build the intelligence layer on the LLM and data the team already owns. That split keeps the differentiating work in-house while the platform layer carries the coordination burden of the response loop.

What integration looks like in practice

Integration only works when the platform layer arrives already split along the same service boundaries the response loop demands. Otherwise, the coordination burden shifts rather than lifts.

Tavus is the human computing company, building Personified Application Layers (PALs) that see, hear, understand, remember, and respond in real-time conversations. A PAL is a real-time application you talk to and build a relationship with, with a behavioral stack already cut along perception, flow, reasoning, and response.

The Conversational Video Interface (CVI) exposes that stack as composable application programming interface (API) services, with each behavioral model owning one phase of the loop:

• Raven-1 fuses emotional and attentional signals, running audio perception under 100ms.  
• Sparrow-2, the conversational flow model, posts 55ms median floor-prediction latency, with 100% precision, 100% recall, and 0 interruptions across 28 challenging real-world conversational samples.  
• The LLM layer reasons about what to say and do next, and teams can drop their own OpenAI-compatible LLM into the intelligence slot.  
• Phoenix-4.5, the real-time facial behavior engine, renders 10+ controllable emotional states, active listening behavior, and emergent micro-expressions in full-duplex.

Each service holds its own contract, so teams keep the intelligence layer they own without rebuilding the behavioral loop around it. Final Round AI runs mock interviews for over 100K job candidates on CVI. When a candidate describes a launch they led, Knowledge Base pulls the competency rubric mid-answer, keeping the follow-up close to the beat a human interviewer would take.

Presence is decided inside the response loop

Dana rehearsing an objection and a candidate defending a launch never see any of this architecture. They experience whether the face across from them noticed, and whether it answered on the beat a person would. That is the outcome the four services exist to produce.

Tavus builds human-like AI agents for face-to-face conversations, cutting the stack along the same boundaries the response loop demands. The behavioral services stay coordinated so engineering teams can focus on the intelligence and data that differentiate their product.

See it for yourself. Book a demo.