Introducing Memories: Built for Ongoing Relationships

Introducing Memories: long-term memory for Tavus PALs, built around the relationship between a PAL and the person talking to it.

Each relationship has its own evolving memory of who that person is, what matters to them, and what has happened across their conversations. That understanding carries forward when they return, so they can continue from where they left off rather than explain themselves again.

We rebuilt memory around the demands of ongoing human conversation. Being able to find an old fact is useful, but a PAL also needs to understand when that fact should change the conversation: which question to ask, which explanation to use, or which suggestion would make sense for this particular person.

You might mention an upcoming board meeting in passing while talking about something else. Then, when you return a few days later, the PAL asks, “How did the board meeting go?” You didn’t ask it to remember the meeting or bring the subject back up. It connected something you shared before with the conversation happening now.

That is a small interaction, but it represents a fundamental shift in the relationship: from a tool you have to keep directing to a counterpart you can work with. The PAL brings shared context into the exchange and can help move it forward. You can work on something together over time without having to carry all of that context yourself.

We designed Memories around that kind of continuity, taking inspiration from how people remember one another. It is selective, shaped by the PAL’s role, and updated as the relationship develops. It tracks changes, consolidates older conversations, and can return to the original history when an exact detail matters. What it learns is also inspectable, correctable, and deletable.

Human Truth: Remembering Is Understanding

Conversation with someone you know starts with a lot already understood. You have built up shared context: what you’re working on, what happened last time, the people in your life, and the things that matter to you. That is part of what makes the conversation flow. You can get straight into something because you don’t need to establish all of that again.

“It finally worked.” “I’m still worried about it.” “Let’s try the other approach.”

Those sentences make sense because of what came before. The shared history is part of the conversation, both explicitly in what you reference and implicitly in how you respond. It shapes the explanation someone chooses, the question they know to ask, and the suggestion they know would not work for you.

When an AI does not carry that context forward, you have to rebuild it. You explain the reference, repeat the constraint, or remind it why something matters. The conversation becomes work that would be unnecessary with someone who already knows you.

But knowing someone is not the same as recording everything they say. Your doctor and your closest friend might hear about the same difficult week and remember different things. What matters depends on the relationship. That selectivity was a starting point for Memories.

And the understanding has to keep changing. A concern gets resolved. A goal becomes more specific. Something that mattered a month ago no longer does. Memory needs to reflect that progression, not leave the person permanently defined by an earlier conversation.

Why We Rebuilt Memory Around the Relationship

A common approach to AI memory is retrieval-first: extract information from previous interactions, store it, and retrieve relevant pieces when answering a new request. Modern systems can combine semantic search, keyword matching, entity relationships, and temporal information to improve what gets retrieved.

That is valuable. Our own Knowledge Base uses retrieval-augmented generation, or RAG, because finding relevant information in a body of documentation is exactly the problem it needs to solve.

But answering a question about the past is not the same as knowing how the past should influence a conversation.

Suppose you mentioned last week that your board meeting is Friday and the hiring section of your presentation still needs work. On Thursday, you return and say, “I’ve got half an hour. What should we work on?”

The PAL can respond, “Your board meeting is tomorrow. Let’s use the time to finish that hiring section.”

The meeting date helps it recognize a priority. The earlier conversation tells it where to start. You didn’t have to ask when the meeting was or remind it what was unfinished. That information shaped a useful suggestion within the conversation.

Retrieval can support that interaction. But PALs also operate under the constraints of real-time conversation, where milliseconds matter. We need useful relationship context available without adding a search step before every response, and without repeatedly processing an ever-growing history. Keeping memory compact enough to use during a live conversation is part of the engineering problem.

So we separated ongoing relationship context from on-demand historical lookup. The PAL starts with a maintained understanding of the person and their shared history, then looks back when it needs more detail. Three design choices make that practical.

Three Things Define Memories

1. Built around the relationship, not just individual facts

Each PAL-person relationship has its own Memory Store, organized into two complementary parts.

The Profile represents the person: preferences, goals, communication style, and longer-term context.

The Timeline represents what has happened: conversations, events, decisions, commitments, and changes over time.

Both are maintained as the relationship develops and provided to the PAL when a conversation begins.

Together, they give the PAL more than background information. They help it understand where things stand and what would be useful next.

For example, a career coach’s Profile might reflect that you want a role with more autonomy. Its Timeline captures the interviews you prepared for and the offer you just accepted. In the next conversation, it can move from interview preparation to helping you think through your first week. It can also connect that new role to the reason you wanted a change in the first place.

The goal still matters. The situation has moved forward. Having both the person and the history in context helps the PAL respond to that progression instead of treating each fact as a separate piece of information.

2. Selective memory, shaped by the PAL’s role

A tutor and a coach should not build identical memories of the same person. Their roles make different information important.

Memories uses the PAL’s role to shape what gets retained. A tutor’s memory can emphasize recurring difficulties and useful explanations. A coach’s can emphasize goals, commitments, and the circumstances affecting progress.

That memory is maintained through consolidation, not accumulation. After a conversation, the system considers the new interaction alongside its existing memory and generates an updated Profile and Timeline. It revises its understanding rather than simply adding another summary to an ever-growing list.

Changes can retain their history, too. When someone changes jobs, the useful memory is not two competing job titles. It is an understanding that one was true, something changed, and another became current.

Recent conversations stay detailed. Older interactions are progressively consolidated into broader summaries, preserving highlights while keeping the immediate context manageable. In a long-running coaching relationship, last week’s check-in might remain detailed while months of earlier conversations become a record of recurring obstacles, decisions, and progress. The PAL does not need every sentence from those conversations to understand how things have developed.

Compaction can lose detail. When that detail matters again, the PAL can look back through saved conversation history. It may remember that you discussed a repair without retaining the exact quote, then look back when you ask, “What did I say it was going to cost?”

3. Automatic memory, with control over what carries forward

People should be able to understand and shape what their PAL remembers. Developers should also be able to provide useful context before the first conversation.

Memories makes that learned state readable and editable. You can inspect the Profile and Timeline, correct a misunderstanding, or delete information that should no longer be part of the PAL’s memory.

Developers can also supply Pinned Memories: explicit context that takes priority over learned memory and cannot be rewritten by the learning process. A tutor can begin with notes from a teacher. A support PAL can start with context from a customer handoff. The person does not need to repeat information the application already has.

Control matters because a misunderstood preference can keep shaping future suggestions. These controls give both users and developers a way to correct what carries forward, rather than repeatedly work around its effects. Tavus maintains the underlying infrastructure, so using memory does not require building it from scratch.

What This Unlocks

The value becomes clearer when a remembered detail changes the conversation. Here is what that can look like.

A K-12 tutor can connect lessons through what has actually helped a student learn. Last week, drawing pizza slices helped a student understand fractions, although they still struggled with denominators. When introducing percentages, the PAL can return to that same explanation: “Remember how three out of four slices was three-quarters? That’s also 75%.” It has a reason to choose that explanation, and context for which parts need more practice.

A patient-education PAL can follow someone’s journey before and after an operation. Before surgery, a patient mentions being worried about managing at home and arranging for their sister to help. In a later, post-operative conversation, the PAL can ask, “Was your sister able to stay with you?” It can continue the care team’s educational material with that context, rather than repeating pre-operative questions or overlooking a concern the patient already shared.

A support PAL can remember the practical details that determine whether its advice is usable. A customer mentioned last time that they don’t have a screwdriver. When troubleshooting continues, the PAL can say, “Since you don’t have a screwdriver, let’s start with the checks we can do without opening the device.” A small detail from an earlier call changes the path through the problem.

A coach can follow up on the obstacle, not just the goal. Someone wants to make more time for focused work, but their mornings keep filling with meetings. They agree to protect Tuesday morning. The next conversation can begin with, “Were you able to keep Tuesday free, or did something get scheduled over it?” The response gives the coach a specific place to continue.

A sales or customer-success PAL can pick up at the actual point of a decision. A buyer has already explained that data-retention questions are holding up security approval. On the next call, the PAL can ask, “Were you able to resolve those questions, or is that still blocking the pilot?” It can focus on the unresolved issue instead of restarting a product pitch or asking the buyer to explain the decision process again.

In each example, memory changes the question, explanation, or recommendation. The person’s history makes the current conversation more useful.

How Memories Works

Developers connect conversations using a consistent participant_tag. The backend combines the account, PAL, and tag to resolve the appropriate Memory Store, keeping each relationship separate. The tag is a lookup key, not something the PAL sees or uses to recognize a person.

After a conversation ends, an LLM processes a text record of the conversation alongside the full existing memory state and generates an updated Profile and Timeline. This happens asynchronously after the call. The Profile update determines what to retain, expand, or supersede. The Timeline organizes interactions into progressively broader periods: recent conversations, days, weeks, and months.

At the start of the next conversation, the Profile, Timeline, and Pinned Memories are assembled into context after the PAL’s persona prompt. Basic relationship context is available without requiring a separate search for each fact.

When more detail is needed, the PAL can call a retrieval agent to search saved conversation history. The agent selects search terms, reads relevant excerpts or conversations, and returns the information. Recovering an interview date, for example, might involve searching for “interview,” “rescheduled,” and “moved to” to distinguish the original plan from the latest one.

The central engineering challenge is preserving useful information across repeated updates while keeping immediate memory small enough for a live conversation. Consolidation manages that growth; conversation-history lookup provides a path back to details outside the maintained memory.

Evaluating Memory

See our full LongMemEval results here

We built Memories around ongoing human conversation. We also wanted to test it against established measures of long-term recall. A PAL cannot make useful connections if it confuses events, misses a constraint, or treats an old fact as current.

We evaluated Memories on LongMemEval-S, a benchmark for long-term conversational memory. Our evaluation included 500 questions with 38 to 62 historical sessions behind each question, totaling 23,867 sessions. The questions cover factual recall, preferences, knowledge updates, temporal reasoning, and reasoning across conversations.

Memories reached 93.4% in our evaluation, using our memory architecture.

For context, published results from dedicated memory providers include ByteRover at 92.8% on LongMemEval-S and Mem0 at 94.4% on LongMemEval. These are contextual comparisons using different models and evaluation procedures, not controlled head-to-head tests.

The evaluation also highlighted an important distinction: remembering information and using it correctly are separate problems. The benchmarking process allowed us to iterate and improve our system on over 20,000 conversations. Maintaining inspectable memory with an auditable record lets us distinguish what’s captured, later dropped, not retrieved, or retrieved but misinterpreted.

That is the combination we built for: accurate recall and relationship context that is ready for a live conversation. Memory maintenance happens after the call, while the resulting context is available during the next one. The PAL can bring that history into a follow-up, explanation, or suggestion without a separate lookup for each piece of basic context.

Beyond a Single Conversation

Human Computing is about building machines that can work with us as counterparts. That requires more than understanding what we are saying right now. They need to understand what we are working toward, what has already happened, and how the situation has changed.

If you have to explain your goals, repeat your constraints, and reconstruct your history every time, you are still carrying all of that work yourself. Memory lets the PAL bring existing understanding to the interaction and build on it with you.

That changes how we can work with a PAL. A goal can span multiple conversations. A lesson can build on a previous breakthrough. A project can move forward without starting over. The shared understanding develops alongside the work.

Every conversation should build on the last.

Try Memories with your PAL, then come back and pick up where you left off.