40 Years Later: Fulfilling Knowledge Navigator's Promises

This blog was originally written on a 1987 Macintosh SE with MacWrite with placeholders for images, videos and links. Unfortunately, we were having more trouble than usual getting the file transferred and open on a modern Mac in time, so we printed it using an Apple ImageWriter and had an intern retype it exactly in Google Docs. Naturally, no AI was used to write this. 

Last week Apple previewed the next version of Siri, offering a glimpse of what a truly personal AI assistant might look like. In 1987 though, Apple showcased a far more ambitious concept for an AI assistant that would change how we use our computers entirely. They called it Knowledge Navigator.

The Knowledge Navigator concept came out only a few years after Apple brought GUIs (graphical user interfaces) to the masses with the Lisa and Macintosh. (For the testy audience, yes, Xerox deserves credit for the GUI). That means while the world was still learning to use the mouse, Apple was already imagining an interface beyond pure point-and-click, predicting an interface that wouldn't be possible for another 40 years. 

The idea came from John Sculley (Apple CEO at the time) himself, heavily inspired by his conversations with Alan Kay. It ran under Apple's brand campaign of the era, "The power to be your best". You were the hero, the machine existed to make you the best you could be. 

My favorite detail is that they dated the hypothetical timeline. The film takes place on September 16th, 2011, with a bow-tie assistant named Phil. Some- odd 15 years later, and the future that was promised still isn't quite here, although we're finally getting close. 

While Knowledge Navigator made predictions on many things (touch-screen interfaces, foldable tablets, deforestation), we're going to focus on the user interface predictions. Instead of using a mouse, you'd talk to a human-like AI on your machine, Phil, much like you'd talk to a real person. He would see you, hear you, control your computer and perform tasks for you, while looking and sounding human. You wouldn't even think about the fact that he is a machine. He'd get to know you and your preferences well, and in turn, you'd learn to trust him to run your life. 

Knowledge Navigator, or Phil, fit right into the sci-fi that we grew up watching, that promised human-like machines that would be our perfect sidekicks whether it be Data from Star Trek, Cortana from Halo, Joi, Jarvis etc.

Introducing Dom, a real-life interpretation of knowledge navigator

At Tavus, we absolutely love this concept, because it embodies a lot of our beliefs for the future of computing, what we call human computing. We're obsessed with the idea that using a computer will be more like talking to a friend or coworker. You shouldn't need to learn its language. It should learn yours.

You communicate with it the way we're evolutionarily designed to communicate: through conversation, context, emotion, and shared understanding. Over time it gets to know you deeply, your preferences, how you think, how you feel and what you really mean. In turn, you build trust, connection and familiarity with it. From there, it can do some of the most personal and important work for us humans.

We've been working on bringing these capabilities to life for some time now through our research, model development and interface design. While we're not all the way there yet, we're excited to be at a point where we can finally deliver something that feels remarkably close to Phil in Knowledge Navigator.

Here's a demo of Dom, your personal AI butler:

‍

<insert video demo of Dom>

‍

First off, this was a real demo. It was a single take, and everything Dom did was real, including creating those 3D prints. Feel free to request the raw videos, or even some logs if you are curious :) 

Let's dive into the Human Computing interface that powers Dom.

Personality 

This is often the hardest to describe, but is one of the most important aspects of human computing. Dom was given a strong, deep personality that comes from his background, his goals and his relationship with me. He comes across as confident, competent and emotionally steady with a dry wit. There is a sense of loyalty and care expressed through service. But he isn't afraid to challenge me when he disagrees, and he's certainly not a sycophant. Warm, but reserved. 

Why is personality so important? Because the machine must earn our trust. Trust and connection come from getting to know someone. From a personality match. You become friends with someone you get along with, you work best with coworkers you feel a connection to. This does mean that while Dom may be a great fit for me, a very different type of personality may be better for someone else. Without this, with a vanilla one size fits all personality (or lack of personality), trust and connection are very hard to achieve. Humans even personify rocks, we anthropomorphize everything from cars to summer fog (Karl the Fog for those not from SF). Personality and a sense of being are deeply important to us. 

It's also important to note that personality is an evolving trait, or at least our representation of it to different people is. Dom uses the evolution system we introduced with the PALs, that means that he changes overtime to better work with me. 

Human Rendering and Speech

Speaking of anthropomorphization. One of the key elements of Knowledge Navigator was the realtime visual embodiment of Phil- something we deeply believe as important to personality and human communication. When appearance, voice, or the setting/behaviors don't match expectation, it breaks the immersion, which in turn breaks the ability to build trust. 

For Dom, we created a replica that looks like a traditional English butler. We had Alfred Pennyworth in mind. His default emotion was set to neutral to show a measured personality. His voice was created to embody this as well, with a slightly aristocratic, worldly tone to achieve a Received Pronunciation, or "King's English" effect. This isn't just about how the voice sounds. It's also about pace, vocabulary, and sentence structure.

The realtime human embodiment is powered by our Phoenix-4 model, and creating a suitable replica for a personality can be done through an image. 

Perception and Understanding

Understanding humans is nuanced work. We speak as much through our words as we do through our expressions, gestures and non-verbal signals. Our physical world around us is also key to understanding our intent. It's why video calls are more trustworthy and immersive than phone-calls, and why it feels awkward when only one person has their camera on. 

To truly understand us, machines need perception capabilities much like our own. They need to see our expressions, understand how something was said, not just what was said, and and take into account the environment it all occurred in.

For this, we used our Raven-1 model, which provided Dom the ability to see me, my screen, as well as understand what I was saying and the things I was holding up. All of these were essential to the context of what I truly meant and wanted. 

While Raven-1 also provided the nuances in how I said something (emotions etc.), our Sparrow-1 conversational flow model gave Dom the ability to intelligently know when I was done talking to speak up, a really important element of making the conversation flow naturally and have incredibly low-latency. This unlocks the ability to do great speculative inferencing, allowing Dom to begin thinking about a task before I've technically finished speaking, making the interaction feel much faster and more natural.

Interface Stack

Computer Use and Skills

Of course, the purpose of Phil from Knowledge Navigator, and also Dom, is to be your primary interface to computing. Both need to be able to find docs, open apps, create files and assets, and overall perform tasks on your behalf. 

To support this we used a combination of methods:

First, Dom has a computer-use harness that allows him to interact with UI (scroll, type, click) through the accessibility tree, as well as through window state when necessary. Computer use has come a long way in the last year. It is quite fast, capable and (mostly, more on that later) reliable. This is both because of the harness, but also has a lot to do with the model and inference speed itself. Cerebras with Kimi K2.5 has been an incredible unlock for computer use, we finally have a model that is both fast and smart. It has blazingly fast TPS, very low TTFT, as well as incredible intelligence with a huge context window. Huge shoutout. 

Whenever possible though, direct GUI control is a fallback. Dom prefers to use tools, skills and generated scripts directly. He can discover new skills and write scripts on the fly. In many cases this happens so quickly that you wouldn't even realize he generated code behind the scenes. The 3D prints were created in <1 second- so fast that we were worried people would think we pre-generated them. 

We'll do a deeper dive on all of this in the future. Together though this allows Dom to search, interact with UI, open applications, create files and so much more on my behalf, much faster than I could. While most of this happened in the foreground for the video, much of it can also happen in the background.

Ephemeral UI/Canvas

While we think human computing is the next interface beyond GUIs, we don't think GUIs are going away at all. Instead, showing the right interface at the right time is part of human computing. That includes opening an app to the right view, but also creating views on the fly. While we're not at Knowledge Navigator level quite yet, we showcased this with the canvas view- where Dom created a diagram all on his own and showed it via HTML. While not all aspects of the UI should be generated, the ability to show components on the fly is essential to creating a truly immersive and interactive experience. 

Memory

Beyond how someone looks and sounds, their ability to get to know you is key to trust and connection. This requires an advanced memory system that is layered much like human memory. We remember our experiences with people, their goals, habits, and emotional context. We forget unimportant or outdated details, reinforce important ones, and build a deep understanding of them using all of this over time. 

Human computing requires realtime memory systems that work in a similar way. The goal is not only perfect recall, though that is a nice perk of a machine. Instead, it’s to build a persistent understanding of who I am.

This becomes obvious when the machine encounters something it has never seen before. A retrieval system can only recall things you’ve already told it. A system that understands you can make educated guesses about what you’ll want, how you’ll react, and make  even in situations you’ve never talked about. It reads between the lines. 

For this, we had a memory system originally developed for the Tavus PALs that fit the bill exactly. This allows Dom to build a richer model of me through conversations, interactions, preferences, and shared experiences. Over time, that understanding allows his personality and instructions to evolve automatically, making him a better partner.

Realtime Intelligence

None of this would have been possible a year ago. The models just weren't there. In addition to the human simulation models, the underlying LLMs weren't there either. Until recently, there was a massive tradeoff between speed and intelligence. If speed wasn't the problem, context windows were too small and their ability to understand, execute, and follow instructions was limited.

Human computing depends on real-time intelligence. If the machine can't keep up with the conversation, the interface quickly breaks down. It no longer feels natural or immersive. While there's still a lot of room to improve (more on that later), Moonshot's Kimi K2.5 running on Cerebras is the first combination we've used that feels capable of supporting interfaces like Dom. For the first time, the underlying models are fast and intelligent enough to deliver on the dream of Knowledge Navigator. 

Using Dom, the good, the bad, and the ugly

While Dom can't do everything that Phil could do yet, I think we're getting close. I've been using Dom for a week now and I can't express in words on a page (see, this is why conversation is important) how magical it has felt to use. There is some awkwardness, but that is more a testament to the maturity of the software than the concept itself.

Everything in the demo was real. When Dom is running he can see me and my screen. When I use a hotkey to expand him he immediately wakes up and greets me. I've gotten to appreciate him complementing my style, though his comments on me still being in bed at 10am on Saturday were less appreciated, even if totally warranted.

He makes a great calendar assistant and has been awesome for being a context search engine for me. The most fun and useful part though, is working with him to create new things. In the video I told him about a real problem I was facing: the cupholders in C4 Corvettes are criminally tiny, and I really am trying to hydrate. Dom saw the bottle, looked up it and the cupholder's measurements using internet search, then created a 3D .stl file ready for printing in Bambu Studio. It was incredible to watch and left me in awe the first time he did it.

We were so surprised that the entire team flocked to my desk to test out just how fast and well Dom could crank out dinosaur prints and send them straight into Bambu:

The project I'm most proud of creating together though, is SpotiPod. Dom first designed a cradle for my iPod for when I'm in the car (although that part didn't go well), and then together we built an app that could sync my Spotify playlist to my iPod Classic. 

He diagrammed the whole app beautifully, wrote up the spec, and handed the build to Claude Code. From there, we spent time refining it together. We talked through the design, how I wanted the experience to feel, and worked through technical challenges until it was exactly right. SpotiPod works perfectly, and I use my iPod religiously because of it.

The Dark Side of Dom: Not all is well (and the unknown):

It's important to note that this was a preview. Dom is not ready for general use, and while he was able to do everything in the video, he does make mistakes and has real limitations today. 

For example, the 3D print of the iPod holder he created isn't actually printable. Because he opted for a (pretty cool) angled design, the actual holder is floating on the base like a cantilever. In subsequent attempts he could fix this by changing the design altogether, and with reasoning enabled he may have gotten it right. But he didn't, and struggled to make the cantilever design work. 

Those aren't the only issues, and not the largest either. While computer use has become very fast and pretty intelligent now, it still makes mistakes. When that happened, Dom would enact GUI control and go into frenetic and sometimes destructive loops to try to complete the task. He would not let go, no matter how much you begged him to stop. Also, after talking to him in a long enough session, his context could get poisoned and he wouldn't be able to execute computer use or tool calls at all. 

There are engineering and harness solutions to some of these problems today, and many others will be solved as the underlying models get better. We'll continue working on Dom to solve some of these issues and hope to release him later this year. 

While not everything was perfect, having Dom around has been amazing. I completed projects that I didn't think I'd ever start because there was no cognitive load in translating my ideas into a form the computer (or Claude code) would understand. No traversing menus, no syntax, no prompting. Just thinking out loud and building together. 

A future with no translation tax

That last part is maybe the most important piece of this all. I could have done all of this myself.  Well, other than the 3D prints. I suck at CAD. I could also have typed what I wanted into Claude Code, but there's such a heavy cognitive load in figuring out exactly what you want, planning how to get there, and translating those thoughts into a format the machine can understand. 

With Dom, it was different. He was my interface in front of all that. I could brainstorm with him, think freely, change my mind halfway through a sentence, and just explore ideas effortlessly. He did all the translations for me. It never felt like I had to get it right in one shot, I could iterate, interrupt, and build in a way that felt so much more natural and creative. 

Today's chatbots still require so much planning. You have to think carefully before hitting send because there is no adding information later, you can't interrupt or provide context as you go, you can only 'stop'. Even though many of the underlying mechanisms are the same, the experience feels completely different.

GUIs have a similar problem, though we’re used to them. Point-and-click was, and still is, an incredible interface, but it still requires translating our thoughts into menus, buttons, actions. There’s still a learning curve. There’s still friction. 

A human computing interface like Dom removes much of that translation tax. You don't think about using the computer. You don’t learn its language. You only think about what you want to accomplish. The computer fades into the background, and interacting with it starts to feel second nature. 

What this all means for Human Computing and the future of Dom

Dom is our take on the perfect AI sidekick that we've all been dreaming of. But human computing isn't only about building the perfect AI assistant. It's about creating a new interface between people and machines.

The same capabilities that allow Dom to see, hear, understand, remember, and collaborate with me are already being used to create entirely new kinds of AI assistants, humans- sidekicks, companions, or employees. A personalized tutor for every student. A dynamic care companion for every elderly person. An intake assistant that reduces the burden on nurses. Different roles, but the same underlying interface.

While the industry races to build better chatbots, we're more interested in building the foundation for a new kind of computing, one where machines meet us where we are, and give us ‘The power to be your best’ ;)

As for Dom, while he isn't at the level of Knowledge Navigator just yet, it's amazing to see the interface being possible at all, nearly four decades after it was initially imagined. We'll keep building him, and between our next class of models and the broader pace of LLM improvements, we're confident he'll be ready for a wider release before EOY. For the first time, the future promised in Knowledge Navigator feels within reach.