Human computing: the future that never came
Hassaan Raza, CEO and Co-Founder
[I’m writing this on an IBM 5150, the predecessor to the modern PC.]
In the 60s, computers were large enough to fill a whole floor and required a team of specialists to operate. By the late 70s, they had become small enough for people to have at home; the personal computer revolution had started. These computers were still for the select few though, those who could learn to speak their language, understand the syntax and navigate command lines. Then came the graphical user interface. By the mid 80s, computers like the Macintosh replaced commands with icons, windows and a mouse, objects more tangible and familiar to people. Suddenly, millions more people could operate a computer with point and click.
Each of these revolutions further democratized computing and made them more accessible. But the fundamental relationship between human and machine remained the same.
Computers were something you operated, something you used. We learned their commands, navigated their menus: what to click, what to type. We learned to ask for what we wanted in the way the machine could understand.
Sci-fi promised us something different. Computers weren’t something you operated, they were someone that you worked with. The Star Trek Computer or Doctor, KITT from Knight Rider, Cortana from Halo. These computers, or ‘AIs’ understood you deeply, picked up on the joke, and earned your trust. They could reason and act, but they also connected with you on a level that felt, well, human. They were partners, not just tools.
That trust and connection made them someone you could turn to: your trusted AI doctors, teachers, assistants, or therapists.
And it wasn’t just science-fiction imagining this future. The great minds that created the foundations of the very machines we use today were imagining a wonderful future of human-machine collaboration.
In 1960 J.C.R. Licklider, one of the forefathers of modern computing, envisioned ‘Man-Computer Symbiosis’, a future where man and machine would form a deeply coupled partnership that would expand the boundaries of what we as humans could accomplish and create.
In 1987, just a few years after the release of the original Macintosh, Apple was already imagining a future beyond point and click. Under the guidance of John Sculley, who was inspired by conversations with Alan Kay, Apple created the Knowledge Navigator concept. It imagined a new kind of interface that met humans where they were. Instead of navigating windows, menus, and desktops yourself, you interacted face-to-face with Phil, a virtual butler who lived on your computer and could carry out tasks on your behalf. Phil understood you and your intent deeply, he wasn’t just doing what he was told, you trusted him enough to act on your behalf.
It was the natural evolution to computing becoming second-nature, invisible, instinctual. The graphical user interface made the computer an abstraction of the physical workspace. Knowledge Navigator imagined the computer as an abstraction of us.
I could continue to write a whole book on the incredible ideas, dreams and concepts on the future of computing that saw machines transform from being our tools to our partners. That’s not the point of this post though. The point is, we’ve been imagining this future for a very long time, and that widely predicted future never came.
We still don’t have Phil from Knowledge Navigator though. We don’t have Jarvis, or Cortana. We don’t have true AI coworkers or AI sidekicks. We see the current LLM systems as tools, not as partners. The progress in AI has been incredible, yet an intelligent system isn’t emergently one that we can trust, connect and converse with like a human.
That requires a deep understanding of how humans work, our timing, expressions, tone: the things we say and we don’t say and the world context in which all that occurs. It requires knowing you, building trust with you and being able to communicate with you in real time, face to face.
At Tavus, we want to deliver on this future that never came, and teach machines the art of being human. A future where the machine meets us where we are, and using a computer feels as natural as talking to a friend or coworker.
We call this human computing.
We revive the definition of human computing from the 2006 paper ‘Human Computing and Machine Understanding of Human Behavior’ by Maja Pantic, Alex Pentland, Anton Nijholt and Thomas Huang. In this paper they described human computing as the next-generation of interface built around the human, with machines able to understand and respond to our social, behavioral and emotional signals.
I read this paper as an undergrad, ten years after it was published, and it gave language to something I, and many of us here at Tavus, had been fascinated with for years: a computer that didn’t feel like a computer at all, but instead, like one of us.
How to solve the human computing problem
There are a few things we believe are essential for human communication:
We are evolutionarily designed to communicate face-to-face. It gives us a rich, familiar way to communicate, and our brains are wired to prefer it.
Meaning lives in nuance: we speak as much through words as we do through expressions, tone, gestures, timing and all sorts of non-verbal communication.
Understanding our true intent depends on context and memory. The same sentence can mean something different depending on how it is said, who says it, and what happened before.
Personality match is essential to building trust and relationships, and we build familiarity and trust over time. We get along with certain people for their specific traits.
Think about the best teacher. They take the time to get to know every student, they notice when you hesitate, when you lose understanding, or even when you say ‘I understand’ but still don’t. They pause, ask a question, or try another way to explain. They learn what worked for you and remember where you struggled before. They make you feel comfortable and heard.
That ability to understand and adapt is a huge part of what makes them a good teacher. It is also part of what makes it so difficult for machines to understand us.
Now imagine, every student had the perfect teacher. The perfect tutor. With human computing, we believe we can deliver on that dream. Human communication is a dance, and our job at Tavus is to teach machines how to waltz.
We believe the next-generation interface for using a computer is the human computing interface. An interface that sees, hears, remembers, responds and even looks like we do, and can talk to you in real time, just like a human would.
In order to do that, we must build models that understand and simulate human behavior in real time. The smallest of nuances of conversation: expressions, gestures, reactions and responses. It must learn the relationship between what was said, the reaction, and the context in which it occurred. We call these Human Interaction Models (HIMs).
These models are continuous conversation systems, they must be full duplex across audio and video: continuously seeing and hearing you while speaking and expressing. Everything is input, your expressions, your tone, the world around you, even silence. They are constantly thinking, constantly processing, deciding. Should I speak up? Should I wait? Should I interrupt? What should I do with this information? Based on what the model sees and hears it must shape the right words, voice, expression, gesture, all faster than real time.
This requires training on large amounts of conversational video data, and results in a model that is essentially a world-model, but for human understanding and simulation. A model powerful enough is indistinguishable from a real human in conversation.
These models will power the human computing interface. From that interface emerges a new application type: PALs.
PALs are not an application that you use, but an application that you work with. PALs learn from us, collaborate with us, and develop an understanding of us. In return, we build trust and connection with them, allowing them to become trusted assistants, sidekicks, companions and coworkers. Each PAL has a name and a distinct personality, expressed through its face, voice, and the way it interacts with you.
PALs combine the ability to see, hear, respond, understand and look human with personality, memory, knowledge, and the ability to act. They bring emotional intelligence to the interaction, use tools to get things done, and learn and evolve through shared experience.
A tutor PAL named Julie could notice when you’re struggling, try another explanation, and remember what helped for next time. Tom, the solutions engineer PAL, could open a product, walk you through a workflow, and adapt the demo as you ask questions.
And we could finally deliver on the promise of Knowledge Navigator: Phil, an assistant that understands what you’re trying to accomplish, remembers what matters to you, and can act on your behalf. An assistant you come to know and trust through working together.
We believe PALs will become commonplace. We will hire them for their skills and work alongside them every day. They will be computers, and we will always know that. But working with them will feel second nature. We will spend less time thinking about how to use the machine and more time on what we can accomplish together.
Let’s deliver the future that never came.
Tavus, the human computing company
