Featured

Introducing Phoenix & the Tavus Developer Platform Public Beta

Julia Szatar
March 12, 2024
min read
Contributors
Build AI video with Tavus APIs
Get Started Free
Share

From the beginning, our mission has been to build a product that lets users create personalized 1:1 video content, and deliver it at scale. 

We’re excited to share the next big steps in that journey, following our $18 million Series A funding led by Scale Venture Partners, Sequoia Capital, Y Combinator, HubSpot, and others.

Today, we’re launching the Public Beta of:

  1. Phoenix, our breakthrough model that delivers realistic digital replicas and text-to-video
  2. Tavus Developer Platform with our models packaged in easy-to-use APIs

Now, product teams can easily leverage our AI models and APIs to build novel engagement experiences with AI generated replica videos. 

With Phoenix, end-users can generate entire videos of themselves, from just a script. No more pain of recording, or re-recording. Being able to rapidly make videos, opens up a world of experimentation and makes hyper personalization of video content possible.

Sign up to our developer platform.

The power of Phoenix

Phoenix generates realistic talking head videos from just a script, complete with natural lip, cheek, nose, eyebrow, chin movements and expressions synchronized with your script and generated voice.

Developed by our team, this novel approach bypasses traditional methods and constructs dynamic, three-dimensional facial scenes using neural radiance fields (NeRFs).  

Most models out there use lip sync methods and operate in the 2D space. The Phoenix model takes a more advanced approach using 3D reconstruction, resulting in a digital replica that looks and speaks like you.

Access Phoenix via our APIs

We’re thrilled to be making our APIs available to developers and product teams on our new Developer Platform. Startups and large companies have been requesting access, and we see limitless potential for our technology's application.

Our APIs are easy to use, requiring one or few API calls and only one or a few parameters. We do everything else for you under the hood.

Today, you can access the Replica API in beta, with the rest of the suite coming soon.

Replica API (Beta)

The Replica API uses the full capability of the Phoenix model. Users can create a digital replica of themselves with just 2 minutes of training footage, and generate realistic videos from a script in minutes. Currently, the replica takes ~9 hours to process. 

The model moves the entire face leading to realism, naturalness, and quality.

  • Multi-language support, generate in 30 languages
  • A video is generated from text in just one API call
  • A replica is trained in one API call with one parameter (video URL)
  • Tavus handles end-to-end pipeline
  • Tavus handles privacy and security 

Great for high-touch use cases where realism is important and the user is too busy to record. 

Sign up to the developer platform.

Lip Sync API (Coming Soon)

The Lip Sync  API is powered by a derivative of the lip sync portion of the Phoenix model, called Hummingbird. It allows a user to lip sync parts of, or an entire video.

  • Lower model training time than Replica API
  • Tavus handles privacy and security 

Great for video editing use cases like making in-place changes to words or short phrases, or doing personalization with variables.

Sign up for waitlist.

Dubbing API (Coming Soon) 

The Dubbing API uses the same technology as lip sync and a multi-language voice cloning model. A video is translated from one language to another. The final result looks and sounds like the original speaker is talking in the target language.

  • Lower model training time than Replica API
  • Tavus handles privacy and security 

This could be used to translate any sort of video for example, training videos or TED-style talks into different languages to broaden the audience.

Sign up for waitlist.

Video Campaign API (Coming Soon)

The Video Campaign API allows developers to provide an end-to-end video campaign experience out of the box, within their solutions. It's a bundle of our Replica API, along with campaign tooling including:

  • Video hosting 
  • Variable mapping and management
  • Personalized landing pages and thumbnails
  • Analytics

This is great for CRM platforms, as it provides the ability for templated video personalization that maps to CRM data directly, and gives end to end hosting and analytics solutions.

Sign up for waitlist.

What will you build?

We’re excited to see what you dream up with replica videos.

Here are some pretty cool end-user use cases we’ve already seen from developer partners:

  • Airline Loyalty Program: A high-touch experience for frequent flyers. Their digital journey starts with a personalized (generated) video from the pilot on flight status, menu options, and other info based on their upcoming flight.
  • Customer engagement: Customer teams generate videos for outreach, relationship management, and more. Reaching more customers with video with less resources.
  • Doctors are getting into video! Doctors can make video sequences to help build a relationship with patients and improve comfort and care between appointments.
  • Making training and compliance fun. Personalize compliance content with employees as stars in the content. Jack from IT, your CEO or Jackie from Design speaking in any language. 

Video will just be different

The Phoenix model allows users to completely reinvent how they use video.

You can now generate 10s, 100s, or 1000s of videos of yourself, without having to record and re-record, in minutes. It’s not just volume. You can now create videos and experiment in ways that were previously impossible. For example, having you speak in a language native to the viewer. Or having busy people make a lot of video content, like pilots, CEOs, doctors, lawyers.

You only make replicas of you

Tavus is built with privacy and security front-of-mind so developers can focus on user experience.

A person is only able to create a replica of themselves. Our technology cannot be used to replicate others. As part of our APIs, we employ a suite of safety checks including voice identification and user consent to ensure only a user can make a replica of themselves and holds the keys to their likeness. The process is handled on behalf of users by Tavus. 

Your feedback

We’re looking forward to hearing your feedback on the Beta.There is still much work to do, and we want to learn from you. 

We’re already working on a few things including reducing the time it takes to train a replica, and continuously improving our documentation.

What are you going to build with AI video? 


Sign up www.tavus.io/developer

Read documentation https://docs.tavusapi.com/sections/introduction

Research initiatives

The team is at the forefront of AI video research and pushes model updates every two weeks based on the latest research and customer needs.

Product
5
min read
This is some text inside of a div block.
min read

How to do Text to Video for AI Replicas

An introduction to how to make an AI video with Tavus' video API.
Product
min read
This is some text inside of a div block.
min read

Build Your First Real Time Conversational Digital Twin in Five Minutes

A tutorial on how you can build a conversational AI in just a few minutes with details of what each of the parameters do.
Product
5
min read
This is some text inside of a div block.
min read

Build a Custom Personality for Real Time Video AI

Customize your conversational AI's personality using a custom persona with system prompts, context, and your own LLM.
Product
5
min read
This is some text inside of a div block.
min read

How to do Text to Video for AI Replicas

An introduction to how to make an AI video with Tavus' video API.
Product
min read
This is some text inside of a div block.
min read

Build Your First Real Time Conversational Digital Twin in Five Minutes

A tutorial on how you can build a conversational AI in just a few minutes with details of what each of the parameters do.
Product
5
min read
This is some text inside of a div block.
min read

Build a Custom Personality for Real Time Video AI

Customize your conversational AI's personality using a custom persona with system prompts, context, and your own LLM.

AI video APIs for digital twins

Build immersive AI-generated video experiences in your application