Tavus taps generative AI to power personalized videos with voice and face cloning

Generative AI already looks like a major tech trend for 2023. The ability to generate new content via algorithms has penetrated the public consciousness with things like ChatGPT, a chatbot-style technology trained on large scale language models (LLMs). Essays, poems, lyrics, news articles, and even computer programs. And then there’s DALL-E from the same Microsoft-backed OpenAI that spawned ChatGPT. It serves a similar purpose, but is intended for visual creation instead.

Some argue that ChatGPT marks the arrival of AI into the mainstream, but the truth of the matter is a new era of AI-powered applications that will transform nearly every aspect of the industry, including consumer search and inventory. is just the beginning. From photography to real estate to content marketing.

A fledgling startup called Tavus is trying to make its mark by allowing companies to create “unique” videos tailored to specific individuals, but it’s based entirely on a single initial recording. increase.

For example, the idea is that sales and marketing teams can publish an endless stream of video pitches to potential customers based on the text data they submit through an online form. Alternatively, headhunters may use the Platform to send multiple personalized videos to potential candidates using data collected from their LinkedIn profile.

Tavus, a Y Combinator (YC) alumnus founded in San Francisco in 2020 by CEOs Hassaan Raza and Quinn Favret, today announced it has raised $6.1 million in a seed round led by Silicon Valley investor Sequoia. . – Meet backers such as Accel Partners, Index Ventures, Lightspeed Ventures and YC Continuity.

Usage

Companies looking to create multiple personalized videos know it’s an incredibly time consuming and iterative process. Record the same message, with virtually the same content, but tweak it for different clients and candidates. What Tavus is trying to address by allowing users to create their own AI video his template in minutes and generate unlimited versions of the video from that original source.

The initial onboarding process requires users, such as recruiters and salespeople, to record a 15-minute video based on a Tavus-provided script that will be used to train the AI. Users then record a template for each campaign they create.

Using a web-based editor, users can select the elements of the video they want to personalize, specify each variable (company, executive name, location, etc.) and add a call to action.

Tavus supports longer-form variables via ChatGPT-powered snippets, allowing for more personalized referrals. According to Favret, this was highly requested by users. For example, the base his script on which the video is made can be configured to include a one-sentence introduction of him generated from a specific individual’s girlfriend’s LinkedIn profile.

In short, Tavus is trying to replicate what mass marketing software has been doing in the email realm for years and bring it to the visually stunning world of video. In fact, this could set a red flag for some — are people so receptive to personalized pitches when they realize the sender isn’t doing so. TRUE Did you make an effort to make a video just for them? But more than that, AI-generated videos that use personal information gleaned from databases can be a little spooky. Personal You may end up coming across an incredible inhumane When the user finds out how it was made.

Questions like this will continue to be raised by scenarios as AI penetrates more deeply into our daily lives.and a phablet While sales and marketing are clearly a use case for the company’s technology, it’s quick to emphasize otherwise. purely For these verticals, we are seeing hiring from “diverse user groups,” including recruiters, university deans, and C-level executives.

“There’s a common misconception that Tavus only works with sales and marketing teams,” says Favret. “This is our focus, but we have users who are applying Tavus in innovative and powerful ways across the customer journey. We are broadly applying Tavus across our organization, including related functions.”

and WHO Are you suggesting cloning, exactly?

“Usually users clone themselves, but it’s also common for key figures such as executives and publicists to record videos to give the company a consistent look,” Favret said. added. “Tavus was designed to make it easy for any type of user to clone themselves in minutes.”

Internally, Tavus says it uses machine learning to train a model with facial gestures and lip movements to create a system that realistically mimics these movements in sync with synthetic speech.

In terms of deployment, enterprises can access Tavus in two main ways. Half of the users do it from her web dashboard, while others integrate her Tavus into their system via her API or natively.

“Sales teams often use and deploy Tavus directly through the platform because they can efficiently generate large volumes of videos for their planned campaigns,” says Favret. “Other teams will use his Tavus in a more programmatic way and integrate directly within their systems. This will allow the user to generate and send his Tavus video after a trigger is ‘event-driven’.” You can create workflows for ”

As an “event-driven” example, when a sales prospect submits a form on the company’s website, the company automatically generates a custom Tavus video, using data entered by the prospect himself, and Can be sent to prospects.

“This allows companies to take advantage of the right moment to maximize conversions without waiting for team members to record videos,” continues Favret.

abuse

At the top of the generative AI spectrum, companies like Microsoft and Google are vying to be the fastest to get their smart technologies into the hands of businesses and consumers. It’s a battle that Microsoft seems to be winning. the current. At the same time, a number of generative AI startups have emerged, such as GlossAi, which uses AI to help companies easily create shareable marketing skits, and Typeface does the same with marketing copy and image generation. going. at the enterprise.

Unique to Tavus, there have been similar companies for several years already, such as Windsor, which is focused on e-commerce but doing something similar. And London-based Synthesia, backed by a bevy of high-profile investors, focuses on creating digital avatars from text for use in training and how-to videos.

So even before the generative AI hype of the past few months, it was clear that there was a growing movement in that direction from the startup world, and Tavus is now looking to capitalize on it. In fact, Tavus has already amassed some pretty big clients in a short time so far, including his CoStar, a real estate data company, and AB Tasty, a French technology scale-up company.

However, it’s worth considering the potential misuse of this kind of technology. For example, is there anything that prevents someone from uploading a video of someone else speaking and creating a new video from that template? Indeed, examples of deepfake scams from across the video and audio spectrum Never run out. And some companies, such as South Park creator startup Deep Voodoo, have already raised VC funding for their deepfake efforts. As this kind of technology takes root in society and becomes standardized, more questions will arise about the ethical implications of all this, even if the underlying intentions were well-meaning.

According to Favret, Tavus has a built-in feature that makes it more difficult to trick the system, requiring users to perform voice authentication and record live on the Tavus platform.

“This means users cannot upload videos of other people speaking,” said Favret. “Furthermore, users have all the rights to their data and likeness, and they can delete or remove their videos at any time. We are also very intentional about how Tavus is used. We screen each use case before to make sure it meets our community guidelines and is ethical.Security and ethics are very important to us, especially given the youth of this technology in our society. is.”

show me money

In terms of business model, Tavus offers a basic deployment plan for small businesses, which costs $275/month and has limits such as the number of videos you can create. A custom “Business” plan removes these restrictions, but there is no set price as advertised. Basically, we adjust the price according to the method used by the company.

“Tavus plans are tailored to a company’s unique use cases and needs, but at a high level they operate on a usage-based model, where users It’s the videos they generate,” said Favret.

Prior to this latest seed round, Tavus had raised a small amount of money as part of its participation in the 2021 YC program. The full roster of investors in the seed round includes Sequoia, Accel Partners, Index Ventures, Lightspeed Ventures and YC. Continuity, SV Angel, Hack VC, Remus Capital, Mantis Capital, Liquid2 Ventures, Zillionize, Soma Capital, GTMfund, Terra Nova, and several undisclosed angel investors.