“No lights, no cameras, all action.” You knew it was coming. One of the major companies behind the Stable Diffusion image generator has unveiled an amazing AI video creation and editing tool that operates like his DALL-E for videos.
Runway AI is working on a number of extraordinary next-gen creative AI projects, but the newly released Gen-1 video tool is a true confrontation of where these things are and how fast they are progressing. snapshot. Let’s take a quick look at the state of creative AI in 2015.
Now let’s see what Gen-1 can do. This is not a full text to video generator. You can’t ask them to make a Hitchcock-style dog food commercial. Well, not yet. Instead, it asks for an input video and creates different versions of that input video in response to text, image, or video prompts.
So if you shoot something very roughly just to get the basic angles, action and camera movements, ask the Gen-1 to shoot that footage and recreate it in a completely different style. can. You can say bluntly, ‘Make this a film noir scene,’ ‘Make this an underwater scene set in Atlantis,’ or ‘Put these characters on a bus in London.’ Now you can instantly design your own Snapchat filter for him.
Alternatively, you can simply find an image or video example that matches your desired style and upload it. Gen-1 will analyze it and determine what it is, then do its best to reproduce the key elements of the video. in a similar context. Or you can isolate and track the subject and modify it in some way. Alternatively, you can use a broader set of training data to improve the fidelity of your results. check it out:
Gen-1: The Next Step in Generative AI
Yes, like Snapchat filters, it’s a bit crude, flickering and fidgety now, but even in its current form it’s perfectly relevant for music videos, commercials and a wide range of other artistic video projects. .
It doesn’t matter if it’s Gen-1 or something else. It should be clear enough where this is going. The pace of creative AI advancement has been tremendous. With a blink of an eye, an algorithm like this creates the entire movie in his 4K 3D.upload pulp fiction And see it played entirely by dogs. Take a cartoon and generate different live-action versions for each region in which it’s shown, altering the cast’s race, settings, backgrounds and landmarks to make everyone feel at home. Please grow a mustache. Automatically replace product placements. Take Winnie the Pooh off your child’s toy shelf for release in China. Put the butthole back into the cat.
This grows into a super-fast, super-cheap visual effects studio in a box.
The company appears to be still in the research phase of another system called Soundify. Soundify takes a video input, analyzes it to determine what it is, what might be happening, and creates matching audio.
For example, let’s say you upload a scene where someone gets into a car parked in the countryside and drives away. Try to match the background sound to the environment, then try to identify the subject and what the subject is doing, the exact moment when the subject’s activity causes the sound, and where in stereo space those sounds originate. will do. Then generate that sound to match the video. There should be footsteps, doors slamming, engine sounds, tire sounds, whatever the scene demands. Here are some examples.
Soundify sample result A
Again, like Gen-1, Soundify is an early iteration and not ready for prime time yet. But honestly, who would argue against an AI tool at this point? Especially the kind that lets you tweak the output in the same kind of plain language that directors currently provide sound effects teams? ?
These tools are another bittersweet inflection point. They will democratize filmmaking in a way that was unimaginable a few years ago. They also evaporate the entire carrier – in this case dreams creative career.
Soon these tools will start to converge. Descendants of god-like beings like ChatGPT, text generators build on their encyclopedic knowledge of the entire history of the art form, from concept to art style to script, to create precedents that follow current humans. Combining it with the ability not to start creating an entire screenplay. Trends, issues, concerns, language usage, fashion.
Working in conjunction with a DALL-E style image generator, they create a coherent visual style utilizing all the essential pieces of human art since cave painting. Then, in conjunction with filmmaking tools like Gen-1 and Soundify, we re-trained on key parts of every movie humans have ever made to create full movies, ads, Tik Tok, custom Christmas greeting videos, send out propaganda. drift. Any style, any face, any voice, any tweaks, it doesn’t matter.
soundtrack? Have you checked out Google’s MusicLM tool? Fully orchestrated and mixed, in almost any style you can name, in response to a text prompt. The music rises and falls perfectly with the script and the action. It’s easy for a tool like this to pinpoint the emotional climax of a scene and amplify or subvert it with perfectly timed music. The entire system easily accommodates change requests, just as clients expect from today’s video professionals.
It’s hard to understand what parts of the entire movie industry – movie trailers, posters, merchandising – can’t ultimately translate into lightning-fast algorithms. And looking at where this technology is now, it might be fair to say we’re talking about a viable system within a decade.
On a small scale, why not create your own custom Snapchat filters for your live video with images or text prompts? 3 years, top. Heck, it could go down next week, and I don’t think I’ll bat an eyelid at this point.
Buckle up, this can be a bumpy road.
Source: Runway
