Meta has released a new update pushing a new AI tool to its growing arsenal. This time, they’re bringing Hollywood vibes to AI with Movie Gen — a new AI model that turns your text prompts into full HD videos. While the visuals are pretty impressive, what got our attention was the perfectly synced audio and a whole set of personalization tools. Here’s a deep dive into what Movie Gen is about and its current status.
Table of Contents
What Is Movie Gen?
Movie Gen is Meta’s new video-generating AI tool. What sets it apart from similar text-to-video AI tools is its use of 30 billion parameters, the highest in the industry for now. Think of parameters like giving AI some brain cells to learn from the training data. The more parameters a model has, the more finer details it can capture from the same data. For comparison, OpenAI’s SORA has around 20 billion parameters (according to unofficial sources), making Movie Gen a significant step forward.
Currently, the model can create 16-second videos at 16 frames per second, with synchronized 48kHz audio. However, it offers many other capabilities, which we explore below.
Here’s what Meta’s Movie Gen Can Do
Text-to-Video Generation
You can generate a video by simply typing a text prompt. The AI model processes the text and creates a fully rendered video with high-quality visuals including the sounds. The system supports different aspect ratios like 1:1, 9:16, and 16:9 and can generate videos in resolutions up to 1080p.
Produce personalized videos
You can upload an image of yourself or others too and then type a prompt to generate personalized videos. This means you could, for example, place yourself in a scenic landscape or create a video where you appear to be interacting with animated elements. You can also change aspect ratios and resolutions up to 1080p.
Edit Videos with Text
Movie Gen AI tool makes editing existing videos easy like typing text. You can modify existing videos by providing text instructions including adding or changing objects, altering the background, or adjusting other visual elements.
For instance, if you have a video of a beach scene, you could add an instruction like “add a palm tree to the left side” or “change the sky to sunset.”
Video-to-Audio Generation
This is the cool part. Movie Gen creates short videos with matching audio that syncs with the frames. However, if you upload your own videos, Movie Gen can generate background music, sound effects, and ambient noises that align with the video content.
For instance, if the uploaded video is set in a forest, Movie Gen will add the sounds of rustling leaves, chirping birds, and other natural noises. Other than sounds, you can also add music by mentioning “rock guitar music” which will play in sync with visuals so it makes sense to the viewer.
Text-to-Audio
This feature allows you to create realistic soundtracks or effects from text prompts, even if you don’t have a video. For example, you could type “city street during the evening” and Movie Gen AI will generate the appropriate ambient sounds like traffic noise, people talking, and distant honking.
Limitations of Movie Gen
While Movie Gen is impressive, it’s not without its limits. Here is what we found:
- No Public Access: Right now, Movie Gen is available only to Meta’s research teams and select partners. There’s no public access yet, though Meta hints they may release some related data for research use in the future. So most content creators have to wait for the public release.
- Short Videos Only: Currently, the model can create videos up to 16 seconds long. For anything longer, you’ll need another solution. For example, OpenAI claims SORA will support up to 60 seconds.
- Frame Rate: It generates videos at 16 frames per second, which works well for short clips but isn’t ideal for long or high-quality videos that demand higher frame rates.
Until Movie Gen becomes publicly available, its performance in real-world scenarios remains uncertain. However, given Meta’s track record with open-source AI models, it’s likely that Movie Gen will also become open-source and free to use, making its advantages more accessible. This open-source nature could also foster the development of more advanced models with higher frame rates and longer videos.
Can Movie Gen Change the Game
Currently, we can only rely on Meta’s claims, and based on that, this tool appears to be the best video-generating tool on paper. It can generate videos from scratch or using your images, edit videos with just text prompts, and even create audio. If it becomes open source like Meta’s Llama models, it could significantly impact content creation for everyone. However, its real-world performance is still yet to be tested.
