What is Sora and how does it work? A guide to OpenAI’s text-to-video AI tool

OpenAI has introduced a new video-making tool called Sora. With Sora, you can make realistic and creative videos by giving it written instructions. This model transforms text into videos, and you can create videos that look real and imaginative, all within a minute, just by providing prompts.

Sora can make complex scenes with many characters, different movements, and detailed subjects and backgrounds. OpenAI’s blog says that the model understands how things exist in the real world. It can also understand objects, use them correctly, and create characters that show strong emotions.

What is Sora?

Sora, which means sky in Japanese, is a text-to-video diffusion model capable of creating minute-long videos that are hard to tell from the real thing.

Sora can create videos of up to 60 seconds featuring highly detailed scenes, complex camera motion, and multiple characters with vibrant emotions,” OpenAI said in a post on the X platform, formerly Twitter).

The company says that their new model can make videos look real using pictures or videos you already have.

We’re teaching AI to understand and simulate the physical world in motion, with the goal of training models that help people solve problems that require real-world interaction

How can you try Sora?

Many of us will need to wait a bit before we can try out the new AI model. Even though the company revealed the text-to-video model on February 15, it’s still being tested and fine-tuned.

Red teaming is a practice in which a team of experts, known as the red team, simulates real-world use to identify vulnerabilities and weaknesses in the system.

“We are also granting access to a number of visual artists, designers, and filmmakers to gain feedback on how to advance the model to be most helpful for creative professionals,” the company said.

The company has, however, shared multiple demos in the blog post, with OpenAI’s CEO sharing videos of prompts requested by users on X.

https://t.co/rmk9zI0oqO pic.twitter.com/WanFKOzdIw

— Sam Altman (@sama) February 15, 2024

welcome to bling zoo! this is a single video generated by sora, shot changes and all. https://t.co/81ZhYX4gru pic.twitter.com/rnxWXY71Gr

— Bill Peebles (@billpeeb) February 15, 2024

How does Sora work?

Imagine starting with a static on a TV, noisy picture and slowly removing the fuzziness until you see a clear, moving video. That’s basically what Sora does. It’s a special program that uses “transformer architecture” to gradually remove the noise and create videos.

This tool can create whole videos in one go, not just piece by piece. When you give the program written descriptions, you can control what happens in the video. For example, you can make sure a person remains visible, even if they briefly move out of the camera frame.

Imagine GPT models as tools that create text using words. Sora works in a similar way, but instead of text, it focuses on images and videos. It takes videos and breaks them into smaller parts called patches.

“Sora is based on earlier work in DALL·E and GPT models. It adopts the recaptioning method from DALL·E 3, where it creates detailed captions for the visual training data. This allows the model to better understand and follow the user’s text instructions in the generated video,” explained the company in their blog post.

However, the company has not provided any details on what kind of data the model is trained on.

Read Also:

• How To Build AI Mobile App Using OpenAI APIs?
• 7 Benefits Of Developing An NLP App Using Open AI
• 7 Key Trends In Conversational AI You Must Know