Runway Text-to-Video: Fun Gimmick, or the Future of Video Editing?

Runway is a browser-based video editing tool. On the surface, it doesn’t look that different from popular editors like Premiere Pro or DaVinci Resolve. But what sets Runway apart are its AI-powered tools. Sure, it has filters, text, multitrack audio, keyframe animations, multiple layers, and basic editing features, but its real differentiation is in its video text fill tools, rotoscoping and content. In a recent tweet, Runway showed off a promotional video for its upcoming text-to-video converter tool. But it remains to be seen if it will be just a gimmick with a strong marketing push, or if it will be a useful tool for publishers.

Text to video

Runway’s newest feature, currently only available by joining a waitlist, is text over AI video. This tool promises to edit and generate videos using descriptions written in natural language. Their promotional video shows what they plan to do with the tool. It shows a user writing commands to import a video of a city street, make it cinematic (by applying a color hint), remove an object (presumably using its Inpainting content-aware fill tool) . The tool is then shown generating images of a lush garden, browsing through different styles. It is not clear if these are images taken from an image search or if they are generated by artificial intelligence tools such as the recently popular DALL-E or Midjourney. The editor is then shown converting basic text commands into actual edits. This apparent ability to understand and act on natural language prompts is a very interesting and promising technological advancement. This can make advanced editing more accessible to non-professionals.

The first question here is whether this is faster than normal editing. Seasoned editors get faster with practice and by using keyboard and mouse shortcuts, as well as macros. For some actions, entering the command may take longer than actually performing the action.

The second question (which is related to the first) is whether these commands are accurate. Even just looking at the controls in this video, it’s clear there might be a lot of tweaks that might not even save time – and it certainly won’t be as instantaneous as promised.

  • Import – Does the example above import stock video or something that was imported manually? If the latter, did the AI ​​scan the content of the imported video and add this metadata?
  • Scaling over time – How long is the animation? How much does it zoom? Is the scale approaching the center point or a different point?
  • Fade in text – What type of font and what color is the text? How long does the fade take?
  • Blur the background – How much blur is applied? What style of blur is it?

Of the commands listed, the one that seems likely to work as expected is character green filtering. Making the background black and white is also probably easy. But at this point, is it faster than typing “black and white” into a list of filters and applying it? It might be quicker to tap the action to apply something like a blur and then adjust from that baseline if you’re not happy with the results.

All in all, the text to video functionality seems like a very interesting leap in technology. But whether this is an interesting showcase of AI technology or if it will help publishers speed up their workflow remains to be seen.

Luckily, Runway has other useful tools that might be worth your time. And you can use them now instead of joining a waiting list.

Green screen

Runway’s green screen tool (commonly referred to as rotoscoping) has high profile users with significant successes. One of the biggest is the graphics team for The Late Show starring Stephen Colbert. They claim to have reduced their rotoscoping workflow from hours to minutes. Rotoscoping is clearly Runway’s strength, easily and intelligently identifying people and objects. And if you need to combine that with a simple modification, the tool can do that. It’s easy to imagine Runway as part of a larger editing workflow, but big productions probably won’t be using it as their sole editing tool in the near future.

Compared to other popular rotoscoping methods in software like After Effects, Runway is competitive even with Adobe’s newest artificial intelligence tool, Rotobrush 2.0 (check out our Rotobrush 2.0 video tutorial here!). But an argument could be made for just keeping your workflow within the Adobe environment, as Green Screen and Rotobrush 2.0 have their own shortcomings. In the 2021 video below, the VFX Corridor YouTube channel tests various rotoscoping methods. Even in the video below, it’s not quite the one-click solution that Runway seems to promise. With a moving subject that changes orientation, several clicks are required to keep the subject selected. But even with that in mind, it was much faster than the other alternatives, including Rotobrush 2.0. Rotobrush was a very close second, especially compared to the more manual routes, but they still paled in quality compared to Runway. To quote one of the Corridor Crew: “Usually with AI…you trade speed for detail, but with this you get both.”

Paint

Runway’s Inpainting feature is a content-aware infill tool. This means that you can select an object in your footage and Runway will remove it while filling the space depending on the context. This is a feature in various software, and it won’t work every time. But it’s a fantastic tool for getting rid of distracting objects and passers-by that have wandered into your shot. Simply paint over the object you want to get rid of, and the tool does the rest. This kind of feature works great on simple backgrounds with minimal movement, but it can be a lifesaver in many scenarios.

Trick or change of life?

With the text-to-video still behind a queue and no actual demos still in the wild, there are a lot of questions regarding the effectiveness of Runway’s next feature. It’s still not clear. But the technology has a lot of potential! It still needs a lot of human intervention, so it won’t steal too many jobs. And the time saved will depend on the accuracy of the tool. We’ll look forward to seeing the results when the video text goes public, but for now, you should keep a healthy dose of skepticism when watching Runway’s promotional videos.

Leave a Comment