Adobe has been in the test kitchen cooking up AI-powered capabilities for video and is finally on the brink of serving them up to users. At IBC 2024, the company gave us a look at new Firefly enterprise offerings for video and audio with an enhanced speech API and dubbing and lip sync API for producing pro-looking and pro-sounding video. It also gave us a preview of the all-new Firefly Video Model, with generative AI capabilities that will be available later this year.
What do we think? Adobe is moving at warp speed when it comes to Firefly, devising and updating its various Firefly-powered models to deliver super-fly capabilities to the company’s various product lines. There’s no doubt that it took Adobe years to develop the training and initial tech and models, but now it is buzzing right along, having delivered the Firefly Image Model, Vector Model, Design Model, and soon, the Video Model. So far, the company’s work has been impressive, and Adobe continues to train on licensed content, so artists are able to use the tools to generate content that is safe to use and does not infringe on creator rights—a badge of honor Adobe points to often. That is not a bad thing, either, as this issue is a big concern to users, especially in this industry.
Adobe previews Firefly for video
It seems as if generative AI takes center stage whenever and wherever hardware or software announcements are made these days. Adobe has been on the cutting edge in this regard, introducing a family of AI models it calls Firefly, focusing on each of its industry segments at a time, starting with beta versions of its products before launching them into general availability.
Since last year, Adobe has teased us with early inspirations of the Firefly-inspired work it was doing in video. At IBC 2024, we got to see additional progress.
First, Adobe detailed recent updates to its video ecosystem, with a focus on its Premiere Pro (beta) and After Effects as we await the October arrival of Version 25 of those two applications. This included a look at new Firefly enterprise offerings for video and audio that broadcasters, content creators, distributors, and others in the media and entertainment industry can use to expand their marketing presence.
Firefly Services (beta) for enterprise customers conprise an Enhanced Speech API and a Dubbing and Lip Sync API. Adobe users are likely familiar with Enhanced Speech from Premiere Pro and Adobe Podcast; now it is available in Firefly Services so enterprises can more easily produce professional audio and video content for a broader audience. The Enhanced Speech API repairs dialog audio at scale, removes unwanted background noise and improves dialog quality, and batch processes raw audio files, alleviating tedious audio cleanup. The Dubbing and Lip Sync API, meanwhile, automates localization of videos at scale and translates spoken dialog into different languages while preserving the original speaker’s voice and ensuring precise lip synchronization.
In other Firefly AI-related video news at IBC, Adobe provided a test flight of its upcoming Firefly Video Model, which provides editors and video pros with tools that help them achieve their creative vision faster and more thoroughly than ever before through capabilities that support creative ideation, fill gaps in their timeline, and add new elements to existing footage.
This latest Firefly addition extends the company’s family of foundational generative AI models for imaging, design, and vector creation to video. Earlier this year, Adobe provided a brief look at its progress, and now, the wait is almost over, as the Firefly Video Model prepares to launch in beta “later this year,” according to Adobe. (Will that happen at next month’s Adobe Max?)
The Firefly Video Model will be integrated into Adobe workflows across Creative Cloud, Experience Cloud, and Adobe Express. It will include features such as:
- Text to Video capabilities to generate video from text prompts; access a wide variety of camera controls such as angle, motion- and zoom to finetune videos; and reference images to generate B-roll that seamlessly fills gaps in a person’s timeline.
- Image to Video capabilities to bring still shots or illustrations to life by transforming them into live-action clips.
- Generative Extend, a new Firefly-powered feature integrated into Premiere Pro, is designed to tackle several editorial challenges, including extending insufficient frames at the beginning or end of a clip, giving the user extra footage to hold on a shot for another beat or to cover a transition, or creating seamless audio edits by generating “room tone” where none exists.
The Video Model supports a broad range of use cases, including creating atmospheric elements like fire, smoke, dust particles, and water against a black or green background that can then be layered over existing content using blend modes or keying inside Adobe’s tools like Premiere Pro and After Effects.
Users who would like to be notified when the Firefly Video Model becomes available in beta can join the Adobe waitlist.