Adobe researchers unveiled VideoGigaGAN, a generative AI model that can upscale blurry videos into crisp, smooth video that looks up to 8x sharper.
We’ve had really good image upscalers for a while now, but making a good video upscaler is exponentially more difficult.
Video Super Resolution (VSR) is the process of taking individual frames of a video, upscaling the resolution and detail, and fitting the frames together to recreate the video.
Doing this well involves solving two conflicting challenges. Current VSRs either generate video that is smooth and blurry, or sharp and glitchy.
Adobe’s VideoGigaGAN upsamples blurry video to produce a video that is both temporally consistent (smooth frame transitions) and has high-frequency details.
Here’s an example of what VideoGigaGAN can do.
Adobe research drops VideoGigaGAN
It allows you to upsample video by 8x with enhanced details.
Paper in comments 👇 pic.twitter.com/7uEiU7bYqw
— Kris Kashtanova (@icreatelife) April 22, 2024
As the name suggests, Adobe’s method relies on GigaGAN, an advanced generative adversarial network (GAN).
GANs are great at upsampling images, and GigaGAN is one of the best at image super-resolution. So why not simply use GigaGAN on each frame to upscale the image and then put them all together to make the video?
When Adobe’s researchers tried that they achieved great video resolution but the resulting video was temporally inconsistent and flickered.
By adding temporal convolutional and attention layers to the GigaGAN the temporal inconsistency was fixed but the flickering was still an issue.
VideoGigaGAN addresses this by separating low-frequency and high-frequency elements in each frame and processing these differently.
The low-frequency feature map is smoothed to remove high-frequency details, which can be sources of noise and flickering.
Using Skip connections, the finer details in high-frequency components are retained by bypassing the middle layers in the model that would otherwise be lost in processing.
You can read more about the technical details in Adobe’s paper.
The demos on Adobe’s GitHub are very impressive. Adobe hasn’t hinted at a release date but let’s hope they let us use it soon.
Imagine what a tool like this could do for historical archival footage, classic movies, or even upscaling your favorite old TV shows into HD.
