Google Research has developed an algorithm that resizes videos for different formats without cropping, stretching or ‘letterboxing’.
The aim is to make a video filmed for one format – 4:3, say, or 16:9 – that appears on another without important bits of the picture going missing or black bars appearing down the sides of the screen.
“Our approach uses all of the screen’s precious pixels, while striving to deliver as much video-content of the original as possible. The result is a video that adapts to your needs, so you don’t have to adapt to the video,” say Matthias Grundmann and Vivek Kwatra of Google Research.
The algorithm works by identifying the really important elements of the image – figures, faces or structured objects such as buildings – and preserving the aspect ratios of these.
“We cannot change this content beyond uniform scaling without it being noticeable,” say Grundmann and Kwatra. “On the other hand, non-salient content, such as sky, water or a blurry out-of-focus background can be squished or stretched without changing the overall appearance or the viewer noticing a dramatic change.”
The technique, developed in conjunction with Georgia Tech, has been dubbed discontinuous seam carving, as it modifies the video by adding or removing disconnected seams, or chains, of pixels.
Key features, says Google, include a method of maintaining temporal continuity of the video as well as its spatial structure, space-time smoothing for automatic as well as user-guided relevant content selection, and sequential frame-by-frame processing, which means it could even be used for streaming video.
“The outcome is a scalable system capable of retargeting videos featuring complex motions of actors and cameras, highly dynamic content and camera shake,” say Grundmann and Kwatra.
There’s more information on the project,here.