Image/Video Deblurring using a Hybrid Camera

Yu-Wing Tai

Hao Du

Michael S. Brown

Stephen Lin

Abstract—We describe a novel approach to reduce spatially-varying motion blur in video and images using a hybrid camera system. A hybrid camera is a standard video camera that is coupled with an auxiliary low-resolution camera sharing the same optical path but capturing at a significantly higher frame rate. The auxiliary video is temporally sharper but at a lower resolution, while the lower-frame-rate video has higher spatial resolution but is susceptible to motion blur.

Our deblurring approach uses the data from these two video streams to reduce spatially-varying motion blur in the high-resolution camera with a technique that combines both deconvolution and superresolution. Our algorithm also incorporates a refinement of the spatially-varying blur kernels to further improve results. Our approach can reduce motion blur from the high-resolution video as well as estimate new high-resolution frames at a higher framerate. Experimental results on a variety of inputs demonstrate notable improvement over current state-of-the-art methods in image/video deblurring.

Publications:

Presentation Slides (ppt)

Related Project:
Richardson-Lucy Deblurring for Scenes under Projective Motion Path

BibTex:

@inproceedings{Tai08cvpr,
author = {Yu-Wing Tai and Hao Du and Michael S. Brown and Stephen Lin},
title = {Image/Video Deblurring using a Hybrid Camera},
booktitle = {CVPR},
year = {2008}
}

@article{Tai09pami,
author = {Yu-Wing Tai and Hao Du and Michael S. Brown and Stephen Lin},
title = {Correction of Spatially Varying Image and Video Motion Blur using a Hybrid Camera},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
volume = {},
year = {2009},
number = {},
pages = {}
}

Acknowledgements:

We thank Bennett Wilburn for providing the high speed cameras used in our implementation.

In this project, we study the tradeoff between resolution and exposure time and propose an approach to deblur high resolution motion-blurred images/videos.

Global Invariant Motion Blur (Motion blur from hand shaking):
Input
Result from [Fregus et. al, Siggraph'06]
Result from [Ben-Ezra and Nayar, CVPR'03]
Result from Back projection
Our Result
Ground Truth

　Rotational Motion:

Input
Result from [Shan et. al, ICCV'07]
Result from [Ben-Ezra and Nayar, CVPR'03]
Result from Back projection
Our Result
Ground Truth

Translational Motion:

Input
Result from [Fregus et. al, Siggraph'06]
Result from [Ben-Ezra and Nayar, CVPR'03]
Result from Back projection
Our Result
Ground Truth

Zoom-in Motion:

Input
Result from [Fregus et. al, Siggraph'06]
Result from [Ben-Ezra and Nayar, CVPR'03]
Result from Back projection
Our Result
Ground Truth

Out-of-plane Rotation:

Input
Result from [Ben-Ezra and Nayar, CVPR'03]
Result from Back projection
Our Result using first low res. frame as reference
Our Result using last low res. frame as reference
Ground Truth

Dual Input Deblurring:

Input 1
Input 2
Our result from Input 1
Our result from Input 2
Our result from using both Input 1 and Input 2
Ground Truth

Video Deblurring (Vase):

Video deblurring with out-of-plane rotational motion. The moving object is a vase with a center of rotation approximately aligned with the image center.

First Row: Input video frames.
Second Row: Close-ups of a motion blurred region.
Third Row: Deblurred video.
Forth Row: Close-ups of deblurred video using the first low-resolution frames as the reference frames.
Fifth Row: Close-ups of deblurred video frames using the fifth low-resolution frames as the reference frames.

The final video sequence has higher temporal sampling than the original high-resolution video, and is played with frames ordered according to the red lines.

Videos for download:
1. Input High Resolution Video
2. Input Low Resolution Video
3. Deblurred High Resolution Video
4. Deblurred High Resolution Video with temporal upsampling

Video Deblurring (Tossed Box):

Video deblurring with a static background and a moving object. The moving object is a tossed box with arbitrary (in-plane) motion.

First Row: Input video frames.
Second Row: Close-up of the motion blurred moving object.
Third Row: Extracted alpha mattes of the moving object.
Fourth Row: The deblurred video frames using the first low-resolution frames as the reference frames.
Fifth Row: The deblurred video frames using the third low-resolution frames as the reference frames.

The final video with temporal upsampling is played with frames ordered as indicated by the red lines.

Videos for download:
1. Input High Resolution Video
2. Input Low Resolution Video
3. Video Matte of the tossed box
4. Deblurred High Resolution Video
5. Deblurred High Resolution Video with temporal upsampling

Video Deblurring (Car):

Video deblurring in an outdoor scene. The moving object is a car moving towards the camera which exhibits both translation and zoom-in effects.

First Row: Input video frames.
Second Row: Closed-up of the moving car.
Third Row: The extracted alpha mattes of the moving object.
Forth Row: The deblurred video frames using the first low-resolution frames as the reference frames.
Fifth Row: The deblurred video frames using the third low-resolution frames as the reference frames.

The final video consists of frames ordered as indicated by the red lines. By combining results from using different low-resolution frames as reference frames, we can increase the frame rate of the deblurred video.

Videos for download:
1. Input High Resolution Video
2. Input Low Resolution Video
3. Video Matte of the moving car
4. Deblurred High Resolution Video
5. Deblurred High Resolution Video with temporal upsampling