Soft Color Segmentation and Its Applications

Yu-Wing Tai

Jiaya Jia

Chi-Keung Tang

Abstract—We propose an automatic approach to soft color segmentation, which produces soft color segments with an appropriate amount of overlapping and transparency essential to synthesizing natural images for a wide range of image-based applications. Although manystate-of-the-art and complex techniques are excellent at partitioning an input image to facilitate deriving a semantic description of the scene, to achieve seamless image synthesis, we advocate a segmentation approach designed to maintain spatial and color coherence among soft segments while preserving discontinuities by assigning to each pixel a set of soft labels corresponding to their respective color distributions. We optimize a global objective function, which simultaneously exploits the reliability given by global color statistics and flexibility of local image compositing, leading to an image model where the global color statistics of an image is represented by a Gaussian Mixture Model (GMM), whereas the color of a pixel is explained by a local color mixture model where the weights are defined by the soft labels to the elements of the converged GMM.Transparency is naturally introduced in our probabilistic framework, which infers an optimal mixture of colors at an image pixel. To adequately consider global and local information in the sameframework, an alternating optimization scheme is proposed to iteratively solve for the global and local model parameters. Our method is fully automatic and is shown to converge to a good optimal solution. We perform extensive evaluation and comparison and demonstrate that our method achieves good image synthesis results for image-based applications such as image matting, color transfer, image deblurring, and image colorization.

Publications:

BibTex:

@article{Tai07pami,
author = {Yu-Wing Tai and Jiaya Jia and Chi-Keung Tang},
title = {Soft Color Segmentation and its Applications},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
volume = {29},
year = {2007},
number = {9},
pages = {1520-1537}
}

@inproceedings{Tai05cvpr,
author = {Yu-Wing Tai and Jiaya Jia and Chi-Keung Tang},
title = {Local Color Transfer via Probabilistic Segmentation by Expectation-Maximization},
booktitle = {CVPR},
year = {2005},
pages = {I: 747--754}
}

In this project, we model the image global color statistics using GMM and segment the image according to the estimated GMM which each segment corresponds to a Gaussian distribution.

Synthetic Example:
Segmentation using a synthetic image. (a) Here, the observed color of a pixel may be explained by a mixture of as many as six colors. (b) The resynthesized image generated by compositing L and C obtained upon the convergence of our AO algorithm. (c) Image difference between (a) and (b). (d) The soft labels L of sample pixels. (e) The soft segments.

　Comparisons to other segmentation algorithms:

Input image shown on the left. Results produced by (a) Mean Shift segmentation, (b) k-means clustering with k = 3, (c) Normalized Cuts, (d) Watershed algorithm, (e) JSeg method, (f) statistical region merging, (g), (h), and (i) our AO algorithm. The three segmented regions correspond to the three basic color components underlying the image.
We compare our multiscale results with those by Galun et al. [12] (top) and DDMCMC [35] (bottom). Using the multiscale prior to (26), our method automatically converges to two soft segments, that is, l1 and l2. (a) Input images. (b) l1 images. (c) l2 images. Our multiscale method deals with textures and produces soft segments with appropriate boundary transparency and spatial coherence. The results in [12] and [35] are shown in (d). Note that our method works in RGB and does not work better in intensity images with one gray-scale channel.

Other results:

Segmentation of a satellite image of a hurricane. (a) Input image. (b), (c), and (d) Our segmentation results displayed as li(x,y)ci(x,y): 1 <= i <= 3, (e) Hard segmentation result. (f), (g), and (h) The corresponding li images for (b), (c), and (d). Our approach segments the land from the hurricane. The l image in (h) indicates the cloud density of the hurricane.
Segmentation of a nebula image. (a) Input image, Messier Object M20. (e), (f), and (g) The soft labels li(x,y) corresponding to (b), (c), and (d). Our algorithm segments the red and blue nebulas with fractional boundaries. Note that the bright stars are not smoothed out due to the discontinuity-preserving property of our MRF formulation.

Comparisons with matting approaches:

Comparison with image matting. Although our results are not as good as matting approaches in natural image matting which use a user-specified trimap or other user-supplied hints, our method is fully automatic.

Applications:

Comparison of color transfer using our approach and that in [24]. (a) The target image (the source image is shown above). (b), (c), and (d) Soft color segmentation results. (e) The color-transfer result using our approach in which soft color segmentation is performed before transferring the colors. (f) The color-transfer result without soft color segmentation [24]. (g) The color-transfer result by histogram equalization. The results in (f) and (g) show undesirable mixture of colors of the leaves and the river.
Comparison of color transfer on a natural scene using our approach and that in [24]. (a) An old photograph of a downtown scene captured on an overcast day. (b) The target image captured on a sunny day. (c) Our transfer result. (d) Transfer result generated in [24] where an unacceptable mixture of colors is present. (e) Transfer result generated by histogram equalization in which undesirable saturation is observed.

 

Image deblurring using color transfer with/without soft color segmentation. (a) Source image. (b) Target image. (c) Result using our approach. (d) Result using a single Gaussian model. Note that the colors in result (d) are not preserved.

 

Comparison on image denoising. (a) Nonflashed source and flashed target images. Results and zoom-in views obtained by (b) our approach, (c) by that in [21], and (d) by that in [11]. Our method makes use of the strong spatial relationship between the source and target images given by the soft color segmentation. Therefore, the red shade of the sofa, the bottles, and the stones are not mixed up. Such an
undesirable mixture is observed in the result in (d). The result using joint
bidirectional filter [21] in (c) is still very noisy.

 

Color transfer to a gray-scale image. (a) The source image. (b) The target image. (c) The result by our approach. (d) The result by that in [24].