Find me a sky : a data-driven method for color-consistent sky search & replacement

Abstract

Replacing overexposed or dull skies in outdoor photographs is a desirable photo manipulation. It is often necessary to color correct the foreground after replacement to make it consistent with the new sky. Methods have been proposed to automate the process of sky replacement and color correction. However, many times a color correction is unwanted by the artist or may produce unrealistic results. We propose a data-driven approach to sky-replacement that avoids color correction by finding a diverse set of skies that are consistent in color and natural illumination with the query image foreground. Our database consists of ∼1200 natural images spanning many outdoor categories. Given a query image, we retrieve the most consistent images from the database according to L2 similarity in feature space and produce candidate composites. The candidates are re-ranked based on realism and diversity. We used pre-trained CNN features and a rich set of hand-crafted features that encode color statistics, structural layout, and natural illumination statistics, but observed color statistics to be the most effective for this task. We share our findings on feature selection and show qualitative results and a user-study based evaluation to show the effectiveness of the proposed method.

 

#

Major Contributions

  • A novel approach for aesthetic enhancement of an image with a dull background by replacing it's sky using a data-driven method for sky search and replacement.
  • Curated dataset of 1246 images with interesting and appealing skies.
  • Identified relevant features helpful in color consistent sky search.
  • Quantative and qualitative proof for the hypothesis that images with matching foreground color properties can have interchangeble backgrounds.

Related Publications

  • Saumya Rawat, Siddhartha Gairola, Rajvi Shah, and P J Narayanan Find me a sky : a data-driven method for color-consistent sky search & replacement, International Conference on Multimedia Modeling (MMM), February 5-7, 2018. [ Paper ] [ Slides ]

Dataset:

The dataset consists of 1246 images with corresponding binary masks indicating sky and non-sky regions in the folders image and mask respectively. It has been curated from 415 Flickr images with diverse skies (collected by [1]) and 831 outdoor images curated from the ADE20K Dataset[2]. ADE20K dataset consists of ∼ 22K images with 150 semantic categories like sky, road, grass. The images with sky category were first filtered to a set of ∼6K useful images for which the sky region made > 40% of the total image. These images were manually rated between 1 to 5 for aesthetic appeal of the skies by two human raters and only the images with average scores higher than 3 were added to the final dataset
Download : Dataset

Bibtex

If you use this work or dataset, please cite :
Rawat S., Gairola S., Shah R., Narayanan P.J. (2018) Find Me a Sky: A Data-Driven Method for Color-Consistent Sky Search and Replacement. In: Schoeffmann K. et al. (eds) MultiMedia Modeling. MMM 2018. Lecture Notes in Computer Science, vol 10704. Springer, Cham

Associated People

  • Saumya Rawat
  • Siddhartha Gairola
  • Rajvi Shah
  • P J Narayanan

References:

  1. Y.-H. Tsai, X. Shen, Z. Lin, K. Sunkavalli, and M.-H. Yang Sky is not the limit: Semantic aware sky replacement. ACM Trans. Graph, 35(4), 2016.
  2. Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba Scene parsing through ade20k dataset, In Proc. IEEE CVPR, 2017.