Skyline Segmentation Using Shape-constrained MRFS

Rashmi Tone Vilas (homepage)

MRF energy minimization has been used for image segmentation in a wide range of applications. Standard MRF energy minimization techniques are computationally expensive. Besides, incorporating higher order priors such as shape and parameters related to it is either very complex or computationally expensive or requires prior information such as shape location. Furthermore, semantic understanding is not achieved using pure MRF formulation, i.e. information about the structure of a skyline such as depth cannot be known through output. Standard semantic segmentation methods using geometric context information is restricted to very few geometric classes or the ones which exploit specific “tiered” structure is computationally exponential in number of labels.

Our aim is to extract the detailed structure of a skyline, i.e. individual buildings and their depth. In this case, there is no restriction on the number of labels. The problem is challenging due to numerous reasons such as complex occlusion patterns, large number of labels and intra-region color and texture variations, etc. We propose an approach for segmenting the individual buildings in typical skylines. Our approach is based on a Markov Random Field (MRF) formulation that exploits the fact that such images contain overlapping objects of similar shapes exhibiting a “tiered” structure. Our contributions are the following:

  • We introduce a dataset Skyline-12 consisting of 120 skyline images from the 12 cities all over the world. All the images are manually annotated with addition of meta-data like initial boundaries and seeds.
  • We include an analysis and integration of low-level features such as color, texture and shape very useful for the segmentation of skylines.
  • We propose a fast, accurate and robust method to extract individual buildings of a skyline exploit- ing “tiered” structure of a skylines and incorporating rectangular shape prior in MRF formulation.

For simple shapes such as rectangles, our formulation is significantly faster to optimize than a standard MRF approach, while also being more accurate. We experimentally evaluate various MRF formulations and demonstrate the effectiveness of our approach in segmenting skyline images.

We propose both Interactive and Automatic methods for segmenting skylines. While interctive set- ting gives an accurate output and a fast approach to segment skylines given input seeds from user, automatic setting provides about 25% improvement over state-of-art low level automatic segmentation methods. Our approach can be generalized to different shapes as well as detailed structure of a skyline can be used in many applications such as 3D reconstruction of a skyline from single image.


Year of completion:  January 2015 
 Advisor : Prof. C. V. Jawahar

Related Publications





    Recent Updates

    8th Summer School on Artificial Intelligence With focus on Computer Vision and Machine Learning
    IIIT-H is happy to announce the commencement of the 8th episode of its Summer School Series, an exciting opportunity...
    Jul, 2024

    The domain of 3D Vision deals with acquisition and analysis of 3-dimensional objects/scenes with wider applications in...
    Apr, 2023

    IT Minister Visit IIIT-H
    As part of the silver jubilee celebrations at IIIT-H, IT Minister KT Rama Rao visited the campus for a fireside chat...
    Jan, 2023