Geometry-aware methods for efficient and accurate 3D reconstruction

Rajvi Shah

Abstract

Advancements in 3D sensing and reconstruction has made a huge leap for modeling large-scale environments from monocular images using structure from motion (SfM) and simultaneous localization and mapping (SLAM) algorithms. SfM and SLAM based 3D reconstruction has applications for digital archival and modeling of real-world objects and environments, visual localization for geo-tagging and information retrieval, and mapping and navigation for robotic and autonomous driving applications. In this thesis, we address problems in the area of large-scale structure from motion (SfM) for 3D reconstruction and localization. We introduce new methods for improving efficiency and accuracy of state-of-the-art pipeline for structure from motion. Large-scale SfM pipeline deals with large unorganized collections of images pertaining to a particular geographical site. These image collections are formed by either retrieving relevant images using textual queries from the Internet, or can be captured for the specific purpose of 3D modeling, mapping, and navigation. Internet image collections tend to be more noisy and present more challenges for reconstruction as compared to datasets captured with specific intention to reconstruct. In this thesis, we propose methods that help with organizing these large, unstructured, and noisy images into a structure that is useful for SfM methods, a match-graph (or a view-graph). We first propose a geometry-aware two stage approach for pairwise image matching that is both more efficient and superior in quality of correspondences. We then extend this idea to SfM pipeline and present an iterative multistage framework for coarse to fine 3D reconstruction. Finally, we suggest that a key to solving many of the reconstruction problems is to address the problem of filtering and improving the view-graph in a way that is specific to the underlying problem. To this effect, we propose a unified framework for view-graph selection and show its application to achieve multiple reconstruction objectives.

Year of completion:	December 2020
Advisor :	P J Narayanan

Related Publications

Rajvi Shah, Visesh Chari and P.J. Narayanan - View-graph Selection Framework for SfM European Conference on Computer Vision (ECCV), 2018, Munich, Germany [PDF]
Saumya Rawat, Siddhartha Gairola, Rajvi Shah and P.J. Narayanan - Find Me a Sky: A Data-Driven Method for Color-Consistent Sky Search and Replacement International Conference on Multimedia Modeling 2018: 216-228 [PDF]
Ishit Mehta, Parikshit Sakurikar, Rajvi Shah, P J Narayanan - SynCam: Capturing sub-frame synchronous media using smartphones IEEE International Conference on Multimedia and Expo (ICME-2017 ).[PDF]
Aditya Singh, Saurabh Saini, Rajvi Shah and P. J. Narayanan - Learning to hash-tag videos with Tag2Vec Proceedings of the Tenth Indian Conference on Computer Vision, Graphics and Image Processing. ACM, 2016. [PDF]
Aditya Singh, Saurabh Saini, Rajvi Shah and P J Narayanan - From Traditional to Modern : Domain Adaptation for Action Classification in Short Social Video Clips 38th German Conference on Pattern Recognition (GCPR 2016) Hannover, Germany, September 12-15 2016. [PDF]
Rajvi Shah, Vanshika Srivastava, P.J. Narayanan - Geometry-aware Feature Matching for Structure from Motion Applications Proceedings of the IEEE Winter Conferenc on Applications of Computer Vision, 06-09 Jan 2015, Waikoloa Beach, USA. [PDF]
Rajvi Shah, Aditya Deshpande, P.J. Narayanan - Multistage SFM: Revisiting Incremental Structure from Motion Proceedings of the International Conference on 3D Vision,08-11 Dec 2014, Tokyo, Japan.[PDF]
Rajvi Shah and P. J. Narayanan - Interactive video manipulation using object trajectories and scene backgrounds IEEE Transactions on Circuits and Systems for Video Technology 23.9 (2013): 1565-1576. [PDF]
Rajvi Shah and P.J. Narayanan - Trajectory based Video Object Manipulation Proceedings of IEEE International Conference on Multimedia and Expo (ICME 2011),11-15 July, 2011, Barcelona, Spain. [PDF]
Rajvi Shah, P. J. Narayanan and Kishore Kothapalli - GPU-Accelerated Genetic Algorithms Proceedings of The 3rd Workshop on Parallel Architectures for Bio-inspired Algorithms(WPABA) in conjunction with Parallel Architectures for Compilation Techniques (PACT'10),11-15 Sep. 2010,Vienna, Austria. [PDF]