3D Computer Vision

Overview

We are interested in computer vision and machine learning with a focus on 3D scene understanding, reconstruction etc. In particular, we deal with problems where human body is reconstructed from 2D images and analysed in 3D, registration of point clouds of indoor scenes captured from commodity sensors as well as large outdoor scenes captured from LIDAR scanners. As 3D data is represented in various formats like mesh, pointclouds, volumetric representations to name a few, we also design novel algorithms for making 3D data compatible with deep learning framework. We investigate how complex prior knowledge can be incorporated into computer vision algorithms for making them robust to variations in our complex 3D world. Below links are detailed descriptions of our individual works

HumanBody Capture

Surface Reconstruction

People Involved: Abbhinav Venkat, Chaitanya Patel, Yudhik Agrawal, Avinash Sharma

Recovering a 3D human body shape from a monocular image is an ill-posed problem in computer vision with great practical importance for many applications, including virtual and augmented reality platforms, animation industry, e-commerce domain, etc. [See More]

PeeledHuman: Robust Shape Representation for Textured 3D Human Body Reconstruction

People Involved: Sai Sagar Jinka*, Rohan Chacko* Avinash Sharma P.J. Narayanan

We introduce PeeledHuman - a novel shape representation of the human body that is robust to self-occlusions. PeeledHuman encodes the human body as a set of Peeled Depth and RGB maps in 2D, obtained by performing raytracing on the 3D body model and extending each ray beyond its first intersection. This formulation allows us to handle self-occlusions efficiently compared to other representations. Given a monocular RGB image, we learn these Peeled maps in an end-to-end generative adversarial fashion using our novel framework - PeelGAN. We train PeelGAN using a 3D Chamfer loss and other 2D losses to generate multiple depth values per-pixel and a corresponding RGB field per-vertex in a dual-branch setup. In our simple non-parametric solution, the generated Peeled Depth maps are back-projected to 3D space to obtain a complete textured 3D shape. The corresponding RGB maps provide vertex-level texture details. We compare our method with current parametric and non-parametric methods in 3D reconstruction and find that we achieve state-of-theart-results. We demonstrate the effectiveness of our representation on publicly available BUFF and MonoPerfCap datasets as well as loose clothing data collected by our calibrated multi-Kinect setup. [See More]

Volumetric Reconstruction

People Involved: Dr.Avinash Sharma, Sai Sagar J, Abbhinav Venkat, Chaitanya Patal, Anubhab Sen, Saurabh Rajguru, Neeraj Bhattan, Yudhik Agrawal, Himansh Sheron   

As we are aware of humans often as the subject of photographs, detecting them and analyzing their shape and pose in 3D is significant for applications such as AR/VR, motion capture etc. Our research objective in this work is to recover 3D human body from a single RGB image. [See More]

Classification of 3D data

People Involved: Dr.Avinash Sharma, Sai Sagar J and Raj Manvar

Apart from acquisition of 3D shapes, analyzing these shapes is also an important task. There are several shape analysis tasks like retrieval, classification, dense and sparse correspondence of two shapes, segmentation etc. The core of the 3D shape analysis lies at the 3D shape descriptors. Descriptor construction is usually application dependent. One expects the descriptor to be discriminative, invariant to some transformations or noise and compact i.e. low dimensional. Recently, few deep learning models for shape descriptors have appeared. Nevertheless, many of the existing deep learning works didn’t exploit the fact that 3D shapes are boundary based i.e. much of the information essential for classification is hinged on boundary. Performing convolution in volumetric space hinders the computing efficiency. In the following work, we address these particular drawbacks and construct novel descriptor that accounts for local geometry aware global characterization for rigib 3D shapes. [See More]

3D Action Retrieval

People Involved:Neeraj Battan, Abbhinav Venkat, Avinash Sharma

3D Human Action Indexing and Retrieval is an interesting problem due to the rise of several data-driven applications aimed at analyzing and/or reutilizing 3D human skeletal data, such as data-driven animation, analysis of sports bio-mechanics, human surveillance etc. [See More]

Point Cloud Analysis

People Involved: Dr.Avinash Sharma, Ashish kubade and Nikhilendra

Working on a point cloud involves tackling the inherent challenges as sparseness, lack of order among points, invariance to view ports etc. For LiDAR scans, additionally, we have to handle the scale of the data as a typical outdoor LiDAR scan has point in the order of millions. Breaking such scene, into smaller chunks might would be an initial guess, however, working on such small chunks unlimately results in loss of global semantics of the scene.[See More]