Estimating 3D Human Pose, Shape, and Correspondences from Monocular Input

Amogh Tiwari

Abstract

In recent years, advances in computer vision have opened up multiple applications in virtual reality, healthcare, robotics, and many other domains. One crucial problem domain in computer vision, which has been a key research focus lately, is estimating the 3D human pose, shape, and correspondences from monocular input. This problem domain has applications in various industries like fashion, entertainment, healthcare, etc. However, it is also highly challenging due to various reasons like large variations in the pose, shape, and appearance of humans and clothing details, external and self-occlusions, challenges with ensuring consistency etc. As part of this thesis, we tackle two key problems related to 3D human pose, shape, and correspondence estimation. First, we focus on the problem of temporally consistent 3D human pose and shape estimation from monocular videos. Next, we focus on dense correspondence estimation across images of different (or the same) humans. We show that despite receiving a lot of research attention lately, existing methods for these tasks still perform sub-optimally in many challenging scenarios and have significant scope for improvement. We aim to overcome some of the limitations of existing methods and advance state-of-the-art (SOTA) solutions to these problems. First, we propose a novel method for temporally consistent 3D human pose and shape estimation from a monocular video. Instead of using the traditionally used, generic ResNet-like features, our method uses a body-aware feature representation and an independent per-frame pose and camera initialization over a temporal window followed by a novel spatio-temporal feature aggregation by using a combination of self-similarity and self-attention over the body-aware features and the per-frame initialization. Together, they yield enhanced spatio-temporal context for every frame by considering the remaining past and future frames. These features are used to predict the pose and shape parameters of the human body model, which are further refined using an LSTM. Next, we expand our focus to the task of dense correspondence estimation between humans, which requires understanding the relations between different body regions (represented using dense correspondences), including the clothing details, of the same or different human(s). We present Continuous Volumetric Embeddings (ConVol-E), a novel robust representation for dense correspondence-matching across RGB images of different human subjects in arbitrary poses and appearances under non-rigid deformation scenarios. Unlike existing representations, ConVol-E captures the deviation from the underlying parametric body model by choosing suitable anchor/key points on the underlying parametric body surface and then representing any point in the volume based on its Euclidean relationship with the anchor points. This allows us to represent any arbitrary point around the parametric body (clothing details, hair, etc.) by an embedding vector. Subsequently, given a monocular RGB image of a person, we learn to predict per-pixel ConVol-E embedding, which carries a similar meaning across different subjects and is invariant to pose and appearance, thereby acting as a descriptor to establish robust, dense correspondences across different images of humans. We thoroughly evaluate our methods on publicly available benchmark datasets and show that our methods outperform existing SOTA. Finally, we provide a summary of our contributions and discuss the potential future research directions in this problem domain. We believe that this thesis improves the research landscape for the domain of the human body, pose, shape, and correspondence estimation and helps accelerate progress in this direction.

Year of completion:  June 2024
 Advisor : Avinash Sharma

Related Publications

    Downloads

    thesis

    Quality Beyond Perception: Introducing Image Quality Metrics for Enhanced Facial and Fingerprint Recognition

    Prateek Jaiswal

    Abstract

    Assessing the quality of biometric images is key to making recognition technologies more accurate and reliable. Our research began with fingerprint recognition systems and later expanded to facial recognition systems, underscoring the importance of image quality in both areas. For fingerprint recognition, image quality is vital for accuracy. We developed the Fingerprint RecognitionBased Quality (FRBQ) metric, which improves on the limitations of the NFIQ2 model. FRBQ leverages deep learning algorithms in a weakly supervised setting, using matching scores from DeepPrint, a FixedLength Fingerprint Representation Model. Each score is labeled to reflect the robustness of fingerprint image matches, providing a comprehensive metric that captures diverse perspectives on image quality. Comparative analysis with NFIQ2 reveals that FRBQ correlates more strongly with recognition scores and performs better in evaluating challenging fingerprint images. Tested with the FVC 2004 dataset, FRBQ has proven effective in assessing fingerprint image quality. After our success with fingerprint recognition, we turned to facial recognition systems. In facial recognition, image quality involves more than just perceptual aspects; it includes features that convey identity information. Existing datasets consider factors like illumination and pose, which enhance robustness and performance. However, age variations and emotional expressions can still pose challenges. To tackle these, we introduced the Unified Tri-Feature Quality Metric (U3FQ). This framework combines age variance, facial expression similarity, and congruence scores from advanced recognition models like VGG-Face, ArcFace, FaceNet, and OpenFace. U3FQ uses a Regression Network model specifically designed for facial image quality assessment. We compared U3FQ to general image quality assessment techniques like BRISQUE, BLINDS-II, and RankIQA, as well as specialized facial image quality methodologies like PFE, SER-FIQA, and SDD-FIQA. Our results, supported by analyses such as DET plots, expression match heat maps, and EVRC curves, show U3FQ’s effectiveness. Our study highlights the transformative potential of artificial intelligence in biometrics, capturing critical details that traditional methods might miss. By providing precise quality assessments, we emphasize its role in advancing both fingerprint and facial recognition systems. This work sets the stage for further research and innovation in biometric analysis, underlining the importance of image quality in improving recognition technologies.

    Year of completion:  July 2024
     Advisor : Anoop M Namboodiri

    Related Publications

      Downloads

      thesis

      Towards Label Free Few Shot Learning : How Far Can We Go?

      Aditya Bharti

      Abstract

      Deep learning frameworks have consistently pushed the state-of-the-art limit across various problem domains such as computer vision, and natural language processing applications. Such performance improvements have only been made possible by the increasing availability of labeled data and computational resources, which makes applying such systems to low data regimes extremely challenging. Computationally simple systems which are effective with limited data are essential to the continued proliferation of DNNs to more problem spaces. In addition, generalizing from limited data is a crucial step toward more human-like machine intelligence. Reducing the label requirement is an active and worthwhile area of research since getting large amounts of high-quality annotated data is labor intensive and often impossible, depending on the domain. There are various approaches to this: artificial generation of extra labeled data, using existing information (other than labels) as supervisory signals for training, and designing pipeline that specifically learn using only a few samples. We focus our efforts on that last class of channels which aims to learn from limited labeled data, also known as Few Shot Learning. Few-Shot learning systems aim to generalize to novel classes given very few novel examples, usually one to five. Conventional few-shot pipelines use labeled data from the training set to guide training, then aim to generalize to the novel classes which have limited samples. However, such approaches only shift the label requirement from the novel to the training dataset. In low data regimes, where there is a dearth of labeled data, it may not be possible to get enough training samples. Our work aims to alleviate this label requirement by using no labels during training. We examine how much performance is achievable using extremely simple pipelines overall. Our contributions are hence twofold. (i) We present a more challenging label-free few-shot learning setup and examine how much performance can be squeezed out of a system without labels. (ii) We propose a computationally and conceptually simple pipeline to tackle this setting. We tackle both the compute and data requirements by leveraging self-supervision for training and image similarity for testing.

      Year of completion:  January 2024
       Advisor : C V Jawahar,Vineeth Balasubramanian

      Related Publications

        Downloads

        thesis

         

        A Deep Learning Paradigm for Fingerprint Recognition: Harnessing U-Net Architecture for Fingerprint Enhancement and Representation Learning

        Ekta Gavas

        Abstract

        Biometric technology has long relied on fingerprint recognition as a trusted means of identity verification. While fingerprint enhancement shares similarities with general image denoising tasks, the unique properties of biometric data make it crucial to treat fingerprints differently from typical realworld images. Deep learning methods have the capacity to capture and improve complex patterns and subtle details in fingerprint data, offering a comprehensive solution. This thesis proposes a deep learning-based method, specifically leveraging the U-Net architecture to achieve superior fingerprint enhancement. At its core, the approach employs a multi-task deep network with domain knowledge from minutia and orientation fields for the enhancement of rolled and plain fingerprint images. We investigate different configurations of the U-Net architecture, examine the effects of various architectural elements, and present extensive experimental evidence to support the effectiveness of our proposed approach. Subsequently, we explore a self-supervised learning paradigm with fingerprint biometrics for robust feature representation learning. In this direction, we introduce a pre-training technique by utilizing the learned information from the enhancement task. We adopt the enhancement pre-trained encoder for learning fixed-length fingerprint embeddings. We evaluate the performance of the learned embeddings in the verification task compared to standard self-supervised techniques without the explicit need for enhanced fingerprints. By merging the capabilities of deep learning with the specificity of fingerprint features, the proposed paradigm offers significant improvements in the robustness and accuracy of fingerprint verification. Experimental evaluations on standard benchmarks such as the NIST and FVC datasets demonstrate the efficacy of this approach. We present compelling evidence that our methodology not only improves the quality of fingerprint images but also facilitates a more effective embedding extraction, ultimately leading to enhanced recognition performance. This work highlights the promising role that deep learning, tailored to biometrics, can play in advancing the field of fingerprint recognition. In addition, it opens up avenues for future research in biometrics, particularly in optimizing computational efficiency and in tackling challenges related to partial and distorted fingerprints.

        Year of completion:  February 2024
         Advisor : Anoop M Namboodiri

        Related Publications

          Downloads

          thesis

           

          Physical Adversarial Attacks on Face Presentation Attack Detection Systems

          Sai Amrit Patnaik

          Abstract

          In the realm of biometric security, face recognition technology plays an increasingly pivotal role. However, as its adoption grows, so does the need to safeguard it against adversarial attacks. Attacks involve presenting images of a person printed on a medium or displayed on a screen. Detection of such attacks relies on identifying artifacts introduced in the image during the printing or display and capture process. Adversarial Attacks try to deceive the learning strategy of a recognition system using slight modifications to the captured image. Evaluating the risk level of adversarial images is essential for safely deploying face authentication models in the real world. Among these, physical adversarial attacks present a particularly insidious threat to face antispoofing systems. Popular approaches for physical-world attacks, such as print or replay attacks, suffer from some limitations, like including physical and geometrical artifacts. The presence of a physical process (printing and capture) between the image generation and the PAD module makes traditional adversarial attacks non-viable. Recently adversarial attacks have gained attraction, which try to digitally deceive the learning strategy of a recognition system using slight modifications to the captured image. While most previous research assumes that the adversarial image could be digitally fed into the authentication systems, this is not always the case for systems deployed in the real world. This thesis delves into the intriguing domain of physical adversarial attacks on face antispoofing systems, aiming to expose their vulnerabilities and implications. Our research unveils novel methodologies using white box and black box approaches to craft adversarial inputs capable of deceiving even the most robust face antispoofing systems. Unlike traditional adversarial attacks that manipulate digital inputs, our approach operates in the physical domain, where printed images and replayed videos are utilized to mimic real-world presentation attacks. By dissecting and understanding the vulnerabilities inherent in face antispoofing systems, we can develop more resilient defenses, contributing to the security of biometric authentication in an increasingly interconnected world. This thesis not only highlights the pressing need to address these vulnerabilities but also motivates towards a pioneering approach by exploring simple yet effective attack strategy to advancing the state of the art in face antispoofing security.

          Year of completion:  February 2024
           Advisor : Anoop M Namboodiri

          Related Publications

            Downloads

            thesis

            More Articles …

            1. Text-based Video Question Answering
            2. Analytic and Neural Approaches for Complex Light Transport
            3. Learning Emotions and Mental States in Movie Scenes
            4. Revolutionizing TV Show Experience: Using Recaps for Multimodal Story Summarization