A Deep Learning Paradigm for Fingerprint Recognition: Harnessing U-Net Architecture for Fingerprint Enhancement and Representation Learning

Ekta Gavas

Abstract

Biometric technology has long relied on fingerprint recognition as a trusted means of identity verification. While fingerprint enhancement shares similarities with general image denoising tasks, the unique properties of biometric data make it crucial to treat fingerprints differently from typical realworld images. Deep learning methods have the capacity to capture and improve complex patterns and subtle details in fingerprint data, offering a comprehensive solution. This thesis proposes a deep learning-based method, specifically leveraging the U-Net architecture to achieve superior fingerprint enhancement. At its core, the approach employs a multi-task deep network with domain knowledge from minutia and orientation fields for the enhancement of rolled and plain fingerprint images. We investigate different configurations of the U-Net architecture, examine the effects of various architectural elements, and present extensive experimental evidence to support the effectiveness of our proposed approach. Subsequently, we explore a self-supervised learning paradigm with fingerprint biometrics for robust feature representation learning. In this direction, we introduce a pre-training technique by utilizing the learned information from the enhancement task. We adopt the enhancement pre-trained encoder for learning fixed-length fingerprint embeddings. We evaluate the performance of the learned embeddings in the verification task compared to standard self-supervised techniques without the explicit need for enhanced fingerprints. By merging the capabilities of deep learning with the specificity of fingerprint features, the proposed paradigm offers significant improvements in the robustness and accuracy of fingerprint verification. Experimental evaluations on standard benchmarks such as the NIST and FVC datasets demonstrate the efficacy of this approach. We present compelling evidence that our methodology not only improves the quality of fingerprint images but also facilitates a more effective embedding extraction, ultimately leading to enhanced recognition performance. This work highlights the promising role that deep learning, tailored to biometrics, can play in advancing the field of fingerprint recognition. In addition, it opens up avenues for future research in biometrics, particularly in optimizing computational efficiency and in tackling challenges related to partial and distorted fingerprints.

Year of completion:  February 2024
 Advisor : Anoop M Namboodiri

Related Publications

    Downloads

    thesis

     

    Physical Adversarial Attacks on Face Presentation Attack Detection Systems

    Sai Amrit Patnaik

    Abstract

    In the realm of biometric security, face recognition technology plays an increasingly pivotal role. However, as its adoption grows, so does the need to safeguard it against adversarial attacks. Attacks involve presenting images of a person printed on a medium or displayed on a screen. Detection of such attacks relies on identifying artifacts introduced in the image during the printing or display and capture process. Adversarial Attacks try to deceive the learning strategy of a recognition system using slight modifications to the captured image. Evaluating the risk level of adversarial images is essential for safely deploying face authentication models in the real world. Among these, physical adversarial attacks present a particularly insidious threat to face antispoofing systems. Popular approaches for physical-world attacks, such as print or replay attacks, suffer from some limitations, like including physical and geometrical artifacts. The presence of a physical process (printing and capture) between the image generation and the PAD module makes traditional adversarial attacks non-viable. Recently adversarial attacks have gained attraction, which try to digitally deceive the learning strategy of a recognition system using slight modifications to the captured image. While most previous research assumes that the adversarial image could be digitally fed into the authentication systems, this is not always the case for systems deployed in the real world. This thesis delves into the intriguing domain of physical adversarial attacks on face antispoofing systems, aiming to expose their vulnerabilities and implications. Our research unveils novel methodologies using white box and black box approaches to craft adversarial inputs capable of deceiving even the most robust face antispoofing systems. Unlike traditional adversarial attacks that manipulate digital inputs, our approach operates in the physical domain, where printed images and replayed videos are utilized to mimic real-world presentation attacks. By dissecting and understanding the vulnerabilities inherent in face antispoofing systems, we can develop more resilient defenses, contributing to the security of biometric authentication in an increasingly interconnected world. This thesis not only highlights the pressing need to address these vulnerabilities but also motivates towards a pioneering approach by exploring simple yet effective attack strategy to advancing the state of the art in face antispoofing security.

    Year of completion:  February 2024
     Advisor : Anoop M Namboodiri

    Related Publications

      Downloads

      thesis

      Text-based Video Question Answering

      Soumya Shamarao Jahagirdar

      Abstract

      Think of a situation where you put yourself in the shoes of a visually impaired person who wants to buy an item from a store or a person who is sitting in their house and watching the news on the television and wants to know about the content of the news being broadcast. Motivated by many more such situations where creating systems capable of understanding and reasoning over textual content in the videos, in this thesis, we tackle the novel problem of text-based video question answering. Vision and Language are broadly regarded as cornerstones of intelligence. Though each of these has different aims – language has the purpose of communication, and transmission of information, and vision has the purpose of constructing mental representations of the scene around us to navigate and interact with objects. When we study both of these fields jointly, it can result in applications, tasks, and methods that, when combined go beyond the scope compared to when they are used individually. This inter-dependency is being studied as a newly emerging area of a study named “multi-modal understanding”. Many tasks such as image captioning, visual question answering, video question answering, text-video retrieval, and more fall under the category of multi-modal understanding and reasoning tasks. To have a system that can reason over both text-based information and temporal-based information, we propose a new task. The first portion of this thesis focuses on the formulation of the text-based VideoQA task, by first analyzing the current datasets and works and thereby arriving at the need for text-based VideoQA. To this end, we propose the NewsVideoQA dataset where the question-answer pairs are framed on the text present in the news videos. As this is a new task proposed, we experiment with existing methods such as text-only models, single-image scene text-based models, and video question-answering models. As these baseline methods were not originally designed for the task of video question-answering using text in the videos, the need for a video question-answering model that can take the text in the videos into account to obtain answers became the need. To this end, we repurpose the existing VideoQA model to incorporate OCR tokens namely – OCR-aware SINGULARITY, a video question-answering framework that learns joint representations of videos and OCR tokens at the pretraining stage and also uses the OCR tokens at the finetuning stage. In this second portion of the thesis, we look into the M4-ViteVQA dataset which aims to solve the same task of text-based video question-answering but the videos belong to multiple categories such as shopping, traveling, vlogging, gaming, and so on. We perform a data exploratory analysis where we analyze both NewsVideoQA and M4-ViteVQA on several aspects that look for limitations in these datasets. Through the data exploratory experiment, we show that most of the questions in both datasets have questions that can be answered just by reading the text present in the videos. We also observe that most of the questions can be answered using a single to few frames in the videos. We perform an exhaustive analysis on a text-only model: BERT-QA which obtains comparable results to the multimodal methods. We also perform cross-domain experiments to check if training followed by finetuning on two different categories of videos helps the target dataset. In the end, we also provide some insights into creating a dataset and how certain types of annotations can help the community come up with better datasets in the future. We hope this work motivates future research on text-based video question-answering in multiple video categories. Furthermore, the pretraining strategies and combined representation learning from these videos and the multiple modalities that videos provide us will help create scalable systems and drive future research towards better datasets and creative solutions.

      Year of completion:  March 2024
       Advisor : C V Jawahar

      Related Publications

        Downloads

        thesis

        Analytic and Neural Approaches for Complex Light Transport

        Ishaan Shah

        Abstract

        The goal of rendering is to produce a photorealistic image of the given 3D scene description. Physically based rendering simulates the physics of the light as it travels and interacts with objects in the scene before finally reaching the camera sensor. Monte Carlo methods have been the go-to approach for physically based rendering. They are general and robust but introduce noise and are computationally expensive. Recent advancements in hardware, algorithms, and denoising techniques have enabled real-time applications of Monte Carlo methods. However, complex scenes still demand high sample counts. In this thesis, we explore and present the utilization of analytic and neural approaches for physically based rendering. Analytic methods offer noise-free renderings but are less general and may introduce bias. In recent years, neural-based approaches have gained traction, offering a balance between generality and computational efficiency. We compare and contrast the traditional Monte Carlo-based methods and emerging analytic and neural network-based methods. We then propose analytic and neural solutions to two challenging cases: direct lighting with many area lights and efficient rendering of glinty appearances on specular normal-mapped surfaces. Direct lighting from many area light sources is challenging due to variance from both choosing an important light and then a point on it. Existing methods weigh the contribution of all lights by estimating their effect on the shading point. We propose to extend one such method by using analytic methods to improve the estimation of the light’s contribution. This enhancement accelerates the convergence of the algorithm, making it more efficient for scenes with many dynamic lights. The second case deals with the challenge of rendering glinty appearances on normal mapped specular surfaces efficiently. Traditional Monte Carlo methods struggle with this task due to the rapidly changing spatial characteristics of microstructures. Our solution introduces a novel method supporting spatially varying roughness based on a neural histogram, offering both memory and compute efficiency. Additionally, full direct illumination integration is computed analytically for all light directions with minimal computational effort, resulting in improved quality compared to previous approaches. Through comprehensive analysis and experimentation, this thesis contributes to the advancement of rendering techniques, shedding light on the trade-offs between different methods and providing insights into their practical applications for achieving photorealistic rendering.

        Year of completion:  March 2024
         Advisor : P J Narayanan

        Related Publications

          Downloads

          thesis

          Learning Emotions and Mental States in Movie Scenes

          Dhruv Srivastava

          Abstract

          In this thesis, we delve into the analysis of movie narratives, with a specific focus on understanding the emotions and mental states of characters within a scene. Our approach involves predicting a diverse range of emotions for individual movie scenes and each character within those scenes. To achieve this, we introduce EmoTx, a novel multimodal Transformer-based architecture that integrates video data, multiple characters, and dialogues for making comprehensive predictions. Leveraging annotations from the MovieGraphs dataset, our model is tailored to predict both classic emotions (e.g., happiness, anger) and nuanced mental states (e.g., honesty, helpfulness). Our experiments concentrate on evaluating performance across the ten most common and twenty-five most common emotional labels, along with a mapping that clusters 181 labels into 26 categories. Through systematic ablation studies and a comparative analysis against established emotion recognition methods, we demonstrate the effectiveness of EmoTx in capturing the intricacies of emotional and mental states in movie contexts. Additionally, our investigation into EmoTx’s attention mechanisms provides valuable insights. We observe that when characters express strong emotions, EmoTx focuses on character-related elements, while for other mental states, it relies more on video and dialogue cues. This nuanced understanding enhances the interpretability and contextual relevance of EmoTx in the domain of movie story analysis. The findings presented in this thesis contribute to advancing our comprehension of character emotions and mental states in cinematic narratives.

          Year of completion:  April 2024
           Advisor : Makarand Tapaswi

          Related Publications

            Downloads

            thesis

            More Articles …

            1. Revolutionizing TV Show Experience: Using Recaps for Multimodal Story Summarization
            2. Revisiting Synthetic Face Generation for Multimedia Applications
            3. Advancing Domain Generalization through Cross-Domain Class-Contrastive Learning and Addressing Data Imbalances
            4. Real-Time Video Processing for Dynamic Content Creation