Parallelizing Modules to Optimize and Enhance Fingerprint Recognition Systems

Saraansh Tandon

Abstract

Today human identification is a practice followed quite frequently in contexts ranging from forensic labs to high security organisations to almost every domestic household. One of the oldest and most accurate ways to establish the identity of an individual is through fingerprints. It is so because, biologically no two people (not even identical twins) have the same fingerprints. Hence it comes as no surprise that fingerprint biometrics has been a very active domain of research for a long time. Over the years, especially with the emergence of the deep learning era, the computational modules of a fingerprint recognition system have become bigger and bigger. Simultaneously, the actual hardware deploying fingerprint recognition systems has become more and more user friendly. These developments provide a contrast as one tries to incorporate the highly accurate but also extremely heavy modules of a fingerprint recognition system in the palm of our hands. In this thesis, we will perform a couple of studies which explore ways to optimize fingerprint recognition systems. This is done by trying to parallelize modules and eliminate semantically redundant computations. Along the way we will also obtain better performances as compared to the academic state of the art. Typical fingerprint recognition systems are comprised of a spoof detection module and a subsequent matching module, running one after the other. In the first study, we posit that both spoof detection and fingerprint matching are correlated tasks. Therefore, rather than performing the two tasks separately, we propose a joint model for spoof detection and matching to simultaneously perform both tasks without compromising the accuracy of either task. In practice, this reduces the time and memory requirements of the fingerprint recognition system by 50% and 40%, respectively. On the other hand, fingerprint matching task itself can be solved in multiple ways. Fingerprint feature extraction is a core step in the fingerprint matching task and it can be solved by using a global as well as a local approach. State of the art global approaches use huge deep learning models to process the full fingerprint image at once which makes the corresponding approach memory intensive. Whereas local approaches involve minutiae based patch extraction and multiple feature extraction steps which make the corresponding approach time-intensive. But both these approaches provide useful and sometimes exclusive insights for solving the problem. Hence using both kinds of approaches together for extracting fingerprint representations is semantically useful but quite inefficient. Hence, in the second study, we use a convolutional transformer based approach with an in-built minutiae extractor. This study provides an efficient solution for feature extraction that parallely provides a global as well as a local represen tation of a fingerprint. This not only provides a replacement for using a heavy combination of two independent local and global approaches but also provides a further 54.41% speedup over the local approach and a 57.93% conservation in memory as compared to the global approach. Both of these studies work towards the common goal of optimizing fingerprint recognition systems and hence provide a significant advantage in the context of such systems deployed on resource constrained devices like smartphones.

Year of completion:  June 2022
 Advisor : Anoop M Namboodiri

Related Publications

    Downloads

    thesis

    Scene Text Recognition for Indian Languages

    Sanjana Gunna

    Abstract

    Text recognition has been an active field in computer vision even before the beginning of the deep learning era. Due to the varied applications of recognition models, the research area has been classified into diverse categories based on the domain of the data used. Optical character recognition (OCR) is focused on scanned documents, whereas images with natural scenes and much complex backgrounds fall into the category of scene text recognition. Scene text recognition has become an exciting area of research due to the complexities and difficulties such as complex backgrounds, improper illumination, distorted images with noise, inconsistent usage of fonts and font sizes that are not usually horizontally aligned. Such cases make the task of scene text recognition more complicated and challenging. In recent years, we have observed the rise of deep learning. Subsequently, there has been an incremental growth in the recognition algorithms and datasets available for training and testing purposes. This surge has caused the performance of recognizing text in natural scenes to rise above the baseline models that were previously trained using hand-crafted features. Latin texts were the center of attention in most of these works and did not profoundly investigate the field of scene text recognition for non-Latin languages. Upon scrutiny, we observe that the performance of the current best recognition models has reached above 90% over scene text benchmark datasets. However, these recognition models do not perform as well on non-Latin languages as they did on Latin (or English) datasets. This striking difference in the performances over different languages is a rising concern among the researchers focusing on lowresource languages, and it is indeed the motivation behind our work. Scene text recognition in low-resource non-Latin languages is difficult and challenging due to the inherent complex scripts, multiple writing systems, various fonts and orientations. Despite such differences, we can also achieve Latin (English) text-like performance for low-resource non-Latin languages. In this thesis, we look at all the parameters involved in the process of text recognition and determine the importance of those parameters through thorough experiments. We use synthetic data for controlled experiments where we test the parameters as mentioned earlier in an isolated fashion to effectively identify the catalysts of text recognition. We analyse the complexity of the scripts via these synthetic data experiments. We present the results of our experiments on two baseline models, CRNN and STAR-Net models, on available datasets to ensure generalisability. In addition to this, we also propose an error correction module for correcting the labels by utilizing the training data of real test datasets. To further improve the results on real test datasets, we propose transfer learning from English to exploit the abundant data that is available for learning. We show that the transfer from English is not feasible, and it actually lowers the performance of the individual language models. Due to the failure of English transfer experiments, we shift our focus onto just the Indian languages and examine the characteristics of each language via character n-gram plots, visual features like vowel signs, conjunct characters and other word statistics. They also share a resemblance to each other concerning certain other factors. We then propose to apply transfer learning across languages to enhance the performance of the language models. We depict the improvement on real datasets because of the transfers among Indian languages that are visually closer or sometimes better than the individual models. The transfers among languages prove to be much more profitable than transfers from English. We comprehend the significance of the variety and number of fonts during data generation via synthetic data experiments on English test datasets. Synthetic data embodies various fonts to ensure diversity of data to create robust recognition systems. In order to strengthen data diversity, we incorporate over 500 Hindi fonts (including Unicode and non-Unicode fonts) into the synthetic data for improved performance on Hindi real test datasets. We also manifest the process to utilize and incorporate nonUnicode fonts of Indian languages into the training process error-free. In addition to these fonts, we make specific changes to encompass an augmentation pipeline that adds to the diversity of data. We utilize more than nine augmentation techniques to boost the performance of Hindi STR systems. We achieved significant improvements over previous works with our evaluations over natural settings. Through our experiments, we set new benchmark accuracies for STR on Hindi, Telugu, and Malayalam languages from the IIIT-ILST dataset by gaining 6%, 5%, and 2% gains in Word Recognition Rates (WRRs) compared to previous works. Similarly, we also achieved a 23% improvement in WRR for the Bangla language from the MLT-17 dataset. We further improve this result by incorporating the error correction module as mentioned above into the training pipeline. In addition to this, we also released two STR datasets for Gujarati and Tamil datasets, containing 440 scene images, further divided into 500 Gujarati and 2535 Tamil cropped word images. We report a 5% and 3% gain in WRR over our baseline models for Gujarati and Tamil, respectively. We also establish benchmark results for MLT-19 and Bangla datasets with 8% and 4% improvements in WRRs over baselines. Further enriching the synthetic dataset with non-Unicode fonts and multiple augmentations helps us achieve a remarkable Word Recognition Rate gain of over 33% on the IIIT-ILST Hindi dataset. Additionally, we implement a lexicon-based transcription approach that utilizes a dynamic lexicon for each image while testing and presenting the results for languages mentioned above. Keywords – Scene text recognition · transfer learning · photo OCR · multilingual OCR · Indian Languages · Indic OCR · Synthetic Data · Data Diversity

    Year of completion:  June 2022
     Advisor : C V Jawahar

    Related Publications

      Downloads

      thesis

      Summarizing Day Long Egocentric Videos

      Anuj Rathore

      Abstract

      The popularity of egocentric cameras and their always-on nature has lead to the abundance of daylong first-person videos. Because of the extreme shake and highly redundant nature, these videos are difficult to watch from beginning to end and often require summarization tools for their efficient consumption. However, traditional summarization techniques developed for static surveillance videos, or highly curated sports videos and movies are, either, not suitable or simply do not scale for such hours long videos in the wild. On the other hand, specialized summarization techniques developed for egocentric videos limit their focus to important objects and people. In this work, we present a novel unsupervised reinforcement learning technique to generate video summaries from day long egocentric videos. Our approach can be adapted to generate summaries of various lengths making it possible to view even one minute summaries of one’s entire day. The technique can also be adapted to various rewards, such as distinctiveness, indicativeness of the summary. When using the facial saliency-based reward, we show that our approach generates summaries focusing on social interactions, similar to the current state of the art. Quantitative comparison on the benchmark Disney dataset shows that our method achieves significant improvement in Relaxed F-Score (RFS) (32.56 vs. 19.21) and BLEU score (12.12 vs. 10.64). Finally, we show that our technique can be applied for summarizing traditional, short, hand-held videos as well, where we improve the state of the art F-score on benchmark SumMe and TVSum datasets from 41.4 to 45.6 and 57.6 to 59.1 respectively

      Year of completion:  July 2022
       Advisor : C V Jawahar,Chetan Arora

      Related Publications

        Downloads

        thesis

        Counting in the 2020s: Binned Representations and Inclusive Performance Measures for Deep Crowd Counting Approaches

        Sravya Vardhani Shivapuja

        Abstract

        Crowd counting is an important task in security, surveillance and monitoring. There are many competitive benchmark datasets available in this domain. The data distribution in the crowd counting datasets show a heavy-tailed and discontinuous nature. This nature of the dataset is majorly ignored while building solutions to this problem. However, the skew in datasets contradicts few assumptions made by the stages of the training pipeline. As a consequence of the skew in the dataset, unacceptably large standard deviation wrt to the customarily used performance measures (MAE, MSE) is observed. To address these issues, this thesis provides modifications that incorporate the dataset skew in training and evaluation pipelines. In the training pipeline, to enable principled and balanced minibatch sampling, a novel smoothed Bayesian binning approach is presented that stratifies the entire count range. Further, these strata are sampled to construct uniform minibatches. The optimization is upgraded with a novel strata-aware cost function that can be readily incorporated into the existing crowd counting deep networks. In the evaluation pipeline, as an alternative to the customary evaluation MAE, this thesis provides three alternative evaluation measures. Firstly, a strata-level performance in terms of mean and standard deviation gives range specific insights. Secondly, relative error perspective is brought in by using a novel Thresholded Percentage Error Ratio (TPER). Lastly, a localization included counting error metric Grid Average Mean absolute Error (GAME) is used to evaluate the different networks. In this thesis, it is shown that proposed binning-based modifications retain their superiority wrt the novel strata-level performance measure. Overall, this thesis contributes a practically useful training pipeline and detail-oriented characterization of performance for crowd counting approaches.

        Year of completion:  July 2022
         Advisor : Ravi Kiran Sarvadevabhatla,Ganesh Ramakrishnan

        Related Publications

          Downloads

          thesis

          Deep Learning Methods for 3D Garment Digitization

          Astitva Srivastava

          Abstract

          The reconstruction of 3D objects from monocular images is an active field of research in 3D computer vision which is further boosted by advancements in deep learning. In context of human body, modeling realistic 3D virtual avatars from 2D images is a recent trend, thanks to the advent of AR/VR & metaverse. The problem is challenging, owing to non-rigid nature of human body, especially because of the garments. Various attempts have been made to solve the problem, at least for relatively tighter clothing styles, but loose clothing styles still pose a huge challenge. This problem has also sparked quite an interest in the fashion e-commerce domain, where the objective is to model the 3D garments, independent from the underlying body, in order to enable intriguing applications like virtual try-on systems. 3D garment digitization has been garnering a lot of interest in the past few years, as the demand for online window-shopping and other e-commerce activities has increased in the recent years, where the unfortunate crisis of COVID-19 plays a huge role. Though the problem of 3D digitization of garments seems intriguing, solving it is not as straightforward as it looks. There are existing works out there in the field, majority of which are deep learning based solutions. Most of these methods rely on predefined garment templates which makes the task of texture synthesis easier, but restrict the usage to a fixed number of garment styles for which templates are available. Additionally, these methods do not deal with issues like complex poses and self-occlusions which are very common under in-the-wild assumption. Template-free methods are also explored which enables modeling arbitrary clothing styles, however, they lack texture information which is essential for high-quality photorealistic appearance. The thesis aims to resolve aforementioned issues by providing novel solutions. The main objective is 3D digitization of garments from a monocular RGB image of a person wearing the garment, both in template-based and template-free settings. Initially, we address challenges in existing state-of-the-art template-based methods. We aim to handle complex human poses, occlusions etc. by proposing to use a robust keypoint regressor which estimates keypoints on input monocular image. These keypoints define thin-plate-spline (TPS) based warping of texture from input image to the UV space of a predefined template. Then, we utilize a deep inpainting network to handle missing texture information. In order to train these neural networks, we curate a synthetic dataset of garments with varying textures, draped on 3D human characters in various complex poses. This dataset helps in robust training and generalization to real images. We achieve state-of-the-art results for specific clothing styles (e.g. t-shirt and trouser). However, template-based methods cannot model any arbitrary garment style. Therefore, we next aim to handle arbitrary garment styles in a template-free setting. Existing state-of-the-art template-free methods can model geometrical details of arbitrary garment styles up to some extent, but fail to recover texture information. To model arbitrary geometry of garments, we propose to use an explicit, sparse representation introduced for modeling human body. This representation handles self-occlusion and loose clothing as well. We extend this representation by introducing semantic segmentation information for differentiating between various clothing styles (top wear /bottom wear) and human body present in the input image. Furthermore, this representation is exploited in a novel way to provide seams for texture mapping, thereby retaining high-quality textural details and providing way to lot of useful applications like texture editing, appearance manipulation, texture super-resolution etc. The proposed method is the first one to model arbitrary garment styles and recover textures as well. We evaluate our proposed solutions on various publicly available datasets, outperforming existing state-of-the-art methods. We also discuss the limitations in the proposed methods and provide potential solutions that can be explored. Finally, we discuss the future extensions of the proposed methods. We believe this thesis significantly improves the research landscape in 3D garment digitization and accelerates the progress in this direction.

          Year of completion:  August 2022
           Advisor : Avinash Sharma

          Related Publications

            Downloads

            thesis

            More Articles …

            1. Skeleton-based Action Recognition in Non-contextual, In-the-wild and Dense Joint Scenarios
            2. Casual Scene Capture and Editing for AR/VR Applications
            3. Towards Understanding Deep Saliency Prediction
            4. Retinal Image Synthesis