Towards Robust and Scalable Post Hoc Model Adjustment Techniques in Deep Learning Models

Amula Venkat Adithya

Abstract

Despite their high performance, deep learning models often require robust post-hoc adjustments; these interventions serve not just to correct inherent vulnerabilities or biases, but also to adapt the model to evolving requirements that emerge after training. Three main lines of work satisfy these criteria, namely: Machine Unlearning, Concept Based Model Improvement, and Backdoor Defense techniques. These techniques operate with different goals and at different levels of a data hierarchy: sample-level (individual data points), concept-level (abstract features or subgroups within classes), and class-level (entire categories)—each addressing undesirable behaviors at varying degrees of granularity. We aim to answer two main questions in this thesis. To what extent can a unified framework address the diverse requirements of post-hoc model correction? And does the level of granularity in the data hierarchy imply easier scalability to all corrective requirements? Our primary contribution is Prototype Guided Backdoor Defense (PGBD), a novel backdoor defense method. PGBD exploits displacements in the geometric spaces of activations to penalize movements towards the trigger. This is done using a novel sanitization loss of a post-hoc fine-tuning step. This approach scales to all types of attacks and trig- gers, and achieves better performance across settings. We also present the first defense against semantic attacks on a new celebrity face images dataset. As a secondary contribution, we conduct a systematic review of Machine Unlearning, establishing a novel taxonomy of its core ideas, diverse settings, and varied goals. Consequently, we consider the setting of Corrective Unlearning and explore the possibility of scaling Unlearning methods to the problem setting of Backdoor Defense. Results show that while Un- learning methods perform a good job of removing memorization of individual data points, they struggle to mitigate shortcuts learned through trigger features. Alternatively, leveraging PGBD’s configurability, we extend PGBD to the Corrective Unlearning setting and show that it outperforms all existing un- learning methods. Encouraged by these results, we argue that activation space manipulation provides a powerful and flexible blueprint for many post-hoc model adjustment tasks, particularly those involving concept-level or trigger-based vulnerabilities. While gradient-based unlearning remains the superior ap- proach for precise, sample-level forgetting, its application is often specialized. Our findings suggest that configurable, representation-agnostic interventions in activation space offer greater versatility across a broader spectrum of post-hoc corrections.

Related Publications

Downloads

Towards Realistic Scene-Aware Human-Object Interaction in 3D World

Kunal Kamalkishor Bhosikar

Abstract

Modeling realistic human-object interaction in three-dimensional environments is a fundamental challenge in computer vision, with broad implications for robotics, virtual reality, and embodied artifi- cial intelligence. Despite significant progress in human motion generation and 3D scene understanding, existing approaches often treat motion, interaction, and environment as separate problems, limiting their ability to generate physically plausible and context-aware behavior.

This thesis addresses this challenge through a unified perspective on scene-aware human-object in- teraction, grounded in both real-world systems and data-driven modeling. We begin by developing a system for real-time human pose understanding and interaction analysis based on joint-level reasoning, formalized in our published patent application. This system demonstrates how structured representa- tions of human motion can enable robust, interpretable, and deployable interaction-aware applications.

Building on this foundation, we introduce a large-scale dataset and benchmark for full-body human motion with object interaction in realistic 3D scenes. This dataset enables the study of scene-aware grasping and exposes key limitations of existing methods in handling interaction under environmental constraints. We then propose a learning-based framework for generating physically plausible, scene- aware human-object interactions that explicitly models the interplay between body motion, object ma- nipulation, and scene geometry.

Related Publications

Downloads

Textual Attribute Recognition for Documents

Jyothi Swaroopa Jinka

Abstract

Understanding document content requires not only recognizing textual characters but also capturing textual attributes such as bold, italic, underline, and strikeout, which convey emphasis, hierarchy, and semantic intent. These attributes play a critical role in real-world documents, including legal records, administrative forms, educational materials, and historical manuscripts, where formatting often encodes meaning that cannot be inferred from text alone. Despite their importance, Textual Attribute Recognition (TAR) has received limited attention compared to traditional Optical Character Recognition (OCR), and existing approaches remain inadequate for handling the complexity of real-world, multilingual, and layout-rich documents.

A key limitation of prior TAR methods lies in their treatment of words as isolated visual units. At- tribute recognition, however, is inherently context-dependent, as distinctions such as bold versus regular text or underline versus structural lines often require comparison with neighboring words or surround- ing layout elements. Methods that ignore this context suffer from ambiguity and reduced accuracy. Conversely, approaches that attempt to incorporate full-document context using transformer-based ar- chitectures face significant computational challenges, due to the quadratic complexity of attention over large numbers of word instances. These issues are further exacerbated in multilingual settings, where script diversity, font variations, and document noise introduce additional variability.

Related Publications

Downloads

Flash–Non-Flash Fusion for Contactless Fingerprint Recognition and Spoof Detection

Roja Lakshmi Sahoo

Abstract

Contactless fingerprint recognition provides a number of benefits in terms of hygiene and minimizing sensor-induced artifacts, over traditional contact-based systems, but it is also associated with a number of challenges that are currently hindering its adoption. The 2D representation of 3D features, in combi- nation with unconstrained illumination, affects the quality of ridges, resulting in inconsistent features. Also, the lack of interaction information in contactless recognition makes it more prone to presentation attacks. Current techniques for acquisition, enhancement, and detection of spoofing attacks are mostly independent and focused on single images, without exploiting any complementary information for im- proved recognition. As the trend towards contactless biometric recognition is becoming increasingly necessary for usability and cost-effectiveness in terms of deployment in mobile and security-critical applications, lower matching accuracy and increased susceptibility to spoofing attacks are significant challenges. This highlights the need for a method that jointly addresses enhancement, recognition, and spoof detection while improving cross-modal interoperability.

In our work, we present a unified framework for contactless fingerprint recognition by exploiting the flash–non-flash acquisition pair as a lightweight active sensing mechanism that extracts complemen- tary illumination information. We thus present the Flash–Non-Flash Fingerphoto (FNF) Database for a thorough analysis of the illumination impact, and Fusion2Print (F2P), an end-to-end deep learning framework that learns the optimal enhancement in the spatial, frequency, and color domains to enhance the quality of the ridge-valley representations and cross-domain recognition accuracy. Furthermore, we investigate a method for the detection of presentation attacks by exploiting the illumination-dependent photometric differences between genuine and spoofed fingerprints, and a low-cost 3D pipeline for the reconstruction of the fingerprint surface geometry from just two images, facilitating the simulation of digital spoof attacks.

Related Publications

Downloads