Towards Natural Sign Language Generation: Scalable and Fluid Sign Synthesis for Inclusive Communication

Aparna Agrawal Nitin

Abstract

Language is a rich, multilayered system through which humans articulate complex ideas, construct social understanding, and negotiate shared meaning. Yet, despite its central role in human interac- tion, much of the world’s linguistic diversity remains marginalized within mainstream communication technologies. Sign languages, primary communication systems for Deaf communities, illustrate this disparity most clearly, as they remain underrepresented in digital infrastructures despite their linguis- tic sophistication and societal importance. Consequently, large portions of informational and cultural content circulating in society remain effectively inaccessible to Deaf and hard-of-hearing individuals. Live news broadcasts, emergency alerts, televised debates, classroom lectures, government proceed- ings, and a substantial portion of online video media are produced primarily for the hearing public. Live news broadcasts, emergency alerts, televised debates, classroom lectures, government proceedings, and a substantial portion of online video media are produced primarily for the hearing public.

According to the World Health Organization (WHO), approximately 466 million people worldwide are deaf or hard of hearing, with around 72 million using sign language as their primary means of communication [58]. In India, a WHO news report estimates that 63 million people have “significant auditory impairment” [84].1 Sign language is a visually rich and complex mode of communication, incorporating spatial grammar, facial expressions, and contextual cues that are often unfamiliar to non- signers, making it challenging for them to learn and use it effectively.

Related Publications

Downloads

Multimodal Methods for Educational Applications

Megha Mariam K. M

Abstract

Recent advances in large language models (LLMs), vision–language models (VLMs), and text-to- video (T2V) systems have sparked growing excitement about the future of education. Imagine a learn- ing experience where complex ideas can be explained through synchronized text, visuals, narration, and video; where educational content can be generated on demand; and where learning becomes more interactive, accessible, and personalized. While these possibilities are increasingly within reach, an im- portant question remains: do current multimodal AI systems truly understand and connect information across different modalities in ways that meaningfully support learning? This thesis explores that ques- tion through the development of benchmarks and evaluation frameworks for multimodal educational AI.

A familiar challenge for learners is following presentation slides while simultaneously listening to a speaker. Slides are often dense, explanations move quickly, and learners can easily lose track of where to focus their attention. To address this, we explore a method that automatically highlights slide regions relevant to the ongoing narration, helping guide learners toward the most important visual information in real time. Supporting this effort, we introduce a dataset designed to evaluate how effectively models can align spoken language with slide content. The dataset consists of 14 presentation videos and 150 slides, where each slide is paired with its corresponding audio segment and transcript. It captures a wide range of visual and textual elements—including figures, equations, tables, and diverse layouts—making it a challenging and realistic benchmark for multimodal alignment.

Related Publications

Downloads

Deep Learning Based Analysis of Cancer Similarities and Subtyping Using Histopathological Whole Slide Images

Piyush Singh

Abstract

Whole slide images (WSIs) contain rich information for computational pathology, yet systematic evaluations of cross-organ and multi-cohort generalization remain limited. This thesis addresses these challenges through two complementary studies. In the first study, patch-level convolutional neural networks (CNNs) were trained on 9,792 slides from The Cancer Genome Atlas (TCGA), spanning 11 cancer subtypes across seven organs, to distinguish cancerous from normal tissue. Both within-organ and cross-organ inference were evaluated, revealing that cancers such as breast, colorectal, and liver can be reliably detected by models trained on other organs. Strong transferability was also observed between subtypes within an organ, such as kidney and lung. To investigate these patterns, feature similarity, overlap in high-attention regions, and nuclear geometry were analyzed, all of which showed positive correlations with cross-organ transfer performance. The second study focuses on lung cancer in the Indian population through the introduction of IPD-Lung, a curated dataset of adenocarcinoma and squamous carcinoma cases. Benchmark evaluations were established using multiple instance learning (MIL) models, with additional experiments exploring model transferability from publicly available TCGA-Lung dataset. Stain normalization methods, particularly Macenko, reduced domain discrepancies to some extent but did not fully bridge performance gaps, especially for squamous carcinoma. A domain adaptation strategy based on a gradient reversal layer (GRL) similarly yielded limited improvements. In contrast, models trained directly on IPD-Lung achieved substantial performance gains, with further enhancements obtained by training on expert-annotated regions of interest (RoIs). Together, these studies demonstrate that deep learning models can uncover meaningful cross-organ similarities in cancer histopathology. 

Related Publications

Downloads

Continual Learning in Interactive Medical Image Segmentation

Kushal Borkar

Abstract

Automated segmentation of medical image volumes holds immense potential to significantly reduce the time and effort required from medical experts for annotation. However, leveraging machine learning for this task remains a formidable challenge due to variations in imaging modalities, the inherent complexity of medical images, and the limited availability of labeled patient data. While existing interactive segmentation methods and foundational models incorporate user-provided prompts to iteratively refine segmentation masks, they often fail to learn the continuity and inter-related information between consecutive slices in a 3D medical image volume. This limitation leads to inconsistencies, spatial discontinuities, and loss of anatomical coherence, ultimately affecting the reliability of segmentation results in clinical applications.

The work proposes a novel interactive segmentation framework that dynamically updates model parameters during inference using a test-time training paradigm guided by user-provided scribbles. Unlike traditional approaches, our method preserves crucial spatial and contextual information from both previously processed slices within the same medical volume and the training dataset through a studentteacher learning mechanism. By leveraging sequential dependencies across slices, our approach ensures smoother and more anatomically consistent segmentation masks while integrating prior knowledge from the training distribution.

We extensively evaluated our framework across diverse datasets, encompassing CT, MRI, and microscopic cell images, demonstrating its superior performance in both efficiency and accuracy. Our method significantly reduces user annotation time by a factor of 6.72× compared to manual annotation workflows and factor of 1.93× compared to state-of-the-art interactive segmentation methods. Furthermore, when benchmarked against foundational segmentation models, our framework achieves a Dice score of 0.9 within just 3–4 user interactions—substantially improving upon the 5–8 interactions required by existing models. This reduction in required interactions translates to a more streamlined and intuitive annotation process for volumetric CT and MRI scans.

Related Publications

Downloads