Enhancing Driving Visibility via Semantic-Guided Knowledge Distillation Framework for Adverse Weather Removal

Hanvitha Saraswathi Mukkamala, Shankar Gangisetty, Ananya Kulkarni,

Veera Ganesh Yalla, C V Jawahar

IIIT Hyderabad

[Paper] [Code]

 Figure1. Illustration of our framework on ADAS vehicle in a rainy driving scenario for Adverse Weather Removal.

Abstract

Adverse weather such as rain, haze, and low light severely degrades visual perception in Advanced Driver Assistance Systems (ADAS) and autonomous driving, leading to degraded scene understanding and increased safety risks. We propose a unified, semantic-guided knowledge distillation restoration framework that addresses multi- weather removal while preserving semantics. Our method employs a semantic-guided dual-decoder architecture trained via two-stage multi-teacher knowledge distillation, transferring expertise from multiple high-capacity models into a lightweight student model. Segmentation-aware contrastive learning further aligns low-level restoration with high-level semantic structure, enabling robust detection of roads, vehicles, and pedestrians under challenging conditions. Trained on a mix of synthetic and real-world data with segmentation-guided feature refinement, our framework gener- alises effectively to real-world unseen environments. Extensive experiments on multiple benchmarks show competitive or superior performance to state-of-the-art methods, with real-time inference suitable for edge deployment. This makes our approach well-suited for safety-critical perception in autonomous and semi-autonomous systems operating in adverse outdoor environments.

 Methodology

Our method builds a unified lightweight student model for adverse weather removal by distilling knowledge from multiple weather-specific teacher networks. The key idea is to combine image restoration with semantic guidance, so the model not only improves visibility but also preserves important scene structures such as roads, vehicles, and pedestrians.

 Figure2. Overview of our semantic-guided knowledge distil- lation framework. Collaborative distillation from multiple weather-specific teachers (rain, haze) to a unified student via two stages: (1) Knowledge Collation, where soft alignment with teacher reconstructions and segmentation priors, and (2) Knowledge Examination, where enforcing ground-truth consistency with hard constraints. Segmentation maps provide region-aware supervision, emphasizing critical structures like roads and vehicles.

At inference time, only the lightweight student model is used. This makes the framework efficient, memory-friendly, and suitable for real-time deployment without requiring any teacher model during testing.

QUANTITATIVE RESULTS

  Our method achieves strong performance across both synthetic and real-world adverse weather datasets, showing consistent improvements in restoration quality while preserving semantic structure. 

 Table 1:Quantitative evaluation of image dehazing performance on synthetic and real datasets. 

 Table 1:Quantitative evaluation of image dehazing performance on synthetic and real datasets. 

 Table 3:Quantitative comparison on real-world derained and dehazed images from IDD-AW using NIQE and BRISQUE. 

QUALITATIVE RESULTS

 Figure 3: Visual comparison of the proposed and existing methods from real datasets (Raindrop, SPA ,O-HAZE ) for multi-weather restoration. The image can be zoomed in for improved visualization. 

 Figure 4: Visual comparison of the proposed and existing methods from synthetic datasets (Outdoor-Rain , RESIDE) for multi-weather restoration. The image can be zoomed in for improved visualization. 

 Figure 5: Qualitative comparison of real-world rain and haze images with low-light from IDD-AW dataset. 

@in proceedings{Enhancing2025icvgip, 

     author = {Hanvitha Saraswathi Mukkamala, Shankar Gangisetty, Ananya Kulkarni, Veera Ganesh Yalla and C. V. Jawahar}, 
     title = {Enhancing Driving Visibility via Semantic-Guided Knowledge
Distillation Framework for Adverse Weather Removal}, 

     booktitle = {}, 

     series = {}, 

     volume = {}, 

     pages = {}, 

    publisher = {}, 

    year = {2025}, 
}

Sketchtopia: A Dataset and Foundational Agents for Benchmarking Asynchronous Multimodal Communication with Iconic Feedback

Mohd Hozaifa Khan, Ravi Kiran Sarvadevabhatla

IIIT Hyderabad|CVIT

The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2025

[Paper] [CVPR Paper] [Dataset (WIP)] [Demo ]

Audio Overview

Abstract

We introduce Sketchtopia, a large-scale dataset and AI framework designed to explore goal-driven, multimodal communication through asynchronous interactions in a Pictionary-inspired setup. Sketchtopia captures natural human interactions, including freehand sketches, open-ended guesses, and iconic feedback gestures, showcasing the complex dynamics of cooperative communication under constraints. It features over 20K gameplay sessions from 916 players, capturing 263K sketches, 10K erases, 56K guesses and 19.4K iconic feedbacks.

We introduce multimodal foundational agents with capabilities for generative sketching, guess generation and asynchronous communication. Our dataset also includes 800 human-agent sessions for benchmarking the agents. We introduce novel metrics to characterize collaborative success, responsiveness to feedback and inter-agent asynchronous communication. Sketchtopia pushes the boundaries of multimodal AI, establishing a new benchmark for studying asynchronous, goal-oriented interactions between humans and AI agents.

 

Sketchtopia Agents

ACTIONDECIDER: The Asynchronous Controller

The ActionDecider is the core component that enables asynchronous communication. It acts as a lightweight controller, continuously monitoring the game state (sketches, guesses, feedback) and deciding when agents should act and what action they should take. This allows for fluid, human-like interaction without the constraints of turn-taking, mirroring real-world communication dynamics.

ActionDecider: The Brains Behind Asynchronous Interaction

Multi-Modality Sketchtopia Agent Diagram

DRAWBOT: The Sketcher

DRAWBOT visually communicates target word through asynchronous sketching, leveraging state-of-the-art generative models fine-tuned for iterative refinement based on communication context.

• Generates sketches from target concepts, adapting to canvas state.
• Refines drawings using feedback signals.
• Adapts to 👍 👎 ❓.
• Operates asynchronously, deciding when to draw or stay idle.

Drawbot Architecture

Multimodal Architecture

GUESSBOT: The Guesser

GUESSBOT interprets sketches and generates intelligent guesses, using a retrieval-based framework informed by historical interaction data.

• Uses vision models on sketch canvas content using vision models.
• Generates relevant guesses using efficient retrieval and filtering.
• Acts asynchronously, deciding when new information warrants a guess.

Guesserbot Architecture

Multimodal Architecture

 

Evaluating Agent Performance

 

🔀

AAO

Asynchronous Action Overlap

Measures concurrent actions between agents. Close AAO values to human suggests more natural, human-like interaction dynamics.

💬

FRS

Feedback Responsiveness Score

Quantifies how effectively agents adapt to feedback (👍👎) and move towards goal.

⏳

MATS

Multimodal Action Timing Similarity

Compares agent action timing patterns with human interactions to assess the naturalness of pacing.

 

Example Sessions

 

Target: ANGRY

Type: Human-Human

Key Guess: "Angry"

Feedback Given: 👍

Successful communication: Guesser guessed the correct emotion despite the sketch and feedback.

Target: ANGRY

Type: Human-Human

Key Guess: afraid, mute

Feedback Given: ❓

Failed communication: Guesser failed to guess the correct emotion despite the sketch and feedback.

Target: DUSTBIN

Type: Human-Human

Key Guess: "Dustbin"

Feedback Given: No feedback

Successful communication: GUESSBOT guessed the correct target word despite the sketch and feedback.

Target: DUSTBIN

Type: Human-Human

Key Guess: Fail guess:face etc

Feedback Given: 👎

Failed communication: Guesser failed to guess the correct target word despite the sketch and feedback.

Target: WALK

Type: Human-Agent

Key Guess: "Walk"

Feedback Given: No Feedback

Successful communication: Guesser guessed the correct target word despite the sketch and feedback.

Target: WALK

Type: Human-Agent

Key Guess: Fail Guess: man,run etc

Feedback Given: 👎,👍

Failed communication: Guesser failed to guess the correct target word despite the sketch and feedback.

❮ ❯

Authors

👤

Mohd Hozaifa Khan

IIIT Hyderabad

👤

Ravi Kiran Sarvadevabhatla

IIIT Hyderabad

Resources

Interactive Demo - Coming Soon!

Stay tuned for a live demo where you can experience Sketchtopia agents interacting.

In the meantime, explore the Dataset

 

Citation

@in proceedings{khan2025sketchtopia, 

     author = {Sainithin Artham, Avijit Dasgupta, Shankar Gangisetty, and C. V. Jawahar}, 
     title  = {Sketchtopia: A Dataset and Foundational Agents for Benchmarking Asynchronous Multimodal Communication with Iconic Feedback}, 

     booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, 

     series = {}, 

     volume = {}, 

     pages = {}, 

    publisher = {}, 

    year = {2025}, 

    url = {https://sketchtopia25.github.io/}, 

Copy BibTeX

 

Intellectual Property Notice

This work is the subject of a patent application filed in [India / under PCT] and is protected under applicable intellectual property laws. All rights to the underlying technology, including the AI agents for drawing and guessing in a Pictionary-like setting, are reserved.The system is currently under active research and development. Any use, reproduction, or commercial exploitation of this work or its components without prior written consent is prohibited..

Patent Application Status: Patent Pending.

 

 

 

IndicDLP : A Foundational Dataset for Multi-Lingual and Multi-Domain Document Layout Parsing 

ICDAR 2025 (Oral) — 🏆 Best Student Paper Runner-Up Award 

Oikantik Nath¹, Sahithi Kukkala², Mitesh Khapra¹, Ravi Kiran Sarvadevabhatla²

[Paper]       [Code]     [Dataset]

Samples from the IndicDLP dataset highlighting its diversity across document formats, domains, languages, and temporal span. For improved differentiability, segmentation masks are used instead of bounding boxes to highlight regions more effectively.

The above figure illustrates the contributions of 12 languages (left) and 12 document domains (right) in the IndicDLP dataset. The distribution is fairly balanced across both categories, with no single language or domain overwhelmingly dominating the dataset. This ensures a diverse and well-represented collection.

Comparison of modern document layout parsing datasets.

Citation

Please cite our paper if you find this dataset or work useful:

@Inproceedings{10.1007/978-3-032-04614-7_2,
  author       = {Oikantik Nath, Sahithi Kukkala, Mitesh Khapra, Sarvadevabhatla, Ravi Kiran},
  editor      = {Xu-Cheng Yin, Dimosthenis Karatzas, and and Daniel Lopresti  },
  title        = {IndicDLP: A Foundational Dataset for Multi-lingual and Multi-domain Document Layout Parsing},
  booktitle    = {Document Analysis and Recognition -- ICDAR 2025},
  year         = {2026}
  publisher    = {Springer Nature Switzerland},
  address      = {Cham},
  pages        = {23--39},
  abstract     = {Document layout analysis is essential for downstream tasks such as information retrieval, 
extraction, OCR, and digitisation. However, existing large-scale datasets like PubLayNet and DocBank lack 
fine-grained region labels and multilingual diversity, making them insufficient for representing complex documents 
layouts. Human-annotated datasets such as {\$}{\$}M^{\{}6{\}}Doc{\$}{\$}M6Doc and {\$}{\$}{\backslash}text
 {\{}D{\}}^{\{}4{\}}{\backslash}text {\{}LA{\}}{\$}{\$}D4LA offer richer labels and greater domain diversity, 
but are too small to train robust models and lack adequate multilingual coverage. This gap is especially 
pronounced for Indic documents, which encompass diverse scripts yet remain underrepresented in current datasets, 
further limiting progress in this space. To address these shortcomings, we introduce IndicDLP, a large-scale 
foundational document layout dataset spanning 11 representative Indic languages alongside English and 12 common 
document domains. Additionally, we curate UED-mini, a dataset derived from DocLayNet 
and {\$}{\$}M^{\{}6{\}}Doc{\$}{\$}M6Doc, to enhance pretraining and provide a solid foundation for Indic layout 
models. Our experiments demonstrate that fine-tuning existing English models on IndicDLP significantly boosts 
performance, validating its effectiveness. Moreover, models trained on IndicDLP generalise well beyond Indic 
layouts, making it a valuable resource for document digitisation. This work bridges gaps in scale, diversity, and 
annotation granularity, driving inclusive and efficient document understanding.}
 isbn      = {978-3-032-04614-7}

Acknowledgments

We would like to acknowledge the support from Indian Institute of Technology, Madras, India and International Institute of Information Technology Hyderabad, India.

MOTOR: A Multimodal Dataset for Two-Wheeler Rider Behavior Understanding

Varun Paturkar, Shankar Gangisetty, C V Jawahar

IIIT Hyderabad

[Paper] [Code] [Dataset] [Project Webpage]

 Fig: Comparison of traffic contexts and accident statistics across the Global North and South. Top Row: Four-wheelers dominating in the USA vs Two-wheelers in India. Bottom Row: Distribution of vehicles (two-wheeler vs four-wheeler) and fatal accidents across North and South.

Abstract

Two-wheelers account for a disproportionately high share of road fatalities in the Global South. Research on rider behavior, however, lags far behind four-wheelers, where multimodal datasets have driven major advances in Advanced Driver Assistance Systems (ADAS). To address this gap, we present the MOtorized TwO-wheeler Rider (MOTOR) dataset, the first large-scale, multi-view, multimodal resource dedicated to two-wheelers in dense, unstructured traffic. MOTOR com- prises 2,500 sequences (25+ hours) collected from 16 riders and integrates synchronized front, rear, and helmet videos, rider eye-gaze from wearable trackers, on-road audio, and telemetry (GPS, accelerometer, gyroscope). Rich annotations capture traf- fic context, rider state, 12 riding maneuvers spanning conven- tional and unconventional behaviors, and legality labels (Legal, Illegal, Unspecified). We benchmark rider behavior recognition and maneuver legality classification using state-of-the-art video action recognition backbones (CNN and Transformer-based), extended with multimodal fusion, and find that combining RGB, gaze, and telemetry consistently yields the best performance. MOTOR thus provides a unique foundation for advancing safety-critical understanding of two-wheeler riding. It offers the research community a benchmark to develop and evaluate models for behavior analysis, legality-aware prediction, and intelligent transportation systems. Dataset and code will be made publicly available.

 The MOTOR Dataset

 Table: Comparison of 4-wheeler and 2-wheeler behavior datasets. Our dataset is unique as it contains multi-modal, multi-view videos from ego-vehicle and helmet, eye gaze, as well as annotated conventional and unconventional behaviors, and legality-related riding scenarios. Note: CRB indicates conventional riding behaviors, and UCRB means unconventional riding behaviors.

 Fig: Data samples helmet-view. (a) Ego-rider weaves through dense, slow traffic, overtaking multiple vehicles across lanes. (b) Rider squeezes through a narrow gap between a bus and a car, narrowly avoiding the bus. (c) Rider rides in the wrong lane against dense oncoming traffic, disrupting flow. (d) Rider turns head fully toward a roadside building, diverting gaze from the road amid fast-moving traffic.

 Results

 Table: Rider Behavior Classification. Comparison of CNN and Transformer-based baselines on MOTOR dataset across different modality combinations.

 Table: Rider Legality Classification: CNN and Transformer-based baselines on MOTOR dataset across different modality combinations.

@in proceedings{motor2026icra, 

     author = {Varun Paturkar, Shankar Gangisetty, and C. V. Jawahar}, 
     title = {MOTOR: A Multimodal Dataset for Two-Wheeler Rider Behaviour Understanding}, 

     booktitle = {}, 

     series    = {}, 

     volume    = {}, 

     pages     = {}, 

     publisher = {}, 

     year      = {2026}, 
}