Towards Realistic Scene-Aware Human-Object Interaction in 3D World

Kunal Kamalkishor Bhosikar

Abstract

Modeling realistic human-object interaction in three-dimensional environments is a fundamental challenge in computer vision, with broad implications for robotics, virtual reality, and embodied artifi- cial intelligence. Despite significant progress in human motion generation and 3D scene understanding, existing approaches often treat motion, interaction, and environment as separate problems, limiting their ability to generate physically plausible and context-aware behavior.

This thesis addresses this challenge through a unified perspective on scene-aware human-object in- teraction, grounded in both real-world systems and data-driven modeling. We begin by developing a system for real-time human pose understanding and interaction analysis based on joint-level reasoning, formalized in our published patent application. This system demonstrates how structured representa- tions of human motion can enable robust, interpretable, and deployable interaction-aware applications.

Building on this foundation, we introduce a large-scale dataset and benchmark for full-body human motion with object interaction in realistic 3D scenes. This dataset enables the study of scene-aware grasping and exposes key limitations of existing methods in handling interaction under environmental constraints. We then propose a learning-based framework for generating physically plausible, scene- aware human-object interactions that explicitly models the interplay between body motion, object ma- nipulation, and scene geometry.

Related Publications

Downloads