At IIITH, we have been aiming to solve problems on Indian roads and driving conditions using data-driven technologies. With the help of techniques from machine learning, computer vision, computer graphics, computational sensing, and allied areas, we have been creating solutions and transferring these solutions across diverse practical conditions on roads.

Driver Intention Prediction

One of the research studies that was undertaken includes early anticipation of driver actions before the onset of a manoeuvre, such as a turn, a lane change, a sudden halt and so on. When you know the driver in front of you is going to take a right turn without indicating with a signal, you anticipate and get ready for it perhaps by slowing down or changing lanes. In order to predict such driver behaviour, CVIT curated a Driving Action Anticipation Dataset (DAAD) that consists of multiple views – both inside the cabin and outside of heterogenous traffic scenarios that include diverse weather, illumination, and driveway conditions. The dataset captures sequences prior to initiating the manoeuvre and during the manoeuvre as well. The research which culminated in a paper titled, “Early Anticipation of Driving Manoeuvres”was presented at the European Conference on Computer Vision 2024. A natural extension to this research was studied on the 2-wheeler where rider intention is anticipated before the onset of the actual manoeuvre. This is more challenging but the ability to predict rider intention enables riders to react to potential incorrect manoeuvres flagged by advanced driver assistance systems (ADAS). The dataset that was curated was submitted to the International Conference on Pattern Recognition 2024 competition on rider intention prediction.

Other Related Research Directions

In the year 2019, Hyderabad received 700 mm rainfall which was 13% over the long term average for the city. It resulted in damaged roads and road infrastructure. Conventional civil engineering methods of surveying damage take forever. The problem statement that arose was whether we could leverage AI to assess structural damage to the roads. We proposed again the low-cost solution of mounting a mobile phone equipped with camera on any vehicle that could be driven around the city capturing the extent of damage and narrowing down on where attention was required. Around the same time, the government approached us to undertake an automatic survey on the extent of tree cover, or in other words, to count the number of trees along certain stretches of roads. Similarly, we extended the object identification to traffic violations. The moment drivers and riders see a traffic light and a surveillance camera mounted on it, they slow down or quickly wear a helmet, etc. The government wanted an unobtrusive way of capturing the violations, with a mobile phone camera mounted on public vehicles which could automatically issue challans, thereby improving traffic discipline and ensuring compliance to traffic rules.

How To Make Roads Safe?

Alongside the above solutions, we began to explore other areas to really understand roads and driver behaviour. For that the AI needed to shift from perception to anticipation, meaning that it could detect what was present and available but it needed to understand what is absent. In other words, we had to move from a reactive to proactive understanding. All the AI algorithms today are able to tell us what is available, but the need of the hour is to predict what will come next – a looking-forward in time mechanism. That involves bringing back humans into the loop and asking the question – Can we predict humans?

Can We Predict Human Behaviour?

Every person on the road has a unique character from the Swiggy delivery personnel to the auto driver. In order to predict how they react, we conducted some experiments. For example, every second on the road matters. Understanding where the driver is looking can mean the difference between avoiding an accident or causing one. Yet the technology to capture driver gaze has been limited by expensive and bulky equipment like wearable eye trackers with complex camera sets. To address this, we introduced Dashgaze, using dash cams as an accessible, low-cost means for driver gaze estimation. Dashgaze is the largest dataset for mapped driver gaze in naturalistic driving conditions. It consists of 0.9 million frames from 28 participants across 32 drives. Each drive is over 10 minute long, totalling more than 10 hours of video. The dataset includes synchronised views of the road, the driver and the driver’s ego-centric perspective. The setup is itself very simple consisting of a mobile phone which has a camera on both sides. So, if an old lady is crossing the road, it should be able to tell us to focus on her rather than the movie star who has made an appearance. Our roads are full of distractions compelling our attention. So, there ought to be a mechanism to tell us when we are not attentive or what we should be focusing on. And in order to do that, it should not be done on an expensive device. The idea is to eventually be able to do it on a mobile phone. With the help of DataGaze, one should be able to build a machine learning model that is cost-efficient and that can go into a mobile phone.

IDD: Datasets That Fuel Research

A large number of datasets exist today for autonomous navigation but unfortunately they have tended to focus on structured driving environments and adherence to traffic rules. The story in the Indian context is however vastly different. In order to cater to unstructured driving environments, a collaborative venture with Intel and IIITH was kicked off to curate a dataset specific to Indian driving conditions. Data was captured by a front facing camera attached to a car that was driven around Hyderabad and Bangalore cities and their outskirts. The idea was to make the dataset available to anyone from industry, academia or startups interested in developing and validating mobility-related algorithms. They should know what makes our roads and the driving on our roads harder, what is required and what is the gap and what kind of technological innovations are required in bridging this gap. The dataset is public. The download statistics reveals that it has a large number of users from across the world.

Enabler and Catalytic Roles

Apart from the dataset, we brainstormed on what was really needed to make the mobility ecosystem survive. There is a Data Foundation now which hosts a large number of datasets that are available for public use. Apart from datasets, there is also a 4-wheeler and a 2- wheeler data capturing platform equipped with multiple sensors where anybody who wants to experiment with AI, machine learning, computer vision, image processing, pattern recognition and more should be able to do so with ease. The number of 2-wheelers on the roads far outnumbers that of 4-wheelers and yet the advanced driving technologies are all developed for cars and trucks. It was to fill this gap that a platform was created especially for 2-wheelers too.

iRaste

The first version of the India Driving Dataset which was officially released in 2018 now has a number of extensions that include not only object detection but also classification, segmentation, and other technical perception tasks, benchmarks, in 2D and 3D. The original motivation behind curating the dataset may have been to fuel research in autonomous navigation but we realised that there were many pressing practical problems to be addressed first. For instance, in many places, there are no traffic signs or if they are present, they are not visible or not maintained. Infrastructural deficiencies have a major role to play in road accidents. Every year, 1.5 lakh lives are lost on Indian roads which translates to a death every 4 minutes. The state of Telangana along with Nagpur city are accelerating the safe systems approach to road safety with the help of AI. It is the first of it’s kind global initiative to upgrade the 4E framework for road safety by leveraging AI and data analytics. The idea here is to leverage predictive insights generated by AI to prevent accidents even before they happen. Prevention of accidents happens in 3 different ways – 1) Vehicle safety; where Advanced Driver Assistance Systems give driver alerts mere seconds before an imminent accident. This along with driver training was found to reduce accidents by up to 60%. 2) Mobility Analysis; By continuously monitoring dynamic risks on the roads, the goal is to flag potential blackspots and remove them before they claim lives. 3) Infrastructure safety; For existing blackspots, the idea is to design engineering fixes and continuously monitor infrastructure quality to deliver improved rate of investment on road infrastructure maintenance.

Conclusion

The future looks very promising. With AI for driving, we are seeing the emergence of very smart vehicles. For that we will need smart infrastructure, tools, apps and may be even a Chat GPT for driving in the not-so-distant future. But if we want such high level tech in India, we will have to understand the 2-wheelers and the 3-wheelers too that form the bulk of the driving landscape here.

This article was initially published in the April ’25 edition of TechForward Dispatch

By Prof C V Jawahar

Prof C V Jawahar is the head of Centre for Visual Information Technology, Machine Learning Lab, and the Executive Education Group at IIITH. He is also the Dean (R&D) and an Amazon Chair Professor.