Insights about the behavioural and cognitive aspects that underlie the processes of reading/visually examining medical images by medical experts are useful in many areas such as design of displays, improving training to reduce performance errors of medical experts, etc. We are interested in using such insights to develop visual search models and design novel image analysis algorithms.

Our current work uses eye-tracking data from different modalities to understand the processes involved in visual search preceding diagnosis. Specifically, we are interested in understanding the gaze patterns of experts with varied level of expertise. We have also generated two innovative saliency models for diabetic retinopathy images.

Chest X-ray

Retinal Images

Subjects with different knowledge levels use different search strategy for abnormalities. Sample gaze tracks are shown below. The first fixation point is shown in white which along the track turns to blue to finally black as the search progresses. Experts appear to land on an abnormality early whereas novice take a very long route to reach to the same abnomality.


(L) Retinal Image with diabetic retinopathy abnormalities encircled by white boundaries (M) Gaze pattern of medical expert (R) Gaze pattern of a novice

Biologically Inspired Saliency Models:

Unsupervised Model

Here an ensemble of spatially-varying morphological maps is used generate a saliency map for retinal images with diabetic retinopathy. Each of this maps were computed by erosion and dilation operations where domain size of the morphology was varied based on Gaussian to mimic the gaussian field observed in the human visual system.

Supervised Model

A Convolutional Neural Network was devised in the Itti-Koch-like feshion for computation of saliency for diabetic retinopathy. There are 2 stages for generating responses for: (1) color and orientation filters (2) center-surround filters. Half of the weights in both the stages are initialized with standard filters which are fine-tuned during the training and the other half are initialized with random filters which are learned from the scratch. A novel loss function is used for training in order to handle range-mismatch between Ground-Truth and computed saliency maps.


Network architecture

Pairs of original image and computed saliency map. Saliency for dark lesion is shown in blue and for bright lesions in green.