We focuses on the challenging problem of Information Retrieval from large document image collections. We propose to develop algorithms and approaches that are scalable to large datasets. We use and extend ideas from machine learning (ML), information retrieval (IR) and computer vision (CV) for this task. Our results are expected to impact the way retrieval is carried out from document images (documents which have textual content and image format). Effective retrieval systems built over textual content (often crawled from web) have changed the way we look at multimedia collections. Since we work on images, traditional IR solutions are not directly applicable. Popular approach is to recognize (e.g. with an OCR) the images and build a textual representation. However recognizers can be brittle and result in noisy outputs in many practical settings (e.g. historic documents, handwritten documents, Indian language documents etc.). We design representations that can scale to millions of document images seamlessly from a small corpus of annotated datasets.

Bag of Visual Words Representation.

In this work, we present a Bag of Visual Words (BoVW) based approach to retrieve similar word images from a large database, efficiently and accurately. We show that a text retrieval system can be adapted to build a word image retrieval solution. This helps in achieving scalability. We demonstrate the method on more than 1 Million word images with a sub-second retrieval time. We validate the method on four Indian languages, and report a mean average precision of more than 0.75. To address the lack of spatial structure in the BoVW representation, we re-rank the retrieved list.

Key Features

• Language independent system : Demonstrated on 5 different languages.
• Scalable to huge datasets : Demonstrated on 1 Million images.
• Handles noisy document images : Demonstrated on dataset for which Commercial OCRs fail.

In this work, we propose a framework for content level access to the scanned pages of Digital Library of India (DLI). We propose a search scheme which fuses noisy OCR output and holistic visual features for content level access to the DLI pages. Visual content is captured using Bag of Visual Words (BoVW) approach. The fusion scheme improves over the individual methods in terms of mean Average Precision (mAP) and mean precision at 10 (mPrec@10). We exploit the fact that OCR has a high precision while BoVW has a high recall.

Digital Library of India

Sample DLI Word Images from the corpus.

Key Features

• Combining Recognition/Recognition-Free Methods.
• Practical framework for document image search engine for Indian languages.
• Improving BoVW using Query Expansion.
• Support of Text queries.
• Demonstration of methods on more than 100 books from two Indian languages.
• Sub second retrieval from millions of word images.

Architecture

Search Architecture for book page retrieval system.

Performance of the word spotting approaches for document retrieval, heavily depends on the exact or approximate matching of images (in some feature space) to retrieve documents containing the same word. However, the harder problem in information retrieval is to effectively bring semantics into the retrieval pipeline. In this work, we investigate this problem, and suggest a solution by directly transferring the semantics from the textual domain. Figure 1(ii) shows the semantic set of retrieved images along with similar images for the query "GENUINE". The result contains most of the semantically related words like true, actual etc. Some of the key features of the work are:-

Sample results from the experiments conducted on large corpus having more than a million word images.

Key Features

• The system is "trainable" with a smaller collection of labeled examples and extended to perform to a larger collection of unlabeled data.
• We use linguistic resource such as WordNet to learn the semantic relationships between the words.
• We demonstrated the method on two languages – English and Hindi having more than a million word images.
• Building the semantic model is independent on the linguistic resource used. Our method is applicable with any learned statistical language model.

Architecture

Search architecture showing the indexing and retrieval. The input to the system is shown at upper left side as the query image given to semantic index.
The results are given in a ranked manner.

Bag of words (BoW) based retrieval is an efficient method to compare the visual similarity between two images. Recognition free methods based on BoW have shown to out- perform OCR based methods. We further improve the perfor- mance by defining a document specific sparse coding scheme for representing visual words (interest points) in document images. Our method is motivated by the successful use of sparsity in signal representation by exploiting the neighbourhood properties. In addition to providing insights into the design of the coding scheme, we also verify the method on two data sets and compare with the recent methods. We have also developed text query based search solution, and we report performance comparable to image based search.

Locality in Character Space (a) Vector Quantization (VQ) (b) Sparse Coding (SC) (c) Locality Coding. In VQ, each descriptor is assigned to single visual word while in SC, it is assigned to multiple
visual words but locality of visual word is lost. In locality coding, each descriptor is assigned to multiple visual words by considering locality of visual words.

1. Ravi Sekhar and C V Jawahar, Word Image Retrieval Using Bag of Visual words, Proceedings of 10th IAPR International Workshop on Document Analysis Systems 27-29 Mar. 2012, ISBN 978-1-4673-0868-7, pp. 297-301, Queensland, Australia. [Paper] [Poster] [bibtex]
   
2. Praveen Krishnan, Ravi Sekhar, C V Jawahar, Content Level Access to Digital Library of India Pages, Proceedings of the 8th Indian Conference on Vision, Graphics and Image Processing, 16-19 Dec. 2012, Bombay, India. . [Paper] [Slides] [bibtex]
   
3. Ravi Sekhar and C V Jawahar, Document Specific Sparse Coding for Word Retrieval , Proceedings of 12th International Conference on Document Analysis and Recognition (ICDAR), 2013, Washington DC, USA. [Paper] [Poster] [bibtex]
   
4. Praveen Krishnan and C V Jawahar, Bringing Semantics in Word Image Retrieval, Proceedings of 12th International Conference on Document Analysis and Recognition (ICDAR), 2013, Washington DC, USA. [Paper] [Poster] [bibtex]