Fine-Grain Annotation of Cricket Videos


Abstract: 

animation  animation animation

 It's a sample video taken of IPL channel from Youtube

The recognition of human activities is one of the key problems in video understanding. Action recognition is challenging even for specific categories of videos, such as sports, that contain only a small set of actions. Interestingly, sports videos are accompanied by detailed commentaries available online, which could be used to perform action annotation in a weakly-supervised setting.

For the specific case of Cricket videos, we address the challenge of temporal segmentation and annotation of actions with semantic descriptions. Our solution consists of two stages. In the first stage, the video is segmented into ``scenes'', by utilizing the scene category information extracted from text-commentary. The second stage consists of classifying video-shots as well as the phrases in the textual description into various categories. The relevant phrases are then suitably mapped to the video-shots. The novel aspect of this work is the fine temporal scale at which semantic information is assigned to the video. As a result of our approach, we enable retrieval of specific actions that last only a few seconds, from several hours of video. This solution yields a large number of labelled exemplars, with no manual effort, that could be used by machine learning algorithms to learn complex actions.


Contribution:

In this paper, we present a solution that enables rich semantic annotation of Cricket videos at a fine temporal scale. Our approach circumvents technical challenges in visual recognition by utilizing information from online text-commentaries. We obtain a high annotation accuracy, as evaluated over a large video collection. The annotated videos shall be made available for the community for benchmarking, such a rich dataset is not yet available publicly. In future work, the obtained labelled datasets could be used to learn classifiers for fine-grain activity recognition and understanding.


Solution:

 1

In the first stage, the goal is to align the two modalities at a “scene” level. This stage consists of a joint synchronisation and segmentation of the video with the text commentary.


Scene Segmentation:

A typical scene in a Cricket match follows the sequence of events depicted in figure

exscenes2 

Model Learning

It was observed that the visual-temporal patterns of the scenes are conditioned on the outcome of the event. In other words, a 1-Run outcome is visually different from a 4-Run outcome

FourModel2   1RunModel2
 Four Run Model One Run Model

 

While the visual model described above could be useful in recognizing the scene category for a given video segment, it cannot be immediately used to spot the scene in a full-length video. Conversely, the temporal segmentation of the video is not possible without using an appropriate model for the individual scenes themselves. This chicken-and-egg problem can be solved by utilizing the scene-category information from the parallel text-commentary.


Dataset:

Our dataset is collected from the YouTube channel for the Indian Premier League(IPL) tournament.
The Commentary data is collected from CricInfo by web Crawling.

Name Matches No. of Phrases Role
IPL Video Dataset 4 Matches (20Hrs) 960 Phrases Video/Shot Recognition
CricInfo Dataset 300 Matches 1500 Bowler Phrases and 6000 Batsmen Phrases Text Classification

 


Results:

 

improvedExample   figure

 

R Bowler Shot Batsman Shot
2 22.15 39.4
4 43.37 47.6
6 69.09 69.6
8 79.94 80.8
10 87.87 88.95
R is Neighborhood to search for correct Shot
Kernel Vocab: 300 Vocab: 1000
Linear 78.02 82.25
Polynomial 80.15 81.16
RBF 81 82.15
Sigmoid 77.88 80.53

Vocab denotes visual vocabulary Size
Results are after applying CRF

 

 

 

 

 

 

 


Related Publications:

Rahul Anand Sharma, Pramod Sankar, C. V. Jawahar - Fine-Grain Annotation of Cricket Videos Proceedings of the Third Asian Conference on Pattern Recognition 3-6 Nov 2015, Kuala Lumpur, Malaysia. [PDF]


Code and Dataset:

  • Code can be Downloaded here Code
  • Crinfo Commentary Dataset can be Downloaded here Dataset
  • IPL Dataset is available on IPL's official youtube channel IPL
  • Divide Each Match into 10 over chunks and run the above code. Refer README inside code directory.
  • In case of any queries/doubts please contact      This email address is being protected from spambots. You need JavaScript enabled to view it.

Associated People:

Online Handwriting Recognition using Depth Sensors

 

image1


Abstract

In this work, we propose an online handwriting solution, where the data is captured with the help of depth sensors. User writes in air and our method recognizes it online in real time using the novel representation of features. Our method uses an efficient fingertip tracking approach and reduces the necessity of pen-up/pen-down switching. We validate our method on two depth sensors, Kinect and Leap Motion Controller and use state-of-the-art classifiers to recognize characters. On a sample dataset collected from 20 users, we report 97.59% accuracy for character recognition. We also demonstrate how this system can be extended for lexicon recognition with 100% accuracy. We have prepared a dataset containing 1,560 characters and 400 words with intention of providing common benchmark for air handwriting character recognition and allied research.


Datase

data

Character samples from Dataset

To evaluate the performance of our approach, we created a dataset, 'Dataset for AIR Handwriting' (DAIR). The dataset is created using 20 subjects, where each user stands straight in front of the sensor and writes in the air with one finger out. Users are allowed to write at their own speed and writing style. Dataset contains two sections DAIR I and DAIR II. DAIR I  consists of 1248 character samples from 16 users by taking 3 samples per character per user. DAIR II consists of words from a lexicon of length 40. Words in the lexicon are taken from the names of most populous cities and vary in length from 3 to 5. It contains 400 words which totals to 1490 characters.

Our datasets can be downloaded from the following links:

Please mail us at {sirnam.swetha@research, rajat.aggarwal@students}.iiit.ac.in for any queries.


Results

final result

Labels represent the predicted and expected characters/words in each pair. Each pair has the input trajectory and trajectory after normalization respectively. (a) samples correctly classified, (b) mis-classified samples, (c) correctly classified words.

 

image5

Accuracy comparison of characters using Kinect and Leap Motion Controller


Related Publications

Rajat Aggarwal, Sirnam Swetha, Anoop M. Namboodiri, Jayanthi Sivaswamy, C. V. Jawahar - Online Handwriting Recognition using Depth Sensors Proceedings of the 13th IAPR International Conference on Document Analysis and Recognition, 23-26 Aug 2015 Nancy, France. [PDF]


Associated People

 

 

Sports-10K and TV Series-1M Video Datasets

[Project page]


videostrfig


 

About

We introduce two large video datasets namely Sports-10K and TV series-1M to demonstrate scene text retrieval in the context of video sequences. The first one is from sports video clips, containing many advertisement signboards, and the second is from four popular TV series: Friends, Buffy, Mr. Bean, and Open All Hours. The TV series-1M contains more than 1 million frames. Words such as central, perk, pickles, news, SLW27R (a car number) frequently appear in the TV series-1M dataset. All the image frames extracted from this dataset are manually annotated with the query text they may contain. Annotations are done by a team of three people for about 150 man-hours. We use 10 and 20 query words to demonstrate the retrieval performance on the Sports-10K and the TV series-1M datasets respectively.


Downloads

Please mail us at This email address is being protected from spambots. You need JavaScript enabled to view it. for a copy of these two datasets (To be used for research purpose only).


Publications

Anand Mishra, Karteek Alahari and C. V. Jawahar.
Image Retrieval using Textual Cues
ICCV 2013 [PDF]


Bibtex

If you use this dataset, please cite:

@InProceedings{MishraICCV13,
  author    = "Mishra, A. and Alahari, K. and Jawahar, C.~V.",
  title     = "Image Retrieval using Textual Cues",
  booktitle = "ICCV",
  year      = "2013",
}

Related datasets


Contact

For any queries about the dataset feel free to contact Anand Mishra. Email:This email address is being protected from spambots. You need JavaScript enabled to view it.

 

The IIIT 5K-word dataset

[Project page]

About

The IIIT 5K-word dataset is harvested from Google image search. Query words like billboards, signboard, house numbers, house name plates, movie posters were used to collect images. The dataset contains 5000 cropped word images from Scene Texts and born-digital images. The dataset is divided into train and test parts. This dataset can be used for large lexicon cropped word recognition. We also provide a lexicon of more than 0.5 million dictionary words with this dataset.


Downloads

IIIT 5K-word (106 MB)
README
UPDATES


Publications

Anand Mishra, Karteek Alahari and C. V. Jawahar.
Scene Text Recognition using Higher Order Language Priors
BMVC 2012 [PDF]


Bibtex

If you use this dataset, please cite:
@InProceedings{MishraBMVC12,
  author    = "Mishra, A. and Alahari, K. and Jawahar, C.~V.",
  title     = "Scene Text Recognition using Higher Order Language Priors",
  booktitle = "BMVC",
  year      = "2012",
}

Contact

For any queries about the dataset feel free to contact Anand Mishra. Email:This email address is being protected from spambots. You need JavaScript enabled to view it.

The IIIT Scene Text Retrieval (STR) Dataset

[Project Page]

strfig


About

The IIIT STR dataset is harvested from Google and Flickr image search. Query words like coffee shop, motel, post office, high school, department were used to collect the images. Additionally, query words like sky, building were used in Flickr to collect some random distractors (images not containg text). The dataset contains 10,000 images in all. The images are manually annotated to say whether they contain a query word or not. Annotation for all the 50 query words used in our paper is available. Each query word appears 10-50 times in the dataset.


Downloads

IIIT STR (758 MB)
README


Publications

Anand Mishra, Karteek Alahari and C. V. Jawahar.
Image Retrieval using Textual Cues
ICCV 2013 [PDF]


Bibtex

If you use this dataset, please cite:

@InProceedings{MishraICCV13,
  author    = "Mishra, A. and Alahari, K. and Jawahar, C.~V.",
  title     = "Image Retrieval using Textual Cues",
  booktitle = "ICCV",
  year      = "2013",
}

Related datasets


Contact

For any queries about the dataset feel free to contact Anand Mishra. Email:This email address is being protected from spambots. You need JavaScript enabled to view it.