Online Handwriting Recognition using Depth Sensors




In this work, we propose an online handwriting solution, where the data is captured with the help of depth sensors. User writes in air and our method recognizes it online in real time using the novel representation of features. Our method uses an efficient fingertip tracking approach and reduces the necessity of pen-up/pen-down switching. We validate our method on two depth sensors, Kinect and Leap Motion Controller and use state-of-the-art classifiers to recognize characters. On a sample dataset collected from 20 users, we report 97.59% accuracy for character recognition. We also demonstrate how this system can be extended for lexicon recognition with 100% accuracy. We have prepared a dataset containing 1,560 characters and 400 words with intention of providing common benchmark for air handwriting character recognition and allied research.



Character samples from Dataset

To evaluate the performance of our approach, we created a dataset, 'Dataset for AIR Handwriting' (DAIR). The dataset is created using 20 subjects, where each user stands straight in front of the sensor and writes in the air with one finger out. Users are allowed to write at their own speed and writing style. Dataset contains two sections DAIR I and DAIR II. DAIR I  consists of 1248 character samples from 16 users by taking 3 samples per character per user. DAIR II consists of words from a lexicon of length 40. Words in the lexicon are taken from the names of most populous cities and vary in length from 3 to 5. It contains 400 words which totals to 1490 characters.

Our datasets can be downloaded from the following links:

Please mail us at {sirnam.swetha@research, rajat.aggarwal@students} for any queries.


final result

Labels represent the predicted and expected characters/words in each pair. Each pair has the input trajectory and trajectory after normalization respectively. (a) samples correctly classified, (b) mis-classified samples, (c) correctly classified words.



Accuracy comparison of characters using Kinect and Leap Motion Controller

Related Publications

Rajat Aggarwal, Sirnam Swetha, Anoop M. Namboodiri, Jayanthi Sivaswamy, C. V. Jawahar - Online Handwriting Recognition using Depth Sensors Proceedings of the 13th IAPR International Conference on Document Analysis and Recognition, 23-26 Aug 2015 Nancy, France. [PDF]

Associated People