Joint Transformer/ RNN Architecture for Gesture Typing in Indic Languages (2020)

Abstract: Gesture typing is a method of typing words on a touch-based keyboard by creating a continuous trace passing through the relevant keys. This work is aimed at developing a keyboard that supports gesture typing in Indic languages. We begin by noting that when dealing with Indic languages, one needs to cater to two different sets of users: (i) users who prefer to type in the native Indic script (Devanagari, Bengali, etc.) and (ii) users who prefer to type in the English script but want the transliterated output in the native script. In both cases, we need a model that takes a trace as input and maps it to the intended word. To enable the development of these models, we create and release two datasets. First, we create a dataset containing keyboard traces for 193,658 words from 7 Indic languages. Second, we curate 104,412 English-Indic transliteration pairs from Wikidata across these languages. Using these datasets we build a model that performs path decoding, transliteration and transliteration correction. Unlike prior approaches, our proposed model does not make co-character independence assumptions during decoding. The overall accuracy of our model across the 7 languages varies from 70-95%.

## introduction gesture typing @xcite is a fast and convenient method for providing textual input to a touchbased keyboard on small-sized devices like mobile phones. the conventional method of prodding the keys to input a word requires several discrete key presses resulting in impeded performance @xcite . besides, touch-based keyboards do not provide the rich tactile sensation of key boundaries and key presses that mitigates erroneous typing on a regular desktop keyboard @xcite . gesture typing allows the user to input a word by drawing a single continuous trace over the keyboard and the finger needs to be lifted only once a word is completed. it takes advantage of muscle memory @xcite which allows the user to subconsciously memorise input shapes for common words, resulting in improved typing speed and accuracy. the advantages of gesture typing over conventional typing methods are of particular significance in indic language keyboards as indic languages have a larger character set than english (for example,the hindi script contains 52 alphabets as opposed to 26 in english), thus making the keyboard denser and increasing the chances of prodding the wrong character. a solution to this is to use an english character keyboard which receives an input sequence of english characters phonetically resembling the indic word and transliterates them into the indic language word @xcite . besides, prior experience with using english keyboards ensures that minimal effort is elicited from the user while switching to the new keyboard. therefore, in this work, we propose solutions to two tasks: (i) english-to-indic decoding to predict an indic word from gesture input provided to an english character keyboard with phonetic correspondence to the intended indic word and (ii) indic-to-indic decoding to predict an indic word from the gesture input provided to an indic character keyboard. to enable development on both these fronts, we release datasets of word traces simulated using the minimum jerk principle @xcite . in this paper, we propose to jointly use a multi-headed transformer encoder layer @xcite and bidirectional lstm layers to encode a sequential stream of (x, y) coordinates obtained from a trace drawn on a touch-based keyboard. we decode the intended character sequence using an lstm decoder to avoid any co-character independence assumptions. we use an lstm encoder-decoder framework with attention for transliterating the decoded sequence into the intended indic language. we also propose a neural network architecture for spelling correction of generated transliterations that uses character embeddings generated using the elmo neural network @xcite to measure proximity between words. unlike prior work, this model requires a smaller training set and allows easy vocabulary augmentation without model re-training. to sum up, our novel contributions include: ## gesture data generation for this work, we curated two datasets foot_0 for performing the indic-to-indic decoding and englishto-indic decoding tasks across the 7 indic languages used in the study. the first dataset contains indic words obtained from the work by @xcite along with corresponding coordinate sequences depicting the path traced on an indic character smartphone keyboard to input the word. the second dataset contains indic-english transliteration pairs scraped from wikidata along with corresponding coordinate sequences for the path traced on an english character keyboard. we use the principle that human motor control tends to maximise smoothness in movement to develop a synthetic path generation technique that models the gesture typing trajectory as a path that minimises the total jerk @xcite . for simulating the path between two characters, a minimum jerk trajectory is first plotted between the character locations. we then add two types of noise to the path. the first type adds noise to the starting and ending positions of the path. the (x, y) coordinates for these positions are sampled from a 2d gaussian distribution centred at the middle of the corresponding key on the keyboard. the second type adds noise to the path between the characters. to perform this, we first sample a coordinate pair (x , y ) from the uniform distribution of points bounded by the x and y coordinates of the starting and ending points of the path. following this, we sample points on a path of minimum jerk passing through the 3 points uniformly over time @xcite . this process is repeated for every pair of adjacent characters in the word to obtain a sequence of coordinates that describes the trace. figure 1(b) shows the trace created for the word budhi using this method. apart from the (x, y) coordinate pair, we augment the input features for each sampled point with the x and y derivatives at the point and a one-hot vector with value 1 at the index i corresponding to the character on which the point lies on the keyboard. ## results and discussion we observe in besides,we observe that the ctc gesture path decoder accuracies are higher for english-to-indic decoding than indic-to-indic decoding. this is because (i) the module uses two additional bidirectional lstm layers that process the outputs of the transformer encoder, resulting in greater representative power and (ii) indic scripts contain a larger character set than english resulting in a denser keyboard with greater chances of error. nevertheless, we find that the final accuracies for the indic-to-indic decoding task after spelling correction using the transliteration correction module are high which highlights the importance of this module. it also means that the character sequences predicted by the ctc gesture path decoder closely match the expected sequence due to which the nearest word from the vocabulary match the expected word. in the following sections, we analyse the performance of certain architectural variants and also perform a detailed error analysis of each module in our architecture. throughout the following study, we present the results on the english-to-indic decoding task, with hindi as the indic language unless otherwise specified. ## conclusion and future work in this work, we have developed an architecture for gesture input decoding on keyboards supporting indic languages and validated our models across 7 languages. we have shown that a joint transformer/rnn architecture that uses the ctc loss function can decode a gesture input with high accuracy. we have also shown that the character-level elmo network can be used to generate spelling-aware word encodings and perform spelling correction efficiently. going further, we wish to extend our work to support visually impaired users and diversify our dataset to a larger set of languages.