Neural Transduction of Letter Position Dyslexia using an Anagram Matrix Representation (2020)

Abstract: Research on analyzing reading patterns of dyslectic children has mainly been driven by classifying dyslexia types offline. We contend that a framework to remedy reading errors inline is more far-reaching and will help to further advance our understanding of this impairment. In this paper, we propose a simple and intuitive neural model to reinstate migrating words that transpire in letter position dyslexia, a visual analysis deficit to the encoding of character order within a word. Introduced by the anagram matrix representation of an input verse, the novelty of our work lies in the expansion from one to a two dimensional context window for training. This warrants words that only differ in the disposition of letters to remain interpreted semantically similar in the embedding space. Subject to the apparent constraints of the self-attention transformer architecture, our model achieved a unigram BLEU score of 40.6 on our reconstructed dataset of the Shakespeare sonnets.

## introduction dyslexia is a reading disorder that is perhaps the most studied of learning disabilities, with an estimated prevalence rate of 5 to 17 percentage points of school-age children in the us @xcite . counter to popular belief, dyslexia is not only tied to the visual analysis system of the brain, but also presents a linguistic problem and hence its relevance to natural language processing (nlp). dyslexia manifests itself in several forms as this work centers on letter position dyslexia (lpd), a selective deficit to encoding the position of a letter within a word while sustaining both letter identification and character binding to words @xcite . a growing body of research advocates heterogeneity of dyslexia causes to poor non-word and irregular-word reading @xcite . along the same lines @xcite suggest that character transposition effects in lpd are most likely caused by a deficit specific to coding the letter position and is evidenced by an interaction between the orthographic and visual analysis stages of reading. to this end, more recently @xcite managed to significantly reduce migration errors by either altering letter contrast or presenting letters to the young adult sequentially. to dyslectic children not all letter positions are equally impaired as medial letters in a word are by far more vulnerable to reading errors compared to the first and last characters of the word @xcite . children with lpd have high migration errors where the transposition of letters in the middle of the word leads to another word, for example, slime-smile or cloud-could. on the other hand, not all reading errors in cases of selective lpd are migratable and are evidenced by words read without a lexical sense e.g., slime-silme. intriguingly, increasing the word length does not elevate the error rate, and moreover, shorter words that have lexical anagrams are prone to a larger proportion of migration errors compared to longer words that possess no-anagram words. a key observation for lpd is that although words read may share all letters in most of the positions, they still remain semantically unrelated. machine learning tools to classify dyslexia use a large corpus of reading errors for training and mainly aim to automate and substitute diagnostic procedures expensively managed by human experts. @xcite used both lda and naive bayes models and showed an area under curve (auc) performance of about 0.8 that exceeded the quality of clinician-rendered labels. in their study, @xcite proposed a statistical model that predicts dyslectic readers using eye tracking measures. employing an svm-based binary classifier, they achieved about 80% accuracy. instead, our approach applies deep learning to the task of restoring lpd inline that we further formulate as a sequence transduction problem. thus, given an input verse that contains shuffled-letter words identified as transpositional errors, the objective of our neural model is to predict the originating unshuffled words. we use language similarity between predicted verses and ground-truth target text-sequences to quantitatively evaluate our model. our main contribution is a concise representation of the input verse that scales up to moderate an exhaustive set of lpd permutable data. ## anagram matrix using a colon notation, we denote an input verse to our model as a text sequence w 1:@xmath0, . . . , w n ) of n words interchangeably with n collections of letters l 1:@xmath1 we generate migrated word patterns synthetically by anchoring the first and last character of each word and randomly permuting the position of the inner letters (l thus given a word with a character length |l (i) |, the number of possible unique transpositions for each word follows next, we extract a migration amplification factor @xmath2:@xmath3, . . . , m k ). word length commonly used in experiments of previous lpd studies averages five letters and ranges from four to seven letters long, hence migrating to feasible 2, 6, 24, and 120 letter substitutions, respectively. we note that words with 1, 2, or 3 letters are held intact and are not migratable. when forty winters shall besiege thy brow wehn fotry wenitrs sahll bseeige thy borw when froty winrtes slhal begseie thy borw wehn fotry wrenits slahl begisee thy borw when forty wtenirs shall begeise thy brow when froty wtneirs shall bigeese thy brow when ftory weinrts sahll bgiesee thy borw wehn frtoy wirtens slhal bisgeee thy borw wehn froty wterins slahl beeisge thy brow when froty wtiners shlal beesgie thy borw wehn frtoy wnetris shall beisege thy borw to address the inherent semantic unrelatedness between transpositioned words, we define a twodimensional migration-verse array in the form of an anagram matrix @xmath4:k ; , . . . , ; m (n) 1:k ] ∈ r k×n , where m (i) are column vectors, [•; •] is columnbound matrix concatenation, and k and n are the transposition and input verse dimensions, respectively. in ## lpd embeddings models for learning word vectors train locally on a one-dimensional context window by scanning the entire corpus @xcite . through evaluation on a word analogy task, these models capture linguistic regularities as linear relationships between word embeddings. @xcite proposed the skip-gram and continuous-bag-of-words (cbow) neural architectures with the objective to predict the context of the target word and the target word given its context, respectively. notably lpd migrating words tend mostly outside the english vocabulary and thus pretrained word embeddings on large corpora are of limited use in our system. foot_1 figure 1 : a two-dimensional context window of size two drawn from outside context cells of an anagram matrix. the center words are shown in gray for both the normal by-row {w t,t-2 , . . . , w t,t+2 } and transposed column-wise {w t-2,t , . . . , w t+2,t } forms of feeding our neural network. while the essence of our task is formalized as verse simplification, mending lpd relies on robust discovery of word similarities along both the migration and verse axes of the anagram matrix. to this extent, we reshape the context window to train word embeddings from one to a two-dimensional array. in figure 1 , we show a bi-dimensional con-text window of size two that is a visible subset drawn from outside context cells of an anagram matrix. learning word vectors for lpd is a two-pass process in our model. first, the context window w feeds our neural network row-by-row for each transpositioned verse, and then follows by iterating migration vectors m (i) in w t as inputs. ## model our task is inspired by recent advances in neural machine translation (nmt). nmt architectures have shown state-of-the-art results in both the form of a powerful sequence model @xcite @xcite and more recently, the cross-attention convs2s @xcite and the self-attention based transformer @xcite networks. given an unintelligible diction of shuffled-letter words, our model aims to output a verse that preserves the semantics of the input, and uses the transformer that outperforms both recurrent and convolutional configurations on many language translation tasks. stacked with several network layers, the transformer architecture only relies on attention mechanisms and entirely dispensing with recurrence @xcite . in figure 2 , we show a synoptic rendition of the transformer. its inputs consist of a source verse with potentially letter-transpositioned words x i , and a ground-truth target verse of words with unshuffled letters y i . the transformer encoder and decoder modules largely operate in parallel and provide for a source-to-target attention communication, and a softmax layer operates on the decoder hidden-state outputs to produce predicted words ŷi . in lpd, source and target verses are consistently of the same word count n, however, copying tokens from the source over to predictions is inconsequen-tial to the quality of repairing reading errors due to extensive out-of-vocabulary non-migrating words. ## setup to quantitatively evaluate our lpd transduction approach, we chose to mainly report n-gram bleu precision @xcite that defines the language similarity between a predicted text sequence and the ground-truth reference verse. in the bleu metric, higher scores indicate better performance. ## results we ran our model inference on a split test set that comprises randomly selected rows sampled from the entire collection of anagram matrices and further excluded from the train set. we postulate that the use of matrix columns along the migration axis are only beneficial for embedding training. in while bleu scores the output by counting ngram matches with the reference, we also evaluated our model using sari @xcite , a novel metric that correlates with human judgments and designed to specifically analyze text simplification models. sari principally compares system output against both the reference and input verse and returns an arithmetic average of n-gram precisions and recalls of addition, copy, and delete rewrite operations. 2 the transformer is known to be bound by a fixedlength context and thus tends to split a long context to segments that often ignore semantic boundaries. this led to the conjecture that context fragmentation may impact our model performance adversely. the novel transformer-xl network @xcite that learns dependencies across subsequences using recurrence, might be the more effective architecture to perform our task. ## discussion to conduct a baseline evaluation of our model, we hand curated a corpus made of lpd screening tests. targeted screeners are brief performance measures intended to classify at-risk individuals. to the extent of our knowledge, @xcite used for their experiments the largest known screener dataset to date that consisted of 196 loose target words in hebrew. correspondingly, we assembled a screening corpus of 196 english words that are prone to erroneous reading. in our system, these words are recast into a set of anagram matrices, each however reduced to a @xmath5. further downstream, we represented context-less words as one-hot vectors. as expected, on the task of reinstating screener data our sequence model achieved a fairly low 1-gram bleu score of 9.2. counter to nearly 4.4x improvement on the sonnets dataset, when trained using a 2d context window. compared to almost two orders of magnitude larger sonnets dataset, the screening corpus was too small and thus overfitting our transformerbased neural model. in addition, to effectively exploit our proposed anagram matrix representation, rather than disjoint words we require to train our sequence model on a dataset comprised of verses or sentences that provides essential context for learning embeddings. in a practical application framework, our proposed model is rated on successful recovery from lpd reading errors that transpire in a text sequence. we envision our model already pretrained on multiple corpora, each extended to a collection of anagram matrices. every editing instance follows with a dyslectic individual who reads and utters a verse at a time from a text document. fed to the network, the verse is then inferred by our model that returns an amended text sequence the user can compare side-by-side on his display. it is key for the system we presented to perform responsively. ## conclusions in this paper, we presented word-level neural sentence simplification to aid letter-position dyslectic children. we modeled the task after a monolingual machine translation and showed the representation effectiveness of a two-dimensional context window to boost our model performance. future avenues of research include using our model in real-world restoration scenarios of lpd, and exploring the efficacy of the transformer-xl architecture to a non language modeling task like ours. we look forward to leverage the exceptional ability of transformer-xl to perform character-level language modeling and improve mending lpd.