Look Ma, Only 400 Samples! Revisiting the Effectiveness of Automatic N-Gram Rule Generation for Spelling Normalization in F ilipino (2022)

Abstract: With 84.75 million Filipinos online, the ability for models to process online text is crucial for developing Filipino NLP applications. To this end, spelling correction is a crucial preprocessing step for downstream processing. However, the lack of data prevents the use of language models for this task. In this paper, we propose an N-Gram + Damerau-Levenshtein distance model with automatic rule extraction. We train the model on 300 samples, and show that despite limited training data, it achieves good performance and outperforms other deep learning approaches in terms of accuracy and edit distance. Moreover, the model (1) requires little compute power, (2) trains in little time, thus allowing for retraining, and (3) is easily interpretable, allowing for direct troubleshooting, highlighting the success of traditional approaches over more complex deep learning models in settings where data is unavailable.

## introduction filipinos are among the most active social media users worldwide @xcite . in 2022, roughly 84.75m filipinos were online @xcite , with 96.2% on facebook @xcite . hence, developing language models that can process online text is crucial for filipino nlp applications. contractions and abbreviations are common in such online text @xcite . for example, dito (here) can be written as d2, or nakakatawa (funny) as nkktawa, which are abbreviated based on their pronunciation. however, language models like google translate remain limited in their ability to detect and correct such words, as we find later in the paper. hence, we aim to improve the spelling correction ability of such models. in this paper, we demonstrate the effectiveness of a simple n-gram based algorithm for this task, inspired by prior work on automatic rule generation by @xcite . specifically, we (1) create a training dataset of 300 examples, (2) automatically generate n-gram based spelling rules using the dataset, and (3) use the rules to propose and select candidates. we then demonstrate that this model outperforms seq-to-seq approaches. ultimately, the paper aims to highlight the use of traditional approaches in areas where sota language models are difficult to apply due to limitations in data availability. such approaches have the added benefit of (1) requiring little compute power for training and inference, (2) training in very little time (allowing for frequent retraining), and (3) giving researchers full clarity over its inner workings, thereby improving the ease of troubleshooting. ## related work the problem of online text spelling correction is most closely related to spelling normalization -the subtask of reverting shortcuts and abbreviations into their original form @xcite . in this paper, we will use correcting to mean normalizing a word. this is useful for low-resource languages like filipino, wherein spelling is often not standardized across its users @xcite . many approaches have been tried for word normalization in online filipino text: (1) predetermined rules using commonly seen patterns @xcite , (2) dictionary-substitution models for extracting patterns in misspelled words @xcite , or (3) trigrams and levenshtein or qwerty distance to select words which share similar trigrams or are close in terms of edit or keyboard distance @xcite . each method has its limitations which we seek to address. predetermined rules must be manually updated to learn emerging patterns, as is common in the constantly evolving vocabulary of online filipino text (salvacion and limpot, 2022; lumabi, 2020). dictionary-substitution models are limited by the constraint of picking mapping each pattern to only a single substitution, whereas in reality, dif-ferent patterns may need to be applied to different words bearing the same pattern @xcite . trigrams and distance metrics alone may be successful in the context of correcting typographical errors for which the model was developed @xcite , but may not be as successful on intentionally abbreviated words. our work uses a combination of these methods to develop a model that can be easily updated, considers multiple possible candidates, and works in the online text setting. the task is further complicated by the lack of data, which hinders the use of large pretrained language models. previous supervised modeling approaches require thousands of labeled examples @xcite , and even unsupervised approaches for similar problems required vocabulary lists containing the desired words for translation @xcite . since such datasets are not available, our paper revisits simpler models, and finds that they exhibit comparable performance to that of much larger sota models. ## data we use a dataset consisting of facebook comments made on weather advisories of a philippine weather bureau in 2014. we identified 403 distinct abbreviated and contracted words, and had three filipino undergraduate volunteers write their "correct" versions. to maximize the data, we removed hyphens and standardized spacing, then filtered out candidates where all annotators gave different answers. we obtained 398 examples (98.7%) with 83.8% inter-annotator agreement. we then created a 298-100 train-test split; we selected test examples that used spelling rules present in the training set to test the ability of our n-gram model to extract and apply such rules. to test generalizability, we also perform cross-validation. the data and code for our experiments are available at the following repository. foot_0 4 model automatic rule generation we extract spelling rules from pairs (w, c), where w is a misspelled word, and c is its corrected version. the rule generation algorithm slides a window of length k over w and c, and records w[i : i + k] → c[j : j + k] as a rule (i, j are pointers); it returns a dictionary mapping each substring to a list of "correct" substrings (see appendix 1 for algorithm and example). we test substrings of length 1 to 4, and find that lengths 1 / 2 work best. this makes sense as many filipino words are abbreviated by syllable, which typically have 1-2 letters. this is similar to indonesian @xcite and malay @xcite , suggesting possible extensions. we further filter candidates to words present in a filipino vocabulary list developed by gensaya (2018) (mit license), except for when none of the candidates exist in the vocabulary list, in which case we use all the generated words as candidates. candidate generation we recursively generate candidates by replacing each substring with all possible rules in the rule dictionary. if the substring is not present, we keep the substring as is. an example can be found in appendix d. we find that rules involving single letter substrings often occur at the end of a word. hence, we test candidate generation algorithms which either allow single letter rules to be used anywhere when generating (v1), or only for the last letter of a word (v2). we also vary the # of candidates kept at each generation step (ranked by likelihood, see eq 2). ranking candidates we explore two ways of ranking candidates: (1) damerau-levenshtein distance we rank candidates based on their edit distance from the misspelled word using the pyxdameraulevenshtein foot_1 package with standard settings, and (2) likelihood score we compute the likelihood of the output word c given misspelled word w as the product of probability the rules used to generate it, where the probability of a rule is the number of occurrences of @xmath0, 2). ## conclusion in this study, we propose an n-gram + dld model for spelling normalization of filipino online text, and compare it to deep learning benchmarks. the n-gram + dld v1 model achieves the best accuracy and best-case dld, with a 32% improvement in accuracy @ 1 over the next best model (dld). this shows the potential of simpler techniques, especially when data is scarce. besides improved performance, the n-gram + dld model requires little compute power and memory for training and inference. this allows for frequent retraining of the model and addition of new spelling rules as new words emerge. the model also allows researchers to understand how predictions are made, and make appropriate tweaks to the spelling rules, candidate sorting method, or hyperparameters used (e.g. length of substrings). this work has limitations which suggest areas for improvement. first, the current work uses a small dataset limited to the weather domain. using more diverse datasets can improve the comprehensiveness of the rule dictionary. also, more complete dictionaries containing filipino words and their conjugations can help filter down valid candidates before running dld. second, the candidate ranking method can be improved, especially in cases where the target and selected words are similar, as discussed in the section 6.3. for example, words can be ranked by how common they are, or by inferring the correct choice from the context. this has the added benefit of reducing the candidate pool, requiring fewer dld calculations and hence reducing inference time. finally, we only explore correcting misspelled words; combining it with misspelling detection can further boost the practical applications of this work. ultimately, the development of such models will pave the way for improvements in filipino nlp, and enable the development of more applications that can serve the wider online filipino community. linting xue, aditya barua, noah constant, rami al-rfou, sharan narang, mihir kale, adam roberts, and colin raffel. 2022. byt5: towards a token-free future with pre-trained byte-to-byte models. transactions of the association for computational linguistics, 10:291-306.