Building a Linguistic Resource : A Word Frequency List for S inhala (2021)

Abstract: A word frequency list is a list of unique words in a language along with their frequency count. It is generally sorted by frequency. Such a list is essential for many NLP tasks, including building language models, POS taggers, spelling checkers, word separation guides, etc., in addition to assisting language learners. Such lists are available for many languages, but a large-scale word list is still not available for Sinhala. We have developed a comprehensive list of words, together with their frequency and part-of-speech (POS), from a large textbase. Unlike many other such lists, our list includes a large number of low-frequency words (many of which are erroneous), which enables the analysis of such words, including the frequencies of errors. In addition to the main list, we have also prepared a list of linguistically verified words. The word frequency list and the verified word list are the largest collections of words lists that are available for the Sinhala language.

## introduction word frequency lists are useful for analysing the vocabulary of a language @xcite . they generally comprise a list of lexical words sorted by frequency of occurrence in a corpus, as a ranked list. such lists can be used to build a directory, for language learning and for teaching. they are also useful in downstream applications such as part-of-speech (pos) tagging, syntactic parsing, machine translation, speech processing etc. in this paper, first, we present a word frequency list for sinhala. we compiled a list of 2 million unique words by considering published documents from multiple sources, representing diverse domains. the raw corpus contained 127 million word tokens. in addition to frequency, the list provides the pos tag of each word. second, we have compiled a linguistically verified word list of over 280 thousand unique words. these two word lists are currently the largest available word lists for the sinhala language. therefore, they would be very useful linguistic resources for many downstream applications. further, both the word frequency list and verified word list are publicly released foot_0 , for the progression of future research. ## related work word frequency lists have long been in use for languages such as spanish, french, german and english. before the advent of computing, they were generated manually, but have become much easier to generate using a computer @xcite @xcite . the main objectives of many word frequency lists based on written or spoken corpora @xcite were for language teaching and learning. however, word lists are valuable resources for computational linguistics as well. they provide annotated datasets for downstream applications such as pos taggers @xcite , dependency parsers @xcite etc. more recently, frequency lists were generated for non-latin languages such as tamil (kumar, 2019) and russian @xcite . frequency lists were also generated for specific domains such as literary genres and historical periods, and even individual books and authors. for sinhala, a corpus based lexicon were done by @xcite . this contains 35k unique words. they have constructed a word frequency list extracted from publicly available sinhala text documents from multiple domains. however, the list does not appear to be publicly accessible. more recently, google has released a sinhala lexicon with about 42,000 entries @xcite . text documents were crawled from the web at scale @xcite for the progress of research in diverse domains. such crawled corpora are also available for sinhala. to the best of our knowledge, the generated word frequency lists had not considered these recently web crawled data. ## conclusion and future work the word frequency list, comprising over 2 million words, is by far the largest word list for the sinhala language. it is also the only one supplemented with pos information. although the list contains many incorrect words and incorrect word separations, this is a feature, not a drawback, as it allows us to analyse the frequencies of variant spellings and compound words. lemma total words ('000) sample words කර 1,674 කර, කරන, කරමින්... ම 1,124 මම, මට, මා, මමත්, මටත්... ඒ 1,072 ඒ, ඒත්, ඒයි, ඒය, ඒවල... ෙම් 1,067 ෙම්, ෙම්ට, ෙම්ෙග්, ෙම්ටත්... ව 902 වූ, වන්ෙන්, ෙවයි, ෙනොවන... අප 865 අපි, අෙප්, අප, අපිට... ය 738 ය, ගිය, ගිහින්, ගියා, යයි... එක 723 එක, එෙක්, එෙකන්, එකත්... බව 649 බව, බවයි, බැවිනි, බැවිණි... නම 625 නම්, නමුත්, නම, නමක්... as future work, we plan to compile a frequency lists of morphosyntactic suffixes for sinhala. we also plan to compile lists of word bi-grams, trigrams, etc. we also plan to compile domain-specific word lists for government documents, news, textbooks, web, etc. these lists have been used to develop several other tools, including a spelling checker and a sentence generator, and are being used to identify spelling error patterns, etc. it will be a useful tool in many other areas of nlp.