Less is Better: A cognitively inspired unsupervised model for language segmentation (2020)

Abstract: Language users process utterances by segmenting them into many cognitive units, which vary in their sizes and linguistic levels. Although we can do such unitization/segmentation easily, its cognitive mechanism is still not clear. This paper proposes an unsupervised model, Less-is-Better (LiB), to simulate the human cognitive process with respect to language unitization/segmentation. LiB follows the principle of least effort and aims to build a lexicon which minimizes the number of unit tokens (alleviating the effort of analysis) and number of unit types (alleviating the effort of storage) at the same time on any given corpus. LiB’s workflow is inspired by empirical cognitive phenomena. The design makes the mechanism of LiB cognitively plausible and the computational requirement light-weight. The lexicon generated by LiB performs the best among different types of lexicons (e.g. ground-truth words) both from an information-theoretical view and a cognitive view, which suggests that the LiB lexicon may be a plausible proxy of the mental lexicon.

## introduction during language comprehension, we cannot always process an utterance instantly. instead, we need to segment all but the shortest pieces of text or speech into smaller chunks. since these chunks are likely the cognitive processing units for language understanding, we call them cognitive units in this paper. a chunk may be any string of letters, characters, or phonemes that occurs in the language, but which chunks serve as the cognitive units? traditional studies @xcite , for example) often use words as the units in sentence analysis. but speech, as well as some writing systems such as chinese, lack a clear word boundary. even for written languages which use spaces as word boundaries, psychological evidence indicates that the morphemes, which are sub-word units, in infrequent or opaque compound words take priority over the whole word @xcite ; at the same time, some supra-word units such as frequent phrases and idioms are also stored in our long-term mental lexicon @xcite @xcite . the evidence suggests that the cognitive units can be of different sizes; they can be words, or smaller than words, or multi-word expressions. despite the flexible size of the cognitive units, and the lack of overt segmentation clues, infants are able to implicitly learn the units in their caregivers' speech, and then generate their own utterances. arguably, children's language intelligence allows them to build their own lexicons from zero knowledge about the basic (cognitive) units in the particular language the child is learning, and then use the lexicon to segment language sequences. can we mimic this ability of a human language learner in a computer model? this question is often phrased as the task of unsupervised segmentation. several types of computational models or nlp algorithms have been proposed for segmentation, taking different approaches: in general, lexicon models capture only the local statistics of the tokens so they tend to be short-sighted at the global level (e.g. long-distance dependencies). the other two types of models, in contrast, learn how the tokens co-occur globally. yet, the ways grammar models and sequence models learn the global information makes them more complicated and computing-intensive than the lexicon models. in this paper we propose a model that builds a lexicon, but does so by using both local and global information. our model is not only a computational model but also a cognitive model: it is inspired by cognitive phenomena, and it needs only basic and light-weight computations which makes it cognitively more plausible than the grammar-and sequence-learning models mentioned above. we show that our model can effectively detect the cognitive units in language with an efficient procedure. we also show that our model can detect linguistically meaningful units. we further evaluate our model on traditional word segmentation tasks. ## model training we trained the lib model on both english and chinese materials ( in order to examine the unsupervised performance of lib, all spaces in the corpora were removed before training. we 4 model evaluation ## conclusions and future work this paper presented an unsupervised model, lib, to simulate the human cognitive process of language unitization/segmentation. following the principles of least effort, larger-first processing, and passive and active forgetting, lib incrementally builds a lexicon which can minimize the number of unit tokens (alleviating the effort of analysis) and unit types (alleviating the effort of storage) at the same time on any given corpus. moreover, it is able to segment the corpus, or any other text in the same language, based on the induced lexicon. the computations in lib are light-weight, which makes it very efficient. the lib-generated lexicon shows optimal performances among different types of lexicons (e.g., ground-truth words) both in terms of description length and in terms of statistical language model surprisal, both of which are associated with cognitive processing. the workflow design and the computation requirement make lib cognitively plausible, and the results suggest that the lib lexicon may be a useful proxy of the mental lexicon. future work will be to allow skip-gram units in the lexicon. skip-grams may help to capture longer-distance dependencies, and further lessen the cognitive effort by reducing the number of unit types/tokens. furthermore, as the word segmentation results of the current lib are not ideal, we hypothesize that skip-gram units may also benefit the detection of infrequent named entities (e.g., the skip-gram "mr. said" helps to detect "mortimer" in "mr.mortimersaid") and thus improve the word segmentation performance. other future work includes a lib variant that accepts speech input and a semi-supervised lib variant that uses semantic knowledge (e.g., word embeddings) to enhance the language unitization.