Constructivist Tokenization for E nglish (2023)

Abstract: This paper revisits tokenization from a theoretical perspective, and argues for the necessity of a constructivist approach to tokenization for semantic parsing and modeling language acquisition. We consider two problems: (1) (semi-) automatically converting existing lexicalist annotations, e.g. those of the Penn TreeBank, into constructivist annotations, and (2) automatic tokenization of raw texts. We demonstrate that (1) a heuristic rule-based constructivist tokenizer is able to yield relatively satisfactory accuracy when gold standard Penn TreeBank part-of-speech tags are available, but that some manual annotations are still necessary to obtain gold standard results, and (2) a neural tokenizer is able to provide accurate automatic constructivist tokenization results from raw character sequences. Our research output also includes a set of high-quality morpheme-tokenized corpora, which enable the training of computational models that more closely align with language comprehension and acquisition.

## introduction although theoretical linguists have been gradually shifting from lexicalism to constructivism, constructivist theories have been barely adapted by computational linguists and psycholinguists. in this paper, we demonstrate the relevance of constructivist approaches to natural language processing (nlp) in the context of tokenization, specifically for english. though constructivist approaches to text segmentation and treebank annotation have been proposed for some languages such as hebrew @xcite and korean @xcite , english tokenization has been viewed as a long-solved problem in nlp. in some nlp tasks, e.g. neural machine translation, it has even been replaced with purely statistics-based approaches, such as byte pair encoding subword tokenization @xcite . we, however, argue that existing tokenization methods are not sufficient for at least two subfields -semantic parsing and modeling language acquisition. we firstly explore the feasibility of (semi-) automatically converting existing lexicalist annotations, such as those in the penn treebank, into constructivist annotations. we demonstrate that simple heuristic rules are able to utilize gold-standard penn treebank part of speech tags to produce high-quality constructivist annotations even without manual cleaning, thus substantially increasing efficiency of the constructivist tokenization and tagging process. through our rule-based algorithm, we are able to automatically produce a set of silver-standard morpheme-tokenized and tagged corpora from the annotated phrase structure trees of the penn treebank (ptb; @xcite and the childes treebank (ctb; @xcite . however, despite the relatively high levels of accuracy of the silver standard corpora, some level of manual annotation is still required to achieve gold standard accuracy. we then study automatic tokenization for raw, unannotated texts. we built a long short-term memory model (lstm; @xcite that was able to produce highly accurate tokenization outputs from raw character sequences even when trained with a large portion of silverstandard data. the high performance of our lstm model is particularly useful in automatically tokenizing texts when previously existing lexicalist annotations are not available. ## neural tokenisation to fully automate tokenization from raw text inputs, we train a lstm model with our manually cleaned gold-standard data as well as large-scale silver-standard data derived from the ptb and ctb phrase structure trees using our rule-based system. our tokenizer is based on character labeling, in which the b(egin), i(nside), and o(utside) labels are used to encode the positional information of each character in an input sentence in regard to its position in its respective token. experiments indicate that lstm, together with our data, are effective in building a high performing constructivist tokenizer, which obtained an average accuracy of over 99%. ## conclusion and future work this project demonstrated that automatic constructivist tokenization is feasible and can achieve high levels of accuracy, despite the complexity when going beyond words to morphemes. although one might still need to manually clean resulting corpora to achieve gold standard accuracy, our rule-based tokenizer is able to substantially increase the efficiency of producing constructivist corpora. we also demonstrated that the use of deep learning, such as lstm models, can be a promising means of building a tokenizer. it is particularly useful in situations where the complexities / irregularity of a language become too difficult or cumbersome to codify into a rule based algorithm. in future research, it would be interesting to explore how these two types of constructivist tokenizers perform in more complex, morpheme-heavy languages, such as turkish. another area of potential future research is to explore ways to enrich the corpora we produced in this project by adding syntactic and semantic annotations. the new corpora we produced will enable the next phase of research of building computational language acquisition models based on constructivism. our corpora will also allow future research in developing new natural language understanding systems.