Neural Borrowing Detection with Monolingual Lexical Models (2021)

Abstract: Identification of lexical borrowings, transfer of words between languages, is an essential practice of historical linguistics and a vital tool in analysis of language contact and cultural events in general. We seek to improve tools for automatic detection of lexical borrowings, focusing here on detecting borrowed words from monolingual wordlists. Starting with a recurrent neural lexical language model and competing entropies approach, we incorporate a more current Transformer based lexical model. From there we experiment with several different models and approaches including a lexical donor model with augmented wordlist. The Transformer model reduces execution time and minimally improves borrowing detection. The augmented donor model shows some promise. A substantive change in approach or model is needed to make significant gains in identification of lexical borrowings.

## introduction when words shows striking similarity between languages, they may be inherited from a common ancestor, borrowed from one contact language into the other, or less frequently, similar by chance or common creative process, e.g., onomatopoeia. identification of lexical borrowings, transfer of words between languages, is an integral step in the comparative method of historical linguistics @xcite and a vital tool for investigations of language contact, and cultural influences. automated or assisted detection of lexical borrowings is one of many important and difficult tasks remaining in historical linguistics @xcite . we use neural network lexical language models of the phonology and phonotactics of a language to identify lexical borrowings. while language models refer generally to "any system trained only on the task of string prediction" @xcite , our lexical language models operate on monolingual wordlists limited to common or essential concepts, with vocabularies limited to language sound segments (phonemes, allophones). a fixed vocabulary of typically less than 100 coded sound segments eliminates the out-of-vocabulary problem, and a limited wordlist size of at most a few thousand lexical entries, reduces the practical model complexity. we use the recurrent neural network model and competing lexical entropies approach of @xcite as a jumping off point, and incorporate a light-weight forward direction transformer model from @xcite into this approach. we demonstrate our light-weight transformer model over 41 different languages from the world loan database @xcite and contrast performance versus the recurrent model of @xcite . we perform several experiments expanding both the technical and application scope of our lightweight transformer model: 1. develop a direct model approach that discriminates between inherited and borrowed words directly without the need to model inherited and borrowed word entropies separately, 2. enhance competing entropies and direct approaches to account for borrowed words by individual donor language, 3. train multiple borrowed and inherited models, and base decisions on majority vote of model combinations, 4. perform search for best model parameters using experimental sequences of randomly assigned parameter values, 5. explore a limited multi-lingual approach of using an additional donor language table to supplement borrowed words from the recipient language. results are evaluated for borrowing detection and, secondarily, execution time. our light-weight transformer minimally improves borrowing detection and reduces execution time versus the baseline. direct detection of borrowed words performs less well than the competing entropies approach. multiple model combinations and parameter search show little improvement over the transformer model, each with increased execution time. adding a donor language wordlist minimally improves borrowing detection. our contributions are: ## related work when words are borrowed into a language, initially, they may retain the phonology, phonotactics, or other features of the donor language. but over time the borrowed words will adapt to the recipient language -making borrowing detection more difficult @xcite . our monolingual wordlist approach gambles that some remnant of the phonology or phonotactics of the donor language will still be perceptible long after the word is borrowed, and that competing entropies lexical models can be used to detect borrowed words. this is the approach taken by @xcite where curated monolingual wordlists of inherited and borrowed words are modeled with markov chain and recurrent neural network lexical models, and then inherited versus borrowed word decisions are made based on which model estimates the minimum entropy. the approach is modestly successful, yet not sufficient as a tool to determine lexical borrowing. professional linguists also take into account substantial multilingual and cross-linguistic lexical and contextual information. an ambitious and complex borrowed word detection approach by @xcite identifies borrowed words by uyghur from turkish, arabic, russian and chinese donors. word embeddings are calculated using monolingual language models from corpora of uyghur and donor languages, and then cross-lingual embeddings are constructed between uyghur and donor languages with the help of bilingual dictionaries. the cross-lingual embeddings are used to identify candidate borrowed words between uyghur and donor languages. candidate words are tested as borrowed or inherited based on sound similarity and other language features. this approach is not limited to curated wordlists indexed by concepts. future research planning will consider @xcite . @xcite and @xcite repurpose lexical cognate identification methods from multilingual wordlists to the task of identifying borrowed words on a newly developed dataset of south-east asian languages. cognate detection, by searching for full or partial cognates based on sound similarity, is applied to identify language family internal cognate sets, and then "cognate sets are compared across language families and clustered into sets of potentially borrowed words" @xcite . we hope to integrate this approach into future research. another approach to borrowed word detection is to construct phylogenetic models of language families based on wordlists, including also intruder languages that are not necessarily part of the language family. observed discrepancies in the model, in particular lexical items that detract from hierarchical family relations and contribute instead to lateral transfers, are likely due to borrowed words @xcite . ## discussion the purpose of this study was to develop and improve tools for automatic or assisted identification of lexical borrowings in monolingual wordlists. we replicated the recurrent lexical model and competing entropies approach of @xcite for detecting borrowed words, producing similar detection results and capturing a baseline execution time. we developed a light-weight transformer model @xcite and observed that it performed minimally better than par versus the recurrent model in borrowed word detection, and was more responsive with reduced execution times. the light-weight transformer offers a viable base for exploring different lexical borrowing detection approaches. ## conclusion we constructed an effective and responsive lightweight transformer model as a lexical language model for use in the competing entropies approach for detection of lexical borrowings. detection performance was on par to minimally better than that using a recurrent neural network model; execution times were reduced. a direct detection model using a light-weight transformer showed poorer detection of lexical borrowings than the competing entropy models approach. entropy seems to capture important evidence about lexical items that our direct model does not. incorporation of language donor into detection models performed on par with the the corresponding borrowed word models. lack of sufficient data detracts more from the model than within language coherence contributes. majority vote over the cross product of inherited and borrowed word models, and parameter search via random experimental trial design, offer no benefit. this vouches for the relative consistency of model results for current parameter ranges and model architecture. addition of a donor language wordlist, as though borrowed words, reduced recall and increased precision. with better simulated borrowed words, rather than simple adoption of the donor wordlist, recall and precision should improve. this offers a promising avenue of investigation.