Ancestor-to-Creole Transfer is Not a Walk in the Park (2022)

Abstract: We aim to learn language models for Creole languages for which large volumes of data are not readily available, and therefore explore the potential transfer from ancestor languages (the ‘Ancestry Transfer Hypothesis’). We find that standard transfer methods do not facilitate ancestry transfer. Surprisingly, different from other non-Creole languages, a very distinct two-phase pattern emerges for Creoles: As our training losses plateau, and language models begin to overfit on their source languages, perplexity on the Creoles drop. We explore if this compression phase can lead to practically useful language models (the ‘Ancestry Bottleneck Hypothesis’), but also falsify this. Moreover, we show that Creoles even exhibit this two-phase pattern even when training on random, unrelated languages. Thus Creoles seem to be typological outliers and we speculate whether there is a link between the two observations.

## introduction creole languages refer to vernacular languages, many of which developed in colonial plantation settlements in the 17th and 18th centuries. creoles most often emerged as a result of contact between social groups that spoke mutually unintelligible languages, i.e., from the interactions of speakers of nonstandard varieties of european languages and speakers of non-european languages @xcite . some argue these languages have an exceptional status among the world's languages @xcite , while others counter that creoles are not unique, and evolve in the typical manner as other languages @xcite . in this paper, we will present experiments in evaluating language models trained on non-creole languages for creoles, as well as in various control settings. we first explore the following hypothesis: r1: language models trained on ancestor languages should transfer well to creole languages. we call r1 the 'ancestry transfer hypothesis.' our experiments, however, suggest that r1 is not easily validated. we note, though, that ancestor-to-creole training exhibits divergent behavior when training for long, leading to the following hypothesis: r2: language models trained on ancestor languages can, after a compression phase, transfer well to creole languages. we call r2 the 'ancestry bottleneck hypothesis.' while compression benefits transfer, performance never seems to reach useful levels. furthermore, similar effects are observed with creoles when training on non-ancestor languages. our findings here are not relevant to applied nlp, but they shed light on cross-lingual training dynamics @xcite , and we believe they have potential implications for the linguistic study of creoles @xcite , as well as for information bottleneck theory @xcite . our contributions we conduct a large set of experiments on cross-lingual zero-shot applications of language models to creoles, primarily to test whether ancestor languages provide useful training data for creoles (the 'ancestry transfer hypothesis;' r1). our results are a mix of negative and positive results: first negative result: ordinary transfer methods do not enable ancestor-to-creole transfer. first positive result: regardless of the linguistic theories of creole creolists have long debated whether creole languages have an exceptional status among the world's languages @xcite . @xcite argue that creoles are simpler than other languages, and defined by minimal usage of inflectional morphology, little or no use of tone encoding lexical or syntactic contrasts, and generally semantically transparent derivation. others have argued that creoles cannot be be unambiguously distinguished from non-creoles on strictly structural, synchronic grounds @xcite . on this view creole grammars do not form a separate typological class, but exhibit many similarities with the grammars of their parent languages, e.g., the similarities in lexical case morphology between french and haitian creole. we do not take sides in this debate, but observe that the exceptionalist position would explain our results that zero-shot transfer to creole languages is particularly difficult. exceptionalism also aligns well with the heatmaps presented in §5. information bottleneck the information bottleneck principle @xcite is an information-theoretic framework for extracting output-relevant representations of inputs, i.e., compressed, non-parametric and model-independent representations that are as informative as possible about the output. compression is formalized by mutual information with input. a lagrange multiplier controls the trade-off between these two quantities (informativity and compression). being able to compute this trade-off assumes the joint input-output distribution is accessible. the tradeoff is found by ignoring task-irrelevant factors and learning an invariant representation. the intuition behind the 'ancestry bottleneck hypothesis' (r2) is that invariant representations are particularly useful for creoles (see figure 1 for an illustration). ## multilingual training this section sets out to evaluate the 'ancestry transfer hypothesis' (r1). to this end, we evaluate multilingual language models -trained with a bert architecture from scratch, but of smaller size and with less data (dufter and schütze, 2020)on creoles such as nigerian pidgin or haitian creole. we compare two scenarios: 1) a scenario in which the training languages are languages that are said to be parent or ancestor languages of the creole, such as french to haitian, and 2) a scenario in which random, unrelated training languages were selected. to compare against creoles, we also explore these transfer scenarios for two target non-creoles -spanish and danish -training on languages closely related to them (i.e., as typically done in cross-lingual learning). ## training for longer it seems linguistically plausible that training for longer on ancestor languages to learn more invariant representations should better facilitate zero-shot transfer to creole languages. this is the essence of the 'ancestry bottleneck hypothesis' (r2), which we explore in this section. ## creoles through the lens of wals we have observed unique patterns for creoles. namely, multilingual learning of the related languages did not lead to successful transfer to creoles; and that creoles exhibit a unique compression effect. here, we speculate whether there is a link between these observations, and investigate whether typological features can shed lights into our results. to that effect, we use the world atlas of language structures (wals) foot_3 , which has been used to study creoles before @xcite . here, we use the cosine distance between the normalized (full) wals feature vectors as our distance metric. 5 in figure 5 , we present an example heatmap for nigerian pidgin, which shows that nigerian pidgin is less related to ancestor and random languages than any of them internally (except quechua and cherokee). we found this pattern present for each of the creoles. thus, it would seem that creoles' relatively large distance foot_5 from other languages may make cross-lingual transfer a particular challenge for learning creoles. foot_6 ## conclusion we have presented two hypotheses (r1 and r2) about the possibility of zero-shot transfer to creoles, both built on the idea that creoles share characteristics with their ancestor languages. this is not exactly equivalent to the so-called superstratist view of creole genesis, which maintains that creoles are essentially regional varieties of their european ancestor languages, but if the superstratist view was correct, r1 would very likely be easily validated @xcite . our results show the opposite trend, however. zero-shot transfer to creole languages from their ancestor languages is hard. we do not claim that our results favor an exceptionalist position on creoles. while we performed a first analysis of several segmentation approaches (i.e., bert word piece, grapheme-tophoneme, and byte-pair encodings) -which did not change the training dynamics -we believe that a rigorous comparison would be beneficial for future work in ancestor-to-creole transfer. we hope that continued investigation in this direction can shed more light on cross-lingual transfer, especially with regards to creoles, and that this work has demonstrated that not all transfer between related languages is trivial. we would like to thank the reviewers for their feedback on this manuscript. this project has received funding from the european union's horizon 2020 research and innovation programme under the marie skłodowska-curie grant agreement no 801199 (for heather lent and emanuele bugliarello) and the google research award (for heather @xcite .