Incorporating tone in the calculation of phonotactic probability (2021)

Abstract: This paper investigates how the ordering of tone relative to the segmental string influences the calculation of phonotactic probability. Trigram and recurrent neural network models were trained on syllable lexicons of four Asian syllable-tone languages (Mandarin, Thai, Vietnamese, and Cantonese) in which tone was treated as a segment occurring in different positions in the string. For trigram models, the optimal permutation interacted with language, while neural network models were relatively unaffected by tone position in all languages. In addition to providing a baseline for future evaluation, these results suggest that phonotactic probability is robust to choices of how tone is ordered with respect to other elements in the syllable.

## introduction the phonotactic probability of a string is an important quantity in several areas of linguistic research, including language acquisition, wordlikeness, word segmentation, and speech production and perception @xcite @xcite . when the language of interest is a tone language, the question arises of how tone should be incorporated into the probability calculation. as phonotactic probability is frequently computed based on some type of ngram model, this means deciding on which segment(s) the probability of a tone should be conditioned. for instance, using a bigram model, one might compute the probability of the mandarin syllable fāng as p (a|f) × p (n|a) × p (tone 1|n), but could just as well consider p (tone 1|f) × p (a|1) × p (n|a), or any other conceivable permutation of tone and segments. while this issue is occasionally remarked on (e.g. @xcite , there remains no widespread consensus in practice. choice of ordering is sometimes justified based on segment-tone co-occurrence restrictions in the language under study @xcite , but is often presented without justification @xcite , and in some cases tone is simply ignored @xcite . when the space of possibilities is considered, researchers generally select the permutation which maximizes model fit to some external data, such as participant judgments of phonological distance @xcite or wordlikeness @xcite . although extrinsic evaluation is in some sense a gold standard, intrinsic metrics of model fit can also be informative, in part because extrinsic metrics are not always robust across data sets. for instance, participant wordlikeness judgments can vary considerably based on the particulars of the experimental design @xcite @xcite , so the treatment of tone that produces a best-fit model for one dataset may not do so for another. the lexicon of a given language is much more internally stable in terms of how segments and tones are distributed, so intrinsic evaluation may provide a useful baseline for reasoning about the treatment of tone relative to segments both within and across languages. this short paper considers a simple information-theoretic motivation for selecting a permutation: all else being equal, we should prefer a model that maximizes the probability of the lexicon (i.e., minimizes the cross-entropy loss), because this will be the model that by definition does the best job of capturing the phonotactic regularities of the lexicon @xcite @xcite . by treating tone as another phone in the segmental string, we can see whether and to what degree this choice has an effect on the overall entropy of the lexicon. intuitively, any model that can take into account phonotactic constraints will result in a reduction in entropy. thus, even an n-gram model with a sufficiently large context window should in principle be able model segment-tone co-occurrences at the syllable level. however, tone languages differ with respect to tonesegment co-occurrence restrictions (see sec. 2). if a relevant constraint primarily targets syllable onsets, for instance, placing the tonal "segment" in immediate proximity to the onset will increase the probability of the string, even relative to a model capable of capturing the dependency at a longer distance. in all cases, the lexicon was defined as the set of unique syllable shapes in each language. for consistency, the syllable template in all four languages is considered to be (c 1 )(c 2 )v(c)t, with variable positioning of t. offglides were treated as codas in all languages. the syllable lexicons for all four languages are provided in the supplementary materials ( http://doi.org/10.17605/osf.io/na5fb ). ## methods two classes of character-level language models (lms) were considered: simple n-gram models and recurrent neural networks @xcite . in an n-gram model, the probability of a string is proportional to the conditional probabilities of the component n-grams: the degree of context taken into account is thus determined by the value chosen for n. in a recurrent neural network (rnn), the next character in a sequence is predicting using the current character and the previous hidden state. at each step t, the network retrieves an embedding for the current input x t and combines it with the hidden layer from the previous step to compute a new hidden layer h t : the incorporation of the recurrent connection as part of the hidden layer allows rnns to avoid the problem of limited context inherent in n-gram models, because the hidden state embodies (some type of) information about all of the preceding characters in the string. although rnns cannot capture arbitrarily longdistance dependencies, this is unlikely to make a difference for the relatively short distances involved in phonotactic modeling. trigram models were built using the srilm toolkit @xcite , with maximum likelihood estimates smoothed using interpolated witten-bell discounting @xcite . rnn lms were built using pytorch @xcite , based on an implementation by @xcite . the results reported here make use of simple recurrent networks @xcite , but similar results were obtained using an lstm layer @xcite . ## results for brevity, only the main findings are summarized here; the full results are available as part of the online supplementary materials ( http://doi.org/10.17605/osf.io/na5fb ). differences between orderings were then assessed visually, aided by simple analyses of variance. for the trigram lms, perplexity was lowest in mandarin when tones followed codas, while differences in perplexity between other orderings were negligible. for thai, vietnamese, and cantonese, all orderings were roughly comparable except for when tone was ordered as the first segment in the syllable (t|#), which increased perplexity by up to 1 over the mean of the other orderings. for thai, the ordering t|m resulted in significantly lower perplexities compared to all other ## discussion consistent with other recent work in computational phonotactics (e.g. @xcite @xcite , the neural network models outperformed the trigram baselines by a considerable margin (a reduction in average perplexity of up to 2.5, depending on language). neural network models were also much less sensitive to the linear position of tone relative to other elements in the segmental string (cf. @xcite , no doubt due to the fact that the ability of the rnns to model cooccurrence tendencies within the syllable is not constrained by context in the way that n-gram models are. perhaps as a result, however, the rnn models reveal little about the nature of segmenttone co-occurrence restrictions in any of the languages investigated. in this regard, the trigram models, while clearly less optimal in a global sense, are still informative. the fact that the ordering t|# was significantly worse under the trigram model for cantonese, vietnamese and thai but not mandarin can be explained (or predicted) by the fact that of the four languages, only mandarin does not permit obstruent codas, and consequently has no coda-tone co-occurrence restrictions (indeed, the four primary tones of mandarin occur with more or less equal type frequency). in the other three languages, syllables with obstruent codas can only bear a restricted set of tones, and in a trigram model, this dependency is not modeled when tone is prepended to the syllable, since this means it will frequently, though not always, fall outside the window visible to the language model. even a model with a large enough context window to capture such dependencies will assign the lexicon a higher perplexity when structured in this way. the finding that the t|m ordering is always optimal in thai (and by a larger margin than in the other languages) is presumably due to the fact that the distribution of the medials /w l r/ is severely restricted in this language, occurring only after /p p h t t h k k h f/. the distribution of tones after onset-medial clusters is inherently more constrained and therefore more predictable. a similar restriction holds in cantonese, albeit to a lesser degree (the medial /w/ only occurs with onsets /k/ and /k h /). ## conclusion this paper has assessed how different permutations of tone and segments affects the perplexity of the lexicon in four syllable-tone languages using two types of phonotactic language models, an interpolated trigram model and a simple recurrent neural network. the perplexities assigned by the neural network models were essentially unaffected by different choices of ordering; while the trigram model was more sensitive to permutations of tone and segments, the effects on perplexity remained minimal. in addition to providing a baseline for future evaluation, these results suggest that the phonotactic probability of a syllable is relatively robust to choice of how tone is ordered with respect to other elements in the string, especially when using a model capable of encoding dependencies across the entire syllable.