ERAS : Evaluating the Robustness of C hinese NLP Models to Morphological Garden Path Errors (2025)

Abstract: In languages without orthographic word boundaries, NLP models perform _word segmentation_, either as an explicit preprocessing step or as an implicit step in an end-to-end computation. This paper shows that Chinese NLP models are vulnerable to _morphological garden path errors_—errors caused by a failure to resolve local word segmentation ambiguities using sentence-level morphosyntactic context. We propose a benchmark, _ERAS_, that tests a model’s vulnerability to morphological garden path errors by comparing its behavior on sentences with and without local segmentation ambiguities. Using ERAS, we show that word segmentation models make morphological garden path errors on locally ambiguous sentences, but do not make equivalent errors on unambiguous sentences. We further show that sentiment analysis models with character-level tokenization make implicit garden path errors, even without an explicit word segmentation step in the pipeline. Our results indicate that models’ segmentation of Chinese text often fails to account for morphosyntactic context.

## introduction when working with languages that do not mark word boundaries in their writing systems, the task of word segmentation, where texts are parsed into word-level tokens, is non-trivial. nonetheless, in high-resource languages like chinese, the state of the art in word segmentation is strong, with f1 scores close to 100% for common benchmarks @xcite @xcite . word segmentation is subject to structural ambiguity: there may be more than one way to segment a text into words or morphemes. in english, for example, the word unlockable can be understood to mean 'unable to be locked' (un + lockable), or 'able to be unlocked' (unlock + able). in code for our experiments is available at: https:// github.com/winglicopy/eras grammatically valid segmentation 学生 留 心机 处理 友人 the student with caution treats his friends 'the student treats his friends with caution.' ungrammatical segmentation 学生 留心 机 处理 友人 the student takes the dealing his friends note of machine with figure 1: morphological garden paths involve local ambiguity at the level of word segmentation. in the sentence above, the bigram 留心 constitutes a valid word, but segmenting it as a word renders the sentence unparsable. a valid parse is obtained if the character 心 forms a word with the following character, 机. this paper investigates whether chinese nlp models correctly resolve local structural ambiguities caused by morphological garden paths. if they do not, then we expect models to make garden path errors by parsing locally ambiguous substrings incorrectly. to that end, we propose the evaluation of robustness to locally ambiguous segmentation (eras) benchmark, a synthetic test set consisting of locally ambiguous test sentences paired with unambiguous but otherwise identical control sentences. as illustrated in figure 2 , each test sentence contains a canary word that can only appear in a segmentation of the sentence if a locally ambiguous substring is parsed incorrectly. eras can be used in two ways. first, eras can evaluate word segmentation models by comparing the models' segmentations for test and control sentences. if a canary word is parsed as a word, but the equivalent segmentation error is not made for the control sentence, then we assume that a garden path error has been committed. eras can also evaluate sentiment analysis models that use a byte-or character-level tokenization scheme. using a lexicon containing human-annotated word-level sentiment ratings @xcite , we ensure that canary words carry a sentiment polarity that is incongruent with the rest of the sentence. if the model's predicted sentiment for the test sentence is influenced by the sentiment of the canary word, then we assume that an implicit garden path error has been committed. this paper conducts both types of evaluations using eras. we first show that transformer-based @xcite and nonneural word segmentation models make garden path errors, systemically parsing canary words as words (section 3). we show that these errors are mostly caused by a preference for resolving garden paths in a greedy left-to-right manner. we then show that transformer-based sentiment analysis models with a character-level tokenization scheme make implicit garden path errors on eras (section 4). these errors can be mitigated by introducing word boundary information to the model during pre-training or fine-tuning. our findings point to morphological garden paths as a source of error for chinese nlp models, especially for models that do not receive any explicit word boundary information during pre-training or fine-tuning. contributions. (1) we provide a benchmark, eras, for determining whether a chinese nlp model explicitly or implicitly resolves local ambiguities due to morphological garden paths. (2) using eras, we show that word segmentation models are vulnerable to explicit garden path errors, and sentiment analysis models are vulnerable to implicit garden path errors. ## explicit garden path errors our first experiment evaluates word segmentation models according to their vulnerability to explicit segmentation errors due to garden path phenomena. under the hypothesis that word segmentation models may segment canary words as words, we predict that test-sentence test sites will be segmented less accurately than control-sentence test sites. ## implicit garden path errors our second experiment evaluates sentiment analysis models according to their vulnerability to implicit segmentation errors due to garden path effects. our hypothesis is that the sentiment polarity of canary words will contribute to the sentiment labels assigned by the models to test sentences, even though these words should, in theory, be excluded from the semantics of these sentences. we therefore predict that, for example, a sentiment analysis model should label a test sentence with a negativesentiment canary word as being "more negative" on average than the corresponding control sentence. we also predict that providing word boundary information during pre-training or fine-tuning will improve a sentiment model's performance according to the aforementioned metric. occluded test sentence 学生 [mask] 心 机 处理 友人 the caution treats his student friends 'the student treats his friends with caution.' occluded control sentence 学生 [mask] 计谋 处理 友人 the shrewdness treats his student friends 'the student treats his friends shrewdly.' figure 4 : our occlusion study ablates canary words by masking out ("occluding") the character in the test site not belonging to the true word. ## discussion and related work left-branching bias in humans. much work has been done in psycholinguistics on the human processing of chinese morphological garden paths @xcite @xcite @xcite . @xcite , in particular, identify a human left-branching bias, similar to the left-branching bias observed for word segmentation models in section 3. they find that human readers spend less time looking at left-branching morphological garden paths than right-branching ones, all else equal, indicating that the former are easier to process than the latter. on the other hand, @xcite did not observe a left-branching bias when instructing gpt-3.5 models to perform word segmentation. garden paths in word segmentation. rulebased, statistical, and non-neural machine learning methods have been proposed for correctly segmenting morphological garden paths in chinese word segmentation @xcite @xcite @xcite . these techniques mostly involve first identifying locally ambiguous substrings, possibly with the help of a lexicon of locally ambiguous substrings, and then choosing a segmentation for each of those substrings based on the surrounding context. corpus analyses have found that morphological garden paths comprise roughly 4% of chinese text @xcite . targeted evaluation in nlp. minimal pair benchmarks have been used to evaluate language models' compliance with morphosyntactic constraints such as subject-verb agreement or negative polarity item licensing @xcite @xcite . @xcite propose an error analysis framework that compares chinese word segmentation models' performance across inputs with differing values for various features, such as sentence length or word frequency. ## conclusion this paper has proposed a benchmark, eras, that detects garden path errors in chinese nlp models, with or without an explicit word segmentation step in the pipeline. using eras, we have observed that both word segmentation models and sentiment analysis models are vulnerable to garden path errors. in both cases, models seem to inherit this vulnerability from their training distributions. the msr segmentation corpus most effectively defends against garden path errors, while the asap dataset seems to exacerbate susceptibility to garden path errors. we also find that garden path errors are driven by particular kinds of inputs: left-branching inputs in the case of word segmentation, and -/+ inputs in the case of sentiment analysis. it has been argued that character-level tokenization is superior to word-level tokenization for chinese neural nlp models @xcite , and indeed, most transformer language models for chinese use character-or byte-level tokenization. our findings from section 4, however, have shown that a total lack of word boundary information makes a model highly vulnerable to implicit garden path errors. we have shown that injecting word boundary information into the model through whole word masking or joint training on word segmentation can significantly reduce implicit garden path errors, though care must be taken to avoid degradations in model performance. we leave the development of such measures to future work.