Are Neural Networks Extracting Linguistic Properties or Memorizing Training Data? An Observation with a Multilingual Probe for Predicting Tense (2021)

Abstract: We evaluate the ability of Bert embeddings to represent tense information, taking French and Chinese as a case study. In French, the tense information is expressed by verb morphology and can be captured by simple surface information. On the contrary, tense interpretation in Chinese is driven by abstract, lexical, syntactic and even pragmatic information. We show that while French tenses can easily be predicted from sentence representations, results drop sharply for Chinese, which suggests that Bert is more likely to memorize shallow patterns from the training data rather than uncover abstract properties.

## introduction the success of deep learning in many nlp tasks is often attributed to the ability of neural networks to learn, without any supervision, relevant linguistic representations. many works have tried to identify which linguistic properties are encoded in the words and phrases embeddings uncovered during training. for instance, @xcite and @xcite studies the capacity of neural networks to uncover morphological information and @xcite , @xcite or @xcite (among many other) syntactic information. these works are based on the definition and study of linguistic probes: a probe @xcite is trained to predict linguistic properties from the representation of language; achieving high accuracy at this task implies these properties were encoded in the representation. however, as pointed out by @xcite , these approaches suffer from a major drawback: there is no guarantee that the probes' good performances result from the ability of the representations to capture relevant linguistic properties rather than to memorize a large number of labeling decisions. indeed, in most of the tasks considered so far, labels could be deduced directly from surface information, namely the word form (its morphology) and the word position in the sentence. given the huge number of parameters of current models, there is a high risk that they are only able to extract and memorize lexicallymarked patterns from training data with low (if any) generalization power. to shed new light on this question, we consider, in this work, a multilingual linguistic probe, the goal of which is to predict the tense of a sentence (i.e. the location in time of an utterance). we compare the performance of this probe on two languages, french, which expresses tense by the verb morphology, and chinese, in which, as explained in §2, the tense is expressed by a combination of lexical, syntactic and pragmatic cues. if intuitively the tense can be predicted from simple surface patterns in french, predicting the tense of a chinese sentence requires capturing the interaction of all sentence-level factors related to time, and sometimes even the contextual information from the previous utterances. contrasting the performance achieved by the probe on several languages ensures that the linguistic properties we detect are actually captured by the representation learned by bert and not by the probe, and thus to avoid a common pitfall of this kind of approaches @xcite . this work has two main contributions: first, we highlight the interest of contrasting linguistic probes on different languages; second, our experiments ( §3-4) show that bert has a preference for learning lexically marked features over lexically unmarked features and, consequently, is not able to extract an abstract representation that can be applied to (groups of) words that have not been seen at training time. ## tense and aspect languages can roughly be classified under two categories: tense language and aspect language, depending on how they denote time relations @xcite . tense indicates the temporal location of a situation, while aspect, according to @xcite , refers to "the different ways of viewing the internal temporal constituency of a situation" and denotes how speakers view it in terms of its duration, frequency and whether or not it is completed. in tense language, like english or french, the tense and aspect are often encoded simultaneously in verb morphology. for example, the simple past in english locates a situation before the speech time and is often indicated by the -ed inflection(e.g. worked). similarly, the french past tense imparfait is marked by the -ait inflection in il travaillait (he worked). contrary to these two tense languages, mandarin chinese does not have a grammatical category of tense @xcite and the verb morphology never changes. figure 1 presents five sentences with the chinese verb 加班 (jiaban, work overtime): while these sentences have different tenses (simple past, present progressive, habitual present, past progressive and future), the form of the verb is always the same. the notion of tense in mandarin chinese is lexicalized and is often indicated by content words like adverbs of time; aspectual information is systematically conveyed by aspect markers. as an aspect language, the temporal interpretation of a verb is tightly related to the notion of aspect. for instance, as noticed by @xcite , a verb marked by a perfective aspect particle such as 了 (le) often gets a past interpretation, such as in the first sentence of figure 1 . and imperfective aspect privileges a present interpretation: in the example 1.2, the same verb is marked by the imperfective aspect particle 在 (zai), which explains why the sentence gets a present interpretation. however, according to the genre of the text, only 2% to 12% of verbs in chinese have aspect markers ( mcenery and xiao, 2010) foot_0 and the tenses should often be inferred from contextual cues like lexical and syntactic features when there is no explicit aspect marker @xcite . for instance, in the absence of aspect marker in sentence 1.3, the adverb 常 常 (changchang, often) leads to a habitual present interpretation. these contextual cues can even invalidate the default correlation of time and aspect we have previously described, as in example 1.4, in which the verb group gets a past interpretation even if it is marked by an imperfective aspect particle because of the past temporal context introduced by the adverbial clause. finally, in example 1.5, the modal auxiliary 会 (hui) and temporal expression 晚上 (wanshang, tonight/in the evening) lead to a future-tense interpretation. thus, unlike french and english, the time of a chinese sentence can only rarely be deduced from a surface analysis of the sentence (i.e. from the characters composing its words) and in order to determine the tense, it is necessary to take into account both syntactic and even pragmatic information. ## creating a corpus annotated with tense information 2 the tense prediction task we consider in this work requires corpora in which the tense of each verb is identified. to the best of our knowledge, there is no such publicly available corpus. for languages such as french or english, in which tenses are described by verb morphology, it is possible to easily build such corpus from morpho-syntactically labeled treebanks (e.g. from the ud project @xcite ). however, this approach cannot be readily generalized to languages such as chinese, in which, time is not explicitly marked. we propose to leverage parallel french-chinese corpora to obtain tense annotations for chinese sentences automatically. our approach relies on two hypotheses. first, we assume that the tense of a translated sentence (target sentence) is the same as the tense of figure 1 : examples of different ways to express the tense in chinese: tenses are indicated by both aspect markers and a combination of lexical and syntactic cues and not by the verb morphology as in french or english. ipfv describes one of the two imperfective aspect markers and pfv one of the two perfective aspect markers its original (source) sentence ignoring translationese effects @xcite . second, we decided to associate each sentence with the tense of its main clause and not label each verb tense. this assumption allows us to mitigate errors related to the verbal structures identifications and misalignments between chinese and french verbs as labels are defined at the sentence level. we considered, in this work, the french-chinese newscommentary foot_2 parallel corpus @xcite . to extract tense information, we use the stanza pipeline @xcite with its pretrained french model to find the root of each sentence and extract its tense and its mode from its (automatic) morphological analysis. we also use the dependency analysis to identify periphrastic catena expressing future (aller + infinitive) or past (venir de + infinitive). with these information, we can define the tense of each sentence easily by mapping the tense of the root verb to one of the three labels present, past or future foot_3 and the tense of a chinese sentence is defined as the tense of its french translation. we evaluate our tense extraction procedure on the pud corpus foot_4 . it appears that stanza is able to correctly predict the tense of 95% of the sentences (i.e. identify the root of the sentence and correctly predict its morphological information). therefore, we consider that the tense labels are predicted with sufficient quality to measure a model's ability to capture time information. however, most of the prediction errors are due to the same construction: the auxiliary être followed by the past participle of the verb, that can be used to form either the passive voice or the passé composé tense of a verb. as a result, most of our corpus' passive sentence is labeled as past while they are at the present tense. in the end, this procedure results in a corpus of 4,764 documents containing 174,347 chinese and french sentences annotated with tense information. as expected, most of the sentences are in the present tense and the corpus is highly unbalanced: 67% of the examples are labeled present, 27% past and only 6% future. our corpus also confirms the observations reported in section 1 on the limited use of temporal markers in chinese: only 16% of the sentences have an explicit temporal marker. 6 we consider 80% of the data for training, 10% for testing and 10% for the validation set. ## conclusion we have shown that the performance of a tense prediction model varies dramatically depending on how the language expresses time, a result that suggests that bert is more likely to memorize shallow patterns from the train set rather than uncover abstract properties. our work also highlights the interest of comparing linguistic probes across languages which opens up a new avenue for research.