“Chère maison” or “maison chère”? Transformer-based prediction of adjective placement in F rench (2023)

Abstract: In French, the placement of the adjective within a noun phrase is subject to variation: it can appear either before or after the noun. We conduct experiments to assess whether transformer-based language models are able to learn the adjective position in noun phrases in French –a position which depends on several linguistic factors. Prior findings have shown that transformer models are insensitive to permutated word order, but in this work, we show that finetuned models are successful at learning and selecting the correct position of the adjective. However, this success can be attributed to the process of finetuning rather than the linguistic knowledge acquired during pretraining, as evidenced by the low accuracy of experiments of classification that make use of pretrained embeddings. Comparing the finetuned models to the choices of native speakers (with a questionnaire), we notice that the models favor context and global syntactic roles, and are weaker with complex structures and fixed expressions.

## introduction in french, the placement of the adjective is subject to a considerable amount of variation-a phenomenon that has been under close scrutiny among linguists. generally speaking, adjective placement in anteposition or postposition is attributed to many intertwining, linguistic processes, rather than a few rigid grammatical rules. however, the order of the adjective can be crucial to the meaning of the noun phrase; in the titular example, chère maison means "dear house" but maison chère means "expensive house". meanwhile, natural language processing researchers investigate whether language models built by transformer architectures are able to capture some of the inner workings of human language during their learning process. so far, research has shown that the high performance of such models does not imply the understanding of basic concepts such as grammatical order because the transformer architecture is non-sequential by design. we are exploring whether transformer-based language models are capable of perceiving the adjective's position in a sentence with regard to its head noun, with a variety of experiments. our goal is not to set a new state-of-the-art, but to explore if and how this information on word order is learned and used in tandem with the contextual word embedding information. while previous work has shown that transformer models are insensitive to word order @xcite , finetuned models were successful in classifying adjective position @xcite . we also tested variations of finetuning training sizes and the use of attention masks to hide either the context of the sentence or the head noun and adjective. for most adjectives, classifying their position is a relatively easy decision based on frequency; to observe the models' underlying competencies in more complex cases, we carried out an error analysis and additional experiments and visualizations on the pretrained versions of the models. we also had the opportunity to conduct an experiment with native french speakers, to compare their choices in challenging cases of adjective placement to the models' predictions. our findings show that finetuned models are capable of learning word order and efficiently classifying it; this knowledge is fainter in pretrained embeddings, but some layers demonstrate some specialization. finetuning a model helps to learn these variations in adjective position and very successfully select the correct one. camembert models were more successful than flaubert models over all experiments and captured more positional information in the finetuned adjective embedding. however, all transformers models show weaknesses (to different degrees) in complex cases of adjective/noun dependent phrases and fixed expressions. ## position of adjective in french noun phrases while traditional grammar proposes that adjectives in french follow the noun, in a noun phrase, linguistic analysis supports that adjectives are mobile, i.e. can be placed in anteposition or postposition relative to the noun @xcite . however, most adjectives tend to appear in specific positions; adjectives that accept only anteposition, only postposition, and those that alternate position @xcite . for example, ordinal adjectives in -ième (e.g. troisième 'third'), are almost always anteposed to the noun, the adjectives exotique 'exotic', idéal 'ideal', populaire 'popular', moderne 'modern', géant 'giant', naturel 'natural' are always postposed, and the adjectives énorme 'huge', immense 'immense', superbe 'superb' alternate between the two possible position @xcite . the preferred position of an adjective depends on its features and frequency; for example, benzitoun (2014) claims that the adjective prochain 'next' in plural form does not occur in postposition (based on corpora statistics), but the singular does. @xcite calculated that the most frequent adjectives in a corpus of literary works tend to precede the noun. however, chromatic adjectives (e.g. rouge 'red') which are of high frequency are always postposed to nouns when not a part of a multi-word expression. adjectives derived from nouns and adjectives have a very strong tendency to be postposed @xcite . @xcite and @xcite support that the length of the adjective affects its position; short adjectives (e.g. bon 'good', beau 'pretty') tend to antepose, while longer adjectives and derivatives can only be postposed. semantic factors may also affect the position of an adjective with respect to its head word. for example, adjectives with multiple meanings may have different meanings in different positions; e.g. propre when anteposed refers to possession 'own', but when postposed means 'clean' @xcite . @xcite also presents the concept of adjective-noun pairs, where the meaning of the noun influences the position of the adjective. these pairs differ from fixed expressions because it is possible to create a pair with a different order (and different meaning), while fixed expressions are lexicalized and do not allow the existence of a variation with a different meaning. for example, the lexicalized phrase arts premiers (where premier is postposed) has a very specific meaning ('arts of the non-western world') compared to premiers arts 'first arts' where it used in its literal sense and is not a lexicalized phrase. the presence of more dependents in the noun phrase also affects the position of the adjective. the presence of an adverbial modifier to the adjective may force the adjective phrase to postposition or increase the occurrence of the adjective in postposition, or at least allow more flexible positioning of the adjective phrase relative to the noun @xcite . a definitive case of postposition happens when an adjective has a multiword modifier, e.g. a prepositional phrase @xcite . postposition is also favored when there are multiple adjectives defining the noun. @xcite also suggests that elements in the syntactic phrase are ordered by increasing length (known as increasing or relative mass). however, it may not apply to high-frequency adjectives such as magnifique 'magnificent' @xcite . ## word order and transformer models there has been extensive work on analyzing the syntactic and semantic capabilities of transformer models and their pretrained word embeddings, with positive and negative findings on the abilities of these models to capture linguistically salient word relations. in this review, we focus on word position and word order findings. the addition of absolute word order (i.e. the sequential order of words) to the training process of contextual word embedding models has proven quite beneficial. transformer models with bidirectional training, which captures adjacent word order, showed improvement compared to the original self-attention neural networks @xcite . transformer models trained with masked language modeling, such as bert and roberta, are able to learn absolute word positions, but they also learn structural word positions (i.e. phrase position in hierarchical tree structures) and make use of them @xcite ). multiple experiments combine absolute and structural word positions to create better-informed and betterperforming word embeddings @xcite @xcite . however, experiments on already pretrained language models and shuffled word order tell a dif-ferent story. @xcite conducted experiments on bert-based models (bert, roberta, albert) with glue classification tasks, and showed that tasks such as sentiment analysis were not affected by shuffled word order, except for the grammatical correctness task. @xcite conducted experiments on the effect that context variation has on transformer models' usable information, and discovered that word shuffling has a negative effect, whether the shuffling was implemented on short or long distances among words. @xcite conducted similar experiments with glue tasks and observed that model performance was lower on shuffled word orders (in methods that render a sequence ungrammatical and incomprehensible to humans) but close enough to support that models rely more on embedding information rather than sequential context. @xcite confirm that pretrained language models are insensitive to word order in tasks of natural language inference and show that, on some occasions, classification is successful only with certain (random) word order variations of an input sequence. they also conducted experiments on finetuned models and noted finetuning's positive influence on learning word order. finetuning improved performance on tasks of inference and grammaticality as well (even with models pretrained with scrambled word order) @xcite . for french, @xcite conducted experiments, on the transformer models' capacity to capture long-range object-verb agreement and word order (in one of their experiments). they observed that models performed worse with scrambled inputs, and increasingly worse, for increasingly complex relations. ## experiment 2: pre-existing knowledge in pretrained embeddings the previous experiment shows that the finetuned transformer models are quite successful in classifying the adjective's position when asked to distinguish between two possible positions. the following part of this research aims to observe whether this capability is given by the finetuning, or whether the pretrained models had already learned enough information on the adjective's preferred position, with regard to its context. ## discussion previous work on exploring transformer models has supported that their success in nlp tasks is heavily based on their vast training data and efficient learning of frequencies. our experiments, compared to a frequency-based uninformed baseline, show that there are more complex operations in play. transformers were more efficient than sequential-orderlearning neural networks, and were in fact able to differentiate between two sentences with identical tokens and slightly different word order. finetuning is more efficient with a larger training corpus and different domains, but can still be successful with a smaller dataset if necessary. when the models' attention mechanism only has access to the context, and not to the adjective-noun pair itself, they were still quite capable of classi-fying adjective position even without attending to it. this observation is consistent with the linguistic description that supports that adjective position is also determined by context and not solely by the noun phrase. however, the fact that camembert models were extremely successful in identifying position without the use of context, while flaubert models failed completely, is caused by the models' different architectures and choices in the way the tokens are handled. in our more detailed experiments, we saw that camembert models assign an overall higher probability to adjectives, regardless of their position, and that, at least for the ud dataset, the adjective embeddings were, in some layers, very informed on the preferred word position. this knowledge is correlated to the learned contextual word embeddings, rather than the word itself, as we observed a lack of semantic similarity in the visualization. regarding the models' mistakes in the testing phase, they were either caused by low-frequency adjectives, bad parsing, or ambiguous meaning which may be grammatical and acceptable in both adjective positions. however, comparing the models to human performance showed their true strengths and weaknesses; when they are successful, the models tend to follow a more rigid syntactic structure and favor postposition, as it is the most frequent adjective position over all adjectives. they showed severe problems in recognizing some fixed expressions, and were more easily swayed than humans by being primed with the same adjective. in cases where both positions were possible, they usually preferred the more "traditional" postposition. these findings may demonstrate that the models base their decisions on adjectives more on frequency rather than the syntactic and semantic information of a particular adjective, and are impervious to factors that affect speakers' decisions such as length, difficulty of processing with regard to cognitive load, and substantial or subtle semantic differences. ## conclusion in this work, we aimed to study the capabilities of transformer-based language models in understanding word order, specifically the order of adjectives in a noun phrase in french. our findings, for pretrained models, confirmed previous ones which claimed that these models are agnostic to word position. however, the process of finetun-ing and classification with two variations of the sentence (one correct and one with permutated adjective order) was very successful, which proves that the models are capable of learning and becoming sensitive to word order. concerning the use of attention masks, the camembert models were very capable of classifying word order by only attending to the adjective and noun, while for the flaubert models it was impossible. the differences between the two architectures were also reflected in our study of the pretrained word embeddings and the adjective probabilities, where we noticed that camembert's adjective embeddings were better informed. the adjective embeddings themselves, for all models, seem to contain more contextual than word-specific information, which makes different iterations of an adjective differ from each other. in our experiment comparing native speakers to the models' preferences, we observed that the models showed weakness in structures with dependents, fixed expressions, and priming, and reverted to the grammatically-established postposition more than humans. therefore, the models' understanding of the position relies both on context and on shallow syntactic roles, but is lacking semantic nuances. we also observed that the information on position is specialized in some layers -and easily learned via finetuning.