Modeling Construction Grammar’s Way into NLP : Insights from negative results in automatically identifying schematic clausal constructions in B razilian P ortuguese (2023)

Abstract: This paper reports on negative results in a task of automatic identification of schematic clausal constructions and their elements in Brazilian Portuguese. The experiment was set up so as to test whether form and meaning properties of constructions, modeled in terms of Universal Dependencies and FrameNet Frames in a Constructicon, would improve the performance of transformer models in the task. Qualitative analysis of the results indicate that alternatives to the linearization of those properties, dataset size and a post-processing module should be explored in the future as a means to make use of information in Constructicons for NLP tasks.

## introduction constructional approaches to language description can be traced back to early work by @xcite , which later gave rise to a myriad of approaches sharing the common assumptions that (a) constructions are learned pairings of form and function related to one another in a network, and (b) grammar does not rely on transformations and derivation, instead it is directly associated with function @xcite . from the 2000's on, computational implementations of construction grammar started being built both in terms of language resources comprising of collections of constructions called constructicons @xcite @xcite @xcite , and proofs of concept, namely constructional parsers @xcite . as a natural consequence of the focus of constructionist analysis on families of constructions, constructicons typically start by modeling the same kind of phenomena, leaving more schematic and foundational language structures, clausal and phrasal constructions, respectively, for later. these kinds of constructions represent a challenge for both constructicography, that is, the process of de-scribing and modeling constructions in a resource @xcite , and for constructional parsing, since schematic clausal constructions, as opposed to idioms, are typically difficult or impossible to describe in terms of the presence of distinctive lexical fillers. moreover, it is common for those constructions to share constituency properties. as an example, consider (1) and (2), both sentences share the same syntactic structure in brazilian portuguese, but express opposite types of semantic events (controlled × uncontrolled activity), thus representing instances of distinct constructions, namely intransitive and unaccusative. because this difference is not derived from specific lexical fillers, if the verbs in the examples were to be changed to dance and slip respectively, the same constructions would be used to describe the sentences. in this paper, we discuss insights from negative results obtained in an experiment for identifying schematic clausal constructions and their construction elements in brazilian portuguese (pt-br) by using a combination of multilingual bert @xcite with the computational representations of such constructions in the framenet brasil constructicon (fn-br ccn) @xcite @xcite . qualitative analysis of the results indicate that alternatives to the linearization of the constructional properties modeled in resources, number of annotated sentences and a post-processing module should be explored in the future as a means to make use of information in constructicons for nlp tasks. in the remainder of this paper, we present, in section 2, how constructions are represented in the fn-br ccn. next, in section 3, we go through the steps needed to convert the fn-br ccn representations into a dataset that could be used for proposing the construction identification model in section 4. sections 5 and 6 describe the experimental setup used to evaluate the model and the results. discussion of the results is carried out in section 7, with quantitative and qualitative analyses. section 8 presents final considerations. ## construction representation in the fn-br constructicon the fn-br constructicon @xcite is built as part of the framenet brasil language resource, meaning that, similarly to lexical units, constructions in this database can have their meaning import represented in terms of frames. therefore, the semantics of the intransitive construction licensing (1) can be represented as the intentionally_act frame in figure the database structure of fn-br allows for construction elements (ces) to be mapped to frame elements (fes), when relevant. hence, the subject and the predicate ces in the intransitive construction can be respectively mapped to the agent and act fes in the intentionally_act frame. moreover, the fn-br ccn allows for other types of information to be represented. first, ces can be defined in terms of phrasal constructions licensing them. for the intransitive, the sub-ject ce is a determined_noun_phrase, while the predicate is a non_complement_taking_verb_phrase. furthermore, the information that the verb ce of this last construction has to be filled by a frame that inherits intentionally_act can also be recorded. if instead, this slot was constrained by a child frame of undergoing, then this would be an unaccusative construction. formal properties of the construction can also be represented, such as the fact that the subject ce usually comes before the predicate, and that the first corresponds to the nsubj relation in the universal dependencies tag set (de marneffe et al., 2021), while the latter would correspond to the root. all the information associated to the intransitive construction in the fn-br ccn, together with the fact that it inherits a general subject_predicate construction are shown in figure 2 . in addition to the two clausal constructions mentioned so far, work by almeida (2022) has modeled 22 other clausal constructions and 22 phrasal and pos constructions licensing the ces in them in the fn-br ccn. many of those ces share the same name (e.g. predicate), but are fully separate entities in the database, each belonging to a single construction. for that reason, when applying these structures to an experiment for automatically identifying construction in corpora, the ces can be treated as the actual labels. a model working with ces is, arguably, more informative and easier to interpret, despite being more complex. moreover, the frame information, used to represent the semantic part of a construction, is not lost because the ces are directly related to the fes of frames. the full dataset that includes pieces of the fn-br ccn and setup used in this work are described next. ## dataset the dataset used in the experiments had to be built step by step because one of our research goals was to assess the impact of universal dependencies (ud) and frame information embedding into a neural model for ce labeling, which is not a traditional nlp task. the corpus consists of 673 sentences annotated for ud, clausal constructions (and their ces) and frames. subsections 3.1, 3.2 and 3.3 describe how each type of data was integrated into the same dataset. ## model figure 4 shows the general architecture of the proposed model. the system was designed and implemented in a way such that some components could be switched or just removed to facilitate testing of the various scenarios.the most important elements in the model are described next. bert word piece tokenizer [cls] tok 1 tok l tok 2 tok 1 tok 2 tok m ud tokenized input + frames ... ... index reduce+mean ud token indices time-distributed e 1 frm 1 rel 1 e 2 rel 2 e m rel m frm 2 frm m ... conditional random field (crf) ce 1 ce 2 ce m embedding one hot bi-lstm transformer block or cased multilingual bert ## experiments in order to understand the impacts of ud and frame data in ce labeling, 9 different experimental setups were proposed, 5 variations using lstm and 4 using transformers. for each of those options, the effectiveness of the crf was evaluated, with and without ud and frame data. the lstm was the only model where the bfs sorting of tokens was tested, hence it has one more variation. this type of ordering was considered only for this architecture because, in theory, the way information vanishes in the cell states is influenced by the order of the elements in the input sequence. the transformer can handle this problem by simply adjusting attention weights. one of the challenges of working with the dataset described in section 3 was the number of samples annotated for each construction. for elapsed_time, for example, only two examples were found in the 673 sentences. this variation in construction frequency is expected and leads to the fact that a much larger dataset would be needed to find a reasonable amount of examples for that construction. we have decided to consider only the two most fre-quent constructions in the ud portuguese gsd treebank, namely direct_transitive and attributive_nominal_predicative, as the models could not perform consistently for the ones with less samples. this resulted in a dataset with 443 sentences and 4 ce labels that was split into train and test sets in a 8:2 ratio. considering the relative effectiveness of bert's fine-tuning and that constructional information can be extracted from it, this dataset can still be used to predict ce labels @xcite @xcite . in all variations, the networks were implemented to predict a single ce label (or none) for each position corresponding to a token in the sequence, despite the fact that it is possible for more than one label to be true. this was done because, in our first implementation tests, we verified that only one instance of the dataset had a token with two labels. moreover, convergence was slow during training, even after adjusting parameters, without any performance gains. for that reason, we used a softmax activation function and cross-entropy as the loss function. when the final layer was a crf, loss is computed using the log-likelihood. for training, we used an adam optimizer with learning rate set to 3e -5. due to gpu memory limitations, batch size was set to 16 samples and the maximum number of epochs to 20, which was not a problem because of the reduced size of the training dataset. to prevent over-fitting, the loss over the validation set was monitored and after 3 epochs without any improvements, training would be stopped, resulting in less than 20 epochs per training run. every model variation was trained 10 times so that their performance and generalization could be better analyzed. the bert model's weights were adjusted, i.e. fine-tuned, in each of those runs and, in order to prevent tests from influencing one another, memory was cleaned up between executions. ## results after the execution of all of the training algorithms, model results were computed and compiled into to better understand the variations between different training iterations, the average (with standard deviation) and the best f1 scores for each configuration were observed. in terms of averages, the lstm model with a crf, but without frames or uds had the best performance. this configuration also had the smallest standard deviation, indicating that training is somewhat consistent. in terms of best results, the base lstm model without crf and the complete transformer model have the highest f1, with a score of .767. of the two, the lstm is a considerable smaller model, as shown in the worst configuration, on average, was the lstm model where tokens were reordered using the dependency tree bfs results. in constrast, every model achieved better average f1 scores when a crf layer was added. ## discussion as previously stated, the lstm models performed better, especially when no additional frame or ud data was embedded into the inputs. however, when using transformers, the same type of data can increase performance. one possible explanation is that the latter has a considerably larger number of parameters, making it easier to integrate the additional information, but, at the same time, being more complex to train and, thus, having worse performance than lstms. also, the difference in the results is likely affected by the small size of the training dataset, it is possible that the quality of the predictions could improve if more samples were processed by the networks. the lstm where the order of the tokens was changed also provides good insight on how this model used the information to make predictions and why it has the lowest average f1 score. when analyzing the average f1 scores for each construction element, it was noted that final f1 was mostly influenced by the subject ce of the attributive_nominal_predicative. in the model using a crf, the f1 for the subject was .628, for the one with the bfs ordering, it was .388. this is a strong evidence that when optimizing with few samples, for a ce that has a relatively strict position in a sentence, positional information is relevant. it also shows that the model was not able to compensate for the absence of this kind of information using only uds and a different ordering. this type of problem can be potentially avoided by using a network architecture designed to handle graphs or trees. we have also decided to carry out a qualitative analysis of the predictions made by the best transformer model. all of the predictions made over the test set were transformed back to ce spans, which where then aligned to the original sentences and paired with the original human annotations. making this side-by-side comparison, notes were taken for each record. during this process, we observed that some types of errors were way more frequent than others. for instance, 26% of the sentences had only the head words of the constructions elements labeled, while 13% had a problem of discontinuity in the ce span. these numbers agree with the f1 results displayed on more importantly, these errors seem to originate from an overgeneralization made by the model over the pos of words. despite the fact that pos tags are not part of the input to the model, this information is arguably embedded into bert. more evidence of that is found on examples where the model labeled some word with the incorrect ce. although rare, when it happens, the ce predicted for that word is of the same pos of a head of that ce. for example, many verbs are labeled as the predicate of a direct_transitive construction, even when they are part of other type of construction. the same happens for adjectives and the attributive_nominal_predicative. this happened in 8.7% of the analyzed sentences. the problem of overgeneralization also occurs with the conj relation in this model. interestingly, this seems to be the only ud relation that clearly influenced the predictions of the test set. in 8,7% of the sentences, the model labeled the tokens of a conjunct despite the fact that they are not related to a subject or predicate. in a deterministic approach, this type of error can be easily verified using the dependency trees, as the ce span nodes would not be connected. ## final considerations the experiment reported on in this paper aimed at testing whether ud and frame information extracted from a constructicon could positively influence the performance of transformer-based mod-els for schematic construction identification in sentences. however, the most effective models in our tests are still the smaller lstms, without any extra information. furthermore, we have identified that the trained models were overgeneralizing certain aspects of the data, causing performance to degrade. one of the limitations of our experiments is in the architecture of the network itself when these various types of information are used. for both lstm and transformers, there are gaps that could be filled if they were implemented to fully handle tree and graph structures instead of their linearized, thus, simplified, versions. other changes in the network structure and training procedure are needed to prevent the overgeneralization discussed in section 7. another course of action to better understand how neural networks can be used to classify constructions effectively is to expand the dataset, both in number of samples, but also in representation of different clausal structures. for example, if the models were to be trained to also label the ces of the unaccusative construction, they would have to learn the semantic boundaries that differentiate an unaccusative from a transitive verb. finally, in spite of their limitations, it is clear that performance could be improved by using postprocessing algorithms that could either find anomalous outputs (e.g. the incorrect conjucts and even expand the spans of the ces based on the dependency relations. this type of procedure is adequate because the models are already able to identify most ce heads. chi sun, xipeng qiu, yige xu, and xuanjing huang. 2019. how to fine-tune bert for text classification? in china national conference on chinese computational linguistics, pages 194-206. springer. kai sheng tai, richard socher, and christopher d. manning. 2015. improved semantic representations from tree-structured long short-term memory networks. in proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: long papers), pages 1556-1566, beijing, china. association for computational linguistics. harish tayyar madabushi, laurence romain, dagmar divjak, and petar milin. 2020. cxgbert: bert meets construction grammar. in proceedings of the 28th international conference on computational linguistics, pages 4020-4032, barcelona, spain (online). international committee on computational linguistics. tiago timponi torrent, ludmila meireles lage, thais fernandes sampaio, tatiane da silva tavares, and ely edison da silva matos. 2014. revisiting border conflicts between framenet and construction grammar: annotation policies for the brazilian portuguese constructicon. constructions and frames, 6(1):34-51. tiago timponi torrent, ely edison matos, alexandre diniz costa, maucha andrade gamonal, simone peron-corrêa, and vanessa maria ramos lopes paiva. forthcoming. a flexible tool for a qualia-enriched framenet: the framenet brasil webtool. language resources and evaluation. tiago timponi torrent, ely edison matos, ludmila meireles lage, adrieli laviola, tatiane da silva tavares, vânia gomes de almeida, and natália sathler sigiliano. 2018. towards continuity between the lexicon and the constructicon in framenet brasil. in benjamin lyngfelt, lars borin, kyoko ohara, and tiago timponi torrent, editors, constructicography: constructicon development across languages, pages 107-140. john benjamins, amsterdam. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n gomez, łukasz kaiser, and illia polosukhin. 2017. attention is all you need. advances in neural information processing systems, 30. alexander ziem and hans boas. 2017. towards a constructicon for german. in the aaai 2017 spring symposium on computational construction grammar and natural language understanding technical report ss-17-02, pages 274-277, palo alto, ca. aaai press.