P rosper AM net at F in C ausal 2020, Task 1 & 2: Modeling causality in financial texts using multi-headed transformers (2020)

Abstract: This paper introduces our efforts at the FinCasual shared task for modeling causality in financial utterances. Our approach uses the commonly and successfully applied strategy of fine-tuning a transformer-based language model with a twist, i.e. we modified the training and inference mechanism such that our model produces multiple predictions for the same instance. By designing such a model that returns k>1 predictions at the same time, we not only obtain a more resource efficient training (as opposed to fine-tuning some pre-trained language model k independent times), but our results indicate that we are also capable of obtaining comparable or even better evaluation scores that way. We compare multiple strategies for combining the k predictions of our model. Our submissions got ranked third on both subtasks of the shared task.

## introduction the dominant strategy for solving natural language processing (nlp) problems these days is by taking some large, task-agnostic pre-trained neural language model and fine-tune it on the set of task-specific training data at disposal. this transfer learning setting has been proven to be highly effective in various nlp problems @xcite @xcite . despite having demonstrated its efficacy, the above described approach also comes with inconveniences inherent to the stochasticity of the fine-tuning procedure. as the performance of the fine-tuned model is dependant on random factors such as the order in which the training instances are considered during fine-tuning or the choice of hyper-parameters of the fine-tuning procedure. one could possibly account for this variability by performing fine-tuning multiple times with different hyper-parameter choices and performing model selection using the development set performance of the independently fine-tuned models. performing fine-tuning @xmath0, however, naturally increases the computational resources required for model selection by a factor of k. the models that we propose in our work are such that they make the training of @xmath1. we achieve this by simultaneously adding k multiple classification heads to the same pre-trained architecture and performing fine-tuning based on the aggregated loss coming from the individual classification heads. our experimental results indicate that the above fine-tuning strategy not only provides a more resource efficient way of training multiple fine-tuned models, but it also has the potential of providing an increased performance for the models, since fine-tuning classifiers in a simultaneous manner likely acts as a regularizer. our source code for replicating our results can be accessed at https://github.com/zsozso21/camuh . ## data the shared task consisted of a sentence classification (sc) and a cause and effect detection (ced) subtask, that we briefly describe below. a more detailed overview of the shared task can be found in the summary paper @xcite . for the sc subtask, the goal was to classify textual utterances whether they convey some form of causality. as such, we treated the sc subtask as a sentence-level binary classification problem for which one can conveniently fine-tune some pre-trained transformer-based model. in the ced subtask, systems were provided (blocks of) sentences and the goal was to detect the correct spans of the causal elements and their consequences. as a concrete example, the spans formatted in italics and bold in the below sentence are considered as the effect and cause constituents, respectively: "zhao found himself 60 million yuan indebted after losing 9,000 btc in a single day (february 10, 2014)". illustrated by the previous example, the ced subtask can be handled as a token-level sequence classification with three classes, i.e. a token can either belong to a constituent of the cause or the effect or it can be none of the above. we shall refer to these three classes by their initials, i.e. c and e for the cause and effect classes and n for those tokens that are neither the cause or the effect constituent of some text. ## system we trained our models using the pre-trained bert models based on the transformers library @xcite . during fine-tuning, we relied on the utilization of the cross-entropy loss for both the sc and the ced subtasks. we treated the sc subtask as a binary sequence-level classification task, whereas we framed the ced subtask as an utterance-level classification problem. ## results and discussion the parameters of the system were optimized by using the shared task's trial dataset for training and the practice dataset for testing 1 . based on our evaluation on the practice dataset, we fixed the batch size to on the sc and the ced subtasks to 8 and 1, respectively. we also applied dropout with probability 0.1 in all of our experiments. those models that we used for making predictions over the evaluation data were trained on the concatenation of the practice and trial datasets and later tested on the evaluation dataset during the post evaluation period. unless stated otherwise, our models described below use averagingbased aggregation strategy of the heads and no span-level auxiliary loss. as neural methods are prone to achieving results with high variance when being trained with different choices of random seed, we evaluated each architectures five times with only their random seeds set differently. we report the individual results obtained by the different choices of the random seed as well as the average and standard deviation of the performance metrics. ## conclusions our experiments demonstrated that the standard fine-tuning mechanisms work well for modeling causality in financial texts as well. more importantly, we introduced our modified fine-tuning architecture that is capable of training multiple classification model on top of a shared transformer architecture. our results demonstrate that we did not only obtain multiple classifiers in a less resource intensive manner, but the classifiers also performed better as if we were training them separately of each other. a potential avenue for improving our proposed fine-tuning architecture could focus on more sophisticated aggregation strategies for the predictions of the simultaneously fine-tuned classification models.