LIORI at the F in C ausal 2020 Shared task (2020)

Abstract: In this paper, we describe the results of team LIORI at the FinCausal 2020 Shared task held as a part of the 1st Joint Workshop on Financial Narrative Processing and MultiLingual Financial Summarisation. The shared task consisted of two subtasks: classifying whether a sentence contains any causality and labelling phrases that indicate causes and consequences. Our team ranked 1st in the first subtask and 4th in the second one. We used Transformer-based models with joint-task learning and their ensembles.

## introduction the financial document causality detection task was devoted to finding causes and consequences in financial news @xcite . this task is relevant for information retrieval and economics. this task was focused on causality associated with a financial event while an event was "defined as the arising or emergence of a new object or context in regard to a previous situation". the shared task consisted of two subtasks: ## dataset the task dataset has been extracted from different 2019 financial news provided by qwam foot_4 . the corpus consists of html-pages of financial news from 2019. it also contains various financial and legal reports from the sec edgar database ticker list, filtered on financial keywords. the texts have been normalized for the research task in the following way: the dataset was split into trial, train and test datasets by the organizers. the trial and train parts contained training labels, while the test part did not include them and was used for ranking. we combined the trial and train parts and used 20% of the combined dataset for validation. ## solution in this work, we went with multitask transformer-based models for both subtasks. it means that we had two inputs and outputs, for each of the tasks respectively. in this work we tried bert @xcite and roberta @xcite based models. bert is a multilingual language model based on self-attention. roberta is a "robustly optimized" bert variant with larger mini-batches and byte-level bpe (byte-pair encodings). in both cases we used english large model variants (bert-large and robertalarge). on top of pre-trained bert and roberta models, we added two linear layers with dropout for each of the tasks. cross-entropy was used for training the models. thus, we had two loss functions (for each of the output layers) that were weighted and concatenated. all used models were provided by hugging face @xcite . our combined loss function can be seen below, where l a is the first subtask loss and l b is the second subtask loss. @xmath0, where λ are scalar weights for the loss functions. all padded words and non-labeled words (and their resulting tokens) were excluded from loss function calculation and not included into n j , while special '[sep]' and '[cls]' tokens were included. while training models for the first subtask we tested a number of weighting schemes ranging between 2 and 0 for sequence labelling subtask loss. however, for the second subtask, the weights for text classification loss were set to zero which makes the model equivalent to a general sequence labelling model. we also tried various sequence labelling formats of the second subtask input: bio (beginning, inside, outside) and bieo (beginning, inside, end, outside). learning rates in the range between 5e -06 and 5e -05 were tested. dropout coefficients were tested from 0.1 to 0.2. for the first subtask, there were also provided the results for ensembles of the best 3, 4 and 5 performing models according to the validation dataset. simple voting ensembles were used. we used a system with 2 nvidia rtx2080 gpus and google colab to train all models. for the first subtask, the organizers used f1-score. for the second subtask, the metric is a weighted average f1 score, where the f1 score of each class is balanced by the number of items in each class (see @xcite ). in the first subtask our final model achieved f1 equal to 0.977 on the leaderboard (the next participant's score is 0.975 f1), in the second subtask our result was 0.826 f1 with the winning solution having 0.947 f1. the results of individual models and their hyperparameters can be seen in tables 1 and 2 for each of the subtasks respectively. as can be seen from paradoxically, for the second subtask bert-based models consistently outperform roberta based ones. moreover, the difference is much larger and constitutes more than 0.7 f1-score points. we did not try ensemble-based models for the second subtask. it also can be seen that all our models tend to overfit to the training and validation datasets. a more robust training scheme such as k-fold cross validation might be of benefit here. ## conclusion this paper describes the results of team liori at the fincausal 2020 shared task held as a part of the 1st joint workshop on financial narrative processing and multilingual financial summarisation. the shared task consisted of two subtasks: classifying whether a sentence contains any causality and labelling phrases which indicate causes and consequences. transformer-based models with joint-task learning were used. in this paper we show that different model architectures perform better for different subtasks and that joint-task learning might improve results for some subtasks. however, it also results in slight overfitting for sequence labelling task and might require further investigation.