A Report on the F ig L ang 2022 Shared Task on Understanding Figurative Language (2022)

Abstract: We present the results of the Shared Task on Understanding Figurative Language that we conducted as a part of the 3rd Workshop on Figurative Language Processing (FigLang 2022) at EMNLP 2022. The shared task is based on the FLUTE dataset (Chakrabarty et al., 2022), which consists of NLI pairs containing figurative language along with free text explanations for each NLI instance. The task challenged participants to build models that are able to not only predict the right label for a figurative NLI instance, but also generate a convincing free-text explanation. The participants were able to significantly improve upon provided baselines in both automatic and human evaluation settings. We further summarize the submitted systems and discuss the evaluation results.

## introduction figurative language such as metaphors, similes or sarcasm plays an important role in enriching human communication, allowing us to express complex ideas and emotions in an implicit way @xcite . however, understanding figurative language still remains a bottleneck for natural language processing @xcite . in spite of the fact that transformer-based language models (lms) get larger @xcite , they are still incapable of comprehending the physical world, cultural knowledge, or social context of figurative language @xcite . in recent years, there have been several benchmarks dedicated to figurative language understanding, which generally frame "understanding" as a recognizing textual entailment (a.k.a natural language inference (nli)) task -deciding whether one sentence (premise) entails/contradicts another (hypothesis) @xcite @xcite . however, similar to general nli datasets, these benchmarks suffer from spurious correlations and annotation artifacts @xcite . these can allow large language models (llms) to achieve near human-level performance on in-domain test sets, yet turn brittle when evaluated against out-ofdomain or adversarial examples @xcite @xcite . to tackle these problems, research in nli has argued that it is not enough to correctly predict the entail/contradict labels, but also to explain the decision using natural language explanations that are comprehensible to an end-user assessing model's reliability @xcite @xcite , leading to novel datasets such as e-snli @xcite . in this paper, we report on the shared task that aim to test the ability of models to not only predict the right label, but also provide a free-text explanation to the instance. this task was conducted as part of the 3rd workshop on figurative language processing (figlang 2022) at emnlp 2022. section 2 provides a description of the shared task, datasets, and evaluation metrics. section 3 contains brief summaries of each of the participating systems whereas section 4 reports a comparative analysis of the participating systems. ## datasets and task description as stated earlier, this shared task is based on the flute dataset that was released by @xcite . flute consists of pairs of premises (literal sentences) and hypotheses (figurative sentences), with the corresponding entailment or contradiction labels (nli instances), along with explanations for each instance ( ## participants and results training phase the shared task started on july 10, 2022, when the training data and the auxiliary scripts were made available to all the registered participants. participants were allowed to choose to partition the training data further to a validation set for tuning the hyper parameters. likewise, they can also elect to use the training data to perform cross-validation. evaluation phase in this phase, test instances for evaluation are released. we released the test data on august 15, 2022. submissions were accepted until august 20, 2022. out of all the submissions, five shared task system papers are accepted to the workshop. predictions are submitted to the codalab site and evaluated against the gold labels of the test instances. we used codalab for the shared task because it is easy to use, provided easy communication with the participants (e.g., allow mass-emailing to the participants), as well as tracks all the submissions updating the leader-board in real-time. we allowed up to five submissions per day for each participant team. we did setup our own gpu-based evaluation using a custom docker architecture. the leader-board displayed the accuracy@60 scores on the descending order. in total we have five participating teams alongside the organizing team of shared task. we describe the participating systems in the following section. baseline @xcite the baseline is the system described in @xcite . this system is trained to predict labels and rationales jointly using a t5-3b model @xcite . unlike other teams @xcite verbalized inputs using natural language instruction: does the sentence "p" entail or contradict the sentence "h"? please answer between "entails" or "contradicts" and explain your decision in a sentence. teamcooldoge @xcite present dream-flute which first uses dream @xcite to generate an elaboration of the situation in the premise and hypothesis (separately), then uses this additional context for classification and explanation generation. they hypothesize that such additional, pertinent details could also improve a model's ability to judge whether it is an entailment or contradiction between the premise and hypothesis. this posit this could be especially helpful for the instances that use figurative language, where the underlying meaning might be opaque to the model and that further elaborating the context can make certain inferences more explicit. they take as @xmath0. while the scene elaboration dimensions from dream can vary across the categories of consequence, emotion, motivation, social norm the winning submission is based on consequence elaboration dimension. it should be noted that the underlying model is similar to the baseline model (ablation without using dream), however the performance differs due to different hyperparameters. rachneet @xcite focus their efforts on the transfer of information from multiple related tasks for improved performance on flute. they compare the effectiveness of sequential fine tuning with that of multitask learning in a context where one of the target tasks is dependent on the other. their final submission which led to the highest acc@60 on the flute test set is a t5 (raffel et al., 2020) based model where the label and rationales are predicted jointly. in particular their best submission is a sequentially fine-tuned model where they first finetune on esnli @xcite followed by impli @xcite and vund (phan et al., 2022) considered both the tasks: the nli task, and the explanation generation task as two seq2seq tasks. they fine-tuned the two tasks separately as a simultaneous computation model. in addition, they also used the attribute about types of figurative language across the data as a predictor and treated it as seq2seq tasks. therefore they have 3 component models based on fine-tuning pre-trained model t5 @xcite : nli predictor, type predictor, and generator. unlike other teams that predict label and rationale jointly here the team uses t5-large model in a pipeline fashion. ## analysis the best performing teams according to both human and automatic evaluation were team-cooldoge, rachneet, and vund ( across types of figurative language, explanations for similes and metaphors achieve higher human scores for the best submissions. this could be explained by the visual nature of comparisons drawing from commonsense property identification which can benefit from elaboration as used in the dream framework used by teamcooldoge. ## conclusion this paper summarizes the results of the shared task on understanding figurative language organized as part of the 3rd workshop on the figurative language processing at emnlp 2022 (figlang 2022). this shared task aimed to not only predict the correct label for a figurative nli instance but also generate a convincing explanation for the same. we provided basic description of each of the participating systems who submitted a shared task system paper (i.e. four qualifying submissions). all of the submitted systems by the participants attain higher accuracy than the baseline. we also conducted human evaluation via mturk platform that shows the quality of explanations generated by the systems is comparable. finally, to conclude, we hope the shared task will promote further exploration into figurative language understanding.