Investigating Robustness of Dialog Models to Popular Figurative Language Constructs (2021)

Abstract: Humans often employ figurative language use in communication, including during interactions with dialog systems. Thus, it is important for real-world dialog systems to be able to handle popular figurative language constructs like metaphor and simile. In this work, we analyze the performance of existing dialog models in situations where the input dialog context exhibits use of figurative language. We observe large gaps in handling of figurative language when evaluating the models on two open domain dialog datasets. When faced with dialog contexts consisting of figurative language, some models show very large drops in performance compared to contexts without figurative language. We encourage future research in dialog modeling to separately analyze and report results on figurative language in order to better test model capabilities relevant to real-world use. Finally, we propose lightweight solutions to help existing models become more robust to figurative language by simply using an external resource to translate figurative language to literal (non-figurative) forms while preserving the meaning to the best extent possible.

## introduction human frequently employ figurative language such as metaphors @xcite and idioms @xcite for effective and/or stylistic communication. thus, dialog models interacting with humans should be equipped to handle these forms of communication. however, understanding figurative language might be challenging for machines since figurative constructions often exhibit noncompositional semantics and may rely on shared cultural and common-sense knowledge @xcite . for example, a powerful gpt2 model fine-tuned on dailydialog dataset is unable to handle the metaphor 'built on the sand' (figure affected by figurative constructs in dialog context. here, the model conflates the metaphorical use of build on the sand with its literal meaning, leading to an inappropriate, atopical response. in this work, we investigate the performance of existing dialog models when faced with inputs containing figurative language use. (1) first, we identify the subsets in existing datasets (such as daily-dialog @xcite and personachat @xcite ) which have figurative language use such as metaphors and similes. we observe that the performance of all the dialog models under consideration is lower on such subsets containing figurative language use compared to the dataset as a whole. (2) second, we gather manually written literal/non-figurative equivalents of the dialog utterances in dailydialog and personachat which exhibit figurative language use. for example, literal version of 'on the sand' can be 'unstable' (figure 1 ). we observe that performance of dialog models improves when using literal equivalents in place of figurative language. we release the resulting datasets, and encourage that new dialog models be tested separately on such datasets to understand and measure their ability to handle figurative language. ## figurative language in open domain dialog we experiment with dailydialog (dd) dataset @xcite , which is an open domain dialog corpus with 13.1k conversations on colloquial topics like tourism, health etc, of which 1k dialogs (6.74k utterances) form the test split. to carry out the desired analysis, we need to first identify the utterances in the dataset which have figurative language use. to achieve high precision labeling, we rely on manual annotations instead of using external figurative language detectors/classifiers. the task was performed manually by two graduate students (native english speakers) studying in a university with english as the primary language of instruction. additionally, we request the annotators to also write down the literal equivalent versions of the utterances containing figurative language. we release the resulting subset of dailydialog dataset as dailydialog-figurative (dd-fig), consisting of those dialog instances which contain figurative language, along with two manually written literal versions of each utterance. though figurative constructs are only mildly frequent at an utterance level (2.2% in dd), their frequency of occurring at least once in a dialog is ≈6 times higher (13.1%). this means that model sensitivity to figurative constructs is more significant an issue than mere utterance-level frequency suggests, since figurative constructs occurring anywhere in a dialog potentially affect all model responses for that dialog. additionally, handling figurative constructs is still critical to robust longtailed performance @xcite , which matters for worst-case behaviour and user satisfaction @xcite . lastly, most the prevalent figurative constructs we observe are metaphors (45.8%), idioms (47.3%), rhetorical questions (5.3%), hyperbole (6.9%), and personification (3.8%). appendix §b explores further properties of figurative prevalence. relative performance of models: we additionally investigate if the relative ranks of the mod- figurative in last utterance bleu-3/4 +98% / +477% na meteor/rouge-l +5.3% / -3.6% na human-rating +12.1% +17.2% figurative anywhere in dialog history bleu-3/4 +13% / +79% na meteor/rouge-l +3.9% / -3.0% na human-rating +13.7% +19.5% does replacing figurative language with semantically equivalent literal translations lead to better performance? above analysis only reports correlation in performance with contexts containing figurative language. however, there could be certain confounding factors involved. thus, to make a more direct comparison, we will next compare results when using figurative contexts versus their literal counterparts. to perform this experiment, we utilize the human written literal versions in dd-fig, and experiment with the gpt2 model (which is the best performing model as per human rating on the overall dataset). we report results under two setups: (1) when figurative language is present in the last utterance of the dialog history, and (2) when figurative language is present anywhere in the dialog history. human ratings are collected using the same procedure as described for experiments with personachat (pc): per-sonachat @xcite is a persona grounded dialog corpus, with 1k dialogs (7.8k utterances) forming the test split. we consider a gpt model fine-tuned on the train split of the pc dataset. we follow the same tagging and annotation procedure for the test split, and refer to the resulting dataset as personachat-figurative dataset (pc-fig). results in we notice that compared to dailydialog, per-sonachat utterances tend to be shorter, more informal, and highly spoken-language like utterances, with fast topic transitions. on replacing figurative language with its literal counterpart into such contexts, the replaced literal text, which is typically english of a more formal and written variety, ends up being much more out of sync with the context in the lexico-syntactic/stylistic sense than it is for dailydialog. this has a slight downward effect on metrics, offsetting some of the gains from replacing away the figurative language. ## mitigation we propose a lightweight mitigation approach wherein we use existing resources to detect and then construct literal translations for two popular figurative constructs: metaphors and idioms. thus, the proposed mitigation approach does not require any retraining of dialog models. metaphor detection through classifier: we train a metaphor detection classifier based on the vua corpus @xcite foot_1 . to better generalize to external data via recent contextual models, we skip using model by @xcite , and instead learn a classifier c met bert by finetuning the bert-base-uncased @xcite checkpoint from @xcite . on vua, c met bert gets a test f1 of 0.724, which is close to @xcite 's value of 0.726. next, we run each test utterance in dilaog dataset through c met bert to get its probability p met of being metaphorical. to retain only more reliable predictions, especially consid-ering domain shift w.r.t vua, we only choose utterances with p @xmath0.9. the set of metaphorical utterances thus identified is d met auto . idiom detection through lexicon: idioms are frequently used expressions, which have a fixed, typically non-compositional meaning understood by most native speakers from cultural precedent. we curate a lexicon of 2048 commonly used idioms (e.g. filling his shoes) from an online source foot_2 -see appendix a.2 for more details). all utterances with at least one lexicon entry as a substring are identified to create the set of automatically detected idiomatic utterances, d idiom auto . we unify the sets detected above to form d f ig @xmath1. |d f ig auto | constitutes 1520 of 6740 utterances for dd (22.5%) and 911 of 7801 utterances for pc (11.7%) respectively. dictionary replacement: wiktionary @xcite -the collaboratively created online dictionary, provides a curation of entries corresponding for phrases with "idiomatic" foot_3 usages. these entries encompass conventionalized metaphors foot_4 , idioms, euphemisms, commonly used similes etc. each entry lists the surface form of the figurative construct paired with a gloss. glosses are for the most part literal interpretations of the figurative construct. however, they often bear other details like dialect("us"), etymology("archaic") etc, which we remove through simple regex-based rules. this allows direct use of the now-cleaned gloss as a literal interpretation in-context. furthermore, we expand entries whose surface forms contain uninflected verb forms or unrealized pronouns indicated by someone, one's etc, spawning one new entry per pronoun-inflection combination foot_5 this yields us a dictionary with 17, 743 tuples of the form {f ig i , lit(f ig i )} foot_6 . finally, for each detected utterance @xmath2, each matched occurrence of f ig i is replaced by lit(f ig i ), ∀1 ≤ @xmath3. ## related work past work has explored fine-grained analysis and understanding of the performance of dialog models @xcite . @xcite analyze open domain dialog systems for skills such as inferring contradictions and determining the topic of conversation inter alia. @xcite analyze the change in perplexity when applying certain perturbations in dialog history. past work has analyzed dialog models from the point of view of safety from toxic language @xcite , and gender biases @xcite . @xcite analyze how well dialog models respond to utterances from infrequent sentence function types (e.g negative declarative utterances like i feel bad today.). @xcite propose to identify the categorical mapping of an indirect response with respect a polar question in a task oriented dialog setup. challenges in handling metaphors and idioms has been explored in prior work on machine translation @xcite @xcite . @xcite propose a method to identify metaphors in english text, and paraphrase them into their literal counterparts before translating to chinese. our work on analyzing dialog models against figurative language contexts is along similar direction, though the task setup and scope of figurative language involved are different. figurative language generation has received reasonable attention such as simile generation @xcite and idiom generation @xcite . compared to them, our focus is on analyzing capability of popular contemporary dialog models when faced with figurative language. ## conclusions in this work, we demonstrate how existing dialog models fall short in handling figurative language use, and propose a light-weight mitigation technique to ameliorate this lacuna. we encourage future research in dialog models to separately analyze and report model performance the mitigation techniques used by us are pretty lightweight, but are not able to capture many occurrences of figurative language. future work could look into improved techniques for figurative language detection. our work is limited to a couple of open domain dialog datasets in english language. similar analyses could be conducted on goal oriented dialog setups and datasets in other languages. et al., 2018) as well as the multiple references dataset from @xcite . we do collect human evaluation ratings using crowd-sourcing. however, we neither solicit, record or request any kind of personal or identity information from the annotators. our work primarily performs experiments on dialog in english @xcite . dialog models are known to suffer from biases learnable from dialog training data, such as gender bias @xcite . however, our work and contribution does not present or release any new models or model checkpoints, and is primarily concerned with more careful evaluation of a particular phenomena (i.e figurative language), and discussion on lightweight mitigation strategy related to the same.