G up S hup: Summarizing Open-Domain Code-Switched Conversations (2021)

Abstract: Code-switching is the communication phenomenon where the speakers switch between different languages during a conversation. With the widespread adoption of conversational agents and chat platforms, code-switching has become an integral part of written conversations in many multi-lingual communities worldwide. Therefore, it is essential to develop techniques for understanding and summarizing these conversations. Towards this objective, we introduce the task of abstractive summarization of Hindi-English (Hi-En) code-switched conversations. We also develop the first code-switched conversation summarization dataset - GupShup , which contains over 6,800 Hi-En conversations and their corresponding human-annotated summaries in English (En) and Hi-En. We present a detailed account of the entire data collection and annotation process. We analyze the dataset using various code-switching statistics. We train state-of-the-art abstractive summarization models and report their performances using both automated metrics and human evaluation. Our results show that multi-lingual mBART and multi-view seq2seq models obtain the best performances on this new dataset. We also conduct an extensive qualitative analysis to provide insight into the models and some of their shortcomings.

## introduction conversation summarization is the process of generating a condensed version of a given conversation while preserving the most salient aspects. with the extensive use of various chat applications such as messaging apps and virtual assistants @xcite , there has been a growing interest in * emails ids of the corresponding authors are, lm4428@nyu.edu, debanjan.mahata@moodys.com, rajivratn@iiitd.ac.in. authors 1,2,3, and 4 have equal contributions. debanjan mahata participated in this work as an adjunct faculty at iiit-delhi. the abstractive summarization of written conversations @xcite @xcite . automatic conversation summarization has potential applications in various fields such as healthcare @xcite , call centers @xcite , education @xcite , and many other areas @xcite . one of the biggest challenges in conversation summarization has been the lack of large datasets with human-annotated summaries. most researchers evaluate their summarization techniques on transcriptions of ami @xcite , or icsi meeting corpus @xcite using the meeting topics as summaries. these corpora are very useful for various speech-related research problems, but they do not represent written conversations in chat applications. recently, @xcite published the samsum corpus, which contains over 16,000 written english conversations and their corresponding manually annotated summaries. though these conversations were not extracted from actual chat applications, they were created by linguists to replicate natural conversations. to our knowledge, this is the largest summarization dataset for written conversations. the samsum corpus is monolingual (english); therefore any model trained on this dataset may not adapt effectively to multi-lingual, especially code-switched conversations where speakers alternate between different languages within the scope of a conversation or even an utterance @xcite @xcite . code-switching is commonly observed during interactions between peers who are fluent in multiple languages. for example, in the indian subcontinent, it is common for people to alternate between english and other regional languages like hindi over the course of a single conversation. code-switching is an integral part of both written and spoken conversations for various multi-lingual communities across the world @xcite . developing models that can accurately process code-switched text is essential to the proliferation of nlp technologies to these communities and contributes towards the diversity and inclusivity of language resources @xcite . however, building such models would require high-quality human-curated datasets. this paper introduces the task of abstractive summarization of open-domain code-switched written conversations. namely, given a multi-party conversation in hi-en, the objective is to generate a summary in english, as shown in ## background in the linguistics community, code-switching typically refers to the change of language or grammatical systems from one utterance to another within the same conversation @xcite . on the other hand, code-mixing refers to the use of linguistic units such as phrases, words, or morphemes of one language in the utterance of another language @xcite . in other words, code-switching is an interutterance, and code-mixing is an intra-utterance phenomenon. however, in this paper, we use the term code-switching to refer to both these concepts. code-switching has started to gain some traction from computational linguists over the last few years @xcite , where they developed datasets for many interesting problems such as language identification @xcite , part of speech tagging @xcite , question answering @xcite , and named entity recognition @xcite . additionally, researchers have also started to develop objective metrics that characterize the complexity of code-switching in a given corpus @xcite . for a thorough review of datasets and other developments in this space, we recommend the review paper from @xcite . most of the code-switched datasets typically contain individual posts or comments from social media applications like twitter and facebook anno-dataset conversational task @xcite language identification and pos tagging @xcite pos tagging @xcite question answering @xcite language identification @xcite language identification @xcite question answering @xcite information retrieval @xcite language identification @xcite hate-speech text classification @xcite question answering @xcite close domain conversation system @xcite question answering @xcite sentiment analysis @xcite named entity recognition. @xcite dependency parsing @xcite accommodation quantification @xcite natural language inference gupshup (this work) ## data collection the goal of our annotation process is to build a hi-en code-switched conversation summarization dataset. we first considered the option of creating summaries for an existing code-switched conversational dataset like dstc2 @xcite . though this dataset is substantially large with over 50,000 utterances, it is not open-domain and only contains human-computer conversations focused on restaurant reservations, thus lacking linguistic diversity. we, therefore, chose the option of manually translating the samsum corpus @xcite from en to hi-en. annotation process -we hired eight annotators who are fluent in both hindi and english. we first explained the concept of code-switching and provided them with a few reference examples annotated by the authors. based on our interactions with the annotators, we observed that hi-en codeswitching was an integral part of their vernacular. they also frequently used code-switching on social media and chat applications. we first provided each annotator with a random sample of ten conversations. we instructed them to first go through the conversation and the corre- after the annotators completed translating the initial random samples, we provided feedback in terms of format and organization of the data. once they were comfortable with the process, we assigned them random batches of conversations, and they worked independently based on their schedules. the contributions of each annotator were mainly driven by their availability. due to time and resource constraints, we only had one translation for a given source conversation. the entire annotation process lasted for around three months, at the end of which we translated 6,831 conversations containing 76,330 utterances. to our knowledge, this is also the largest parallel corpus for hi-en and en languages containing 109,346 sentences, out of which 48,578 are code-switched. ## corpus analysis figure 1 shows the distribution of the number of utterances per conversation. gupshup has 76,330 utterances, where the shortest conversation has one utterance and the longest has 31 utterances. the average length of a conversation is 11.17 utterances. the english version of the corpus had 28.6 words per utterance and 19,252 unique words, whereas gupshup has 31.1 words per utterance and 25,865 unique words. a large portion of the data (73.6% of conversations) had only two participants, 18.7% of the conversations had 3 participants, 5.2% had 4 participants, and the remaining conversations had more than 4 participants. language tagging -since gupshup is a codeswitched corpus, it is essential to analyze and quantify the complexity of code-switching in the corpus, which requires us first to identify the language associated with each token. since this is a parallel corpus, we used information from the source utterances to determine non-english tokens in the translated utterance. more specifically, for a given pair of utterances, we first removed all the punctuation marks, emojis, and numerical tokens. we then identified named entities in the english utterance, and if these entities also appeared in the translated utterance, the corresponding tokens were excluded from language tagging. of the remaining tokens, if any of them appeared in the english utterance, we tagged them as english. we repeated the same process on the remaining tokens but with their lemmatized versions because we observed that sometimes the annotators used the english word from a source utterance in a different part of speech. for e.g. the phrase "a few years after we graduated" was translated to "graduation ke kuch saal baad". here graduation can be accurately tagged as english using lemmatization. lastly, the remaining tokens in the hi-en utterance were tagged as hindi. we observed that this process captured most english tokens in the translated utterances, except when the annotators introduced a new english token. for example, one annotator translated the phrase "i have been thinking for a while" to "main kaafi time se soch rahi hu", where they introduced the token time. however, we observed that this was a rare phenomenon and did not sway the analysis significantly. code-switching statistics -per the language tagging approach described above, 18.15% (13,760) of the utterances were entirely in transliterated hindi, 23.49% (17,804) were entirely in english. the majority of the utterances, 58.86% (43,407), were vocabulary size 26,865 english vocabulary 11,616 hindi vocabulary 12,016 total utterances 76,330 unique utterances 75,791 code-switched utterances 43,407 hindi utterances 13,760 english utterances 17,804 avg. length of utterance 10.07 avg. # of code-switched utterances per conversation 6.35 # of code-switched utterances with hindi matrix 45,644 # of code-switched utterances with english matrix 2,934 # of hindi insertions into english matrix 2,810 # of english insertions in hindi matrix 38,539 based on the matrix language and token-level language tags, we further analyzed the codeswitching complexity using the metrics c avg (gambäck and das, 2016), c c @xcite , and i-index @xcite . these metrics quantify complexity in terms of the number of foreign language tokens and switch points. on our corpus, we estimated that c @xmath0.25, c @xmath1.57, and i-index was 0.14. for reference, @xcite applied these metrics to the different code-switching datasets from @xcite and observed that english-nepalese corpus had the highest levels of switching with c c value of 49.06 and c avg value of 7.98. these metrics show that gupshup has high levels of code-switching complexity. ## empirical benchmarks in our experimental work, we employ the following six abstractive summarization models: gpt-2, bart, pegasus, t5, multitask t5, mbart, and multi-view seq2seq model @xcite . most of these transformer models were pre-trained on english corpora, except for mbart, which was trained on multilingual data. we expect our results to serve as empirical benchmarks for future researchers. we used 5,831/500/500 conversations as our train, dev, and test splits, respectively. we trained all the models for three epochs, with evaluation on the dev set after each epoch. due to constraints on computing resources, we used a smaller mbart model with a 12-layer encoder and a 6-layer decoder. for the t5 model 5 , we tried both fine-tuning and multitask learning approach (t5 mtl); for the latter approach, we trained the model for both summarization and translation. we ran this entire process for three experimental setups: results -the results are summarized in the t5 mtl model outperformed the t5 model across all metrics and also achieved the best me-teor score. we applied the wilcoxon-signed rank 5 we also trained mt5. however, even after training for 30 epochs with early stopping we barely got a rouge-1 score of 31.06, rouge-2 score of 9.41 and rouge-l score of 25.33, which were lesser than what we obtained using t5. on manually analyzing the results, we observed that the summaries produced by mt5 were not grammatically sound, there were lot of repetitions, the entities mentioned in the summaries were wrong, and relevant points from the conversations were missing in the summaries. this is very surprising. since we decided to keep only the top performing models in the final results table and could not understand the reason behind such poor performance of mt5 we did not pursue it further. in the second experimental setup, where english summaries were generated from english conversations, we observed a significant improvement for all the models and across all the metrics. this observation is expected as most of these models have been pre-trained on english data. the multi-view model obtained the best r1, bleu, and meteor scores, whereas the pegasus model obtained the best r2, rl, and bleu scores. once again mbart obtained better scores than bart, but the difference is less pronounced than for hi-en conversations. it is unclear if the superior performance of mbart is due to multilinguality or because it was pre-trained on more english data. in the future, we would like to explore the use of devanagari script for the portions of the conversations that are in hindi as opposed to the romanized script since models like mbart have been exposed to devanagari script. in both these experimental setups, we observed that gpt-2 model obtained the lowest scores, but further analysis helped us realize that the summaries generated by this model were not semantically meaningful and often contained repeated words. this could be attributed to the model's architecture which contains only a decoder and can-not, therefore, create a meaningful representation of the source conversation @xcite . additionally, we also observed minimal variance in bertscore values across the models, suggesting it may not be the most effective metric for measuring the quality of summaries. in the third experimental setup, we generated hi-en summaries from hi-en conversations. here we observed a steep drop in performance of all the models when compared to generating english summaries. however, this drop was less prominent for bart, which obtained the best scores across most metrics. another interesting observation is that, despite being multilingual, mbart generated poor hi-en summaries. overall, from this experiment, we observe that most language models may not be capable of generating hi-en summaries. human evaluation -we conducted a human evaluation of the english summaries generated by all the models for a random selection of 100 hi-en conversations. following @xcite , we had three annotators rate the summaries for the following four metrics foot_4 : consistency, coherence, fluency, and relevance on a likert scale of 1 to 5. the results, average likert score normalized to the range 0 to 1, are summarized in the multi-view model obtained the best scores for consistency and relevance, whereas the pega-sus model obtained the best scores for coherence and fluency. the mbart model also obtained comparable scores. these observations are reasonably consistent with the numbers in ## qualitative analysis error analysis -to get a better insight into the generated summaries, we chose a random sample of 100 conversations both english and hi-en codeswitched, and analyzed english summaries generated by all the models. we classified each summary into one or more of the following four error categories: (a) missing information (mi): the generated summary is missing a salient aspect mentioned in the reference summary, (b) erroneous reference (er): one of the speakers is incorrectly associated with an action or location, (c) incorrect inference (ii): summary contains a statement that cannot be inferred or reasoned from the conversation, and (d) wrong pronouns (wp): summary misrepresents a speaker's gender. these classes were not predetermined, but grew out of the error analysis. as with automatic metrics ( the ii errors have increased significantly for all the models when summarizing hi-en conversations, further demonstrating the inability of the models to understand code-switched conversations. we noticed that even though the models select salient aspects of the conversations, the summaries have incorrect inferences because some of the critical tokens in the conversation were code-switched. summary compression -we measured the compression ratio of the summaries generated by all models for both en and hi-en conversations as show in figure 2. per @xcite , compression ratio is defined as the ratio of the number of tokens in the conversations to that of the summaries. we observe that all the models seem to generate shorter summaries for hi-en conversations. this is understandable considering that most of these models are pre-trained on english data; therefore, when exposed to hi-en conversations, the english content becomes less frequent and frag- pegasus t5 bart mbart gpt multi-view t5-mtl reference 2 4 6 model compression ratio en hi-en figure 2: compression ratios of summaries generated by all the models. pegasus t5 bart mbart multi-view reference 20 40 60 model compression ratio uni-gram bi-gram tri-gram 4-gram mented, resulting in the models under-generating or even stopping early. compared to reference summaries, bart, mbart, and pegasus seem to generate more compressed summaries, whereas t5 and gpt-2 generate longer summaries. this could also explain why t5 has fewer mi errors. quantifying abstractiveness -during our error analysis, we also observed that the summaries generated from english conversations were often more extractive than abstractive, where the models were selecting the most salient phrases in the source conversations. following @xcite , we quantified the abstractiveness of the summaries by calculating the portion of new n-gram generated by the models for english. namely, we report the ratio of n-grams that were part of the summaries but not the source conversations to the total number of n-grams in the summaries. when compared to reference summaries, all the models seem to be more extractive in nature. of the five models here, t5 generated the most novel n-grams despite not introducing many new uni-grams, which is consistent with the low compression ratio of t5. model diversity -since we have employed different language models in our work, we wanted to understand how similar are the summaries generated by these models. to this end, we measured the rouge-1 and rouge-4 scores between the summaries from each pair of models. the results are summarized in figure 5 . we observe that the rouge-1 scores are significantly higher than rouge-4 scores for both en and hi-en conversations suggesting that the models seem to be choosing a similar set of tokens for summarization. however, the word order is very different, thus suggesting that the eventual summaries are different. ## conclusions and future work in this work, we presented the first code-switched conversation summarization dataset -gupshup, which has over 6,800 multi-party open-domain conversations and their corresponding summaries in en clark: hi, tumhaari profile kaafi interesting hai louise: thank you :) clark: yaha jaada john cage ke fans nazar nhi aate louise: mujhe uske music ke fans khi bhi nazar nhi aate clark: sahi baat clark: yaha pr saturday ko ek gig hai louise: tribute to john cage louise: ha mujhe pta hai! clark: do you wanna go? louise: mene phle hi ticket leli hai so sure :) clark: great! :) reference summary: clark and louise are john cage's fans. they will meet at the tribute concert on saturday. louise already has her ticket. bart: clark and louise will go to john cage's tribute gig on saturday. mbart: clark and louise are going to a tribute to john cage gig on saturday. multiview: louise is going to john cage's gig on saturday. clark is going too. pegasus: clark and louise are going to a gig on saturday to pay tribute to john cage. t5: clark and louise are going to john cage's tribute gig on saturday on saturday. they have a ticket for the gig, so they're going to buy it. t5_mtl: clark and louise want to go to a tribute to john cage gig on saturday. louise has a ticket for the gig. and hi-en. we quantified the performance of various state-of-the-art neural models using both automatic metrics and human evaluation. mbart and multi-view models obtained the best scores on automatic metrics, whereas pegasus and mbart obtained best scores on human judgement. the t5 model generated relatively longer summaries and was therefore effective in capturing all the salient aspects of the conversation. rogue-based metrics and bleurt were highly correlated with the human judgements but meteor, bertscore, and bleu proved relatively ineffective for this task. we also observed that multi-task learning setup showed promise and we would like to explore this further. we conducted an extensive qualitative analysis of the summaries, which provided very interesting insights into different language models. in the future, we would like to explore new techniques for overcoming the various challenges that we observed in this work and would like to dive deeper in understanding the failures of different language models when applied to code-switched text.