Multilingual and Code-Switched Sentence Ordering (2024)

Abstract: Sentence Ordering (SO) is a linguistic task which requires re-ordering of shuffled sentences into a coherent paragraph. SO has downstream applications, but also serves as a semantic probe for computational models as this capability is essential for understanding narrative structures, causal and temporal relations within texts. Despite its importance, prior research has been limited to predictable English language structures and has not thoroughly addressed the complexities of multilingual and varied narrative contexts. To fill this gap, we introduce a novel and comprehensive Multilingual Sentence Ordering task that extends SO to diverse narratives across 12 languages, including challenging code-switched texts. We have developed MultiSO, a new benchmark dataset that represents these challenges. Our findings reveal that both specialized sentence ordering models and advanced Large Language Models like GPT-4 face significant challenges with this task.

## introduction advances in language models (lms) have increased focus on general language understanding through increasingly sophisticated tasks requiring a deeper understanding of meaning in text. these advances are underpinned by improved representation learning of core linguistic units (morphemes, words, sentences) via methods like subword tokenization, masked lms, and next sentence prediction -combined with significant increases in model size. at the sentence level, the self-supervised task of re-ordering shuffled tokens and sentences to recover the original sequence has been used, e.g., in bart @xcite . sentence ordering (so) 1 is a task that extends the permutation recovery approach to the paragraph 1 sometimes called sentence arrangement or re-ordering. numerous computational approaches to so have been explored @xcite . however, there are several shortcomings. to our knowledge, all so research has been on english. further, most work uses sentences from paper abstracts or text describing entities, and recent work has shown that these texts have similar and highly regular structures, allowing models to learn simple shallow cues that result in shortcut learning @xcite . to address these gaps, we propose a comprehensive multilingual so task using varied narratives spanning several domains and 12 languages, including challenging code-switched passages. our proposed multilingual so task is depicted in figure 1 . experiments on multiso, a new benchmark dataset that we have created, show that both models trained specifically for so, as well as stateof-art llms (gpt-4), struggle on this task. in sum, our contributions include: sentence ordering is a longstanding task within natural language processing research @xcite . so also has more direct downstream applications in text summarization @xcite , retrieval-dependent qa @xcite , and concept-to-text generation @xcite . more recently, the task has gained attention with the rise of neural language models @xcite . for a comprehensive overview of the work in the area, we refer the reader to the recent survey by @xcite . research on so is nascent, and there is a paucity of benchmark tasks and datasets. datasets such as roc stories @xcite provide well-structured, simple narratives composed of five sentences, purposely crafted to model coherent story progression in a strictly monolingual (english) context. similarly, datasets based on abstracts from nips, acl, and arxiv papers @xcite focus on the logical sequence of scientific ideas, yet are confined to english language scholarly texts. these datasets predominantly support tasks that require understanding simple, linear narrative structures in solely english contexts. contrasting this, our work extends beyond the monolingual framework by introducing a novel, multilingual dataset that includes code-switching, addressing the complexities of interlaced linguistic elements. additionally, our dataset encompasses a broader spectrum of intricate narration styles, thereby challenging models to grasp and generate more sophisticated narratives. while so is intrinsically interesting, it is also relevant to research on using lms to generate semantic representations of text: @xcite find that so is an important pretraining task for downstream task performance in a monolingual setting. we hypothesize that multilingual so, particularly a code-switched variant, may help align semantic representations across languages. our work tries to address some of the above shortcomings by proposing a new multilingual so task, and developing a new corresponding dataset (multiso) to further research in this area. ## multilingual sentence ordering to address current gaps in the literature, we design a new so task which is more challenging. we focus on the following areas: ## conclusion and future work we proposed a multilingual so task and dataset, and showed that it is challenging, particularly for code-switched data. although based on a well explored monolingual so task, our research is the first to address the gap in research covering non-english and code-switched languages. our multiso dataset uses narratives describing stories from creative works, making it varied and providing a challenge for language models. this dataset is the first to explore the multilingual and mixed-language directions. we expect this task and data will facilitate research in several areas. the task enables evaluation of lm representations and model reasoning over longer language units and sequences. each task also covers multiple languages, making it possible to study cross-lingual transfer using multiso. in future work, we plan to: (1) expand the dataset with more languages and narrative types to further provide researchers with valuable resources for enhancing multilingual language models (2) perform a deeper investigation on using models to solve the task, in particular llms.