Plot Retrieval as an Assessment of Abstract Semantic Association (2024)

Abstract: Retrieving relevant plots from the book for a query is a critical task, which can improve the reading experience and efficiency of readers. Readers usually only give an abstract and vague description as the query based on their own understanding, summaries, or speculations of the plot, which requires the retrieval model to have a strong ability to estimate the abstract semantic associations between the query and candidate plots. However, existing information retrieval (IR) datasets cannot reflect this ability well. In this paper, we propose PlotRetrieval, a labeled dataset to train and evaluate the performance of IR models on the novel task Plot Retrieval. Text pairs in PlotRetrieval have less word overlap and more abstract semantic association, which can reflect the ability of the IR models to estimate the abstract semantic association, rather than just traditional lexical or semantic matching. Extensive experiments across various lexical retrieval, sparse retrieval, dense retrieval, and cross-encoder methods compared with human studies on PlotRetrieval show current IR models still struggle in capturing abstract semantic association between texts. PlotRetrieval can be the benchmark for further research on the semantic association modeling ability of IR models.

## introduction we propose a new task, plot retrieval, which retrieves the relevant plots from the book for a query. the task is a spontaneous process in humans' daily lives. when reading a book or coming across other life events that remind a plot, humans naturally require to find the target plot. as a result, plot retrieval is a common and natural scenario but has not been well-studied in nlp. although plot retrieval can be formalized as an information retrieval (ir) task, the key challenge odqa: ms marco query: how bits in a pixel? passage: the bit depth is the number of bits used to encode each color channel. today, most digital images use a bit depth of 8, meaning that each color channel in a pixel is represented by 8 bits ... ## plotretrieval we introduce collection, filtering, translation, annotation, and statistics for plotretrieval in this section. more details are introduced in appendix. ## experiments in this section, we evaluate various ir models on plotretrieval and perform human studies. all of them are fine-tuned on ms marco. dense retrieval. (1) dpr (karpukhin et al., 2020), a classical dense retrieval model based on bi-encoder and trained with bm25 hard negatives and in-batch contrastive loss. (2) ance (xiong et al., 2021), it dynamically updates negatives during training. (3) tas-b (hofstätter et al., 2021) is trained with supervision from cross-encoder. (4) berm (xu et al., 2023a,b), a plug-and-play method to enable dense retrieval models to learn representations that are more suitable for matching. (5) ernie-search (lu et al., 2022) trains dense retrieval model by cascade distillation from colbert (khattab and zaharia, 2020) and crossencoder. all of the above baselines are fine-tuned on ms marco. there are also some methods first pre-train models on large-scale datasets by self-supervised ir signal. (6) coco-dr (yu et al., 2022) is pre-trained on beir (thakur et al., 2021). (7) cocondenser (gao and callan, 2022) and (8) retromae (xiao et al., 2022) are pretrained on english wikipedia and bookcorpus. (8) contriever (izacard et al., 2022) is pre-trained on english wikipedia and ccnet. all of these models are fine-tuned on ms marco after pre-training for ir. late-interaction. colbert (khattab and zaharia, 2020) performs late interaction on embeddings of each token to achieve finer-grained interaction than dense retrieval. this model is fine-tuned on ms marco. re-ranking. we use cross-encoder @xcite ) that exploits self-attention for interaction between tokens as re-ranker, which has shown power in book qa tasks @xcite . before reranking, we first use contriever to retrieve top-100 documents for each query as its candidate list. this model is fine-tuned on ms marco. ## conclusion in this paper, we propose a novel task called plot retrieval that retrieves relevant plots from the book for a query. compared with the existing ir datasets, plot retrieval requires the ir models to have the strong ability to capture the abstract semantic association between texts rather than the simple lexical and semantic matching. it is meanly because readers integrate their own understanding, summaries, or speculations of the plot when writing the query. for the plot retrieval task, we propose plotre-trieval, a large labeled dataset with more abstract semantic association and less word overlap between texts, which can be used as a benchmark to train and evaluate the ability of ir models to capture abstract semantic associations between texts. extensive experiments across various lexical retrieval, sparse retrieval, dense retrieval, and crossencoder methods compared with human studies on plotretrieval show that the current ir models still struggle in capturing abstract semantic association between texts and there is a lot of room for improvement in future research.