Q uote R : A Benchmark of Quote Recommendation for Writing (2022)

Abstract: It is very common to use quotations (quotes) to make our writings more elegant or convincing. To help people find appropriate quotes efficiently, the task of quote recommendation is presented, aiming to recommend quotes that fit the current context of writing. There have been various quote recommendation approaches, but they are evaluated on different unpublished datasets. To facilitate the research on this task, we build a large and fully open quote recommendation dataset called QuoteR, which comprises three parts including English, standard Chinese and classical Chinese. Any part of it is larger than previous unpublished counterparts. We conduct an extensive evaluation of existing quote recommendation methods on QuoteR. Furthermore, we propose a new quote recommendation model that significantly outperforms previous methods on all three parts of QuoteR. All the code and data of this paper can be obtained at https://github.com/thunlp/QuoteR .

## introduction a quotation, or quote for short, is a sequence of words that someone else has said or written. quotes, especially the famous quotes including proverbs, maxims and other famous sayings, are quite useful in writing -they can not only help illuminate and emphasize the meaning we want to convey, but also endow our writing with elegance and credibility @xcite . as a result, the use of quotes is very common and, moreover, universal among all languages. however, it is not an easy job for ordinary people to promptly come up with appropriate quotes that fit the current context of writing, due to the huge number of quotes. search engines can provide some help in finding quotes by keyword matching, but it is often not enough. quotes generally "there's an old bible verse my dad used to say all the time that says sufficient unto the day is the evil thereof," pyron said. "in other words -today has its own set of problems, we can't do anything about yesterday, and i don't want to jump too far into tomorrow." express their meanings implicitly by rhetorical devices like metaphor and have different word usages from modern and everyday writing, as illustrated in figure 1 , for which quote search based on keyword matching is ineffective. in addition, some quote repository websites organize quotes by topic. however, even after filtering by topic, there are still too many candidate quotes, and selecting a suitable one remains time-consuming. to tackle these challenges, @xcite introduce the task of quote recommendation, aiming to automatically recommend suitable quotes given the context of writing. 1 afterward, a series of studies propose various approaches to this task @xcite @xcite . however, these studies use different evaluation datasets, and none of them are publicly available. the lack of a standard and open dataset is undoubtedly a serious obstacle to the quote recommendation research. in this paper, to solve this problem, we build a large quote recommendation dataset that is publicly available. this dataset is named quoter (abbreviated from quote recommendataion) and composed of three parts: (1) the english part that comprises 6,108 english quotes with 126,713 contexts; (2) the standard chinese (mandarin) part, which contains 3,004 standard chinese quotes with 40,842 contexts; and (3) the classical chinese (wenyan) part, which comprises 4,438 classical chinese quotes (including classical poems) and 116,537 contexts. any part of this dataset is abso-lutely larger than, or even doubles, previous closedsource counterparts. we conduct a fair and extensive evaluation of existing quote recommendation methods on quoter with a thorough set of metrics. by analyzing these methods and their evaluation results, we find two weaknesses of these methods and propose a new method by making corresponding improvements, which we believe would serve as a strong baseline for quote recommendation. first, most existing methods encode contexts and quotes into vectors for quote-context matching, using lstm @xcite or cnn @xcite as the encoders. these encoders have proven inferior to the pre-trained language models like bert @xcite , which limits the final quote recommendation performance. therefore, we try to utilize a pre-trained language model, specifically bert, as the sentence encoders to learn representations of quotes and contexts. considering the huge compute resulting from the large scale of the dataset and the bert model, it is nontrivial to train the context and quote encoders simultaneously. we design an ingenious training strategy to address this issue. second, it is harder to learn good representations for quotes compared with contexts, because most quotes are quite pithy, and their words usually carry rich semantics, as shown in figure 1 . existing methods, however, do not address this challenge well. to handle this challenge, we incorporate a kind of general lexical knowledge, namely sememes, into the quote encoder, aiming to improve the representations of quotes. a sememe is defined as the minimum semantic unit in linguistics @xcite , and the sememes of a word atomically interpret the meaning of the word. incorporating sememes can bring more semantic information for quote representation learning and conduce to a better quote vector. in experiments, we demonstrate that both the utilization of bert and the incorporation of sememes substantially improve quote recommendation performance. and the sememe-incorporated bertbased model significantly outperforms all previous methods on quoter. moroever, ablation and case studies as well as human evaluation further prove its effectiveness. to conclude, our contributions are threefold: (1) building a large and the first open quote recommendation dataset; (2) conducting an extensive and fair evaluation of existing quote recommendation methods; (3) proposing a quote recommendation model that outperforms all previous methods and can serve as a strong baseline for future research. ## task formulation before describing our dataset and model, we first formulate the task of quote recommendation for writing and introduce several basic concepts, most of which follow previous work @xcite . in the task of quote recommendation for writing, a query context @xmath0; c r ] is given, and the gold quote q c is wanted, where the query context is the context provided by the user and the gold quote is the quote in the quote set that fits the query context best. theoretically, a query context may have more than one gold quote because there are some quotes that convey almost the same meaning. following previous work @xcite , for simplicity, we only regard the quote that actually appears together with the query context in corpora as the gold quote. for a quote recommendation model, given the quote set q, its input is a query context @xmath1; c r ], and it is supposed to calculate a rank score for each candidate quote in q and output a quote list according to the descending rank scores. ## dataset construction in this section, we present the building process and details of the quoter dataset. ## methodology in this section, we elaborate on our proposed quote recommendation model. this model is based on the representative pre-trained language model bert @xcite , but can be readily adapted to other pre-trained language models. ## experiments in this section, we evaluate our model and previous quote recommendation methods on quoter. ## conclusion and future work in this paper, we build a large and the first open dataset of quote recommendation for writing named quoter and conduct an extensive evaluation of existing quote recommendation methods on it. we also propose a new model that achieves absolute outperformance over previous methods, and its effectiveness is proved by ablation studies. in the future, we will try to improve our model in handling classical chinese quotes by using a special classical chinese pre-trained model to encode them. we will also consider boosting the performance of our model in the few-shot and zero-shot situations. yuan zang, fanchao qi, chenghao yang, zhiyuan liu, meng zhang, qun liu, and maosong sun. 2020. word-level textual adversarial attacking as combinatorial optimization. in proceedings of acl. lei zhang, fanchao qi, zhiyuan liu, yasheng wang, qun liu, and maosong sun. 2020a. multi-channel reverse dictionary model. in proceedings of aaai. yuhui zhang, chenghao yang, zhengping zhou, and zhiyuan liu. 2020b. enhancing transformer with sememe knowledge. in proceedings of the 5th workshop on representation learning for nlp. yukun zhu, ryan kiros, rich zemel, ruslan salakhutdinov, raquel urtasun, antonio torralba, and sanja fidler. 2015. aligning books and movies: towards story-like visual explanations by watching movies and reading books. in proceedings of iccv.