D eep CT -enhanced Lexical Argument Retrieval (2024)

Abstract: The recent Touché lab’s argument retrieval task focuses on controversial topics like ‘Should bottled water be banned?’ and asks to retrieve relevant pro/con arguments. Interestingly, the most effective systems submitted to that task still are based on lexical retrieval models like BM25. In other domains, neural retrievers that capture semantics are more effective than lexical baselines. To add more “semantics” to argument retrieval, we propose to combine lexical models with DeepCT-based document term weights. Our evaluation shows that our approach is more effective than all the systems submitted to the Touché lab while being on par with modern neural re-rankers that themselves are computationally more expensive.

## introduction lexical retrieval models like bm25 @xcite or dirichletlm @xcite are the basis of many of the early argument retrieval approaches @xcite @xcite and also were the most common choice of many participants of the touché lab's shared task on argument retrieval for controversial questions @xcite . a few neural rankers like k-nrm @xcite and cedr @xcite were used by the task participants but showed to be less effective than the task's official dirichletlm-based baseline. interestingly, also newer neural retrieval models like colbertv2 @xcite and laprador @xcite are less effective than bm25 on the touché subset of the beir benchmark for zero-shot retrieval @xcite . in this paper, we propose to improve the effectiveness of lexical argument retrieval models by adding a semantic document expansion step that uses term weights calculated by deepct @xcite . for term weighting, deepct utilizes contextualized word representations generated by bert @xcite and is then finetuned to predict whether a document term is likely to appear in "relevant" queries. at the inference step, a fine-tuned model is applied to every document in the retrieval collection independently from the test queries. hence, an advantage of deepct is that the inference can be done offline before indexing the corpus. another advantage is that deepct does not necessarily require relevance judgments either for training or for inference making deepct beneficial for retrieval tasks in specialized domains that have no or little training data. for our experiments, we use the lexical retrieval models bm25 and dirichletlm and their combination with the query expansion model rm3 @xcite . we test these models on the touché 2020 and 2021 test collections consisting of 49 and 50 test queries on controversial topics respectively, the args.me corpus @xcite as a document collection (about 400,000 documents, i.e., english arguments crawled from online debate portals), and 6,000 graded relevance judgments (not relevant, relevant, and highly relevant) from touché. additionally, we expand the documents in the retrieval corpus based on the term weights predicted by fine-tuned deepct models. to fine-tune the deepct models (originally trained on the ms marco dataset @xcite ) specifically for the argument retrieval task, we make use of the args.me structured documents, consisting of the argument premises, the conclusion, and the main debate topic. we use either the conclusions or the debate topics combined with the conclusions as ground truth terms in the reference field of deepct. afterwards, we apply the finetuned deepct model to the whole args.me corpus and expand the document's premises by repeating terms based on the learned deepct term weights. we compare our approaches with the following baselines: (1) the most effective touché systems (that use bm25 and dirchletlm combined with query and document processing) and (2) bm25 combined with neural re-rankers: (a) a contextualized late interaction model colbert (khattab and zaharia, 2020), (b) pointwise crossencoders monobert (nogueira and cho, 2019) and monot5 @xcite , and (c) a zeroshot listwise re-ranker lit5 @xcite . to evaluate the retrieval effectiveness, we use ndcg@5 @xcite , the official evaluation measure of the touché task. to account for missing relevance judgments (up to 60%; see the evaluation results show that our deepctenhanced lexical argument retrieval approach is more effective than all the systems submitted to the touché task while being on par with modern neural re-rankers that are more computationally expensive. our findings thus may indicate the potential of combining lexical models with semantic document expansion for specialized retrieval tasks like argument retrieval, where little or no training data (in terms of relevance judgments) is available. foot_0 ## related work retrieving relevant arguments from the web is essential to support discussions on controversial topics like 'should bottled water be banned?' @xcite . until now, lexical retrieval models like bm25 @xcite and dirich-letlm @xcite have been the most effective retrievers for this task @xcite . for instance, argument search engines args.me @xcite , argu-mentext @xcite , and targer @xcite , all use bm25 for initial retrieval. however, even though neural retrievers like col-bertv2 @xcite , laprador @xcite , or coco-dr @xcite have led to effectiveness improvements in many domainspecific retrieval tasks as evaluated in the beir benchmark @xcite , for argument retrieval (e.g., the touché subset of beir), lexical retrievers still outperform neural models. studying argument retrieval approaches was also carried out as part of the touché lab's shared tasks on argument retrieval for controversial questions @xcite . most of the participant's approaches used lexical retrieval models (i.e., bm25 and dirichletlm) for initial document retrieval combined with various query processing and reformulation techniques. the initial document candidates were further re-ranked based on the estimated document argumentativeness (i.e., the presence of conclusions and premises) and argument quality. several tested neural rankers, like k-nrm @xcite and cedr @xcite were less effective (measured with ndcg@5) than the lexical models. lexical retrieval models (that rely on an exact match between the query and document terms), conversely, may suffer from "ignoring" the semantic similarity between the query and document terms. hence, we propose to combine lexical retrievers (that are effective for argument retrieval) with document expansion based on estimated semantic term importance (term weights) predicted by deepct @xcite . the deepct model exploits the bert @xcite fine-tuning paradigm by fine-tuning a pre-trained bert model to predict the importance of words in documents w.r.t. reference terms (e.g., query terms). fine-tuning aims to minimize the mean square error between the predicted term weights and the ground truth term weights. the ground truth labels can be generated using documents only, relevance labels, or pseudo-relevance feedback. in our work, we use the documentsonly approach which does not require manual relevance labels for fine-tuning deepct (cf. section 3 for more details). the fine-tuned deepct is then applied to the documents and predicts the document term importance scores. finally, the documents are modified by repeating terms proportionally to predicted weights (w * 100, where term weights @xmath0, 1]), thus boosting the term frequency of the repeated terms in the inverted index. thus, lexical retrieval models that rely on the term frequency as a relevance signal can benefit from repeated "relevant" document terms. it has been shown that combining deepct with lexical models improves the effectiveness of ad hoc retrieval @xcite on general-domain document collections, e.g., ms marco @xcite or wikipedia articles @xcite . thus, we aim to test a combination of lexical re-trieval models with semantic document expansion for the argument retrieval task. for the evaluation of argument retrieval approaches, several datasets @xcite @xcite and the touché test collections @xcite ) emerged. by far the largest and hence one of the most frequently used document collections, the args.me corpus, contains about 400,000 arguments crawled from online debates on controversial topics @xcite . the touché shared tasks on argument retrieval for controversial questions also used the args.me corpus. additionally, the task organizers created and published manual relevance judgments and runs (ranked results) submitted by the task participants. our experiments use the touché data from the years 2020 and 2021. ## data and approach data. in this work, we use the datasets from the touché 2020 and 2021 tasks on controversial argument retrieval @xcite . the task was to retrieve and rank relevant argumentative documents for queries addressing socially important (and often controversial) topics like 'should bottled water be banned?'. the document collection was the args.me corpus @xcite containing about 400,000 arguments crawled from different online debate portals. each document is structured and contains a debate topic field (e.g., 'pollution'), an argument conclusion (e.g., 'plastic bottles should be banned'), and a main content containing several premises (i.e., reasons, opinions, or evidence that support or attack the conclusion). we access all the data inside the pyterrier framework @xcite via ir_datasets (macavaney et al., 2021), including queries, document collection, and available document-level manual relevance judgments (the participants' systems are available at the touché task website https://touche.webis.de ). deepct-based term weighting. our pilot experiments using the original deepct model pretrained on the ms marco dataset showed that the retrieval effectiveness of lexical models degrades (cf. dirichletlm + deepct achieves ndcg@5 of 0.59 vs. 0.83 that dirichletlm achieves on the unmodified args.me corpus). this is likely due to the document domain change (general domain vs. argument retrieval). we thus opt for fine-tuning deepct for the argument retrieval task on the 19) being manufactured, purchased, used, and discarded. to fine-tune deepct, we use three variants of the args.me corpus: (1) all documents in the corpus; and to analyze the effect of possible train-test leakage: (2) judged documents from the touché 2020 and 2021 tasks are removed, and (3) top-50 documents from all systems submitted to touché 2020 and 2021 are removed from args.me. after expanding the passages using the fine-tuned deepct models, the passages are concatenated back into complete documents. the original args.me corpus is then modified with the three differently fine-tuned deepct models, resulting in three corpus variants. retrieval models. for every variant of the modified corpus, we test the effectiveness of bm25 and dirichletlm and their combination with the query expansion model rm3. we select the model's parameters using grid search and two-fold crossvalidation (each fold is either the touché 2020 or 2021 relevance judgments) implemented in pyterrier @xcite . ## evaluation we compare our approaches (lexical retrieval models with deepct-based corpus transformations) with the most effective systems at touché 2020 (49 queries, and 2,298 relevance judgments) and 2021 (50 queries, and 3,711 judgments) as well as with four strong neural retrieval baselines implemented in tirex @xcite . due to the high portion of missing judgments for systems not in the touché's original pool (cf. column 'judged@5' in at touché 2020, the most effective system (highest ndcg@5 and highest bpref; cf. we complement the best systems at touché (which are all based on lexical retrieval) with four neural re-rankers: (1) colbert @xcite , a contextualized late interaction model that uses bert @xcite , (2-3) monobert (nogueira and cho, 2019) and monot5 (nogueira et al., 2020), two pointwise cross-encoder models based on bert and t5 @xcite , and (4) lit5 @xcite , a zero-shot listwise re-ranker using t5. all four models were used in a re-ranking setting using tirex @xcite , to re-rank the top-1000 documents retrieved by bm25. results. with respect to both ndcg@5 and bpref, our approach of using deepct for semantic document term weighting improves over the touché best systems when using the bm25 retrieval model. when using dirichletlm, the deepct term weighting does not outperform the participants' systems on the touché 2021 data. we also find that the best neural baseline, monot5 as a re-ranker, is also more effective than the best touché systems of 2021, while the other neural re-rankers fall back behind. our most effective deepct-based approach does not outperform monot5; yet, it is on par with monot5 for touché 2020 data (significantly equivalent to the best system within @xmath3.1 band, see furthermore, in at least half of the retrieval scenarios, fine-tuning deepct on the args.me documents after removing the judged ones, results in the highest evaluation scores. thus, we do not observe strong evidence of the train-test leakage influence on the retrieval results. moreover, combining an argument conclusion with a debate topic for fine-tuning deepct often benefits the retrieval effectiveness of lexical models. ## conclusion in this paper, we proposed to combine lexical retrieval models with semantic document expansion for argument retrieval. specifically, to calculate the term weights, we fine-tuned deepct on the args.me corpus. the main advantages of deepct are that the calculation of term weights can be done in an offline fashion before document indexing and that its training does not require manual relevance judgments. this is especially important in the specialized domains (e.g., argument retrieval), where no or little training data is available. furthermore, at query time only lexical retrieval models are used on the expanded documents that require less computational resources than neural models. our evaluation results showed that adding some "semantics" to strong lexical argument retrieval approaches improves the overall effectiveness over the lexical retrieval alone. additionally, we showed that our approach is on par with modern neural rerankers, which themselves can be more computationally expensive. however, we also indicated that for a more robust conclusion, further experiments should be conducted, where the missing relevance judgments are filled. another potentially interesting future direction can be to include the argument mining step in the document expansion process, for instance, using only argumentative parts (conclusions and premises) of documents for fine-tuning deepct.