Twente- BMS - NLP at P erspective A rg 2024: Combining Bi-Encoder and Cross-Encoder for Argument Retrieval (2024)

Abstract: The paper describes our system for the Perspective Argument Retrieval Shared Task. The shared task consists of three scenarios in which relevant political arguments have to be retrieved based on queries (Scenario 1). In Scenario 2 explicit socio-cultural properties are provided and in Scenario 3 implicit socio-cultural properties within the arguments have to be used. We combined a Bi-Encoder and a Cross-Encoder to retrieve relevant arguments for each query. For the third scenario, we extracted linguistic features to predict socio-demographic labels as a separate task. However, the socio-demographic match task proved challenging due to the constraints of argument lengths and genres. The described system won both tracks of the shared task.

## introduction the perspective argument retrieval shared task @xcite addresses the challenge of incorporating socio-cultural factors into argument retrieval. it is based on the x-stance dataset @xcite and includes three scenarios: baseline, explicit, and implicit. queries in the baseline scenario are provided without sociodemographic requirements on extracted arguments. in the explicit and implicit scenarios, one sociodemographic feature is provided along with the query as an additional requirement to retrieve arguments that match the label, (e.g.{gender: male}). an extracted argument is considered a true candidate only if it is relevant to the query text and matches the socio-demographic label provided with the query in explicit and implicit scenarios. the difference between the explicit and implicit scenarios is that in the explicit scenario, the sociodemographic information of argument authors is available in the corpus, whereas in the implicit scenario it is not. this means that in the implicit scenario, systems must predict or match the socio-demographic features in addition to determining relevance to the query. in our system, we combine bi-encoder and crossencoder models to retrieve relevant arguments. additionally, we predict socio-demographic features from argument texts in the implicit scenario, using sentence embeddings, n-gram of part-of-speech (pos) tags and stop words, and token length distributions as input features. the system performed best in both tracks of the shared task, relevance and diversity. ## related work previous studies have employed several strategies to model query and argument sentences before they can be matched subsequently: simple statistical features like token n-grams or part-of-speech (pos) n-grams @xcite , tf-idf @xcite , or word2vec @xcite . word embeddings with mean pooling or other techniques to generate sentence embeddings of fixed lengths @xcite , sentence embedding models such as sentence-bert @xcite or labse @xcite , and ensemble approaches combining models from the aforementioned categories @xcite . sentence embedding models typically use a biencoder architecture, such as a siamese neural network. the relevance of two sentences is often measured with the cosine similarity of their embeddings. bi-encoders are more suitable for symmetric searches where two sentences are interchangeable @xcite . asymmetric scenarios like answering a specific question often benefit more from cross-encoder models. @xcite pointed out the challenge in training bi-encoders to represent two asymmetric sentences within a shared vector space. beyond encoder architectures, @xcite proposed a method that employs decoders for sentence embeddings and semantic search. ## dataset analysis in the training and development datasets, we observed that the queries and candidates are structured as follows: the same query is provided in three languages: german, french, and italian. though as separate query entries, each language version shares the same list of candidate arguments. options for addressing this cross-lingual setting include using or fine-tuning cross-lingual sentence embedding models or translating different languages into one. additionally, we noted that an argument candidate only appears under one query (or the same query of three language versions). this suggests that the dataset might have been created from an existing set of query arguments (in the x-stance dataset), with arguments randomized and compiled into a mixed arguments corpus, rather than through annotations of argument relevance for each query. this setup could pose challenges for traditional semantic search tasks: if an argument could validly answer two different queries, it is still tied to only the query with which it was originally associated, and appearing under the other query would be considered a false retrieval, even if it might be correct. a potential strategy to address this issue is to evaluate each candidate's relevance across all queries, assigning it to only one query. however, this approach has a drawback: if an argument is incorrectly assigned to one query, it precludes the possibility of it being correctly assigned to another query. ultimately, we adopted a hybrid strategy. for queries in the development set, we only considered candidates that had not appeared under training queries. similarly, for each test set, we only considered candidates that had not appeared in either the training or development queries, which helps narrow down the argument pools and potentially enhance retrieval accuracy. this strategy was implemented across all test sets and scenarios. nevertheless, we also provide unfiltered results in this paper for broader comparison with the baselines. ## results and discussion figure 1 presents the comparison between our system and sentence-bert. the first row shows the ndcg scores of three test sets in the baseline scenario. our system performs similarly well in test figure 1 : relevance results (ndcg scores) for retrieved argument candidates from the top 4 to top 20 across three scenarios and three test sets. for implicit scenarios (the third row in figure 1 ), our system shows no difference from sentence-bert, except in test 3, which is significantly better with the filtering procedure but much worse without filtering. the implicit scenario shows that the benefits of utilizing the 'important issues' classifier to filter first are not evident. the predicted important issues' might also be decoded by sentence-bert, and overall low precision may result from the mismatch of other demographic factors. despite performing less well in the implicit scenario than in the other two, our results still achieved first place among all participated teams in tests 2 and 3, and second in test 1. ## conclusion this study demonstrates the advantages of combining bi-encoder and cross-encoder models over solely using the bi-encoder (sentence-bert). we also found that perspective argument retrieval or inferring socio-demographic features from short arguments remains challenging, accompanied by the disadvantage of the number and quality of labels.