Evaluating Unsupervised Argument Aligners via Generation of Conclusions of Structured Scientific Abstracts (2024)

Abstract: Scientific abstracts provide a concise summary of research findings, making them a valuable resource for extracting scientific arguments. In this study, we assess various unsupervised approaches for extracting arguments as aligned premise-conclusion pairs: semantic similarity, text perplexity, and mutual information. We aggregate structured abstracts from PubMed Central Open Access papers published in 2022 and evaluate the argument aligners in terms of the performance of language models that we fine-tune to generate the conclusions from the extracted premise given as input prompts. We find that mutual information outperforms the other measures on this task, suggesting that the reasoning process in scientific abstracts hinges mostly on linguistic constructs beyond simple textual similarity.

## introduction scientific reasoning involves pairing conclusions with premises, which encompasses information such as pre-existing knowledge, observations, and experimental results @xcite . this reasoning process is inherently directional: while inductive reasoning establishes logical links from the causal premises to the resulting conclusions @xcite , abductive reasoning aligns the most plausible premises for given conclusions @xcite @xcite . the goal of mining scientific arguments is to discover the argumentative structure within academic papers @xcite . despite the great success in recent studies @xcite , a crucial aspect of evaluating the alignment quality involves assessing the logical strength and quality of arguments @xcite figure 1 : the evaluation pipeline for argument aligners. first, the structured abstract is split into premise and conclusion sentences. then, the argument aligner uses nearest neighbor search to find relevant premises for conclusions. finally, a trained language model generates conclusions from the selected premises. the best aligner is the one that selects the most sufficient premises for generated conclusions with the highest rouge score, compared to the original conclusions. in this work, inspired by previous studies on text alignment @xcite , we investigate the sufficiency of premises aligned by various unsupervised argument aligners, i.e. normalized point-wise mutual information (npmi, @xcite ; padmakumar and he (2021)), normalized perplexity (nppl, miaschi et al. ( 2021 )), and semantic (cosine) similar-ity (csim, @xcite ). drawing inspiration from @xcite and @xcite , we assess the sufficiency of premises by evaluating the extent (measured with rouge score) to which a language model can generate the paired conclusion from them. our main contributions are: 1) we constructed a dataset named arg-align, which comprises more than 17k pairs of premises and conclusions aggregated from structured scientific abstracts from the pubmed central open access (pmcoa) corpus; 2) we assessed the sufficiency of the aligned premises by reconstructing the corresponding conclusions using language models; 3) we highlighted that premises in scientific abstracts may contain redundant information in terms of the drawn conclusions. ## unsupervised argument aligners given an abstract that contains a premise segment of @xmath0, unsupervised argument aligners compute alignment scores d(p, c) between each premise sentence p and the entire conclusion segment c. we set ourselves the goal of finding the @xmath1, as judged by a language model. we consider the conclusion segment c as a single text rather than as a list of individual sentences because a paper typically has one primary research finding that is stated over possibly multiple conclusion sentences. the argument aligners therefore should identify premise sentences that are relevant to inferring c as a whole. in contrast to previous studies that focused on inductive argument alignment, where c is identified based on p @xcite , we focus on abductive argument alignment, where p is identified based on c. this choice is motivated by the fact that the conclusion sentences in structured abstracts can be easily located by searching for the conclu-sions discourse section using regular expressions, whereas premise sentences are distributed across all discourse sections and therefore more difficult to identify. to abductively align a premise sentence p with the conclusion segment c, we explore four unsupervised argument aligners with different alignment scores: to calculate nppl and npmi scores with low computational cost, we use a simple pre-trained gpt-2 model (|v | = 50, 257, @xcite ) and compute the log likelihoods by taking the logits of the last decoder layer. ## methodology in line with the concept presented by @xcite , our objective is to investigate the extent to which the premises, when aligned with the conclusions using our argument aligners, can effectively contribute to the reconstruction of those conclusions. ## results and discussion we present conclusion generation results for different argument aligners in we found that all argument aligners selected premise sentences of encouraging sufficiency, evident from their average rouge-2 scores consistently exceeding 10. interestingly, premises aligned using npmi consistently generated the best conclusion, suggesting that npmi captures well the dichotomy of premises and conclusions in scientific arguments. somewhat surprisingly, we found that full (unrestricted) premises tended to degrade the generated conclusions, as evidenced by lower rouge scores. perhaps, full premises may contain irrelevant content in relation to the conclusions that overshadows conclusion generators finally, the fine-tuned bart-large conclusion generator outperformed the 425 times larger zeroshot gpt-3.5 generator. we suggest that because llms such as gpt-3.5 excel at generating text of low perplexity @xcite , it is likely that gpt-3.5 has a preference to use less common vocabulary and expressions when generating the conclusion, resulting in lower rouge scores. however, we noticed that after fine-tuning, gpt-3.5 has acquired the ability to incorporate words more typical of scientific language, leading to improved rouge scores. ## related works computational argument sufficiency was first studied by @xcite . they viewed argument sufficiency as a binary classification task and trained a cnn classifier to predict whether an argument is sufficient or not. later, the concept of argument sufficiency was extended to include argument strength, with strong arguments steering conversations towards more crucial topics compared to weak arguments. @xcite proposed assessing the strength of deductive arguments by probabilistically modeling the necessity and sufficiency of premises for claims with a defeasible logic. their four-dimensional probabilistic measures of argument strength provided a theoretical foundation of computational argument evaluation. computational argument evaluation often involves utilizing language models for assessing premise-conclusion pairs. for example, conclusion generation focuses on the challenge of inferring conclusions from a provided collection of premises, approaching it as a text generation task @xcite @xcite . @xcite investigated the effectiveness of seq2seq models in generating conclusions from random clinical trials (rcts), indicating the capability of these models to perform scientific reasoning. other works focused on generating sentence-and paragraph-level counter-arguments, with carefully designed control mechanisms ( hua et al. (2019); schiller et al. (2021); saha and srihari (2023); alshomary and wachsmuth (2023) ) such that the generated conclusions contain more detailed information. ## conclusions in this study, we explored semantic similarity, text perplexity, and mutual information as unsupervised argument aligners. we quantified these metrics on the task of pairing premises with conclusions in pmcoa paper abstracts. our primary objective was to probe the sufficiency of aligned premises by using them to reconstruct the conclusions. our findings indicate that semantic similarity, often considered a straightforward measure of text relevance, did not emerge as the best criterion for premise-conclusion alignment. this surprising result suggests that the process of scientific reasoning within abstracts is not solely driven by textbased similarity, but rather encompasses nuanced perspectives involving the cohesiveness of premise sentences amongst each other, as captured by p (p). this study highlights the need for a deeper understanding of the intricacies involved in the construction of well-aligned argument pairs in scientific papers. our research sheds light on the multifaceted nature of scientific reasoning and the importance of exploring alternative approaches that better capture the underlying connections between premises and conclusions. as we move forward, it becomes evident that refining the techniques for aligning arguments will contribute to more accurate and insightful representations of scientific discourse, with the potential of improving the information dissemination and knowledge synthesis within the scientific community.