DREAM : Deployment of Recombination and Ensembles in Argument Mining (2023)

Abstract: Current approaches to Argument Mining (AM) tend to take a holistic or black-box view of the overall pipeline. This paper, in contrast, aims to provide a solution to achieve increased performance based on current components instead of independent all-new solutions. To that end, it presents the Deployment of Recombination and Ensemble methods for Argument Miners (DREAM) framework that allows for the (automated) combination of AM components. Using ensemble methods, DREAM combines sets of AM systems to improve accuracy for the four tasks in the AM pipeline. Furthermore, it leverages recombination by using different argument miners elements throughout the pipeline. Experiments with five systems previously included in a benchmark show that the systems combined with DREAM can outperform the previous best single systems in terms of accuracy measured by an AM benchmark.

## introduction a well-known and open challenge in argument mining (am) is that approaches do not generalize well across domains @xcite . thus, a single system will not be able to solve the task of extracting arguments from publications across multiple research fields. therefore, we investigate the use of ensemble methods @xcite to find combinations that are expected to help alleviate the issue. furthermore, the overview of am approaches presented by @xcite shows that papers typically introduce novel techniques or use methods for am that have demonstrated success in other applications. as new systems tend to take the holistic view of an end-to-end pipeline @xcite , it has become evident that novel approaches rarely investigate improvements of intermediate steps. by introducing a system with ensemble methods and combinations, we aim to improve smaller aspects of the pipeline. moreover, @xcite also take the same line by advocating for a unifying framework to enable the harmonization of all am tasks, including the format of data and results. such a unification would be necessary to combine many systems and facilitate the integration of additional ones. likewise, not every task receives the same amount of attention, with approaches for identifying argumentative relations being sparse @xcite . thus, they cover a smaller range of domains or do not work well across them. by using recombination, we hypothesize to increase coverage and find yet untapped potential. to this end, we formulate the following research question: rq. how can we leverage (re-)combinations of argument mining systems to improve accuracy? thus, we build dream, a system that allows for the deployment of recombination and ensemble methods for argument miners. for this endeavor, we base our approach and the evaluation on bam @xcite , our benchmark for argument mining. we reuse the performance data of five am systems when evaluated by bam as well as its implementation for our purposes. accordingly, we restrict the systems for the initial combinations to these five argument miners and adhere to the definition of the four tasks in the am pipeline @xcite : sentence classification, boundary detection, component identification, and relation prediction. using these tools, we try to outperform the current best accuracy for every task of the am pipeline by combining systems with the following ensemble methods: voting, stacking, and bagging. finally, we split the am systems into "modules" according to the am pipeline, allowing their recombination to increase accuracy. we present two main contributions in this paper. first and foremost, we build the dream framework to combine am systems using ensemble methods and recombinations. second, we show the value of such combinations, as they outperform some of the state-of-the-art systems used in the am benchmark. the remainder of this paper is structured as follows: section 2 presents background and the related work, and section 3 introduces our methodology. in the ensuing section 4, we describe our experiments and their results before we evaluate them in section 5. then, section 6 discusses limitations and future work. finally, we draw conclusions in section 7. ## background in this section, we lay the foundations by describing argument mining and presenting specific related work. ## methodology in this section, we first lay out our evaluation hypotheses and then describe the concept of the dream (re-)combination framework. ## experiments in this section, we discuss conducted experiments. first, we describe the setup used. then, we explain the implementation of the experiments and the subsequent evaluation. ## limitations and future work the major limitation of this work is that we implement post-hoc combinations. the reasons for not re-training the systems are two-fold. first, out of practicality to facilitate the addition of new am systems and existing ensembles. second, to set the scope of this research as opposed to works that look to explicitly fuse models such as neural networks by entangling the final classification layer such as described in @xcite . the latter opens up the future work of applying these techniques to the current five am systems and mixing their latent representations, as opposed to only their outputs. another limitation is that all the included systems take a holistic view of the am pipeline, and none is explicitly split into the four modules we infer for the ensemble methods. given the success of (re-)combinations of components in other domains, this paper can, hence, be seen as a call to action to systematically explore the effectiveness of functional components of the am pipeline and share these for re-use by others. indeed, more broadly, the limited availability of am systems and benchmark datasets hampered our ability to systematically compare a larger design space of system (component) combinations and limits the generalizability of our findings to other domains/datasets. the plans for future efforts in this direction include two main points. as the next step, we aim to conduct an error analysis and explore the influences of the systems on the results. this will help identify the strengths and weaknesses of the individual systems and may provide insights about current am systems' common weaknesses. also, the new analysis can incorporate the spatial and temporal costs of the recombinations, which was omitted in this paper. in the future, should the number of argument mining systems considerably increase, the framework could be extended to include a predictor to choose the sets and sequences of argument miners for a given document that lead to an optimal accuracy improvement. this would involve developing a cost function. ## conclusions this paper presented dream, a framework for the deployment of recombination and ensemble methods for argument miners. our work focuses on improving accuracy in argument mining (am) and addresses the need for incremental improvements as opposed to current approaches, which tend to provide all-new solutions @xcite . with the dream framework, we implemented a flexible and automated approach for (re)combining am systems. it offers the vertical integration (using ensemble methods for a single task), the horizontal integration (using different systems throughout the pipeline), and, finally, the combined integration (allowing sets of systems for the intermediate data). our findings confirmed the hypotheses formulated in this work. we showed that ensemble methods @xcite could be used to improve accuracy for specific tasks in the am pipeline. furthermore, we demonstrated that recombination by using intermediate data from the most accurate system could lead to higher accuracy in the subsequent task. finally, we highlighted the potential of deploying ensemble methods and recombination for am. we hope this work will contribute to the further improvement of state-ofthe-art and better generalizing am systems across domains, a prevalent and well-acknowledged problem @xcite .