D ebate S um: A large-scale argument mining and summarization dataset (2020)

Abstract: Prior work in Argument Mining frequently alludes to its potential applications in automatic debating systems. Despite this focus, almost no datasets or models exist which apply natural language processing techniques to problems found within competitive formal debate. To remedy this, we present the DebateSum dataset. DebateSum consists of 187,386 unique pieces of evidence with corresponding argument and extractive summaries. DebateSum was made using data compiled by competitors within the National Speech and Debate Association over a 7year period. We train several transformer summarization models to benchmark summarization performance on DebateSum. We also introduce a set of fasttext word-vectors trained on DebateSum called debate2vec. Finally, we present a search engine for this dataset which is utilized extensively by members of the National Speech and Debate Association today. The DebateSum search engine is available to the public here: http://www.debate.cards

## introduction and background american competitive debate's increasingly technical nature leads its competitors to search for hundreds of thousands of pieces of evidence every year. while there are many types of competitive debate that can be assisted by argument mining technologies, some types of formal debate are easier to automate than others. in the united states, the national speech and debate association (nsda) organizes the majority of competitive debates held in secular high schools. the nsda sanctions four different types of debate and many different speaking events. the nsda-sanctioned debate format most suited to being assisted by natural language processing technologies is called "cross examination debate" (cx) or "policy debate". this is because policy debate is significantly more popular and evidence-intensive than the other debate forms that the nsda offers. unlike other forms of debate, which have narrow topics that rotate per tournament or per month, policy debate maintains one extremely broad topic over a whole year. this encourages extremely deep and thorough amounts of preparation. significantly more evidence (and subsequently, training data) is produced by nsda policy debaters than by other types of debaters. most debaters are encouraged to keep their cases and evidence secretive and hidden from their opponents. however, due to the extreme research burdens which policy debaters face, many universities hold "debate camps" which students attend to prepare for the year's topic. the primary goal of debate camp is for attendees to produce as much evidence as they can before the competitive season starts. these debate camps attract thousands of coaches, competitors, and staff, and function as an effective crowed sourcing platform. at the end of the summer, these debate camps release all evidence gathered by the attendees together on the open evidence project foot_1 . the open evidence project hosts thousands of debate cases and hundreds of thousands documents. the open evidence projects extensive case library gives any policy debater access to a wide variety of debate cases, allowing for novices and competitors with limited amounts of preparation time to present effective arguments. the open evidence project is a fantastic hidden-gem resource for argument mining. a diverse range of highly motivated high school students and their coaches produce high quality arguments and evidence to support each argument made. policy debate does not focus on the speaking style or delivery of the speech as much as other types of debate do. policy debates are instead extremely intricate and technical and most unexpectedly to lay-people, the debaters deliver them fast foot_2 . since rounds are usually decided by technical details of evidence, competitors are encouraged to present the maximal amount of evidence for their position in their speeches. this evidence is always available to be reviewed later by the judge before a decision is made. this leads to a phenomenon known as speed-reading (colloquially referred to as "spreading" within the debate community) which is done by the majority of serious competitors for strategic benefits. to casual observers, spreading seems completely absurd, but the competitive advantages that it confers are significant. the desire to present as much evidence as possible motivates competitors to research extremely deeply and to produce/deliver the maximum amount of evidence possible. it is due to these factors that debatesum is such a large dataset. figure 1 : an example of an argument-evidencesummary triplet from the debatesum dataset as presented in its original form before parsing. the argument is highlighted in blue (lines 1-4), and would be presented by debaters as part of their case. metadata, such as the date, title and author of the evidence are highlighted in green (lines 5-9). the evidence consists of all text after the blue and green-highlighted sections. the extractive summary consists of all underlined text within the document. the highlighted sections of the underlined text are the extracts which the debater chooses to read outloud alongside their argument. note that the argument can also be used an abstractive summary or as a query in query-focused summarization debaters can quickly search for evidence by keywords. they can pick evidence to view in detail with the eyeball icon, or move it to the "saved" column with the arrow icon. when a debater has moved all of the evidence that they need to the "saved" column, they can click the download button (not shown) to download a properly formatted word document with all of the saved evidence in it. policy debaters extensively use this download feature before or in debate rounds to compile evidence for their case. this uniquely technical style of debate has thus far been largely unassisted by natural language processing technologies. debaters would find that effective information retrieval, summarization (especially query-focused summarization), and classification systems could automate, supplement, and/or assist them in their research work. the primary reason for this lack of effective systems within competitive debate was the lack of domain specific training data available. debatesum remedies this problem by introducing the largest (to our knowledge) dataset of document-argument-summary triplets in existence and we believe that it is one of the largest argument mining datasets ever gathered. ## innovations introduced we introduce three innovations: the debatesum dataset (and summarization models trained on it), de-bate2vec word embeddings, and the "debate.cards" argument search engine. open evidence stores tens of thousands of debate cases as word documents. some debate cases can have thousands of documents within them. debatesum was gathered by converting each word document into an html5 file using pandoc foot_3 . this allows for easy parsing of documents, allowing for them to have their arguments, document, and summary extracted. debatesum consist of argument-evidence-summary triplets. policy debaters colloquially refer to these triplets as "cards". each card consist of a debater's argument (which acts as a biased abstractive summary of the document), a supporting piece of evidence and its citations, and a word-level extractive summary produced by "underlining" and/or "highlighting" the evidence in such a way to support the argument being made. figure 1 shows an example of one of these cards before parsing. thousands of competitors and coaches manually produced and annotated debatesum's triplets. we train transformerbased token-level extractive summarization models to form strong baselines and to act as a blueprint for performance benchmarking summarization models on debatesum. debate2vec is a set of fasttext @xcite word vectors produced for study on word analogy and word similarity tasks. it is trained on the evidence in debatesum along with additional evidence (which was not included in debatesum due to missing or malformed arguments or summaries). debate2vec is trained on 222485 documents. debate2vec is 300 dimensional, with a vocabulary of 107555 lowercased words. it was trained for 100 epochs with a learning rate of 0.10. no subword information is included to keep the memory consumption of the model down. debate2vec is particularly well suited for usage in domains related to philosophy, law, government, or politics. debate.cards is an argument search engine which indexes debatesum (and additional evidence gathered by college level debaters or contributed by users later in the year). debate.cards was designed as a research tool for policy debaters. prior to debate.cards, competitors would have to painstakingly search through potentially thousands of word documents for specific pieces of evidence. debate.cards allows competitors to search for evidence by keyword match within the argument, evidence or the citation. figure 2 shows a sample of the debate.cards search engine. ## prior work we are not the first to utilize natural language processing techniques to assist debaters with summarization. abstractive summarization of political debates and arguments has been studied for quite some time @xcite , @xcite . the most well-known application of argument mining to debate comes from ibm's project debater. recent work from ibm research has shown the impressive capabilities of automatic argument summarization @xcite . they utilize a crowd-sourced argument dataset of 7000 pro/con claims scraped from the internet and then they abstractivly synthesize key-points which summarizes the arguments made by the participants. our work and dataset focuses instead on word-level extractive summarization of debate documents. there is work related to retrieving extractive "snippets" in support of an argument, but this work is query-independent and extracts summaries at the sentence level rather than the word level @xcite . other work related to debate summarization exists, but is trained on limited datasets or restricted to significantly less technical debating styles and formats @xcite . one notable feature that would be extremely useful to members of the policy debate community is the ability to generate an extractive summary of their evidence which is biased towards supporting their argument. some evidence will make arguments for both sides, but only the portions which support a particular position would ideally be read aloud. some authors have explored this unique problem which they call query focused or query based summarization, notably @xcite , @xcite and @xcite . these systems are somewhat similar to our work, but they deal with abstractive rather than extractive summarization, and are trained on comparatively small datasets like the duc 2007 or debatepedia dataset. luckily, a queryable word-level extractive summarization system exists and is used among the community foot_4 . this summarizer is called "cx_db8" (to celebrate the "cross examination" debate community) and is unsupervised, which makes it unable to be directly trained on debatesum (though it can use word embedding's which are trained on it). it also gets inferior rouge scores compared to supervised models. the summarization models we train on debatesum are not queryable, but they follow the tradition of supervised extractive summarization models being benchmarked using rouge scores. before our work, there were (to our knowledge) no pre-trained word embedding's or language models publically available for policy debate. the closest pre-trained language model that we could find to our domain is the publically available law2vec foot_5 set of legal word embedding's. a significant proportion of policy debaters end up becoming lawyers, in no small part due to the similarity between competitive debate and litigation. our debate2vec foot_6 word embedding's are trained on the entirety of the debatesum document dataset. they are (to our knowledge) the largest publically available set of word vectors trained on a non-legal argumentative corpus finally, there is prior work related to argumentation search engines and information retrieval systems for debaters. websites such as debate.org, procon.org, and debatepedia.org serve as useful search engines for arguments in support or opposition of a topic. argument search engines which match or exceed the scale of debatesum's arguments have been built by crawling these sorts of websites @xcite , but these indexed arguments do not have corresponding evidence and extracts associated with them. no dedicated search engine for competitive policy debate evidence existed prior to our work. furthermore, we believe that no dedicated search engine for any type of debate argument-evidencesummary triplets exists that matches the scale or breadth of the debate-cards search engine. ## analysis the debatesum dataset consists of 187,386 unique documents that are larger than 100 words. there are 107,555 words which show up more than 10 times in the corpus. there are 101 million total words in debatesum. each document consists of on average 520 words. each argument is on average 14 words long, and each summary consist of 198 words on average. the mean summary compression ratio is 0.46 and the mean argument compression ratio is 0.06. debatesum is made from evidence pertaining to each of the yearly policy debate resolutions. since debatesum consists of 7 years of evidence, there are 7 resolutions which it covers. affirmative debate cases almost always advocate for a particular "plan" which implements a resolution. there are potentially an infinite number of topical plans. negative teams can read a potentially infinite number of counter-plans or counter-advocacies alongside evidence for why the affirmative plan is bad. debaters will be expected to prepare cases for both the affirmative and negative side and debate each side an equal number of times throughout a tournament. as a result, a considerable amount of evidence gathered in a particular year will be only tangentially related to the resolution as it must be generic enough to answer any type of plan or counterplan. there is a consistent "metagame" of popular and strong arguments which are crafted to be used for any topic on either side. many of them have their roots in philosophical critiques of the plan or even the state itself which may appear to have little or no relevance to the resolution. one can get insights into the specific list of arguments made available by looking at the corresponding years page of open evidence and inspecting the debatesum debate cases in their original form. a table which ## experiments and result we train transformer architecture neural network language models on word-level extractive summarization using the simple-transformers foot_7 framework. we formulate this problem as a token classification problem between "underlined" and "not-underlined" tokens. we fine-tune the existing weights for 5 epochs using early stopping and the adam @xcite optimizer. fp16 is enabled for training. we evaluate our models on a test split of 18,738 documents. the rouge metric is used for measuring summarization quality. we evaluate using the default settings of py-rogue foot_8 on our models. we report the rouge f1 scores of these transformer models. ## conclusion in this paper, we presented a new large-scale argument mining and summarization dataset called de-batesum. each row of debatesum consists of a piece of debate evidence, a word-level extractive summary of that evidence, and the argument made with the evidence. the argument can also be used as an abstractive summary or as a query in tandem with the extract for query-focused extractive summarization. we also trained word vectors on the debate evidence subset of debatesum. we showcased an innovative search engine for debatesum which is extensively utilized by competitive debaters today. finally, we fine-tuned several transformer models on word-level extractive summarization of debatesum documents and measured their performance using the rouge metric. we observe that longformer is superior to competitor models, likely because of the significantly larger sequence length allowing longrange context to assist in choosing which tokens to include in a summary. we release all code and data to the public.