A useful first step m document summausation is the selection of a small number of 'meamngful' sentences from a larger text Kupiec et al (1995) describe tim as a clasmficatlon task on the basis of a corpus of technical papers with summaries written by professional abstractors, their system ldent~fies those sentences m the text which also occur in the summary, and then acquires a model of the 'abstract-worthiness' of a sentence as a combination of a hmlted numbel of properties of that sentence We report on a rephcatlon of thin expernnent with different data summaries for our documents were not written by professional abstractors, but by the authors themselves Tins produced fewer allguable sentences to tram on We use alternative 'meaningful' sentences (selected by a human judge) as training and evaluation material, because tlns has advantages for the subsequent automatic generation of more flexible abstracts We quantitatively compare the two ¢hfferent strategies for training and evaluation (vm ahgnment vs human judgement), we also chscnss qualitative chfferences and consequences for the generatlon of abstracts 1 Introduction A useful first step m the automatic or semiautomatic generation of abstracts from source texts m the selection of a small number of 'meamngful' sentences from the source text To achieve tins, each sentence m the source text is scored according to some measure of importance, and the best-rated sentences are selected Thin results m collections of the N most 'meamngful' sentences, m the order m wlnch they appeared m the source text -we will call these excerpts An excerpt can be used to give readers an idea of what the longer text m about, or It can be used as input into a process to .produce a more coherent abstract It has been argued for almost 40 years that it m posmble to automatically create excerpts which meet bamc reformation compresmon needs (Luhn, 1958) Since then, different measurements for the importance of a sentence have been suggested, m particular stochastic measurements for the mgmficance of -key words or phrases (Lulm, 1058, Zechner, 1995 Other research, starting with (Edmundson, 1969) , stressed the Importance of heuristics for the location of the candidate sentence m the source text (Baxendale, 1958) and for the occurrence of cue phrases (Palce and Jones, 1993, Johnson et al, 1993) Single heunstms tend to work well on documents that resemble each other m style and content For the more robust creation of excerpts, combinations of these heuristics can be used The eruclal question m how to combine the ¢hfferent heuristics In the past, the relative usefulness of single methods had to be balanced manually Kupmc et al (1095) use supervised learnmg to automatically adjust feature w~ghts, using a corpus of research papers and corresponding summaries Humans have good intuition about what makes a sentence 'abstract-worthy', I e suitable for inclusion in a summary Abstract-worthiness m a lughlevel quality, comprising notions such as semantic content, relative importance and appropriateness for representing the contents of a document • .For the automatic evaluation of the quality of machine generated excerpts, one has to find an operational approximation to this subjective notion of abstractworthiness, 1 e a defuntlon of a desired result We will call the criteria of what constitutes success the In Kupiec et al (1995), a gold standard sentence is a sentence m the source text that zs matched ruth a summary sentence on the basra of semantic and syntactic snnflanty In thear corpus of 188 engineermg papers with summaries written by professional abstractors, 79% of sentences occurred m both summary and source text with at most minor moddica~ tzons However, our collection of papers, whose abstracts were written by the authors themselves, shows a szgnh~cant difference these abstracts have $1~nl~-. cantly fewer ahgnable sentences (31 7%) This does. not mean that there are fewer .abstract-worthy sentenees m the source text We used a simple (labourintensive) way of defimng thin alternative gold standard, vzz aslang a human judge to identify additional abstract-worthy sentences in the source text Our mare question was whether Kuplec et al's methodology could be used for our kind of gold standard sentences also, and if there was a fundamental chfference in extraction performance between sentences in both gold standards or between documents with higher or lower alignment We also conducted an experiment to see how additional training matehal would influence the statistical model The remainder of this paper is organized as follows in the next section, we s-mmanze Kuplec et al's method and results Then, we describe our data and dmcuss the results from three experiments with dflferent evaluation strategies and tralmng material Differences between our and Kuplec et al's data with respect to the ahgnablhty of document and summary sentences, and consequences thereof are conmdered m the discussion In Kupzec et al's experiment, the gold standard sentences are those summary sentences that can be aligned with sentences m the source texts Once the alignment has been carried out, the system tries to determine the characteristic properties of ahgned sentences according to a number of features, wz presence of particular cue phrases, location in the text, sentence length, occurrence of thematic words, and occurrence of proper names Each document sentence receives scores for each of the features, resuiting m an estimate for the sentence's probabihty to also occur m the summary This probabihty is calculated as follows 59
The Cue Phrase approach focuses on words that are used to determine the appropriate sentences for the summary such as "this paper", "propose", and "concluding" REF .
Sentence Extraction As A Classification Task
Workshop On Intelligent Scalable Text Summarization
Computer Science