In this paper we examine how the differences in modelling between different data driven systems performing the same NLP task can be exploited to yield a higher accuracy than the best individual system . We do this by means of an experiment involving the task of morpho-syntactic wordclass tagging . Four well-known tagger generators ( Hidden Markov Model , Memory-Based , Transformation Rules and Maximum Entropy ) are trained on the same corpus data . After comparison , their outputs are combined using several voting strategies and second stage classifiers . All combination taggers outperform their best component , with the best combination showing a 19.1 % lower error rate than the best individual tagger . In all Natural Language Processing ( NLP ) systems , we find one or more language models which are used to predict , classify and/or interpret language related observations . Traditionally , these models were categorized as either rule-based / symbolic or corpusbased / probabilistic . Recent work ( e.g. Brill 1992 ) has demonstrated clearly that this categorization is in fact a mix-up of two distinct Categorization systems : on the one hand there is the representation used for the language model ( rules , Markov model , neural net , case base , etc . ) and on the other hand the manner in which the model is constructed ( hand crafted vs. data driven ) . Data driven methods appear to be the more popular . This can be explained by the fact that , in general , hand crafting an explicit model is rather difficult , especially since what is being modelled , natural language , is not ( yet ) wellunderstood . When a data driven method is used , a model is automatically learned from the implicit structure of an annotated training corpus . This is much easier and can quickly lead to a model which produces results with a ' reasonably ' good quality . Obviously , ' reasonably good quality ' is not the ultimate goal . Unfortunately , the quality that can be reached for a given task is limited , and not merely by the potential of the learning method used . Other limiting factors are the power of the hard-and software used to implement the learning method and the availability of training material . Because of these limitations , we find that for most tasks we are ( at any point in time ) faced with a ceiling to the quality that can be reached with any ( then ) available machine learning system . However , the fact that any given system can not go beyond this ceiling does not mean that machine learning as a whole is similarly limited . A potential loophole is that each type of learning method brings its own ' inductive bias ' to the task and therefore different methods will tend to produce different errors . In this paper , we are concerned with the question whether these differences between models can indeed be exploited to yield a data driven model with superior performance . In the machine learning literature this approach is known as ensemble , stacked , or combined classifiers . It has been shown that , when the errors are uncorrelated to a sufficient degree , the resulting combined classifier will often perform better than all the individual systems ( Ali and Pazzani 1996 ; Chan and Stolfo 1995 ; Tumer and Gosh 1996 ) . The underlying assumption is twofold . First , the combined votes will make the system more robust to the quirks of each learner 's particular bias . Also , the use of information about each individual method 's behaviour in principle even admits the possibility to fix collective errors . We will execute our investigation by means of an experiment . The NLP task used in the experiment is morpho-syntactic wordclass tagging . The reasons for this choice are several . First of all , tagging is a widely researched and well-understood task ( cf . van Halteren ( ed . ) 1998 ) . Second , current performance levels on this task still leave room for improvement : ' state of the art ' performance for data driven automatic wordclass taggers ( tagging English text with single tags from a low detail tagset ) is 96 - 97 % correctly tagged words . Finally , a number of rather different methods are available that generate a fully functional tagging system from annotated text . Our experiment shows that , at least for the task at hand , combination of several different systems allows us to raise the performance ceiling for data driven systems . Obviously there is still room for a closer examination of the differences between the combination methods , e.g. the question whether Memory-Based combination would have performed better if we had provided more training data than just Tune , and of the remaining errors , e.g. the effects of inconsistency in the data ( cf . Ratnaparkhi 1996 on such effects in the Penn Treebank corpus ) . Regardless of such closer investigation , we feel that our results are encouraging enough to extend our investigation of combination , starting with additional component taggers and selection strategies , and going on to shifts to other tagsets and/or languages . But the investigation need not be limited to wordclass tagging , for we expect that there are many other NLP tasks where combination could lead to worthwhile improvements .

Challenge: Different data driven approaches tend to produce different errors and their qualities are limited due to the learning method and available training material.
Approach: They propose to combine four different modelling methods for the task of morpho-syntactic wordclass tagging by using several voting strategies and second stage classifiers.
Outcome: All combinations outperform the best component, with the best one showing a 19.1% lower error rate and raising the performance ceiling.