The most-frequent-sense and the predominant domain sense play an important role in the debate on word-sensedisambiguation. This discussion is, however, biased by the way sense-tagged corpora are built. In this paper, we argue that current sense-tagged corpora neglect rare senses and contexts and, as a result, do not represent a good corpus for training and testing word-sensedisambiguation. We defined three quality criteria for sense-tagged corpora and a methodology to satisfy these criteria with minimal effort. Following this method, we built a Dutch sense-tagged corpus that tried to meet these criteria. The corpus was evaluated by deriving word-sensedisambiguation systems and testing these on different subsets of the corpus in different ways. The performance of our systems and the quality of the derived data are equal to state-of-the-art English systems and corpora. Finally, we used the systems to create a Dutch corpus of over 47 million sense-tagged tokens spread over a large variety of genres, domains and usages of Dutch. The results of the project can be downloaded freely from the project website. The DutchSemCor project was an NWO Humanities medium investment subsidy project with a subsidy period of September 2009 -August 2012. We would like to thank NWO for making it possible to carry out the project. Word Sense Disambiguation (WSD) research in the last decade demonstrated a number of important insights (Agirre and Edmonds, 2006): 1. evaluation results are strongly dependent on the corpus and the lexicons used, 2. the most-frequentsense derived from SemCor (Miller et al., 1993 ) is a strong baseline that is not easy to beat in evaluations like SensEval or SemEval and 3. predominant senses in specific domains give the best WSD results by far (McCarthy et al., 2007) . From these observations, one may conclude that we need to collect large sets of (sense-tagged) domain-and probably genre-specific corpora to determine predominant senses. Obtaining sufficient data without ignoring rare or low-frequent senses, however, requires an enormous effort. Manually tagged data is still very sparse and evaluation results vary from task to task, hence we still do not know where we stand in the area of WSD. This raises the question: how should the ideal sense-tagged corpus for WSD look like, to enable detection of any sense in any type of corpus? Existing sense-tagged corpora have different design properties that make them good corpora in some aspects but not in others. In this paper, we will define quality criteria for sense-tagged corpora and will describe a novel method for building a largescale sense-tagged Dutch corpus that meets these criteria with as little manual effort as possible. We argue that an ideal sense-tagged corpus should be balanced for the different senses, for the different contexts and should provide information on sensefrequencies, preferably across a wide range of domains and genres. In the DutchSemCor 1 project we tried to meet these three criteria by using large corpora that cover a wide range of language-use, including spoken and written language, Flemish and Dutch standard language and dialects, and numerous genres and domains. Furthermore, we tagged these corpora through a mixture of manual and automatic annotations and selections of word tokens. We first aimed at a corpus that represents the meanings of an existing lexicon including sufficient examples for rare senses. Secondly, we extended this corpus to acquire a wider representation of contexts when needed and, finally, in order to acquire sense-distributions, the full corpus was annotated automatically applying three WSD systems. The resulting annotations (both manual and automatic) were tested for all three criteria. As a side result, 1 http://www2.let.vu.nl/oz/cltl/dutchsemcor/ 710
As a response to sense-skewed datasets, REF created a balanced sense corpus in the DutchSemCor project in which each sense gets an equal number of examples.
DutchSemCor: in quest of the ideal sense-tagged corpus
RANLP
Computer Science