An experiment with recent test suite and grammar (engineering) resources is outlined: a critical assessment of the EU-funded tsnlp (Test Suites for Natural Language Processing) package as a diagnostic and benchmarking facility for a distributed (multi-site) large-scale hpsg grammar engineering effort. This paper argues for a generalized, systematic, and fully automated testing and diagnosis facility as an integral part of the linguistic engineering cycle and gives a practical assessment of existing resources; both a flexible methodology and tools for competence and performance profiling are presented. By comparison to earlier evaluation work as reflected in the Hewlett-Packard test suite data, released exactly ten years before tsnlp, it is judged where testsuite-based evaluation has improved (and where not) over time. Contemporary lexicalized constraint-based grammars (e.g. within the hpsg framework) with wide grammatical and lexical coverage exhibit immense conceptual and computational complexity; as the grammatical framework aims to eliminate redundancy and factor out generalizations, the interaction of lexicon and phrase structure apparatus can be subtle and make it hard to predict how even modest changes to the grammar affect system behaviour. Additionally, in a distributed grammar engineering setup (i.e. for a project where several * Part of the research reported presently was funded by the German National Science Foundation ( people or even sites contribute to a single grammatical resource) it becomes necessary to assess the impact of individual contributions, regularly evaluate the quality of the overall grammar, and compare it to previous versions. Besides concise coverage (i.e. competence) judgments, in most application scenarios efficiency and resource consumption play an increasingly important role; hence, processing components typically provide a (potentially) large inventory of control parameters and preference settings. When tuning the analysis component to improve system performance, grammar writers often rely on introspection, knowledge of the grammar, and personal experience; yet, without systematic profiling and performance analysis, processor optimization amounts to guessing parameter settings and constant experimentation. In general, given the similarity in setup (parsing with large-scale unification grammars), it is to be expected that the observations of Carroll (1994) (see above) hold for hpsg-type systems too. Despite the growing variety of approaches to NLP, the development of computational grammars still is a prerequisite for NLP systems that rely on (or at least incorporate) what is often called deep (or full) linguistic analysis. To allow developers to achieve the grammatical coverage required for most realistic application contexts, substantial investments into grammar engineering methodology and tools have been made; hence, for most of the contemporary grammatical frameworks in computational linguistics, there is a (sometimes large) number of development environments to choose from. At the same time, the improved methodology and strict formalization, declarativity, and modularity of grammatical resources facilitate the distribution of development across several grammar writers and sometimes even sites. While in the past (e.g. for purely rule-based systems) grammars often were developed and maintained by a single person over years or decades and ultimately connected in life cycle to that one developer (see Erbach and Uszkoreit (1990) ), it is now more fre-
The further use of the resources is supported by a parser, sentence generator and facilities for profiling and regression testing REF .
Towards Systematic Grammar Profiling Test Suite Technology Ten Years After
Special Issue on Evaluation), 411
Computer Science