The Air Travel Information System (ATIS) domain serves as the common evaluation task for ARPA"spoken language system developers. 1 To support this task, the Multi-Site ATIS Data COllection Working group (MADCOW) coordinates data collection activities. This paper describes recent MADCOW activities. In particular, this paper describes the migration of the ATIS task to a richer relational database and development corpus (ATIS-3) and describes the ATIS-3 corpus. The expanded database, which includes information on 46 US and Canadian cities and 23,457 flights, was released in the fall of 1992, and data collection for the ATIS-3 corpus began shortly thereafter. The ATIS-3 corpus now consists of a total of 8297 released training utterances and 3211 utterances reserved for testing, collected at BBN, CMU, MIT, NIST and SRI. 2906 of the training utterances have been annotated with the correct information from the database. This paper describes the ATIS-3 corpus in detail, including breakdowns of data by type (e.g. context-independent, context-dependent, and unevaluable)and variations in the data collected at different sites. This paper also includes a description of the ATIS-3 database. Finally, we discuss future data collection and evaluation plans. The ATIS task was first used as a common ARPA spoken language evaluation task in 1990 [1, 2]. In the ATIS task, subjects obtain air travel information such as flight schedules, fares, and ground transportation from a relational database using spoken natural language, and use it to solve air travel planning scenarios. Although the core air travel planning task has remained the same since the beginning, its use in evaluation has gradually evolved over the years with the general objectives of increasing tile match between 1. This paper was prepared under the auspices of the Multi-Site ATIS Data Collection Working group (MADCOW). In addition to the authors, many other people, listed under the Acknowledgments section, made important contributions to this work. the evaluation and a real task as well as increasing the accuracy of the metric. The first official evaluation took place in February of 1991, following a dry run in June of 1990. In the 1991 evaluation, context-independent (Class A) queries as well as dialog pairs (D1) were evaluated. The score for a system was the weighted-error metric which included a penalty for incorrect answers as opposed to "No Answer". Further refinements took place in the November 1992 evaluation, where Class D (utterances with context dependencies throughout the dialog) queries were evaluated. Another variation introduced in 1992 was the rain-max criterion, in which the information provided by systems in the answer was required to fall between a minimum and a maximum amount. In the most recent evaluation, December 1993, the main change has been to drop the weighted error metric and report results based on the unweighted error, or 100-%T. The 1993 ATIS spoken language understanding evaluation is the first evaluation based on the ATIS-3 corpus ([5]). The ATIS-3 corpus will also supply test data for tile December 1994 ATIS evaluation. In addition, test data has also been reserved for a dry run of a semantic evaluation [6] . The initial ATIS task was based on a relational database containing air travel information for 11 cities. Three corpora of spontaneous spoken language utterances (ATIS-0, ATIS-1 and ATIS-2) were collected with this database using a variety of paradigms, as described in [3, 4] . As ATIS technology developed, it was felt that the initial ATIS task was unrealistically limited because of the small size of tile database. Consequently, the database was expanded to include air travel information for 46 cities. The expanded database was released in tile fall of 1992, and data collection began shortly thereafter. The new database is based on air travel data obtained from tile Official Airline Guide (OAG) in June 1992 and current at that time. The database includes information for 46 cities 43 and 52 airports in the US and Canada. The largest table in the exp;mded database, the flight table, includes information on 23,457 flights. This compares to 11 cities, 9 airports, and 765 flights in the earlier ATIS databases and clearly represents a significant scaling up of the ATIS task. Despite the fact that the number of flights in the database has been increased by over a factor of thirty, the conversion to the larger database has not caused any serious difficulties for the :sites doing data collection, the sites doing evaluation, or the annotators.This result is encouraging, since it indicates that the SLS technology developed on a small database can scale up to a significantly bigger task. Cities mid airports included in the new database are listed in the Appendix. The AT1S-3 data was collected at BBN, CMU, MIT, NIST, and SRI. NIST participated in ATIS data collection for the first time in this round of data collection, using data collection software from both BBN and SRI. Since the beginning of the ATIS task data collection paradigms have moved toward increasingly automated approaches. The original ATIS-0 corpus was collected using human wizards to both transcribe the subjects' utterances as well as to interpret them (the so-called "wizard of OZ" paradigm). In the ATIS-3 corpus, nearly all transcription and interpretation of the subjects' speech was done by the sites' ATIS systems themselves. The only exception was MIT, which collected data using a transcription wizard instead of a speech recognizer, while using MIT's natural language system to interpret the utterances. Automatic data collection has the advantage of reduced cost. In addition, the data is more realistic in the sense that it is obtained from subjects who are really talking to a computer. The disadvantage of automatic data collection is that imperfect processing by the spoken language system sometimes leads to the presence of artifacts in data collection, such as utterances repeated over and over again. The general process of data collection and annotation as described in [3] has not changed in the ATIS-3 data collection effort. We summarize this process here for convenience. Collected data is transcribed at the collecting site and sent to NIST, where it is logged and potential test data is held out. The data is then released to sites participating in the ATIS evaluations as initial, i.e. unannotated, data, and is simultaneously sent to SRI for annotation. During annotation, the data is classified into three categories: The Principles of Interpretation document is used to categorize utterances in these three classes and also specifics how to interpret vague expressions which occur in utterances in Class A and D. Annotated data is returned from SRI to NIST and released by NIST. A full set of data for a subject session includes the following files: • .wav: speech waveform  The total data collected for the ATIS-3 corpus consists of 12,047 utterances, of which 3,876 have been annotated. The data breaks down across sites as shown in Table 1 . Approximately 20% of submitted data is allocated to the pool from which evaluation test data is drawn. In addition, 500 utterances from the NIST data collection activity have been reserved as test data for a possible dry run for semantic evaluation in ATIS [6] . This table does not include 1440 additional utterances collected at CMU which have not yet been released as initial data. Two subject-scnearios from the ATIS-3 corpus can be seen in Appendix 2. Note in particular the false starts typical of spontaneous speech. Slightly over 1/3, or 36%, of the released initial data has been annotated with the correct answers. Unannotated data includes data reserved for the December 1994 evaluation, which will be annotated just before the evaluation to insure that it is consistent with the Principles of Interpretation current at that time. Other unannotated data includes data from NIST and SRI which was received too late to be annotated. The full corpus of annotated data also includes 667 sentences collected for the November 1992 logfile experiments [4] . Although these utterances were collected using the 11 city database, they were annotated using the expanded database. The rationale for this decision was that the annotators wished to get experience with the expanded database, and at the time, the logfile data was the only data available. The annotated data breaks down into Classes A, D, and X by site as shown in Table 2 . If the annotated data is broken down by site as well as by class, it can be noted that there is a wide range of variation across sites in tile relative proportion of A, D, and X queries, which can be seen in Figure 1 . We believe this is largely attributable to the effects of different data collection scenarios used by the different sites. The practical consequences of this effect are that an understanding of how scenarios achieve this effect might lead to the development of techniques for improving system performance for particular applications. ATIS PLANS Despite the fact that nearly 25,000 ATIS utterances have been collected since 1990, no standard developmeut test data exists for ATIS. Sites have individually constructed development test sets from the training data and evaluation test sets, but this method makes inter-site comparisons difficult. While inter-site comparisons are a major goal of the official evaluation, variations in the test set from year to year make comparisons across years problematic. In addition, if evaluation test data is used after the evaluation as development test data, it is contaminated by system developers looking at it in detail for the purposes of adjudication. The existence of a development test corpus will also extend the usefulness of the ATIS-3 corpus after all training and evaluation test data is released by providing a source of unseen data. For these reasons MADCOW has decided to collect an additional 300-500 utterances from BBN, CMU, MIT, NIST, and SRI, to be designated development test data. This data is to be collected in the spring of 1994 and will have a high priority for early annotation. MADCOW is also interested in exploring evaluation paradigms other than file standard CAS evaluation. These include the end-to-end/logfile approach described in [4] , as well as the semantic evaluation paradigm described in [6] .
Much of this work uses the ATIS corpus REF as data and hence is directly comparable.
Expanding The Scope Of The ATIS Task: The ATIS-3 Corpus
Human Language Technology Conference
Computer Science