We describe two approaches to analyzing and tagging team discourse using Latent Semantic Analysis (LSA) to predict team performance. The first approach automatically categorizes the contents of each statement made by each of the three team members using an established set of tags. Performance predicting the tags automatically was 15% below human agreement. These tagged statements are then used to predict team performance. The second approach measures the semantic content of the dialogue of the team as a whole and accurately predicts the team's performance on a simulated military mission. The growing complexity of tasks frequently surpasses the cognitive capabilities of individuals and thus, often necessitates a team approach. Teams play an increasingly critical role in complex military operations in which technological and information demands require a multi-operator environment. The ability to automatically predict team performance would be of great value for team training systems. Verbal communication data from teams provides a rich indication of cognitive processing at both the individual and the team level and can be tied back to both the team's and each individual team member's abilities and knowledge. The current manual analysis of team communication shows promising results, see for example, Bowers et al. (1998) . Nevertheless, the analysis is quite costly. Hand coding for content is time consuming and can be highly subjective. Thus, what is required are techniques for automatically analyzing team communications in order to categorize and predict performance. In the research described in this paper we apply Latent Semantic Analysis (LSA), to measure free-form verbal interactions among team members. Because it can measure and compare the semantic information in these verbal interactions, it can be used to characterize the quality and quantity of information expressed. This can be used to determine the semantic content of any utterance made by a team member as well as to measure the semantic similarity of an entire team's communication to another team. In this paper we describe research on developing automated techniques for analyzing the communication and predicting team performance using a corpus of communication of teams performing simulated military missions. We focus on two applications of this approach. The first application is to automatically predict the categories of discourse for each utterance made by team members during a mission. These tagged statements can then be used to predict overall team performance. The second application is to automatically predict the effectiveness of a team based on an analysis of the entire discourse of the team during a mission. We then conclude with a discussion of how these techniques can be applied for automatic communications analysis and integrated into training. Our corpus (UAV-Corpus) consists of 67 transcripts collected from 11 teams, who each completed 7 missions that simulate flight of an Uninhabited Air Vehicle (UAV) in the CERTT (Cognitive Engineering Research on Team Tasks) Lab's synthetic team task environment (CERTT UAV-STE). CERTT's UAV-STE is a threeteam member task in which each team member is provided with distinct, though overlapping, training; has unique, yet interdependent roles; and is presented with different and overlapping information during the mission. The overall goal is to fly the UAV to designated target areas and to take acceptable photos at these areas. The 67 team-at-mission transcripts in the UAVCorpus contain approximately 2700 minutes of spoken
In REF automatic analysis of the semantic content of team communication and automatic verbal behavior labeling was used to assess team performance in a command and control task with an unmanned aerial vehicle in a simulated environment.
Automated Team Discourse Annotation And Performance Prediction Using LSA
Human Language Technology Conference And Meeting Of The North American Association For Computational Linguistics - Short Papers
Computer Science