We explore the problem of resolving the second person English pronoun you in multi-party dialogue , using a combination of linguistic and visual features . First , we distinguish generic and referential uses , then we classify the referential uses as either plural or singular , and finally , for the latter cases , we identify the addressee . In our first set of experiments , the linguistic and visual features are derived from manual transcriptions and annotations , but in the second set , they are generated through entirely automatic means . Results show that a multimodal system is often preferable to a unimodal one . The English pronoun you is the second most frequent word in unrestricted conversation ( after I and right before it ) . 1 Despite this , with the exception of Gupta et al . ( 2007b ; 2007a ) , its resolution has received very little attention in the literature . This is perhaps not surprising since the vast amount of work on anaphora and reference resolution has focused on text or discourse -mediums where second-person deixis is perhaps not as prominent as it is in dialogue . For spoken dialogue pronoun resolution modules however , resolving you is an essential task that has an important impact on the capabilities of dialogue summarization systems . Besides being important for computational implementations , resolving you is also an interesting and challenging research problem . As for third person pronouns such as it , some uses of you are not strictly referential . These include discourse marker uses such as you know in example ( 1 ) , and generic uses like ( 2 ) , where you does not refer to the addressee as it does in ( 3 ) . ( 1 ) It 's not just , you know , noises like something hitting . ( 2 ) Often , you need to know specific button sequences to get certain functionalities done . ( 3 ) I think it 's good . You 've done a good review . However , unlike it , you is ambiguous between singular and plural interpretations -an issue that is particularly problematic in multi-party conversations . While you clearly has a plural referent in ( 4 ) , in ( 3 ) the number of its referent is ambiguous . 2(4 ) I do n't know if you guys have any questions . When an utterance contains a singular referential you , resolving the you amounts to identifying the individual to whom the utterance is addressed . This is trivial in two-person dialogue since the current listener is always the addressee , but in conversations with multiple participants , it is a complex problem where different kinds of linguistic and visual information play important roles ( Jovanovic , 2007 ) . One of the issues we investigate here is how this applies to the more concrete problem of resolving the second person pronoun you . We approach this issue as a three-step problem . Using the AMI Meeting Corpus ( McCowan et al . , 2005 ) of multi-party dialogues , we first discriminate between referential and generic uses of you . Then , within the referential uses , we distinguish between singular and plural , and finally , we resolve the singular referential instances by identifying the intended addressee . We use multimodal features : initially , we extract discourse features from manual transcriptions and use visual information derived from manual annotations , but then we move to a fully automatic approach , using 1-best transcriptions produced by an automatic speech recognizer ( ASR ) and visual features automatically extracted from raw video . In the next section of this paper , we give a brief overview of related work . We describe our data in Section 3 , and explain how we extract visual and linguistic features in Sections 4 and 5 respectively . Section 6 then presents our experiments with manual transcriptions and annotations , while Section 7 , those with automatically extracted information . We end with conclusions in Section 8 . We have investigated the automatic resolution of the second person English pronoun you in multi-party dialogue , using a combination of linguistic and visual features . We conducted a first set of experiments where our features were derived from manual transcriptions and annotations , and then a second set where they were generated by entirely automatic means . To our knowledge , this is the first attempt at tackling this problem using automatically extracted multimodal information . Our experiments showed that visual information can be highly predictive in resolving the addressee of singular referential uses of you . Visual features significantly improved the performance of both our manual and automatic systems , and the latter achieved an encouraging 75 % accuracy . We also found that our visual features had predictive power for distinguishing between generic and referential uses of you , and between referential singulars and plurals . Indeed , for the latter task , they significantly improved the manual system 's performance . The listeners ' gaze features were useful here : in our data set it was apparently the case that the speaker would often use the whiteboard / projector screen when addressing the group , thus drawing the listeners ' gaze in this direction . Future work will involve expanding our dataset , and investigating new potentially predictive features . In the slightly longer term , we plan to integrate the resulting system into a meeting assistant whose purpose is to automatically extract useful information from multi-party meetings .

Challenge: Although the word "you" is frequently used and has several possible meanings, such as reference or generic, it is not well studied yet.
Approach: They first manually automatically separate the word "you" between generic and referential, then later use a multimodal system for automation.
Outcome: They show that visual features can help distinguish the word "you" in multi-party conversations.