professor a: We 're not recording yet , are we ? grad g: Well , I don't think phd f: No , that that wasn't recorded . phd f: I guess I have to d stop doing this sigh of contentment , you know , after sipping cappuccino or something . phd f: I was just noticing a big s professor d: So are we recording now ? Is this phd e: Yeah . professor d: So , what were we gonna talk about again ? So we said we said data collection , which we 're doing . Do we do th do you go around the room and do names or anything ? grad g: I think that phd e: It 's a good idea . grad g: u usually we 've done that and also we 've s done digits as well , but I forgot to print any out . Besides with this big a group , phd b: You can write them on the board , if you want . I it 'd be even better with this big grad g: it would take too much time . phd e: Mari ? postdoc h: What professor a: What ? professor d: It 's not that long . professor d: w u phd e: So that thing the little th that part should be pointing upwards . professor a: It 's kind of professor d: Oh , yeah , the element , yeah , n should be as close to you your mouth as possible . So what we had was that we were gonna talk about data collection , and , you you put up there data format , professor a:  . professor d: and other tasks during data collection , professor a: So , I think the goal the goal was what can we do how can you do the data collection differently to get professor d: and professor a: what can you add to it to get , some information that would be helpful for the user - interface design ? Like grad g: especially for querying . postdoc h: Well , one thing that came up in the morning in the morning was the , i  , if he I ,  if he has s I I don't remember , Mister Lan - Doctor Landry ? grad g: Landay . postdoc h: La - Landay ? So he has , these , tsk note - taking things , professor a:   postdoc h: then that would sort of be a summary which you wouldn't have to solicit . professor a: Well , if if you actually take notes as a summary as opposed to n take notes in the sense of taking advantage of the time - stamps . So action item or  , reminder to send this to so - and - so , blah - blah - blah . grad g: But if we had the CrossPads , we could ask people , you know , if if something comes up write it down and mark it somehow , postdoc h:   And so , you know phd f: the down - side to that is that he sort of indicated that the , quality of the handwriting recognition was quite poor . I don't think there 'd be so many that you couldn't have someone clean it up professor a: So grad g: pretty easily . We also could come up with some code for things that people want to do so that for frequent things . So , if that you know , if if we just keep it simple then maybe it 's still useful . professor d: I just realized we skipped the part that we were saying we were gonna do at the front where we each said who we were . professor a: But anyway , shall we do the roll call ? professor d: No , not a No , I just My mind went elsewhere . So , yeah , I 'm Morgan , and where am I ? I 'm on channel three . postdoc h: Should we have used pseudo - names ? Should we do it a second time with pseudo No . professor d: I 'm Rocky Raccoon on channel phd e: Let me , turn that off . grad g: And , do you want to do the P D As and the P Z phd e: Oh . phd e: PDM - right , PZA - right PDA - right , PDA - left . grad g: Yeah , and eventually once this room gets a little more organized , the Jimlets will be mounted under the table , and these guys will be permanently mounted somehow . postdoc h: I have a question on protocol in these meetings , which is when you say " Jimlet " and the person listening won't know what that is , sh shou How how do we get Is that important information ? You know , the Jimlet  , the box that contains the professor d: Well , suppose we broaden out and go to a range of meetings besides just these internal ones . There 's gonna be lots of things that any group of people who know each other have in column common that we will not know . professor d: we we were originally gonna do this with VLSI design , and and and the reason we didn't go straight to that was because immediately ninety percent of what we heard would be jargon to to us . Should we do it and , if so , what 'll we have them do ?  , do we have s people write summaries ? Everybody or one person ? And then , do we ask people for how they would query things ? Is that phd f: There 's there 're sub - problems in that , in that where or when do you actually ask them about that ? professor a: Right . phd f: that was One thing I was thinking about was is that Dan said earlier that , you know , maybe two weeks later , which is when you would want to query these things , you might ask them then . phd f: But there 's a problem with that in that if you 're not If you don't have an interactive system , it 's gonna be hard to go beyond sort of the first level of question . professor d: There 's there 's another problem grad g: And professor d: which is , we certainly do want to branch out beyond , recording meetings about Meeting Recorder . And , once we get out beyond our little group , the people 's motivation factor , reduces enormously . And if we start giving them a bunch of other things to do , how you know , we we did n you know another meeting here for another group and and , they were fine with it . But if we 'd said , " OK , now all eight of you have to have to come up with , the summar " grad g: Well , I asked them to and none of them did . grad g: So , I I asked them to send me ideas for queries after the meeting postdoc h:   grad g: So I didn't track them down and say " please do th do it now " . professor d: I I 'm worried that if you did even if you did push them into it , it it it might be semi - random , professor a: Right . professor d: as opposed to what you 'd really want to know if you were gonna use this thing . grad g: I just don't know how else to generate the queries other than getting an expert to actually listen to the meeting and say " that 's important , that might be a query " . Well , there is this other thing which y which you were alluding to earlier , which is , there are certain key words like , you know , " action item " and things like that , which could be used in , t to some degree finding the structure . phd e: Although professor a: W postdoc h: And and I also , was thinking , with reference to the n  , note - taking , the advantage there is that you get structure without the person having to do something artificial later . And the fir third thing I wanted to say is the summaries afterwards , I think they should be recorded instead of written because I think that , it would take so long for people to write that I think you wouldn't get as good a summary . professor a: How about this idea ? That normally at most meetings somebody is delegated to be a note - taker . professor a: And So why don't we just use the notes that somebody takes ? postdoc h: Yeah . grad g: that gives you a summary but it doesn't really How do you generate queries from that ? phd e: Well . But , maybe a summary is one of the things we 'd want from the output of the system . phd b: Actually And so grad g: James and I were talking about this during one of the breaks . And the problem with that is , I 'm definitely going to do something with information retrieval even if it 's sort of not full full - bore what I 'm gonna do for my thesis . And so if someone wants to do that , that 's fine , but it 's not gonna be me . professor d: Well , I think that we  , the the f the core thing is that you know once we get some of these issues nailed down , we need to do a bunch of recordings professor a: Well professor d: and send them off to IBM and get a bunch of transcriptions even if they 're slightly flawed grad g: Yep . professor d: And then , i i we can start l looking and thinking , what do we want to know about these things and at the very least . So , if you could sense just when people are writing , and you tell them not to doodle , or try not to be using that for other purposes , and each person has a note pad . Then you c you can just have a fff plot of wh you know , who 's writing when . But I bet that 's that will allow you to go into the sort of the hot places where people are writing things down . phd b: you can tell when you 're in a meeting when everybody stops to write something down that something was just said . phd b: It may not be kept in the later summary , but at that point in time is was something that was important . phd b: And that wouldn't take any extra postdoc h: That 's a nice idea . phd b: Or someone could just pu you could just put your hand on the pad professor d: It professor c:   It 's professor d: That 's a good idea but that doesn't Maybe I 'm missing something , but that doesn't get to the question of how we come up with queries , right ? professor a: Well , what it does phd b: Well , then you can go to the points where the you could actually go to those points in time and find out what they were talking about . phd b: And grad g: y professor a: I I think it 's an interesting thing . I don't think it gets at the the queries per - se , but it does give us an information fusion sort of thing that , you know , you wanna i say " what were the hot - points of the meeting ? " phd b: Yeah . professor d: That that 's what  , is that I think it gets at something interesting but if we were asking the question , which I thought we were , of of of , " how do we figure out what 's the nature of the queries that people are gonna want to ask of such a system ? " , knowing what 's important doesn't tell you what people are going to be asking . phd e: Well , see , there are th professor a: I think you could say they 're gonna ask about , when  , when did so - and - so s talk about blah . phd b: At least you can find the locations where there are maybe keywords professor d: Maybe . grad g: i this would tell you what the hit is , phd b: and grad g: not what the query is . grad g: And so you could you can generate a query from the hits , professor a: Right . grad g: but phd b: We don't even know what  , if you want to find out what any user will use , that might be true for one domain and one user , professor d:   professor d: Yeah , but we 're just looking for a place to start with that phd b:  . professor d: because , you know , th what what what James is gonna be doing is looking at the user - interface and he 's looking at the query in in i We we have five hours of pilot data of the other stuff but we have zero hours of of of queries . So he 's just sort of going " where where do I where do I start ? " professor a: w Well , th you could do I think the summaries actually may help get us there , professor d: OK . One , if you have a summary if you have a bunch of summaries , you can do a word frequency count and see what words come up in different types of meetings . professor a: So " action item " is gonna come up whether it 's a VLSI meeting , or speech meeting , or whatever . So words that come up in different types of meetings may be something that you would want to query about . professor a: the second thing you could possibly do with it is just run a little pilot experiment with somebody saying " here 's a summary of a meeting , what questions might you want to ask about it to go back ? " grad g: Yeah , I think that 's difficult because then they 're not gonna ask the questions that are in the summary . professor a: Well grad g: But , I think it would give professor a: That 's one possi one possible scenario , though , is you have the summary , grad g:   So that that is another use of Meeting Recorder that we haven't really talked about , which is for someone else , as opposed to as a remembrance agent , which is what had been my primary thought in the information retrieval part of it would be . But , I guess if you had a meeting participant , they could use the summary to refresh themselves about the meeting and then make up queries . phd e: But th there is this ,  There is this class of queries , which are the things that you didn't realize were important at the time but some in retrospect you think " oh , hang on , didn't we talk about that ? " And it 's something that didn't appear in the summary but you professor a:   phd e: And that 's kind of what this kind of , complete data capture is kind of nicest for . phd e: Cuz it 's the things that you wouldn't have bothered to make an effort to record but they get recorded . So ,  And th there 's no way of generating those , u u until we just until they actually occur . phd e: But  , it 's difficult to sort of say " and if I was gonna ask four questions about this , what would they be ? " Those aren't the kind of things that come up . postdoc h: I also think that w if if you can use the summaries as an indication of the important points of the of the meeting , then you might get something like y So if th if the obscure item you want to know more about was some form of data collection , you know , maybe the summary would say , you know , " we discussed types of na data collection " . If you if you had the the larger structure of the of the discourse , then if you can categorize what it is that you 're looking for with reference to those l those larger headings , then you can find it even if you don't have a direct route to that . postdoc h: I think that grad g: That 's a pretty fine grain that the note - taker will have to take . I think you got to have somebody who knows the pro knows the topic or you know , whose job it is delegated to be the note - taker . phd b: No , but someone who can come sit in on the meetings and then takes the notes with them that the real note - taker grad g: But they phd b: And that way that one student has , you know , a rough idea of what was going on , and they can use it for their research .  , this isn't really necessarily what you would do in a real system , grad g:   But if he has some students that want to study that then they should sort of get to know the people and attend those meetings , professor d:   Their sort of note - taking application stuff they 've been doing for the last couple of years , and I don't think anyone is still working on it . grad g: And so the question is " is there such a person ? " And I think right now , the answer is " no " . professor a: Well professor d: I 've b been thinking grad g: We 'll just have to see . professor d: I 've been thinking about it a little bit here about the  , th this , e  I think that the now I 'm thinking that the summary a summary , is actually a reasonable , bootstrap into this into what we 'd like to get at . It 's it 's not ideal , but we you know , we we have to get started someplace . So I was I was just thinking about , suppose we wanted to get w We have this collection of meeting . So now we have five hours of meetings and , you ask me , " Morgan , what d you know , what kind of questions do you want to ask ? "  , I wouldn't have any idea what kind of questions I want to ask . So in fact if I looked at summary of it , I 'd go " oh , yeah , I was in that meeting , I remember that , what was the part that " And and th I think that might then help me to think of things even things that aren't listed in the summary , but just as a as a as a refresh of what the general thing was going on in the meeting . One , as sort of a refresh to help bootstrap queries , professor d:   professor a:  phd b: So how does the summary get generated ? professor a: Well , i i ? phd b: I 'm not against the idea of a summary , grad g: By hand . phd b: but I wanted to think carefully about who 's generating it professor a: Or , d o phd b: and how because the summary will drive the queries . professor a: What I I think , you know , in most meetings , phd b: So professor a: this one being different , but in most meetings that I attend , there 's somebody t explicitly taking notes , frequently on a laptop  , you can just make it be on a laptop , phd b:   professor a: so then yo you 're dealing with ASCII and not somebody you don't have to go through handwriting recognition . postdoc h: I I I think that , there 's we 're using " summary " in two different ways . postdoc h: And what I originally thought was , if you asked someone " what was the meeting about ? " phd b: OK . postdoc h: And then they would say " well , we talked about this and then we talked about that , and so - and - so talked about " And then you 'd have , like I e My thought was to have multiple people summarize it , on recording rather than writing because writing takes time and you get irrelevant other things that u take time , that professor a:   postdoc h: Whereas if you just say it immediately after the meeting , you know , a two - minute summary of what the meeting was about , I think you would get , with mult See , I I also worry about having a single note - taker because that 's just one person 's perception . And , you know , it it 's releva it 's relative to what you 're focus was on that meeting , professor d:   professor d: A postdoc h: So , my proposal would be that it may be worth considering both of those types , you know , the note - taking and a spontaneous oral summary afterwards , professor a: OK . postdoc h: no longer than two minutes , professor d: Adam , you can postdoc h: from multiple people . professor d: but but , my impression was that , pretty much , true that the meetings here , nobody sits with a w  , with a laptop grad g: Never . Does anyone ? phd b: Dan ? professor d: I grad g: Dan is the one who who most frequently would take notes , phd e: Yeah . Most of the meetings we have are four or five people grad g: That 's true are four or five people . professor d: and you 're not you don't have somebody sitting and taking minutes for it . professor a: So , I think it depends on whether it 's a business meeting or a technical discussion . grad g: The IRAM meeting , they they take notes every professor d: Do they ? grad g: There 's  a person with a laptop at each meeting . phd b: I don't know how , but for instance , the outline is sort of up here and that 's what people are seeing . phd e: We sh we should grad g: I agree , but but you you just you g end up with video , postdoc h: I agree . phd e: f u I think for this data capture , it would be nice to have a digital camera grad g: Yeah , different phd b: y phd e: just to take pictures of who 's there , where the microphones are , and then we could also put in what 's on the board . grad g: But don't you think that 's Don't you think that But postdoc h: I agree .  , I I just think  , I think that right now we don't make a record of where people are sitting on the tables . phd e: Not not as part of the not as a part of the data that you have to recover . phd e: Just just in terms of phd b: We should just Like archiving it or storing it . postdoc h: It 's i because discourse is about things , phd b: Because someone postdoc h: and then you have the things that are about , and it 's recoverable . phd b: someone later might be able to take these and say " OK , they , you know at least these are the people who were there phd e: So phd b: and here 's sort of what they started talking about , and " and just postdoc h: Yes . professor d: Li postdoc h: Like you said , three snapshots professor d: L L L postdoc h: and professor d: Liz , you postdoc h: Just to archive . professor d: u  , Liz , you sa you sat in on the , subcommittee meeting or whatever phd e: Actually professor d: on you on the subcommittee meeting for for at the ,  that workshop we were at that , Mark Liberman was was having . They they they they h must have had some discussion about video and the visual aspect , and all that . But I 'm just saying first of all there 's a whole bunch of fusion issues that DARPA 's interested in . And there 's also I think a personal interest on the part of Mark Liberman in this kind of in storing these images in any data we collect professor d:   phd b: And professor d: I think you  , that the key thing there is that this is a description of database collection effort that they 're talking about doing . professor d: And if the database exists and includes some visual information that doesn't mean that an individual researcher is going to make any use of it . professor d: So ,  grad g: But that it 's gonna be a lot of effort on our part to create it , and store it , and get all the standards , and to do anything with it . So we 're gonna So we 're gonna do what we 're gonna do , whatever 's reasonable for us . phd b: I think even doing something very crude professor d: But having phd b: Like I know with ATIS , we just had a tape recorder running all the time . phd b: And later on it turned out it was really good that you had a tape recorder of what was happening , even though you w you just got the speech from the machine . So if you can find some really , you know , low , perplexity , grad g: Low fidelity . And if it 's simple as  , as simple as just the digital phd b: Otherwise you 'd you lose it . professor d: Well , minimally , what what Dan is referring to at least having some representation of the p the spatial position of the people , postdoc h: Yeah . professor d: so ,  grad g: Well , once the room is a little more fixed that 's a little easier professor d: Yeah . grad g: But phd b: Also CMU has been doing this and they were the most vocal at this meeting , Alex Waibel 's group . And they have said , I talked to the student who had done this , that with two fairly inexpensive cameras they they just recorded all the time professor d:   phd b: and were able to get all the information from or maybe it was three from all the parts of the room . So I think we would be we might lose the chance to use this data for somebody later who wants to do some kind of processing on it if we don't collect it at all . The problem with it is you 'll have more people who don't want to be filmed than who don't want to be recorded . grad g: So that there 's going to be another group of people who are gonna say " I won't participate " . phd b: Or you could put a paper bag over everybody 's head grad g:  phd b: and not look at each other and not look at boards , and just all be sitting talking . phd b: That would be an interes Bu postdoc h: Well professor d: Great idea . But I think y she 's we 're just proposing a minimal preservation of things on boards , phd b: Yeah . postdoc h: sp spatial organization And you could anonymize the faces for that matter . You know , this is grad g: But , you know , that 's a lot of infrastructure and work . postdoc h: We can talk about the grad g: To set it up and then anonymize it ? postdoc h: It 's just one snapshot . phd b: No , it wa n not ,  professor a: No , no , no , no . Maybe we don't want to spend that much more time discussing it , phd f: Did they store it digitally , or ? postdoc h: but phd b:   I think they just phd f: or just put it on videotape ? phd b: I think they just had the videotapes with a c you know , a counter or something . Well , I think for  , for our purposes we probably will d phd b: I 'm not sure . professor d: we we might try that some and and we certainly already have some recordings that don't have that , which , you know , we we 'll we 'll get other value out of , I think . postdoc h: Th The thing is , if it 's easy to collect it it th then I think it 's a wise thing to do because once it 's gone it 's gone . And phd b: I 'm just The community If LDC collects this data u  , and L - if Mark Liberman is a strong proponent of how they collect it and what they collect , there will probably be some video data in there . phd b: And so that could argue for us not doing it or it could argue for us doing it . The only place where it sort of overlaps is when some of the summarization issues are actually could be , easier made easier if you had the video . professor d: I think at the moment we should be determining this on the basis of our own , interests and needs rather than hypothetical ones from a community thing . professor d: As you say , if they if they decide it 's really critical then they will collect a lot more data than we can afford to , and and will include all that . professor d: professor a: e professor d: I I I 'm not worried about the cost of setting it up . In other words , it 's it it 'd be kind of silly to collect it all and not look at it at all . And so I I I think that we do have to do some picking and choosing of the stuff that we 're doing . But I I am int I do think that we m minimally want something we might want to look at at some some , subsets of that . Like for a meeting like this , at least , take a Polaroid of the of the of the boards , phd b: Of the board . phd b: Or at least make sure that the note - taker takes a sh you know , a snapshot of the board . professor d: a and know the position of the people phd b: That 'll make it a lot easier for meetings that are structured . phd b: otherwise later on if nobody wrote this stuff on the board down we 'd have a harder time summarizing it or agreeing on a summary .  , this is shared knowledge among all the participants , and it 's a shame to keep it off the recording . grad g: except in postdoc h: s grad g: er , if we weren't recording this , this this would get lost .  , I just think that the grad g: The point is that we 're not saving it anyway . postdoc h: Well professor a: What do you mean we 're not saving it anyway ? I 've written all of this down and it 's getting emailed to you . That would be the other alternative , to make sure that anything that was on the board , is in the record . postdoc h: Well professor a: Well , that 's why that 's why I 'm saying that I think the note - taking would be I think in many for many meetings there will be some sort of note - taking , in which case , that 's a useful thing to have  , we  , we don't need to require it . professor a: so so we won't worry about requiring these things , but the more things that we can get it for , the more useful it will be for various applications . professor d: So So , departing for the moment from the data collection question but actually talking about , you know , this group and what we actually want to do , so I guess that 's th the way what you were figuring on doing was was was , putting together some notes and sending them to to everybody from from today ? OK . professor d: So so the question that that we started with was whether there was anything else we should do during during th during the collection . professor d: And I guess the CrossPads was certainly one idea , and we 'll get them from him and we 'll just do that . Right ? And then the next thing we talked about was the was the summaries and are we gonna do anything about that . professor a: So , if I 'm collecting data then there is this question of do I use CrossPads ? professor d: Yeah . professor a: So , I think that if we really seriously have me collect data and I can't use CrossPads , it 's probably less useful for you guys to go to the trouble of using it , unless you think that the CrossPads are gonna n I 'm not I 'm not sure what they 're gonna do . But but having a small percentage of the data with it , I 'm not sure whether that 's useful or not . professor d: I guess the point was to try again , to try to collect more information that could be useful later for for the UI stuff . professor d: So it 's sort of Landay supplying it so that Landay 's stuff can be easier to do . professor d: So it it Right now he 's g operating from zero , professor a: Nothing . professor d: and so even if we didn't get it done from UW , it seems like that would could still You shou professor a: OK . phd b: I think it 'd be useful to have a small amount of it just as a proof of concept . grad g: And and they seem to not be able to give enough of them away , so we could probably get more as well . So if it if it seems to be really useful to you guys , we could probably get a donation to me . I think it it it will again depend on Landay , and if he has a student who 's interested , and how much infrastructure we 'll need . grad g: but if it requires a lot of our time , we probably won't do it . professor d: I guess a lot of the stuff we 're doing now really is pilot in one sense or another . That if if people have something in front of them phd b: I 'd be sort of cool . So , I j I think we should just say this is not we don't want to put any extra burden on people , but if they happen to generate minutes , could could they send it to us ? grad g: Yeah . What I was gonna say is that I don't want to ask people to do something they wouldn't normally do in a meeting . professor d: So who who does this summarization ? postdoc h: Yeah , I 'm thinking that grad g: People in the meeting . grad g: You know , just at at the end of the meeting , before you go , postdoc h:  - huh . phd e: People with radio mikes can go into separate rooms and continue recording without hearing each other . phd b: Well , then you should try them a few weeks later postdoc h: How fascinating . phd b: and They have all these memory experiments about how little you actually retain grad g: And see score them ? phd e: That 's right . phd b: and wasn't phd e: If we do if we collect four different summaries , you know , we 're gonna get all this weird data about how people perceive things differently . professor a: But but again , like the CrossPads , I don't think I would base a lot of stuff on it , professor d: Ru grad g: I d yeah , I don't know how you would do it , though . professor a: because I think I know when I see the the clock coming near the end of the meeting , I 'm like inching towards the door . grad g: Maybe e Is email easier ? postdoc h: Well , I think if grad g: I when you first said do do it , spoken , what I was thinking is , oh then people have to come up postdoc h:   postdoc h: Yeah , I 'd just try Well , however the least intrusive and and quickest way is , and th and closest to the meeting time too , cuz people will start to forget it as soon as they l leave . But , I don't think that they 'll necessarily you 'll you 'll get many people willing to stay . professor a: But , you know , if you get even one postdoc h: w professor d: Well , I think it 's like the note - taking thing , postdoc h: I would s Yeah . professor d: that that y that you can't certainly can't require it or people aren't gonna want to do this . postdoc h: And I 'm also wondering , couldn't that be included in the data sample so that you could increase the num you know , the words that are , recognized by a particular individual ? If you could include the person 's meeting stuff and also the person 's summary stuff , maybe that would be  , phd e: Yeah . postdoc h: Under the same acoustic circumstance , cuz if they just walk next door with their set - up , nothing 's changed , phd e: Right . postdoc h: just phd f: So I have a question about queries , grad g: God , that 's bugging me . phd f: which is , grad g: Can we turn that light off ? postdoc h: You turn phd f:  grad g: If can we turn that just that that let professor d: The fl the fluorescent light is flickering . grad g: Yeah , there 's a phd b: Oh , it is it is like OK . professor a: That and y Too much caffeine and really tired , grad g: Too much caffeine . phd f: the question I had about queries was , so what we 're planning to do is have people look at the summaries and then generate queries ? Are are we gonna try and o grad g: We we 've just been talking , how do we generate queries ? phd f: Yeah . phd f: so , the question I had is is have we given any thought to how we would generate queries automatically given a summary ?  , I think that 's a whole research topic un unto itself , professor d: Mmm . phd f: n phd b: Shouldn't Landay and his group be in charge of figuring out how to do this ? professor d: Yeah . phd b: They 're the expert phd e: Mari ? professor a: Yeah ? phd e: Someone wants to know when you 're getting picked up . Is someone picking you up ? professor a: what 's our schedule ? professor d: Well , you still wanted to talk with Liz . professor a: Let 's see , you and I need dis  , no , we did the Liz talk . professor a: We need to finish this discussion , and you and I need a little time for wrap - up and quad chart . So , grad g: And what ? professor a:  professor d: I 'm at your disposal . professor a: what what 's the plan for this discussion ? We should professor d: I think we should be able to wind up in another half - hour or something , you think ? grad g: At least . professor a: So , I think phd b: It 's interesting that he 's got , like , this discussion free professor d: Well , we still haven't talked about the action items from here and so on . professor d: And professor a: e e why don't you say five - thirty ? I don't phd e: OK , five - thirty . grad g: Well , in answer to " is it Landay 's problem ? " , he doesn't have a student who 's interested right now in doing anything . So I don't think he wants to do information retrieval , query generation , that sort of stuff . professor d: Yeah , well there 's gonna be these student projects that can do some things but it can't be , yeah , very deep . u I I actually think that that , again , just as a bootstrap , if we do have something like summaries , then having the people who are involved in the meetings themselves , who are cooperative and willing to do yet more , come up with with with queries , could at least give give Landay an idea of the kind of things that people might want to know .  , ye Right ? If he doesn't know anything about the area , and the people are talking about and and ,  phd b: But the people will just look at the summaries or the minutes and re and sort of back - generate the queries . phd f: Well , I 'm not sure I 'm not sure that 's a solved problem . phd f: Right ? Of how to how to generate queries from a phd b: How to do this from the summary . phd b: So what you want to h to do is , people who were there , who later see , minutes and s put in summary form , which is not gonna be at the same time as the meeting . phd b: and , like , make up some stuff to which these notes would be an answer , or or a deeper Yeah . phd b: You postdoc h: I 'm also wondering if we could ask the the people a a question which would be " what was the most interesting thing you got out of this meeting ? " Becau - in terms of like informativeness , phd b: That 's a good one . postdoc h: it might be , you know , that the summary would would not in even include what the person thought was the most interesting fact . professor a: But actually I would say that 's a better thing to ask than have them summarize the meeting . postdoc h: Because you get , like , the general structure of important points and what the what the meeting was about . postdoc h: So you get the general structure , the important points of what the meeting was about with the summary . But with the " what 's the most interesting thing you learned ? "  , so the fact that , I know that Transcriber uses Snack is something that I thought was interesting phd b: Going to see the kids . So , you could ge pick up some of the micro items that wouldn't even occur as major headings professor a:   professor c: Are you thinking about just asking one participant or all of them ? grad g: As many are willing to do it . Cuz you 'll get cuz you 'll get very different answers from everybody , right ? professor c: and then Yeah . phd e: So grad g: Well , maybe one thing we could do is for the meetings we 've already done  , I we didn't take minutes and we don't have summaries . professor a: But actually it would be an easy thing to just go around the room and say what was the most interesting thing you learned , grad g: Mmm . postdoc h: And that I think it would pick up the micro - structure , the some some of the little things that would be hidden . professor d: Boy , I I don't know how we get at this postdoc h: That would be interesting . professor c: Yeah , but when you go around the room you might just get the effect that somebody says something grad g: Or want to get up and leave . professor c: and then you go around the room and they say " yeah , me too , I agree . professor c: So phd e: On the other hand people might try and come up with different ones , right ? professor d: Well phd e: They might say " oh , I was gonna say that one but now I have to think of something else " . grad g: Well , you have the other thing , that that they know why we 're doing it . We 'll  , we 'll we 'll be telling them that the reason we 're trying to do this is is to d generate queries in the future , so try to pick things that other people didn't say .  , It seemed The kind of , interest that I had in this thing initially was , that i basically the form that you 're doing something else later , professor a:   So it 's really the imp the the list of what 's important 's in the something else professor a: Right . professor d: in fact if if it was really major , if it 's the thing that really stuck in your head , then you might not need to go back and and and check on it even . So it 's it 's that you 're trying to find You 're you 've now You weren't interested Say I I said " well , I wasn't that much interested in dialogue , I 'm more of an acoustics person " . professor d: But but thr three months from now if for some reason I get really interested in dialogue , and I 'm " well what is what was that part that that that , Mari was saying ? " grad g: Yeah , like Jim Bass says " add a few lines on dialogue in your next perf " professor a:  professor d: Yeah . professor d: And then I 'm trying to fi  , that 's that 's when I look in general when I look things up most , is when it 's something that didn't really stick in my head the first time around and but for some new reason I 'm I 'm I 'm interested in in in the old stuff . professor a: Well , I That 's hard to generate professor d: So , I don't I don't know . phd f: Do we professor a: and and I think that 's half of what i I would use it for . But I also a lot of times  , make you know , think to myself " this is interesting , professor d:   professor a: So , things that I think are interesting , I would be , wanting to do a query about . And also , I like the idea of going around the room , because if somebody else thought something was interesting , I 'd kind of want to know about it and then I 'd want to follow up on it . That that might get at some of what I was I was concerned about , being interested in something later that w  , I didn't consider to be important the first time , which for me is actually the dominant thing , because if I thought it was really important it tends to stick more than if I didn't , but some new task comes along that makes me want to look up . grad g: But But what 's interesting to me may not b have been interesting to you . grad g: Yeah , I I think you can't get at all of it , professor d: Yeah . phd f: The question the question then is h h how much bias do we introduce by you know , introduce by saying , you know , this was important now and , you know , maybe tha something else is important later ? professor d:   But postdoc h: Well , and and one thing , we we 're saying " important " and we 're saying " interesting " . But I I I guess that 's the question , really , is that  , postdoc h:   professor d: W phd f: does building queries based on what 's important now introduce an irreversible bias on being able to do what Morgan wants to do later ? professor d: Well , irreversible . phd f: That 's that 's professor d: I I  , I guess what I what I I keep coming back to in my own mind is that , the soonest we can do it , we need to get up some kind of system postdoc h: Yeah . professor d: so that people who 've been involved in the meeting can go back later , even if it 's a poor system in some ways , and ,  and ask the questions that they actually want to know . If you know , if  , as soon as we can get that going at any kind of level , then I think we 'll have a much better handle on what kind of questions people want to ask than in any anything we do before that . postdoc h: I will say that that I I chose " interesting " because I think it includes also " important " in some cases . postdoc h: Well , and and also i it puts a lot of burden on the person to to evaluate . You know , I think inter " interesting " is is non - threatening in professor a: OK - OK . professor a: In the interest of , grad g: Importance ? postdoc h: Yeah . professor a: No , i in the interest of generating some minutes here , and also moving on to action items and other things , let me just go through the things that I wrote down as being important , that we at least decided on . CrossPads we were going to try , if Landay can get the ,  get them to to you guys , and see if they 're interesting . professor a: getting just , digital pictures a couple digital pictures of the the table and boards to set the context of the meeting .  , and then going around the room at the end to just say qu ask people to mention something interesting that they learned . So rather than say the most interesting thing , something interesting , postdoc h: k professor a: and that way you 'll get more variety . " And then the last thing c would be for those people who are willing to stay afterwards and give an oral summary . professor a: OK ? Does that pretty much cover everything we talked about ? That well , that we want to do ? postdoc h:   grad g: Yeah , that 's like n I think that 's gonna predominantly end up being whoever takes down the equipment then . postdoc h: And and that would also be that the data would be included in the database . phd e: there is still this hope that people might actually think of real queries they really want to ask at some point . phd e: And that if if that ever should happen , then we should try and write them down . grad g: Give them a reward , a dollar a query ? phd e: Yeah , really . So professor d: Well , and again , if we can figure out a way to jimmy a a a a very rough system , say in a year , then  , so that in the second and third years we we actually have something to postdoc h: Yeah .  , the level of the query could be , you know , very low - level or very high - level . And it gets fuzzier and fuzzier as you go up , right ? grad g: Well , we 're gonna phd b: So you need to have some sort of if you start working with queries , some way of identifying what the you know , if this is something that requires a a one - word answer or it 's one place in the recording versus was there general agreement on this issue of all the people who ha phd e:  . phd b: In fact , they 're very meaningful cuz they 're very high - level . phd b: It But it may well grad g: But professor d: Because  , b because it depends on , what our goal is . grad g: Really ? professor d: If our goal is Wizard of Oz - ish , we might want to know what is it that people would really like to know about this data . professor d: And if it 's if if it 's something that we don't know how to do yet , th great , phd e: Yeah . grad g: Yeah , I was thinking about Wizard of Oz , but it requires the wizard to know all about the meetings . phd e: Well just imagine if professor d: But but just phd e: Get people to ask questions that they def the machine definitely can't answer at the moment , professor d: Yeah . w Just " what would you like to know ? " phd e: but grad g: Yep . postdoc h: I I was wondering if if there might be one s more source of queries which is indicator phrases like " action item " , professor a: OK . postdoc h: Yeah , that 's something to be determined , something to be specified , phd b: Well , probably if you have to sit there at the end of a meeting and say one thing you remember , it 's probably whatever action item was assigned to you . phd b: So , in general , that could be something you could say , right ? I 'm supposed to do this . Well , but then you could you could prompt them to say , you know , " other than your action item " , you know , whatever . phd e: Well postdoc h: But but the action item would be a way to get , maybe an additional query . Well , but you know , but you could get again @ @ professor a: Well , we 're piloting . professor a: or maybe we should wait until the summary of this until this meeting is transcribed and then we will hav professor d: We we had  , phd e: Yeah . professor d: somewhere up there we had milestones , but I guess Did y did you get enough milestone , from the description things ? professor a: I got Yeah . In fact , why don't you hand me those transparencies so that I remember to take them . professor d: And , you know , there 's obviously detail behind each of those , as much as is needed . What I have down for action items is we 're supposed to find out about our human subject , requirements . professor a: people are supposed to send me U R for their for web pages , to c and I 'll put together an overall cover . We professor a:  ? phd e: we need to look at our web page professor a: And and you also need to look at your web page phd e: and make one that 's that 's p professor a: and clean it up by mid - July . And so ,  professor a: How about if I just c  , Right now all I want I personally only want text data . I think the only thing Jeff would do anything with right now But I 'm just speaking fr based on a conversation with him two weeks ago I had in Turkey . professor d: Wh postdoc h: You could email to both of us , just  , if you wanted to . postdoc h: I don't think either of us would mind recei professor a: OK . postdoc h: but but in any case I 'd be happy to send you the professor a: And your email is ? professor d: i postdoc h: Edwards at ICSI . professor a: And then professor d: In in our phone call , before , we we ,  It turns out the way we 're gonna send the data is by , And ,  and then what they 're gonna do is take the CD - ROM and transfer it to analog tape and give it to a transcription service , that will grad g: Oh , is this IBM ? professor d: Yeah . postdoc h: Yeah , using foot pedals professor d: Yeah , foot foot pedals postdoc h: and professor d: and grad g: so do they How are they gonna do the multi - channel ? professor d: See , that 's a good question . professor d: probably about like you did , grad g: Mix ? professor d: and then there will be some things you know , many things that don't work out well . And that 'll go back to IBM and they 'll they 'll ,  they run their aligner on it and it kicks out things that don't work well , which you know , the overlaps will certainly be examples of that . professor d: We 'll give them all the the multi - channel stuff grad g: So we 'll give them all sixteen channels professor d: and grad g: and they 'll do whatever they want with it . phd b: But you also should probably give them the mixed You know , equal sound - level professor d: Yeah . professor d: It 's also won't be adding much to the data to give them the mixed . phd f: But w phd b: I phd f: It 's not professor a: Right . phd b: You should that may be all that they want to send off to their transcribers . Related to to the conversation with Picheny , I need to email him , my shipping address and you need to email them something which you already did . I I m emailed them the Transcriber URL , the on - line , data that Adam set up , The URL so they can click on an utterance and hear it . And I was gonna m email them the which I haven't yet , a pointer to to the web pages that we that we currently have , cuz in particular they want to see the one with the the way the recording room is set up postdoc h: Good . grad g: And then p possibly postdoc h: I C - I CC ' ed Morgan . grad g: Not an immediate action item but something we do have to worry about is data formats for for higher - level information . We were gonna grad g: Well , or d or not even higher level , different level , prosody and all that sort of stuff . professor a: That 's postdoc h: Well ,  professor a: W My my u feeling right now on format is you guys have been doing all the work phd e: Well  , yeah . So , what n important thing professor a: but Cuz I 'm not actually touching the data , postdoc h: Well , it c phd e: Right . professor d: So a key thing will be that you we tell you postdoc h: Great . phd f: and " professor d: We also had the ,  that we were s  , that you were gonna get us the eight - hundred number professor a: Oh , yeah . professor d: and we 're all gonna we 're gonna call up your Communicator thing and and we 're gonna be good slash bad , depending on how you define it , users . professor c: Now , something that I mentioned earlier to Mari and Liz is that it 's probably important to get as many non - technical and non - speech people as possible in order to get some realistic users . Cuz we don't want people who already know how to deal with dialogue systems , professor a: Yeah . Or , like if you have a professor c: who know that you shouldn't hyper - articulate , for instance , and things like that . professor a: Or , like if you have somebody who makes your your plane reservations for you , professor c: So . e You know , it could result in some good bloopers , which is always good for presentations . professor d: My mother would have a very interesting conversation with it grad g: He would never use it . So I have the professor d: We talked about that we 're getting the recording equipment running at UW . And so it depends , w e e e they 're you know , they 're p m If that comes together within the next month , there at least will be , major communications between Dan and UW folks phd e: Yeah .  , professor a: I 'm I 'm shooting to try to get it done get it put together by the beginning of August . professor d: as to phd e: we should talk about it , but postdoc h: Mmm . professor a: So , you know , if professor d: But we have it it 's it 's pretty We don't know .  , he he s  , he said that it was sitting in some room collecting dust professor a: We don't know . professor d: and and so we don't know , professor a: i It 's probably unlikely that we 'll pull this off , professor d: i e professor a: but a at least it 's worth trying .  , and I will email these notes  , I 'm not sure what to do about action items for the data stuff , although , then somebody I guess somebody needs to tell Landay that you want the pads . professor d: and he also said something about outside there that came up about the outside text sources , that he he may have grad g:   professor a: Oh ! professor d: some text sources that are close enough to the sort of thing that we can play with them for a language model . phd e: Yeah , that was  , that was What he was saying was this he this thing that , Jason had been working on finds web pages that are thematically related to what you 're talking about . But it 's p it might be phd e: so professor a: But but that 's actually what I wanna do . grad g: Un - unfortunately Landay told me that Jason is not gonna be working on that anymore . He seemed when I asked him if he could actually supply data , he seemed a little bit more reluctant . If I don't get something grad g: Who ? Landay or Jason ? professor a: Landay . professor a: you know , otherwise , if you guys have any papers or I could I could use ,  I could use your web pages . You 've got all the web pages on the Meeting Recor professor d: Yeah , why search for them ? professor a: Yeah ! professor d: They 're we know where they are . professor d: There 's there 's some , Carnegie Mellon stuff , right ? On on meeting recording , grad g: Yep . professor a: So , there 's there 's ICSI , Xerox , professor d: and phd b: And there 's You should l look under , like , intelligent environments , professor d: And Xerox . phd b: smart rooms , grad g: the " Georgia Tech Classroom Two Thousand " is a good one . J There 's th That 's where I thought you would want to eventually be able to have a board or a camera , professor a:   phd b: because of all these classroom grad g: Well , Georgia Tech did a very elaborate instrumented room . I think that 's good enou that 's that 's pretty much all I can think of . postdoc h: Can I ask , one thing ? It relates to data data collection and I and I 'd and we mentioned earlier today , this question of  , so , I s I know that from with the near - field mikes some of the problems that come with overlapping speech , are lessened . But I wonder if  , is that sufficient or should we consider maybe getting some data gathered in such a way that , u w we would c  , p have a meeting with less overlap than would otherwise be the case ? So either by rules of participation , or whatever . postdoc h: Now , you know , it 's true , we were discussing this earlier , that depending on the task so if you 've got someone giving a report you 're not gonna have as much overlap . phd f: Adam ! postdoc h: But , i i  , so we 're gonna have s you know , non - overlapping samples anyway . But , in a meeting which would otherwise be highly overlapping , is the near - field mike enough or should we have some rules of participation for some of our samples to lessen the overlap ? professor d:  . phd e: turn off professor a: I don't think we should have rules of participation , but I think we should try to get a variety of meetings . That 's something that if we get the the meeting stuff going at UW , that I probably can do more than you guys , postdoc h: OK . Just want to be sure there 's enough data to professor a: phd b: They 're still gonna overlap , postdoc h: OK , good . phd b: but Mark and others have said that there 's quite a lot of found data from the discourse community that has this characteristic and also the political Y you know , anything that was televised for a third party has the characteristic of not very much overlap . professor d: Wasn - but w I think we were saying before also that the natural language group here had less overlap . postdoc h: because you know , it is true people can modify the amount of overlap that they do if if they 're asked to . postdoc h: Not not entirely modify it , but lessen it if if it 's desired . But if if that 's sufficient data I just wanted to be sure that we will not be having a lot of data which can't be processed . So I 'm just writing here , we 're not gonna try to specify rules of interaction but we 're gonna try to get more variety by i using different groups of people postdoc h: Time . And I you know , I I know that the near f near - field mikes will take care of also the problems to s to a certain degree . professor a: Cuz if I recorded some administrative meetings then that may have less overlap , because you might have more overlap when you 're doing something technical and disagreeing or whatever . Well , I just as as as a contributary eh , so I I know that in l in legal depositions people are pr are prevented from overlapping . They 'll just say , you know you know , " wait till each person is finished before you say something " . professor a: Well , the reason why I didn't want to is be why I personally didn't want to is because I wanted it to be as , unintrusive as possi postdoc h:   I just want to be sure we don't that we 're able to process , i u  , you know , as much data as we can . Did they discuss any of that in the the meeting they had with L Liberman ? phd b:   professor d: What phd b: And there was a big division , professor d: What what do they phd b: so Liberman and others were interested in a lot of found data . phd b: So there 's lots of recordings that They 're not close - talk mike , professor d: Yeah . phd b: but And and there 's lots of television , you know , stuff on , political debates and things like that , congre congressional hearings .  , and then the CMU folks and I were sort of on the other side in cuz they had collected a lot of meetings that were sort of like this and said that those are nothing like these meetings . And , I guess we just left it as @ @ that if there 's found data that can be transformed for use in speech recognition easily , then of course we would do it , professor d:   Is that is that going to be publicly available , phd b: As far as I know , they h have not . grad g: It 's also it 's not it 's not near - far , right ? phd b: I 'm not sure .  , if people were interested they could talk to them , but I I got the feeling there was some politics involved . grad g: Cuz I had thought they 'd only done far - field , phd b: I think you need to talk to Waibel and grad g: intelligent - room sorts of things . phd b: But they had multiple mikes and they did do recognition , and they did do real conversations . Well , once we send out postdoc h: Yeah , I think professor d: we still haven't sent out the first note saying " hey , this list exists " . But but , once we do that professor a: Is that an action item ? professor d: Yeah . We should at least check that everybody here ? grad g: I think everyone here is on the list . grad g: I I added a few people who didn't who I knew had to be on it even though they didn't tell me . grad g: You are on it , aren't you ? postdoc h: Yeah , I am . So " found data " , they mean like established corpora of linguistics and and other fields , right ? phd b: What they mean is stuff they don't have to fund to collect , postdoc h: It sounds like such a t phd b: and especially good postdoc h: Yeah , OK . It 's things occur without any You know , the pe these people weren't wearing close - talking mikes , but they were recorded anyway , like the congressional hearings and , you know , for legal purposes or whatever . Of course it 's not " found " in the sense that at the time it was collected for the purpose . phd b: But what he means is that You know , Mark was really a fan of getting as much data as possible from you know , reams and reams of stuff , of broadcast stuff , postdoc h: That 's interesting . phd e: We should go around and say something interesting that happened at the meeting ? professor a: Oh . phd b: Rrrh ! grad g: Now , I was already thinking about it , so professor d: Oh ! Good man . phd b: I grad g: So , I really liked the idea of what I thought was interesting was the combination of the CrossPad and the speech . So , can you determine the interesting points by who 's writing ? Can you do special gestures and so on that that have , special meaning to the corpora ? I really liked that . postdoc h: Well , I I just realized there 's another category of interesting things which is that , I I found this discussion very , i this this question of how you get at queries really interesting . And and the and I and the fact that it 's sort of , nebulous , what what that what kind of query it would be because it depends on what your purpose is . So I actually found that whole process of of trying to think of what that would involve to be interesting . I just sort of thought we we went around a nice discussion of the factors involved there , which I thought was worthwhile . grad g: Did you take pictures of the boards ? phd e: Not that I postdoc h: Yeah .  , I will take pictures of them , but postdoc h: That 's a good point . phd f: I 'm gonna pass because I can't  , of the Jane took my answer . But I will say  , I will actually  , a spin on different slightly different spin on what you said , this issue of , realizing that we could take minutes , and that actually may be a goal . So that that may be kind of the test in a sense , test data , the the template of what we want to test against , generating a summary . I think the question of how to generate queries automatically was the most interesting question that came up , and it 's something that , as you said , is a whole research topic in itself , so I don't think we 'll be able to do anything on it because we don't have funding on it , in this project . professor d: Well , being more management lately than than research , I think the thing that impressed me most was the people dynamics and not any of the facts . phd e: How are we gonna find that in the data ? grad g: Well , if we had people wearing the wireless mikes all the time phd e: Oh , yeah . grad g: Yeah , I think phd f: Well , one thing you could search for is were people laughing a lot . phd e: How happy were they ? professor d: I 'd probably search for something like that . postdoc h:  h Do we need do I need to turn something off here , or I do unplug this , or ? professor d: Now these we turn off