grad c: Yeah , we had a long discussion about how much w how easy we want to make it for people to bleep things out . phd d: It it doesn't grad c: Did did did it ? I didn't even check yesterday whether it was moving . phd d: So I don't know if it doesn't like both of us grad c: Channel three ? Channel three ? phd d: You know , I discovered something yesterday on these , wireless ones . grad c:  -  ? phd d: You can tell if it 's picking up breath noise and stuff . So if you yeah , if you breathe under breathe and then you see AF go off , then you know it 's p picking up your mouth noise . phd f: In fact , if you listen to just the channels of people not talking , it 's like " @ @ " . It 's very disgust grad c: What ? Did you see Hannibal recently or something ? phd f: Sorry . So , grad c:  phd f: I was gonna try to get out of here , like , in half an hour , cuz I really appreciate people coming , and the main thing that I was gonna ask people to help with today is to give input on what kinds of database format we should use in starting to link up things like word transcripts and annotations of word transcripts , so anything that transcribers or discourse coders or whatever put in the signal , with time - marks for , like , words and phone boundaries and all the stuff we get out of the forced alignments and the recognizer . So , we have this ,  I think a starting point is clearly the the channelized output of Dave Gelbart 's program , which Don brought a copy of , grad c: Yeah . phd f: which phd d: Can I see it ? grad c: And so the only question is it the sort of thing that you want to use or not ? Have you looked at that ?  , I had a web page up . So , grad c: So phd f: I actually mostly need to be able to link up , or I it 's it 's a question both of what the representation is and grad c: You mean , this I guess I am gonna be standing up and drawing on the board . grad c: and then you can have lots of different sections , each of which have I Ds attached to it , and then you can refer from other sections to those I Ds , if you want to . I don't e I don't remember exactly what my notation was , phd a: Oh , I remember seeing an example of this . grad c: Yeah , " T equals one point three two " ,  And then I I also had optional things like accuracy , and then " ID equals T one , one seven " . And then , I also wanted to to be i to be able to not specify specifically what the time was and just have a stamp . grad c: Yeah , so these are arbitrary , assigned by a program , not not by a user . And then somewhere la further down you might have something like an utterance tag which has " start equals T - seventeen , end equals T - eighteen " . grad c: Right ? But it ends at this T - eighteen , which may be somewhere else . We don't know what the t time actually is but we know that it 's the same time as this end time . Right ? So you could you could have some sort of other other tag later in the file that would be something like , oh , I don't know , " noise - type equals door - slam " . You know ? And then , you could either say " time equals a particular time - mark " or you could do other sorts of references . So or or you might have a prosody " Prosody " right ? D ? T ? D ? T ? T ? phd f: It 's an O instead of an I , but the D is good . grad c: you know , so you could have some sort of type here , and then you could have ,  the utterance that it 's referring to could be U - seventeen or something like that . So , that seems that seems g great for all of the encoding of things with time and , grad c: Oh , well . phd f:  I I guess my question is more , what d what do you do with , say , a forced alignment ? phd a: How - how phd f:  you 've got all these phone labels , and what do you do if you just conceptually , if you get , transcriptions where the words are staying but the time boundaries are changing , cuz you 've got a new recognition output , or s sort of what 's the , sequence of going from the waveforms that stay the same , the transcripts that may or may not change , and then the utterance which where the time boundaries that may or may not change ? phd a: Oh , that 's That 's actually very nicely handled here because you could you could all you 'd have to change is the , time - stamps in the time - line without without , changing the I Ds . And you 'd be able to propagate all of the the information ? grad c: Right . phd a: You 'd have you 'd have phd f: The we we have phone - level backtraces . grad c: Yeah , this I don't think I would do this for phone - level . I think for phone - level you want to use some sort of binary representation phd f:  grad c: because it 'll be too dense otherwise . So , if you were doing that and you had this sort of companion , thing that gets called up for phone - level , what would that look like ? phd a: Why grad c: I would use just an existing an existing way of doing it . But but why not use it for phone - level ? phd f: H h phd a: It 's just a matter of it 's just a matter of it being bigger . But if you have you know , barring memory limitations , or  I w  this is still the m grad c: It 's parsing limitations . I don't want to have this text file that you have to read in the whole thing to do something very simple for . You would use it only for purposes where you actually want the phone - level information , I 'd imagine . phd f: So you could have some file that configures how much information you want in your in your XML or something .  , you 'd y phd f: phd a: You grad c: I I am imagining you 'd have multiple versions of this depending on the information that you want . grad c: I 'm just what I 'm wondering is whether I think for word - level , this would be OK . grad c: For lower than word - level , you 're talking about so much data that I just I don't know . I don't know if that phd f: we actually have So , one thing that Don is doing , is we 're we 're running For every frame , you get a pitch value , phd d: Lattices are big , too . phd f: and not only one pitch value but different kinds of pitch values grad c: Yeah , for something like that I would use P - file phd f: depending on grad c: or or any frame - level stuff I would use P - file . phd d: But what what 's the advantage of doing that versus just putting it into this format ? grad c: More compact , which I think is is better . grad c: if you did it at this phd f:  these are long meetings and with for every frame , grad c: You don't want to do it with that Anything at frame - level you had better encode binary phd f:  grad c: or it 's gonna be really painful .  , b you can always , G - zip them , and , you know , c decompress them on the fly if y if space is really a concern . phd d: Yeah , I was thi I was thinking the advantage is that we can share this with other people . grad c: Well , but if you 're talking about one per frame , you 're talking about gigabyte - size files . These are really  grad c: Right ? Because you have a two - gigabyte limit on most O Ss . But for phone - level stuff it 's perfectly phd f: And th it 's phd a: Like phones , or syllables , or anything like that . So , you know , people don't v Look at it , words times the average The average number of phones in an English word is , I don't know , five maybe ? phd f: Yeah , but we actually phd a: So , look at it , t number of words times five . That 's not that not phd f: Oh , so you mean pause phones take up a lot of the long pause phones . grad c: So I think it it 's debatable whether you want to do phone - level in the same thing . grad c: But I think , a anything at frame - level , even P - file , is too verbose . phd f: I haven't seen this particular format , phd a: I 've I 've used them . phd a: I 've forgot what the str phd d: But , wait a minute , P - file for each frame is storing a vector of cepstral or PLP values , grad c: It 's whatever you want , actually . grad c: So that what 's nice about the P - file It i Built into it is the concept of frames , utterances , sentences , that sort of thing , that structure . So , the only problem with it is it 's actually storing the utterance numbers and the frame numbers in the file , even though they 're always sequential . Is there some documentation on this somewhere ? grad c: Yeah , there 's a ton of it . I I was just looking for something I 'm not a database person , but something sort of standard enough that , you know , if we start using this we can give it out , other people can work on it , grad c: Yeah , it 's not standard . phd f: or Is it ? grad c: it 's something that we developed at ICSI . But ,  phd f: But it 's been used here grad c: But it 's been used here phd f: and people 've grad c: and and , you know , we have a well - configured system that you can distribute for free , and phd d: it must be the equivalent of whatever you guys used to store feat your computed features in , right ? phd f: OK . phd a: Yeah , th we have Actually , we we use a generalization of the the Sphere format . phd a: but Yeah , so there is something like that but it 's , probably not as sophist grad c: Well , what does H T K do for features ? phd d: And I think there 's grad c: Or does it even have a concept of features ? phd a: They ha it has its own  , Entropic has their own feature format that 's called , like , S - SD or some so SF or something like that . grad c: I 'm just wondering , would it be worth while to use that instead ? phd d: Yeah . Th - this is exactly the kind of decision It 's just whatever phd d: But , people don't typically share this kind of stuff , right ? phd a: Right . phd f: Actually , I I just you know , we we 've done this stuff on prosodics and three or four places have asked for those prosodic files , and we just have an ASCII , output of frame - by - frame . phd f: Which is fine , but it gets unwieldy to go in and and query these files with really huge files . I was just thinking if there 's something that where all the frame values are grad c: And a and again , if you have a if you have a two - hour - long meeting , that 's gonna phd f:  ? They 're they 're fair they 're quite large . phd f: and So it 's doable , it 's just that you can only store a feature vector at frame - by - frame and it doesn't have any kind of , phd d: Is is the sharing part of this a pretty important consideration phd f:  phd d: or does that just sort of ,  a nice thing to have ? phd f: I I don't know enough about what we 're gonna do with the data . But I thought it would be good to get something that we can that other people can use or adopt for their own kinds of encoding . phd f: And especially for the prosody work , what what it ends up being is you get features from the signal , and of course those change every time your alignments change . So you re - run a recognizer , you want to recompute your features , and then keep the database up to date . phd f: Or you change a word , or you change a utterance boundary segment , which is gonna happen a lot . And so I wanted something where all of this can be done in a elegant way and that if somebody wants to try something or compute something else , that it can be done flexibly .  , it doesn't have to be pretty , it just has to be , you know , easy to use , and grad c: Yeah , the other thing We should look at ATLAS , the NIST thing , phd f: Oh . phd f:  grad c: I 'm not sure what to do about this with ATLAS , because they chose a different route . Your your file format can know about know that you 're talking about language and speech , which is what I chose , and time , or your file format can just be a graph representation . So what it looked like ATLAS chose is , they chose the other way , which was their file format is just nodes and links , and you have to interpret what they mean yourself . phd f: And why did you not choose that type of approach ? grad c: because I knew that we were doing speech , and I thought it was better if you 're looking at a raw file to be t for the tags to say " it 's an utterance " , as opposed to the tag to say " it 's a link " . grad c: So , but phd f: But other than that , are they compatible ?  , you could sort of grad c: Yeah , they 're reasonably compatible . phd f: Yeah , that 's w So , grad c: So , well , the other thing is if we choose to use ATLAS , which maybe we should just do , we should just throw this out before we invest a lot of time in it . phd f: just sort of how to  , cuz we need to come up with a database like this just to do our work . And I actually don't care , as long as it 's something useful to other people , what we choose . phd f: So maybe it 's maybe oth you know , if if you have any idea of how to choose , cuz I don't . phd a: Do they already have tools ? grad c: I I chose this for a couple reasons . phd f: And you can have as much information in the tag as you want , right ? grad c: Well , I have it structured . So what What NIST would say is that instead of doing this , you would say something like " link start equals , you know , some node ID , phd f: Yeah . So grad c: end equals some other node ID " , and then " type " would be " utterance " . phd f: So why would it be a a waste to do it this way if it 's similar enough that we can always translate it ? phd d: It probably wouldn't be a waste . It would mean that at some point if we wanted to switch , we 'd just have to translate everything . But it se Since they are developing a big phd f: But it but that sounds phd d: But that 's I don't think that 's a big deal . And so it seems to me that if if we want to use that , we might as well go directly to what they 're doing , rather than phd a: If we want to Do they already have something that 's that would be useful for us in place ? phd d: Yeah .  , how stable is their Are they ready to go , grad c: The I looked at it phd d: or ? grad c: The last time I looked at it was a while ago , probably a year ago , when we first started talking about this . Since then , they 've developed their own external file format , which is , you know , this sort of s this sort of thing .  , and apparently they 've also developed a lot of tools , but I haven't looked at them . phd f: would the tools would the tools run on something like this , if you can translate them anyway ? grad c: th what would would would what would worry me is that maybe we might miss a little detail phd a: It 's a hassle phd f: that I guess it 's a question that phd a: if phd f: yeah . phd a: I I think if it 's conceptually close , and they already have or will have tools that everybody else will be using , it would be crazy to do something s you know , separate that phd f: OK . phd f: Actually , so it 's that that would really be the question , is just what you would feel is in the long run the best thing . phd f: Cuz once we start , sort of , doing this I don't we don't actually have enough time to probably have to rehash it out again grad c: The Yep . The other thing the other way that I sort of established this was as easy translation to and from the Transcriber format . But , I suppose that as long as they have a type here that specifies " utt " , grad c: It 's almost the same . phd f: it 's yeah , close enough that grad c: The the the the point is with this , though , is that you can't really add any supplementary information . Right ? So if you suddenly decide that you want phd f: You have to make a different type . phd f: So Well , if you look at it and  , I guess in my mind I don't know enough Jane would know better , about the types of annotations and and But I imagine that those are things that would well , you guys mentioned this , that could span any it could be in its own channel , it could span time boundaries of any type , grad c: Right . And then at the prosody - level we have frame sort of like cepstral feature files , grad c: Yep . And that 's sort of the world of things that I And then we have the aligned channels , of course , grad c: Right . phd a: And then phd f: I I definitely agree and I wanted to find actually a f a nicer format or a maybe a more compact format than what we used before . phd f: Just cuz you 've got ten channels or whatever and two hours of a meeting . phd a: Now now how would you how would you represent , multiple speakers in this framework ? Were You would just represent them as grad c: phd a: You would have like a speaker tag or something ? grad c: there 's a spea speaker tag up at the top which identifies them and then each utt the way I had it is each turn or each utterance , I don't even remember now , had a speaker ID tag attached to it . grad c: And in this format you would have a different tag , which which would , be linked to the link . grad c:  Let 's see , would it be a node or a link ?  And so so this one would have , an ID is link link seventy - four or something like that . grad c: And then somewhere up here you would have a link that that , you know , was referencing L - seventy - four and had speaker Adam . phd f: Actually , it 's the channel , I think , that phd a: Well , channel or speaker or whatever . phd f: w yeah , channel is what the channelized output out phd a: It doesn't grad c: This isn't quite right . phd f: Yeah , but phd a: But but so how in the NIST format do we express a hierarchical relationship between , say , an utterance and the words within it ? So how do you tell that these are the words that belong to that utterance ? grad c: you would have another structure lower down than this that would be saying they 're all belonging to this ID . And then each utterance could refer to a turn , phd d: So it 's it 's not hi it 's sort of bottom - up . phd f: And what if you actually have So right now what you have as utterance , the closest thing that comes out of the channelized is the stuff between the segment boundaries that the transcribers put in or that Thilo put in , which may or may not actually be , like , a s it 's usually not  , the beginning and end of a sentence , say . phd f: So , I assume this is possible , that if you have someone annotates the punctuation or whatever when they transcribe , you can say , you know , from for from the c beginning of the sentence to the end of the sentence , from the annotations , this is a unit , even though it never actually i It 's only a unit by virtue of the annotations at the word - level . grad c: And , what phd f: But it 's just not overtly in the phd a: OK . phd f: cuz this is exactly the kind of phd a: So phd f: I think that should be possible as long as the But , what I don't understand is where the where in this type of file that would be expressed . phd f: S so it would just be floating before the sentence or floating after the sentence without a time - mark . grad c: You could have some sort of link type type equals " sentence " , and ID is " S - whatever " . phd a:  grad c: Can you can you say that this is part of this , phd f: See , cuz it 's phd a: Hhh . phd f: it 's phd d: You would just have a r phd f: S grad c: or do you say this is part of this ? I think phd d: You would refer up to the sentence . phd f: But they 're phd a: Well , the thing phd f: they 're actually overlapping each other , sort of . grad c: So phd a: the thing is that some something may be a part of one thing for one purpose and another thing of another purpose . phd a: s  , well , s let 's let 's ta so let 's grad c: Well , I think I 'm I think w I had better look at it again phd f: Yeah . phd a: y So for instance @ @ sup grad c: There 's one level there 's one more level of indirection that I 'm forgetting . phd a: Suppose you have a word sequence and you have two different segmentations of that same word sequence . phd a: I don't know if that 's true or not but let 's as phd f: Well , it 's definitely true with the segment . phd f: That 's what I exactly what I meant by the utterances versus the sentence could be sort of phd a: Yeah . So , you want to be s you want to say this this word is part of that sentence and this prosodic phrase . grad c: I I 'm pretty sure that you can do that , but I 'm forgetting the exact level of nesting . phd a: So , you would have to have two different pointers from the word up one level up , one to the sent grad c: So so what you would end up having is a tag saying " here 's a word , and it starts here and it ends here " . grad c: And then lower down you would say " here 's a prosodic boundary and it has these words in it " . phd f: So you would be able to go in and say , you know , " give me all the words in the bound in the prosodic phrase grad c: Yep . The the o the other issue that you had was , how do you actually efficiently extract ,  find and extract information in a structure of this type ? phd f: OK . phd a: So you gave some examples like phd f: Well , and , you guys might I don't know if this is premature because I suppose once you get the representation you can do this , but the kinds of things I was worried about is , phd a: No , that 's not clear . phd f:  phd a: yeah , you c sure you can do it , phd f: Well , OK . So i if it phd a: but can you do it sort of l l you know , it phd f: I  , I can't do it , but I can  , phd a: y y you gotta you gotta do this you you 're gonna want to do this very quickly grad c: Well phd a: or else you 'll spend all your time sort of searching through very complex data structures phd f: Right . But an example would be " find all the cases in which Adam started to talk while Andreas was talking and his pitch was rising , Andreas 's pitch " .  , that 's gonna be Is the rising pitch a feature , or is it gonna be in the same file ? phd f: Well , the rising pitch will never be hand - annotated . So the all the prosodic features are going to be automatically grad c: But the  , that 's gonna be hard regardless , phd f: So they 're gonna be in those grad c: right ? Because you 're gonna have to write a program that goes through your feature file and looks for rising pitches . So normally what we would do is we would say " what do we wanna assign rising pitch to ? " Are we gonna assign it to words ? Are we gonna just assign it to sort of when it 's rising we have a begin - end rise representation ? But suppose we dump out this file and we say , for every word we just classify it as , w you know , rise or fall or neither ? grad c: OK . grad c: r phd f: So we would basically be sort of , taking the format and enriching it with things that we wanna query in relation to the words that are already in the file , grad c: Right . phd a: You want sort of a grep that 's that works at the structural on the structural representation . There 's a standard again in XML , specifically for searching XML documents structured X - XML documents , where you can specify both the content and the structural position . phd a: Yeah , but it 's it 's not clear that that 's That 's relative to the structure of the XML document , phd f: If phd a: not to the structure of what you 're representing in the document . grad c: It 's it 's you would use that to build your tool to do that sort of search . phd f: But as long as the grad c: It 's a graph , but phd a: That 's different from searching through the text . phd f: But it seems like as long as the features that grad c: Well , no , no , no . phd a:  grad c: So that  th phd f: That 's true if the features from your acoustics or whatever that are not explicitly in this are at the level of these types . phd f: That that if you can do that grad c: Yeah , but that 's gonna be the trouble no matter what . Right ? No matter what format you choose , you 're gonna have the trou you 're gonna have the difficulty of relating the the frame - level features phd f: That 's right . phd f: You know , it Or another example was , you know , where in the language where in the word sequence are people interrupting ? So , I guess that one 's actually easier . phd d: What about what about , the idea of using a relational database to , store the information from the XML ? So you would have XML basically would  , you you could use the XML to put the data in , and then when you get data out , you put it back in XML . phd d: but then you store the data in the database , which allows you to do all kinds of good search things in there . grad c: The ,  One of the things that ATLAS is doing is they 're trying to define an API which is independent of the back store , phd f: Huh . grad c: so that , you could define a single API and the the storage could be flat XML files or a database . grad c: My opinion on that is for the s sort of stuff that we 're doing , I suspect it 's overkill to do a full relational database , that , just a flat file and , search tools I bet will be enough . phd a: But grad c: But that 's the advantage of ATLAS , is that if we actually take decide to go that route completely and we program to their API , then if we wanted to add a database later it would be pretty easy . phd f: It seems like the kind of thing you 'd do if I don't know , if people start adding all kinds of s bells and whistles to the data . And so that might be  , it 'd be good for us to know to use a format where we know we can easily , input that to some database if other people are using it . grad c: I guess I 'm just a little hesitant to try to go whole hog on sort of the the whole framework that that NIST is talking about , with ATLAS and a database and all that sort of stuff , phd f: So grad c: cuz it 's a big learning curve , just to get going . grad c: Whereas if we just do a flat file format , sure , it may not be as efficient but everyone can program in Perl and and use it . grad c: Right ? phd a: But this is grad c: So , as opposed to phd a: I I 'm still , not convinced that you can do much at all on the text on the flat file that that you know , the text representation . e Because the text representation is gonna be , not reflecting the structure of of your words and annotations . It 's just it 's grad c: Well , if it 's not representing it , then how do you recover it ? Of course it 's representing it . You you have to what you have to do is you have to basically grad c: That 's the whole point . grad c: Right ? So what I was saying is that phd a: But that 's what you 'll have to do . And it 's a set of tools that let you specify given the D - DDT DTD of the document , what sorts of structural searches you want to do . So you want to say that , you know , you 're looking for , a tag within a tag within a particular tag that has this particular text in it , and , refers to a particular value . And so the point isn't that an end - user , who is looking for a query like you specified , wouldn't program it in this language . phd f: Is a See , I think the kinds of questions , at least in the next to the end of this year , are there may be a lot of different ones , but they 'll all have a similar nature . They 'll be looking at either a word - level prosodic , an a value , grad c:   But you know , we 'll do something where we some kind of data reduction where the prosodic features are sort o  , either at the word - level or at the segment - level , grad c: Right . They 're not gonna be at the phone - level and they 're no not gonna be at the frame - level when we get done with sort of giving them simpler shapes and things .  , one that Chuck mentioned is starting out with something that we don't have to start over , that we don't have to throw away if other people want to extend it for other kinds of questions , grad c: Right . phd f: and being able to at least get enough , information out on where we condition the location of features on information that 's in the kind of file that you put up there . grad c: And so it seems to me that  , I have to look at it again to see whether it can really do what we want , but if we use the ATLAS external file representation , it seems like it 's rich enough that you could do quick tools just as I said in Perl , and then later on if we choose to go up the learning curve , we can use the whole ATLAS inter infrastructure , phd f: Yeah . phd f: I I don't So if if you would l look at that and let us know what you think . phd f: I think we 're sort of guinea pigs , cuz I I want to get the prosody work done but I don't want to waste time , you know , getting the phd a: Oh , maybe phd f: Yeah ? phd a:  grad c: Well , I wouldn't wait for the formats , because anything you pick we 'll be able to translate to another form . phd a: Well Ma well , maybe you should actually look at it yourself too to get a sense of what it is you 'll you 'll be dealing with , phd f: OK . phd a: because , you know , Adam might have one opinion but you might have another , so grad b: Yeah . phd f: Especially if there 's , e  you know , if someone can help with at least the the setup of the right grad c: Hi , Jane . phd f: the right representation , then , i you know , I hope it won't We don't actually need the whole full - blown thing to be ready , grad c: Can you Oh , well .  , so maybe if you guys can look at it and sort of see what , grad b: Yeah . phd f:  I think we 're we 're we 're actually just grad c: We 're about done . phd f: wrapping up , but ,  Yeah , sorry , it 's a  short meeting , but ,  Well , I don't know . Is there anything else , like  that helps me a lot , grad c: Well , I think the other thing we might want to look at is alternatives to P - file . phd f: but grad c: th the reason I like P - file is I 'm already familiar with it , we have expertise here , and so if we pick something else , there 's the learning - curve problem . phd a: Is there an is there an IP - API ? grad c: And so Yeah . And , phd a: There used to be a problem that they get too large , grad c: a bunch of libraries , P - file utilities . phd a: and so basically the  the filesystem wouldn't grad c: Well , that 's gonna be a problem no matter what . phd a: Maybe you could extend the API to , support , like splitting up , you know , conceptually one file into smaller files on disk so that you can essentially , you know , have arbitrarily long f grad c: Yep . That that most many of them can s you can specify several P - files and they 'll just be done sequentially . phd f: So , I guess , yeah , if if you and Don can if you can show him the P - file stuff and see . grad c: if you do " man P - file " or " apropos P - file " , you 'll see a lot . phd f: Yeah ? phd d: I don't remember what the " P " is , though . grad c: But there are ni they 're The Quicknet library has a bunch of things in it to handle P - files , phd a: Yeah . phd a:  phd f: And that isn't really , I guess , as important as the the main I don't know what you call it , the the main sort of word - level grad c: Neither do I .  , so grad c: Yeah , I 've been meaning to look at the ATLAS stuff again anyway . I guess it 's also sort of a political deci  , if if you feel like that 's a community that would be good to tie into anyway , then it 's sounds like it 's worth doing . grad c: Yeah , I think it it w phd a: j I think there 's grad c: And , w  , as I said , I what I did with this stuff I based it on theirs . So now that they have come up with a format , it doesn't it seems pretty reasonable to use it . grad c: As I said , that phd f: Cuz we actually can start grad c: There 's one level there 's one more level of indirection and I 'm just blanking on exactly how it works . phd f: we can start with , I guess , this input from Dave 's , which you had printed out , the channelized input . Cuz he has all of the channels , you know , with the channels in the tag and stuff like that . And so then it would just be a matter of getting making sure to handle the annotations that are , you know , not at the word - level and , t to import the grad b: Where are those annotations coming from ? phd f: Well , right now , I g Jane would would grad c:   postdoc e: Are you talking about the overlap a annotations ? phd f: Yeah , any kind of annotation that , like , isn't already there . And since we w we I I think it 's important to remain flexible regarding the time bins for now . And so it 's nice to have However , you know , you want to have it , time time  , located in the discourse . So , if we if we tie the overlap code to the first word in the overlap , then you 'll have a time - marking . It won't it 'll be independent of the time bins , however these e evolve , shrink , or whatever , increase , or Also , you could have different time bins for different purposes . And having it tied to the first word in an overlap segment is unique , you know , anchored , clear . postdoc e: Or the ? phd d: I 'm not sure what that @ @ grad c: Well , is that phd d: It probably doesn't matter . phd d: No , I d postdoc e: We don't have to go into the codes . W the idea is just to have a separate green ribbon , you know , and and and let 's say that this is a time bin . This is the first word of an overlapping segment of any length , overlapping with any other , word  , i segment of any length . And , then you can indicate that this here was perhaps a ch a backchannel , or you can say that it was , a usurping of the turn , or you can you know , any any number of categories . But the fact is , you have it time - tagged in a way that 's independent of the , sp particular time bin that the word ends up in . postdoc e: we sh change the boundaries of the units , it 's still unique and and , fits with the format , phd f: Right . phd a: it would be nice  , eh , gr this is sort of r regarding  ,  it 's related but not directly germane to the topic of discussion , but , when it comes to annotations , you often find yourself in the situation where you have different annotations of the same , say , word sequence . phd a: And sometimes the word sequences even differ slightly because they were edited s at one place but not the other . phd a: So , once this data gets out there , some people might start annotating this for , I don't know , dialogue acts or , you know , topics or what the heck . And the only thing that is really sort of common among all the versi the various versions of this data is the word sequence , or approximately . But , see , if you 'd annotate dialogue acts , you don't necessarily want to or topics you don't really want to be dealing with time - marks . phd a: You 'd it 's much more efficient for them to just see the word sequence , right ? phd f:   phd a: most people aren't as sophisticated as as we are here with , you know , time alignments and stuff . So So the the the point is grad c: Should should we mention some names on the people who are n ? phd a: Right . So , the p my point is that you 're gonna end up with , word sequences that are differently annotated . And you want some tool , that is able to sort of merge these different annotations back into a single , version . OK ?  , and we had this problem very massively , at SRI when we worked , a while back on ,  well , on dialogue acts as well as , you know , what was it ?  , phd f: Well , all the Switchboard in it . phd a: Because we had one set of annotations that were based on , one version of the transcripts with a particular segmentation , and then we had another version that was based on , a different s slightly edited version of the transcripts with a different segmentation . So , we had these two different versions which were you know , you could tell they were from the same source but they weren't identical . So it was extremely hard to reliably merge these two back together to correlate the information from the different annotations . But once you have a file format , I can imagine writing not personally , but someone writing a tool that is essentially an alignment tool , that mediates between various versions , phd f:   phd a: and  , sort of like th  , you know , you have this thing in UNIX where you have , diff . phd a: There 's the , diff that actually tries to reconcile different two diffs f based on the same original . phd a: Something like that , but operating on these lattices that are really what 's behind this  , this annotation format . phd a: So grad c: There 's actually a diff library you can use to do things like that that so you have different formats . phd f: You could definitely do that with the phd a: So somewhere in the API you would like to have like a merge or some some function that merges two two versions . phd f: But the one thing that would work here actually for i that is more reliable than the utterances is the the speaker ons and offs . So if you have a good , grad c: But this is exactly what  , is that that the problem i phd f:  Yeah . The problem is saying " what are the semantics , phd f: And grad c: what do you mean by " merge " ? " phd f: Right , right . So so just to let you know what we where we kluged it by , doing  , by doing Hhh . phd a: Both were based on words , so , bo we have two versions of the same words intersp you know , sprinkled with with different tags for annotations . phd a: And that 's how grad c: That 's just wh how I would have done it . But , you know , it had lots of errors and things would end up in the wrong order , and so forth . phd a: it it was a kluge because it was basically reducing everything to  , to  , to textual alignment . grad c: A textual phd a: so phd f: But , d isn't that something where whoever if if the people who are making changes , say in the transcripts , cuz this all happened when the transcripts were different ye  , if they tie it to something , like if they tied it to the acoustic segment if they You know what  ? Then Or if they tied it to an acoustic segment and we had the time - marks , that would help . phd f: But the problem is exactly as Adam said , that you get , you know , y you don't have that information or it 's lost in the merge somehow , postdoc e: Well , can I ask one question ? phd f: so postdoc e: It it seems to me that , we will have o an official version of the corpus , which will be only one one version in terms of the words where the words are concerned . We 'd still have the the merging issue maybe if coding were done independently of the phd a: And you 're gonna get that postdoc e: But but phd a: because if the data gets out , people will do all kinds of things to it . And , s you know , several years from now you might want to look into , the prosody of referring expressions . And so that 's exactly what we should somehow when you distribute the data , say that you know , that have some way of knowing how to merge it back in and asking people to try to do that . postdoc e: Well , then the phd d: What 's what 's wrong with doing times ? I postdoc e: I agree . phd f: yeah , time is the grad c: Well , postdoc e: Time is unique . You were saying that you didn't think we should phd f: Time is passing ! phd a: Time time times are ephemeral . grad c: what if they haven't notated with them , times ? phd f: Yeah . postdoc e: But then couldn't you just indirectly figure out the time tied to the word ? phd f: But still they Exactly . phd d: But can they change the words without changing the time of the word ? grad c: Sure . The the point is , that that they may have annotated it off a word transcript that isn't the same as our word transcript , so how do you merge it back in ? I understand what you 're saying . grad c: And I I guess the answer is , it 's gonna be different every time . grad c: I it 's exactly what I said before , phd f: You only know the boundaries of the grad c: which is that " what do you mean by " merge " ? " So in this case where you have the words and you don't have the times , well , what do you mean by " merge " ? If you tell me what you mean , I can write a program to do it . phd f: And beyond that , all you know is is relative ordering and sometimes even that is wrong . grad c: So so in so in this one you would have to do a best match between the word sequences , phd f: So . grad c: extract the times f from the best match of theirs to yours , and use that . postdoc e: But it could be that they just  , it could be that they chunked they they lost certain utterances and all that stuff , grad c: Right , exactly . phd f: Well , I guess , w I I didn't want to keep people too long and Adam wanted t people I 'll read the digits . phd f: if not , I guess phd a: For th for the for the benefit of science we 'll read the digits