professor b: OK , so We we had a meeting with ,  with Hynek , in in which , Sunil and Stephane ,  summarized where they were and and , talked about where we were gonna go . phd e: D did did you guys get your code pushed together ? phd d: Oh , yeah . professor b: What was the update ? phd a: What was the update ? So there is th then the all the new features that go in . These are the phd e: Is the ,  the CVS mechanism working well ? phd a: Yeah . phd e: Are are people , up at OGI grabbing code  , via that ? phd d: I don't think I don't think phd e: Or ? phd a: I don't know if they use it , but . phd d: Yeah , I I don't think anybody up there is like working on it right now . professor b: I think it more likely that what it means is that when Sunil is up there he will grab it . professor b: But what 'll happen is is he 'll go back up there and , Pratibha will come back from from , the east coast . professor b: And ,  and and I guess actually , after Eurospeech for a little bit , he 'll go up there too . So , actually everybody who 's working on it will be up there for at least a little while . phd e: So has Has anybody tried remotely accessing the CVS using , SSH ? professor b: Yeah .  phd a: I don't know if Hari did that or You d phd d: I can actually do it today .  , I can just log into phd e: Have you tried it yet ? phd d: No , I didn't . phd a: Actually I I tried wh while when I installed the repository , I tried from Belgium . phd a: I logged in there and I tried to import phd e: Yeah ? It worked good ? phd a: Yeah , it works . phd e: Oh , good ! phd a: But it 's So , right now it 's the mechanism with SSH . phd e: Great ! phd a: I don't s I didn't set up You can also set up a CVS server on a new port . phd e: right ? phd a: But I didn't do that because I was not sure about security problems . I I would have to phd e: So w when you came in from Belgian Belgium , using SSH , was it asking you for your own password into ICSI ? So if yo you can only do that if you have an account at ICSI ? phd a: Right . phd e: Cuz there is an a way to set up anonymous CVS right ? phd a: Yeah , you ha in this way you ca you have to set up a CVS server but then , yeah , you can access it . phd e: So the anonymous mechanism phd a: You can access them and mostly if you if y the set the server is set up like this . Because a lot of the open source stuff works with anonymous CVS and I 'm just wondering  , for our transcripts we may want to do that . professor b: Yeah , for this stuff I don't think we 're quite up to that . So ,  phd e: What 's new ? professor b: Well , I think maybe the thing to me might be I me I 'm sure you 've just been working on on , details of that since the meeting , right ? And so phd a: Mmm , since the meeting , well , I I 've been I 've been train training a new VAD and a new feature net . professor b: But I guess maybe the thing since you weren't yo you guys weren't at that that meeting , might be just just to , sort of recap , the the conclusions of the meeting . Cuz that was sort of ,  we we 'd sort of been working up to that , that that , he would come here this week and and we would sort of phd e:  - huh . professor b: Since he 's going out of town like now , and I 'm going out town in a couple weeks , and time is marching , sort of , given all the mu many wonderful things we could be working on , what what will we actually focus on ? phd e:   professor b: And ,  and what do we freeze ? And , you know , what do we ? So ,  . So then there 's something central and there aren't at least a bunch of different versions going off in in ways that differ trivially . professor b: and then within that , I guess the idea was to freeze a certain set of options for now , to run it , a particular way , and decide on what things are gonna be experimented with , as opposed to just experimenting with everything . So we 've been working like six weeks on on the noise compensation and we end up with something that seems reasonable . phd e: Are you gonna use which of the two techniques ? phd a: So finally it 's it 's , Wiener filtering on FFT bins . And it uses , two steps , smoothing of the transfer function , the first step , that 's along time , which use recursion . And after this step there is a further smoothing along frequency , which use a sliding window of twenty FFT bins . And ,  phd e: So this is on the  , before any mel scaling has been done ? phd a: Yeah , yeah . phd e: This is phd a: It was professor b: This this smoothing is done on the estimate , of what you 're going to subtract ? Or on the thing that has already had something subtracted ? phd a: Yeah . So we are going to fix this for the moment and work on the other aspects of the whole system . phd a: So professor b: Actually , let me int eh , Dave isn't here to talk about it , but let me just interject . This module , in principle , i  , you would know whether it 's true in fact , is somewhat independent from the rest of it .  , well you don't I guess you don't re - synthesize speech , but you could phd a: We we do not fo professor b: but you could . phd a: Well well , we do , but we don't don't re - synthesize . In in the program we don't re - synthesize and then re - analyze once again . professor b: But you have a re - synthesized thing that you that 's an an option here . professor b: Yeah , I gu I guess my point is that , i in some of the work he 's doing in reverberation , one of the things that we 're finding is that , it 's it 's for the for an artificial situation , we can just deal with the reverberation and his techniques work really well . But for the real situation  , problem is , is that you don't just have reverberation , you have reverberation in noise . So in fact it might be a very nice thing to do , to just take the noise removal part of it and put that in front of what he 's looking at . And , generate new files or whatever , and and ,  and then do the reverberation part . He 's  , e phd e: I guess he 's busy with professor b: Yeah , prelims , right . professor b: but but , you know , that 'll  , it 's clear that we ,  we are not with the real case that we 're looking at , we can't just look at reverberation in isolation because the interaction between that and noise is is considerable . And that 's  , in the past we 've looked at , and this is hard enough , the interaction between channel effects and and ,  and additive noise , so convolutional effects and and additive effects . And we have , the , LDA stuff that in principle is doing something about convolutional effects . i i There 's all these interactions between these two and that 's part of why these guys had to work so hard on on juggling everything around . But now when you throw in the reverberation , it 's even worse , because not only do you have these effects , but you also have some long time effects . And , so Dave has something which , is doing some nice things under some conditions with with long time effects but when it 's when there 's noise there too , it 's it 's it 's pretty hard . So we have to start Since any almost any real situation is gonna have  , where you have the microphone distant , is going to have both things , we we actually have to think about both at the same time . professor b: So ,  So there 's this noise suppression thing , which is sort of worked out and then , maybe you should just continue telling what what else is in the the form we have . phd a: Yeah , well , the , the other parts of the system are the the blocks that were already present before and that we did not modify a lot . professor b: So that 's again , that that 's the Wiener filtering , followed by , that 's done at the FFT level . And then the mel and then the log , and then the phd a: Then the LDA filter , professor b: LDA filter . phd a: mmm , then the downsampling , professor b: And then  downsample , phd a: DCT , professor b: DCT , phd a: then , on - line normalization , professor b: on - line norm , phd a: followed by upsampling . phd a: And finally frame dropping , which  , would be a neural network also , used for estimated silence probabilities . And the input of this neural network would be somewhere between log mel bands or one of the earlier stages of the processing . So that 's sort of most of this stuff is yeah , is operating parallel with this other stuff . So the things that we , I guess we sort of  , There 's there 's some , neat ideas for V A So , in I think there 's sort of like There 's a bunch of tuning things to improve stuff . There 's questions about various places where there 's an exponent , if it 's the right exponent , or ways that we 're estimating noise , that we can improve estimating noise . But structurally it seemed like the things the main things that that we brought up that , are are gonna need to get worked on seriously are , a a significantly better VAD , putting the neural net on , which , you know , we haven't been doing anything with , the , neural net at the end there , and , the , opening up the second front . phd e: The other half of the channel ? professor b: Yeah , yeah , cuz we we have we have , half the the , data rate that they allow . phd e: That what you mean ? professor b: And , so the initial thing which came from , the meeting that we had down south was , that , we 'll initially just put in a mel spectrum as the second one . We probably will go to something better later , but the initial thing is that cepstra and spectra behave differently , phd e:   phd e: So if you took the system the way it is now , the way it 's fro you 're gonna freeze it , and it ran it on the last evaluation , where it would it be ? phd a:   It , phd e: In terms of ranking ? phd a: Ri - right now it 's second . professor b: Although you you know , you haven't tested it actually on the German and Danish , have you ? phd a: No , we didn't . phd e: So on the ones that you did test it on it would have been second ? professor b: Yeah . Would it  But When you 're saying second , you 're comparing to the numbers that the ,  that the best system before got on ,  also without German and Danish ? phd a: Yeah , yeah . professor b: Well ranking didn't before , but I 'm just asking where this is to where theirs was without the German and Danish , phd a: Yeah . phd e: Where where where were we actually on the last test ? professor b: Oh , we were also esp essentially second , although there were there were  , we had a couple systems and they had a couple systems . And so , I guess by that we were third , but  , there were two systems that were pretty close , that came from the same place . phd e: We 're so this second that you 're saying now is system - wide second ? professor b: See  , no I think it 's also institutional , isn't it ? phd e: Still institutionally second ? professor b: Right ?  , I think both of their systems probably phd a: we are between their two systems . professor b: And and , you know , in some sense we 're all doing fairly similar things .  , one could argue about the LDA and so forth but I I think , you know , in a lot of ways we 're doing very similar things . But what what phd e: So how did they fill up this all these these bits ?  , if we 're u professor b: why are we using half ? Well , so you could you c phd e: Yeah . Or how are they using more than half , I guess maybe is what I professor b: Yeah , so I I think  , you guys are closer to it than me , so correct me if I 'm wrong , but I I think that what 's going on is that in in both cases , some kind of normalization is done to deal with convola convolutional effects . And they seem to comple complement each other enough and be different enough that they both seem to help help us . phd e: So th So professor b: So that if you throw away high modulation frequencies , then you can downsample . professor b: So phd e: So what if you didn't So do you explicitly downsample then ? Do we explicitly downsample ? professor b: Yeah . phd e: And what if we didn't do that ? Would we get worse performance ? phd a:  Yeah , not better , not worse . professor b: I think it doesn't affect it , does it ? phd e: I see . So I think the thing is , since we 're not evidently throwing away useful information , let 's try to put in some useful information . professor b: And , so I you know , we we 've found in a lot of ways for quite a while that having a second stream  , helps a lot . So that 's that 's put in , and you know , it may even end up with mel spectrum even though I 'm saying I think we could do much better , just because it 's simple . And you know , in the long run having something everybody will look at and say , " oh , yeah , I understand " , is is very helpful . phd e: So you would you 're You 're thinking to put the , mel spectrum in before any of the noise removal stuff ? or after ? professor b: Well , that 's a question . It looks like it 'd be straightforward to to , remove the noise , and , phd e: Cuz that happens before the mel conversion , right ? professor b: Yeah . There 's even a question in my mind anyhow of whether th you should take the log or not . phd d: Well , it it it it so it actually makes it dependent on the overall energy of the  , the frame . professor b: If you do or don't normalize ? phd d: If yo if you don't normalize and if if you don't normalize . And then if if normalization helps , then y you have something to compare against , and say , " OK , this much effect "  , you don't want to change six things and then see what happens . And then saying , oh  particularly because we 've found in the past there 's all these these these different results you get with slight modifications of how you do normalization . So , I would think you would wanna have some baseline that says , " OK , we don't normalize , this is what we get " , when we do this normalization , when we do that normalization . phd e: So this second stream , will it add latency to the system professor b: No , it 's in parallel . phd e: or ? grad c: Para professor b: We 're not talking about computation time here . grad c: So with this , new stream would you train up a VAD on both both features , somehow ? phd d: No , I guess the VAD has its own set of features . that 's phd d: which could be this one of these streams , or it can be something derived from these streams . phd a: And there is also the idea of using TRAPS , maybe , for the VAD , which ,  phd d: Yeah , that 's also phd a: Well , Pratibha apparently showed , when , she was at IBM , that it 's a good idea . grad c: Would would that fit on the handset , or ? Oh ! phd a: I have no idea . phd d: Well , it has t  the th phd a: It would have to fit but Yeah . And so I guess the issue there is , are we are we using neural - net - based TRAPS , and and how big are they ? So that 'll that 'll be , you know , an issue . grad c: Cuz she also does the ,  the correlation - based , TRAPS , with without the neural net , just looking at the correlation between professor b: Right . professor b: Or a simple neural net , right ?  , the thing is , if you 're doing correlation , you 're just doing a simple  , dot product , you know , with some weights which you happened to learn from this learn from the data . professor b: And so , putting a nonlinearity on it is , you know , not that big a deal . professor b: So , the question is , how complex a function do you need ? Do you need to have an added layer or something ? In which case , potentially , you know , it could be big . phd e: So the meeting with Hynek that you guys just had was to decide exactly what you were gonna freeze in this system ? Is that ? Or was there ? Were you talking about what t new stuff , or ? professor b: What to freeze and then what to do after we froze . And like I was saying , I think the you know , the basic directions are , there 's lots of little things , such as improve the noise estimator but the bigger things are adding on the neural net and , the second stream . phd d: So , I 'll ,  I 'll actually after the meeting I 'll add the second stream to the VAD and maybe I 'll start with the feature net in that case . It 's like , you 're looking at the VAD , right ? phd a: yeah . phd d: OK , so just figure how to take the features from the final phd a: Yeah . But , yeah , I think there are plenty of issues to work on for the feature net @ @ . phd e: What about the , the new part of the evaluation , the , Wall Street Journal part ? professor b: Right . Well you you may be called upon to help , on account of , all the work in this stuff here has been , with small vocabulary . So what how is the , interaction supposed to happen ?  , I remember the last time we talked about this , it was sort of up in the air whether they were going to be taking , people 's features and then running them or they were gonna give the system out or phd d: Yeah . professor b: Do we already have it ? phd d: Yeah , th I I guess it 's almost ready . phd e: So we 'll have to grab this over CVS or something ? phd d: It - no , it 's just downloadable from their from their web site . professor b: Cuz one of the things that might be helpful , if you 've if you 've got time in all of this is , is if if these guys are really focusing on improving , all the digit stuff , maybe and you got the front - end from them , maybe you could do the runs for the phd e: OK . professor b: and and , you know , iron out hassles that that you have to , tweak Joe about or whatever , phd e: Sure . professor b: S phd d: So I 'll point you to the web site and the mails corresponding . So I phd e: And it but it 's not ready yet , the system ? phd d: I I think they are still , tuning something on that . So they 're like , d they 're varying different parameters like the insertion penalty and other stuff , and then seeing what 's the performance . phd e: Are those going to be parameters that are frozen , nobody can change ? Or ? phd d: w I guess there is , time during which people are gonna make suggestions . phd d: So these sugges these this , period during which people are gonna make suggestions is to know whether it is actually biased towards any set of features or professor b: Yeah , so I th th certainly the thing that I would want to know about is whether we get really hurt , on in insertion penalty , language model , scaling , sorts of things . professor b: in which case , H Hari or Hynek will need to , you know , push the case more about about this . phd e: And we may be able to revisit this idea about , you know , somehow modifying our features to work with professor b: Yes .  , some of that may be , a last minute rush thing because if the if our features are changing  . Yeah , the other thing is that even though it 's months away , it 's starting to seem to me now like November fifteenth is right around the corner . And , if they haven't decided things like this , like what the parameters are gonna be for this , when " deciding " is not just somebody deciding .  , in fact there should be some understanding behind the , deciding , which means some experiments and and so forth . phd e: So wha what 's the significance of November fifteenth ? professor b: That 's when the evaluation is . So , yeah , so after But , you know , they may even decide in the end to push it off . But , due to other reasons , like some people are going away , I 'm I 'm hoping it 's not pushed off for a l a long while . There 's there 's not anybody OGI currently who 's who 's , working with this and and phd e: Is is this part of the evaluation just a small part , or ho how important is this to the overall ? professor b: I I think it 's it 's ,  it depends how badly you do . phd d: b phd e: This is one of those things that will be debated afterwards ? professor b: Yeah . Well , it 's it 's Conceptually , it my impression , again , you guys correct me if I 'm wrong , but my impression is that , they want it as a double check . That you haven't come across you haven't invented features which are actually gonna do badly for a a significantly different task , particularly one with larger vocabulary . professor b: the truth is , most of the applications they 're looking at are pretty small vocabulary . phd e: Seems to me that if it 's a double check , they should give you a one or a zero . Y you passed the threshold or you didn't pass the threshold , and they shouldn't even care about what the score is .  , but in in the current thing , for instance , where you have this well - matched , moderately - matched , and and mis highly - mismatched , the emphasis is somewhat on the on the well - matched , but it 's only a a marginal , phd e: Yeah . professor b: right ? It 's like forty , thirty - five , twenty - five , or something like that . So you still if you were way , way off on the highly - mismatched , it would have a big effect . So again , if you 're if you get If it doesn't help you much , for noisy versions of this of large vocabulary data , then , you know , it may not hurt you that much . professor b: But if it if you don't if it doesn't help you much at all , or to put it another way , if it helps some people a lot more than it helps other people , if their strategies do , then phd e:   So is this ,  ?  , Guenter was putting a bunch of Wall Street Journal data on our disks . phd e: So that 's the data that we 'll be running on ? professor b: Yeah . professor b: well there 's training and test , right ? phd e: I I guess , I 'm not sure . professor b: No , if it 's like the other things , there 's there 's data for training the H M Ms and and data for testing it . phd e: I just professor b: So I wouldn't So it it 's phd e: OK . But I think it 's trained on clean and Is it trained on clean and and test on ? phd d: The Wall Street ? professor b: Yeah . It 's training on a range between ten and twenty DB , I think , and testing between five and fifteen . phd d: It 's ,  It 's like a medium medium - mismatch condition , sort of . phd a: and So the noise is There is a range of different noises also  which are selected randomly and added randomly , to the files . When did they estimate that they would have that system available for download ? phd d: I guess I guess one some preliminary version is already there . phd e: Oh , so there 's w something you can download to just learn ? phd d: Yeah , it 's already there . phd e: OK , phd d: But they 're actually parallel - y doing some modifications also , I think . phd d: So I guess the f final system will be frozen by middle of , like , one more week maybe . grad c: Is this their , SVM recognizer ? phd d: No , it 's just a straightforward  . professor b: You know , their their They have a lot of options in their recognizer and and the SVM is one of the things they 've done with it , but it 's not their more standard thing . phd d: Yeah , this is a g yeah , this i professor b: what ? phd d: yeah . phd e: So , just so that I understand , they 're providing scripts and everything so that basically , you you push a button and it does training , and then it does test , and everything ? Is that the idea ? phd d: I I I think yeah , I I guess something like that . phd d: is what Do they provide all the scripts , everything , and then Just , phd e: I see . Somehow yo there 's hooks to put your features in and phd d: ju Yeah , I th I think . In fact , if you look into it a little bit , it might be reasonable You know Joe , right ? Yeah . professor b: Just to sort of ask him about the issue of , different features having different kinds of , scaling characteristics and so on . So that , you know , w w possibly having entirely different optimal values for for the usual twiddle factors and what 's what 's the plan about that ? phd e: OK . phd d: So sh shall we , like , add Chuck also to the mailing lists ? It may be better , in that case if he 's going to professor b: Yeah . phd d: Yeah , I guess maybe Hari or Hynek , one of them , has to send a mail to Joe . phd d: Well , yeah , to add or maybe wh phd e: I I know him really well . professor b: Do you have Hari 's ,  ? phd e: I have Hari 's professor b: Yeah , so maybe just CC Hari and say that you 've just been asked to handle the large vocabulary part here , and , you know , phd e: OK . Why don't you just ask Joe but CC Hari , and then in the note say , " Hari , hopefully this is OK with you " . professor b: That way you can get started asking Joe quickly while he 's while he 's maybe still , you know , putting in nails and screws and  Yeah . phd d: And there is an , archive of all the mails that has been gon that has gone , between these people among these people . phd d: So , like like , it 's , like professor b: Have you thought about how long would be  , most useful for you to go up to OGI ? phd a: I don't know ,  . professor b: Oh , so you 're you 're imagining more that you would come back here first for a while and then and then go up there ? phd a: I professor b: it 's to you . professor b: I ju you guys are Well , y anyway , you don't have to decide this second but thi think about it about what what you would think would be the the best way to work it . And , I was looking at some of the work that , Sangita was doing on these TRAPS things . So she has ,  she has temporal patterns for , a certain set of phonemes , from from TIMIT , right ? the most common phonemes .  , so she has ,  a TRAP for each one of the phonemes , times fifteen , for each of the fifteen critical bands . And , she does this agglomerative hierarchical clustering which which basically , is a clustering algorithm that , starts with many , many , many different points many different clusters  , corresponding to the number of data , patterns that you have in the data . phd e: And then you can pick , values anywhere along that tree to fix your set of clusters . grad c: Right , usually it 's when ,  when the sol similarity measures , don't go down as much . And what she found was , sh  , was there were five broad ,  broad categories , corresponding to , things like , fricatives and , vocalic , and , stops .  , and , I was thinking about ways to to generalize this because w you 're it 's sort of like a it 's not a completely automatic way of clustering , because yo beforehand you have these these TRAPS and you 're saying that that these frames correspond to this particular phoneme .  , and that 's that 's constraining your your clustering to to the set of phonemes that you already have .  , whereas maybe we want to just take take a look at , arbitrary windows in time , of varying length , and cluster those . grad c: And I 'm thinking if we if we do that , then we would probably , at some point in the clustering algorithm find that we 've clustered things like , OK , thi this is a transition , this is a relatively stable stable point . grad c: and I 'm hoping to find other things of of similarity and maybe use these things as the intermediate ,  intermediate categories that , I 'll later classify . F  , I 'm professor b: Cuz that 's what you 're gonna be using , right ? grad c: Yeah , yeah . I I haven't exactly figured out , the exact details for that but , the the representation of the data that I was thinking of , was using , critical band , energies , over different lengths of time . professor b: Yeah , it seems somehow that needs th  , there 's a couple things that I wonder about with this . professor b: if you 're going for this sort of thing where you have  , little detectors that are looking at narrow bands , then what you 're going to be looking for should be some category that you can find with the narrow bands .  , and then the other thing , is that I wonder about with it , and and don't take this in the wrong way , like I I know what I 'm doing or anything , grad c: Right . professor b: the sort of standard answer about this sort of thing is that if you 're trying to find the right system in some sense , whether you 're trying by categories or or parameters  , and your goal is discrimination , then having choices based on discrimination as opposed to , unsupervised nearness of things , is actually better . professor b: and I don't know if that  , since you 're dealing with issues of robustness , you know , maybe maybe this isn't right , but it 'd be something I 'd be concerned about . Because , for instance , you can imagine , i i if you remember from from ,  from your your quals , John Ohala saying that , " buh " and " puh " differed , not really cuz of voicing but because of aspiration . professor b: So , if you looked if you were doing some coarse clustering , you probably would put those two sounds together . And yet , I would gue I would guess that many of your recognition errors were coming from , pfft , screwing up on this distinction . professor b: So , in fact , it 's a little hard because recognizers , to first order , sort of work . And the reasons we 're doing the things we 're doing is because they don't work as well as we 'd like . And since they sort of work , it means that they are already doing if you go and take any recognizer that 's already out there and you say , " how well is it distinguishing between schwas and stops ? " grad c:   professor b: Boy , I bet they 're all doing nearly perfectly on this , right ? grad c:   professor b: So these these big categories that differ in huge obvious ways , we already know how to do . So , what are we bringing to the party ?  , in fact what we wanna do is have something that , particularly in the presence of noise , is better at distinguishing between , categories that are actually close to one another , and hence , would probably be clustered together .  , I understand that there 's this other constraint that you 're considering , is that you wanna have categories that ,  that would be straightforward for , say , a human being to mark if you had manual annotation . But I think it 's also essential that you wanna look at what are the confusions that you 're making and how can you come up with , categories that , can clarify these confusions . professor b: So , the standard sort of way of doing that is take a look at the algorithms you 're looking at , but then throw in some discriminative aspect to it .  Y y this is more like , you know , how does LDA differ from PCA ?  , they 're the same sort of thing . professor b: But , you know and and , this is a little harder because you 're not just trying to find parameters .  , well W actually , you stopped thinking about it for a long time , but you used to think about it a lot . phd e: I guess I don't I don't  , it 's not clear to me how to reconcile , you know , what you 're saying , which I think is right , with the way I 've been looking at it . But it seems to me that the desire the desirable feature to have is something that , is bottom - up . phd e: And and so I guess what I don't understand is how to do that and still be discriminative , because to be discriminative you have to have categories and the only categories that we know of to use are sort of these human human sig significant categories that are significant to humans , like phonemes , things like that . professor b: Well , here 's a here 's a ,  Here 's a generic and possibly useless thought , which is , what do you really  , in a sense the only s s systems that make sense , are ones that that have something from top - down in th in them . Right ? Because if e even the smallest organism that 's trying to learn to do anything , if it doesn't have any kind of reward for doing or penal penalty for doing anything , then it 's just going to behave randomly . professor b: So whether you 're talking about something being learned through evolution or being learned through experience , it 's gotta have something come down to it that gives its reward or , you know , at least some reinforcement learning , phd e: Right . professor b: right ? phd e: So the question is , how far down ? professor b: And phd e: We could stop at words , but we don't , right ? We go all the way down to phonemes . professor b: Right , but I me I I think that maybe in some ways part of the difficulty is is trying to deal with the with these phonemes . You know , and and and i it 's almost like you want categories if if our if our , metric of of goodness , i if our phd e:   professor b: correction if our metric of badness is word error rate then , maybe we should be looking at words . professor b: for for for very nice , reasons we 've looked for a while at syllables , and they have a lot of good properties , but i i i if you go all the way to words , that 's really  , d w In many applications you wanna go further . You wanna go to concepts or something , or have have have concepts , actions , this sort of thing . And and phd e: Yeah , so the common right , the common wisdom is you can't do words because there 's too many of them , right ? So you have to have some smaller set that you can use , and and so everybody goes to phonemes . But the problem is that we we build models of words in terms of phonemes and these models are are really cartoon - ish , right ? So when you look at conversational speech , for example , you don't see the phonemes that you that you have in your word models . See , so her here 's maybe where If the issue is that we 're trying to come up with , some sort of intermediate categories which will then be useful for later stuff , then maybe it doesn't matter that we can't have enough phd e:   professor b: what you wanna do is is build up these categories that are that are best for word recognition . professor b: And and somehow if that 's built into the loop of what the categories  , we do this every day in this very gross way of of running o a thousand experiments phd e:  Right . In some ways it 's really not a bad bad thing to do because it tells you in fact how your adjustments at the very low level affect the the final goal . professor b: so maybe there 's a way to even put that in in a much more automatic way , phd e: Right . professor b: where you take , you know , something about the error at the level of the word or some other it could be syllable but in some large unit , phd e:  - huh . professor b: and  yeah , you may not have word models , you have phone models , whatever , but you sort of don't worry about that , and just somehow feed it back through . professor b: You know , so that 's , wh what I called a useless comments because I 'm not really telling you how to do it . But  , it 's a it 's it 's , you know it phd e: No , but I think the important part in there is that , you know , if you want to be discriminative , you have to have  , you know , categories . If you can put the words in to the loop somehow for determining goodness of your sets of clusters  professor b: Now , that being said , I think that that if you have something that is ,  i Once you start dealing with spontaneous speech , all the things you 're saying are are really true . professor b: If you have read speech that 's been manually annotated , like TIMIT , then , you know , i i you the phones are gonna be right , actually , for the most part . Yeah , professor b: So so , it doesn't really hurt them to to do that , to put in discrimination at that level . professor b: if you go to spontaneous speech then it 's it 's trickier and and and , the phones are  , you know , it 's gonna be based on bad pronunciation models that you have of phd e:  professor b: and ,  And it won't allow for the overlapping phenomenon phd e: Mmm . So it 's almost like there 's this mechanism that we have that , you know , when when we 're hearing read speech and all the phonemes are there you know , we we deal with that , but but when we go to conversational , and then all of a sudden not all the phonemes are there , it doesn't really matter that much to us as humans because we have some kind of mechanism that allows for these word models , whatever those models are , to be munged , you know , and and it doesn't really hurt , and I 'm not sure how how to build that in . professor b: Yeah , I guess the other thing i is is to think of a little bit  , we when y when you start looking at these kind of results I think it usually is is pretty intuitive , but start looking at  , what are the kinds of confusions that you do make , you know , between words if you want or or or , even phones in in in in read speech , say , when there is noise . You know , so is it more across place or more across manner ? Or is it cor you know , is it ? grad c:   professor b: I know one thing that happens is that you you you , you lose the , low energy phones . And if that if that is if it  , if that turns it into another word or or different you know , or another pair of words or something , then it 's more likely to happen . But , I don't know , I w I would I would guess that you 'd grad c:   Anyway , that 's phd e: I think part of the difficulty is that a l a lot of the robustness that we have is probably coming from a much higher level . phd e: You know , we understand the context of the situation when we 're having a conversation . And so if there 's noise in there , you know , our brain fills in and imagines what what should be there . professor b: but  , even if you do  , diagnostic rhyme test kind of things , you know , where there really isn't an any information like that , people are still better in noise than they than they are in in , than the machines are . If we 're not working on that then we should work on something else and improve it , but especially if it looks like the potential is there