, I can say about just q just quickly to get through it , that Dave and I submitted this ASRU .  , basically we 're dealing with rever reverberation , and , when we deal with pure reverberation , the technique he 's using works really , really well .  , and when they had the reverberation here , we 'll measure the signal - to - noise ratio and it 's , about nine DB . professor b: phd a: You mean , from the actual , recordings ? professor b: a fair amount of phd d: k phd a: It 's nine DB ? professor b: Yeah .  , I know that when you figured out the filters that we 're using for the Mel scale , there was some experimentation that went on at at ,  at OGI .  , but one of the differences that we found between the two systems that we were using , the the Aurora HTK system baseline system and the system that we were the the  , other system we were using , the  , the SRI system , was that the SRI system had maybe a , hundred hertz high - pass . professor b: For some reason , Dave thought it was twenty , phd d:  So the , center would be somewhere around like hundred professor b: but . professor b: But do you know , for instance , h how far down it would be at twenty hertz ? What the how much rejection would there be at twenty hertz , let 's say ? phd d: At twenty hertz . professor b: Yeah , any idea what the curve looks like ? phd d: Twenty hertz frequency Oh , it 's it 's zero at twenty hertz , right ? The filter ? phd c: Yea - actually , the left edge of the first filter is at sixty - four . professor b: It 's actually set to zero ? What kind of filter is that ? phd c: Yeah . professor b: Is this oh , from the from phd c: It This is the filter bank in the frequency domain that starts at sixty - four . professor b: Oh , so you ,  so you really set it to zero , the FFT ? phd d: Yeah , phd c: Yeah . So I wonder , is it @ @ Was there their experimentation with , say , throwing away that filter or something ? And ,  phd d: throwing away the first ? professor b: Yeah . professor b: Right , but the question is , whether sixty - four hertz is is , too , low . phd d: I t I think I 've tried a hundred and it was more or less the same , or slightly worse . professor b: On what test set ? phd d: On the same , SpeechDat - Car , Aurora . professor b: phd d: So it was professor b: and on and on the , TI - digits also ? phd d: No , no , no . That 'd be something to look at sometime because what , eh , he was looking at was performance in this room . professor b: Would that be more like Well , you 'd think that 'd be more like SpeechDat - Car , I guess , in terms of the noise .  , the signal - to - noise ratio , you know , looks a fair amount better if you if you high - pass filter it from this room . professor b: So , the main the the phd a: So that 's on th that 's on the f the far field ones though , right ? Yeah . phd a: So wha what is ,  what 's causing that ? professor b: Well , we got a a video projector in here , and ,  which we keep on during every every session we record , phd a: Yeah . professor b: which , you know , I I w we were aware of phd a:  - huh . professor b: But but ,  So , those are those are major components , I think , phd a: I see . professor b: but , it ,  I guess , I maybe I said this last week too but it it it really became apparent to us that we need to to take account of noise . And , so I think when when he gets done with his prelim study I think one of the next things we 'd want to do is to take this , noise , processing stuff and and , synthesize some speech from it . phd a: When are his prelims ? professor b: And then  , I think in about , a little less than two weeks .  , let 's see , this is the sixteenth , seventeenth ? Yeah , I don't know if he 's before It might even be in a week . I I guessed that they were gonna do it some time during the semester professor b: week and a half . phd a: but they 'll do it any time , huh ? professor b: They seem to be Well , the semester actually is starting up . phd a: Is it already ? professor b: Yeah , the semester 's late late August they start here .  , that that was sort of one  , the overall results seemed to be first place in in in the case of either , artificial reverberation or a modest sized training set . And But if you had a a really big training set , a recognizer , system that was capable of taking advantage of a really large training set I thought that One thing with the HTK is that is has the as we 're using the configuration we 're using is w s is being bound by the terms of Aurora , we have all those parameters just set as they are . So even if we had a hundred times as much data , we wouldn't go out to , you know , ten or t or a hundred times as many Gaussians or anything .  , so , that ,  that seemed to be So , if you have that that better recognizer that can that can build up more parameters , and if you , have the natural room , which in this case has a p a pretty bad signal - to - noise ratio , then in that case , the right thing to do is just do u use speaker adaptation . But I think that that would not be true if we did some explicit noise - processing as well as , the convolutional kind of things we were doing . phd d: That sample was released only yesterday or the day before , right ? phd a: No Well , I haven't grabbed that one yet . phd d: Oh , there is another short sample set phd a: There was another short one , yeah . phd a: And so I haven't grabbed the latest one that he just , put out yet . professor b: Is there any word yet about the issues about , adjustments for different feature sets or anything ? phd a: No , I I d You asked me to write to him and I think I forgot to ask him about that . phd d: Cuz they have it phd a: Maybe I 'll send it to the list . phd d: Cuz they have , already frozen those in i insertion penalties and all those stuff is what I feel . phd d: And , on that , they have run some experiments using various insertion penalties and all those phd a: And so they 've picked the values . professor b: For r w what test set ? phd d: p the one that they have reported is a NIST evaluation , Wall Street Journal . professor b: But that has nothing to do with what we 're testing on , right ? phd c:   So they 're , like  So they are actually trying to , fix that those values using the clean , training part of the Wall Street Journal . phd d: they want to train it and then this they 're going to run some evaluations . professor b: So they 're set they 're setting it based on that ? phd d: Yeah . So now , we may come back to the situation where we may be looking for a modification of the features to account for the fact that we can't modify these parameters . professor b:  but it 's still worth , I think , just since you know , just chatting with Joe about the issue . Do you think that 's something I should just send to him professor b:  phd a: or do you think I should send it to this there 's an a m a mailing list .  , we 're , you know , certainly willing to talk about it with everybody , but I think I think that , it 's probably best to start talking with him just to phd a: OK . professor b:  @ @ you know , it 's a dialogue between two of you about what you know , what does he think about this and what what you know what could be done about it . professor b: if you get ten people in involved in it there 'll be a lot of perspectives based on , you know , how phd a: Yeah . professor b: but if if if there is any , way to move in a way that would that would , you know , be more open to different kinds of features . But if if ,  if there isn't , and it 's just kind of shut down and and then also there 's probably not worthwhile bringing it into a larger forum where where political issues will come in . phd d: Because he there was some mail r saying that it 's may not be stable for Linux and all those . phd d: SUSI phd a: Yeah , SUSI or whatever it was phd d: yeah . professor b: but , I noticed , just glancing at the , Hopkins workshop , web site that ,  one of the thing I don't know Well , we 'll see how much they accomplish , but one of the things that they were trying to do in the graphical models thing was to put together a a , tool kit for doing ,  r  , arbitrary graphical models for , speech recognition . professor b: So And Jeff ,  the two Jeffs were phd a: Who 's the second Jeff ? professor b:  Oh , do you know Geoff Zweig ? phd a: No . professor b: so he did he did his PHD on dynamic Bayes - nets , for for speech recognition . He had some continuity built into the model , presumably to handle some , inertia in the in the production system , and ,  phd a:  .  , so it 's exactly the same approach , but the features that the VAD neural network use are , MFCC after noise compensation . professor b: What was it using before ? phd c: Before it was just P L phd d:  phd c: So . phd c: Yeah , phd d: Yeah , yeah , yeah , phd c: noisy noisy features . phd c:  This is what we get after This So , actually , we , yeah , here the features are noise compensated and there is also the LDA filter .  , and then it 's a pretty small neural network which use , nine frames of of six features from C - zero to C - fives , plus the first derivatives . phd a: Is that nine frames u s  , centered around the current frame ? Or phd c: Yeah . professor b: S so , I 'm I 'm sorry , there 's there 's there 's how many how many inputs ? phd c: So it 's twelve times nine . So I guess about eleven thousand parameters , which actually shouldn't be a problem , even in in small phones . phd a: So , I 'm I 'm s so what is different between this and and what you phd c: It should be OK . So the previous syst It 's based on the system that has a fifty - three point sixty - six percent improvement . The only thing that changed is the n a p eh a es the estimation of the silence probabilities . phd c:  So it 's it 's not bad , but the problem is still that the latency is too large . professor b: What 's the latency ? phd c: Because  the the latency of the VAD is two hundred and twenty milliseconds . And , the VAD is used  , i for on - line normalization , and it 's used before the delta computation . So if you add these components it goes t to a hundred and seventy , right ? professor b: I I 'm confused . You started off with two - twenty and you ended up with one - seventy ? phd c: With two an two hundred and seventy . phd c: If Yeah , if you add the c delta comp delta computation professor b: Oh . I the is this are these twenty - millisecond frames ? Is that why ? Is it after downsampling ? or phd c: The two - twenty is one hundred milliseconds for the  No , it 's forty milliseconds for t for the , cleaning of the speech . phd c: Yeah , and there is the delta at the input which is , professor b: One hundred milliseconds for smoothing . phd c:  phd d: It 's like forty plus forty plus professor b: And then forty phd c: Mmm . phd c: Yeah , there are twenty that comes from There is ten that comes from the LDA filters also . phd d: If you are using professor b:  phd c: Plus the frame , phd d: t If you are using three frames phd c: so it 's two - twenty . phd d: If you are phrasing f using three frames , it is thirty here for delta . So it 's like s five , six cepstrum plus delta at nine nine frames of professor b: And then ten milliseconds for phd d: Fi - There 's an LDA filter . professor b: ten milliseconds for LDA filter , and t and ten another ten milliseconds you said for the frame ? phd c: For the frame I guess . I computed two - twenty Yeah , well , it 's I guess it 's for the fr the professor b: OK . And then there 's delta besides that ? phd c: So this is the features that are used by our network and then afterwards , you have to compute the delta on the , main feature stream , professor b: OK . phd c: which is  , delta and double - deltas , which is fifty milliseconds . No , the after the noise part , the forty the the other hundred and eighty Well , Wait a minute . Some of this is ,  is ,  is in parallel , isn't it ?  , the LDA Oh , you have the LDA as part of the V D -  , VAD ? Or phd c: The VAD use , LDA filtered features also . professor b: so the delta at the end is how much ? phd c: It 's fifty . So phd c: But well , we could probably put the delta , before on - line normalization . It should not that make a big difference , phd a: What if you used a smaller window for the delta ? phd c: because phd a: Could that help a little bit ?  , I guess there 's a lot of things you could do to phd c: Yeah . phd c: Yeah , professor b: So phd c: but , nnn professor b: Yeah . So if you if you put the delta before the , ana on - line If Yeah phd c:   phd c: Cuz i professor b: And then y then you don't have that additive phd c: Yeah , phd d: Yep . phd c: cuz the time constant of the on - line normalization is pretty long compared to the delta window , professor b: OK . And you ought to be able to shove tw ,  sh  pull off twenty milliseconds from somewhere else to get it under two hundred , right ?  phd a: Is two hundred the d professor b: The hundred milla phd c:   It could be eighty and and probably do @ @ phd c: Yeah , phd a: i a hun phd c: yeah . phd a:  Wh - what 's the baseline you need to be under ? Two hundred ? professor b: Well , we don't know . professor b: if it 's two if if it 's ,  if it 's two - fifty , then we could keep the delta where it is if we shaved off twenty . If it 's two hundred , if we shaved off twenty , we could we could , meet it by moving the delta back . phd a: So , how do you know that what you have is too much if they 're still deciding ? professor b: we don't , but it 's just  , the main thing is that since that we got burned last time , and you know , by not worrying about it very much , we 're just staying conscious of it . professor b: And so , th  , if if if a week before we have to be done someone says , " Well , you have to have fifty milliseconds less than you have now " , it would be pretty frantic around here . professor b:  phd a: But still , that 's that 's a pretty big , win . And it doesn't seem like you 're in terms of your delay , you 're , that professor b: He added a bit on , I guess , because before we were we were had were able to have the noise , stuff , and the LVA be in parallel . phd c: Well , but I think the main thing , maybe , is the cleaning of the speech , which takes forty milliseconds or so . phd d: Yeah , the LDA LDA we don't know , is , like is it very crucial for the features , right ? phd c: No . professor b: so you could start pulling back , phd d: S s h professor b: but phd d: Yeah , professor b: But I think you have phd d: l professor b: you have twenty for delta computation which y now you 're sort of doing twice , right ? But yo w were you doing that before ? phd c: Mmm . Well , in the proposal , the input of the VAD network were just three frames , I think . professor b: So , what you have now is fort  , forty for the the noise , twenty for the delta , and ten for the LDA . That 's seventy milliseconds of stuff which was formerly in parallel , phd c:  professor b: right ? So I think , phd c:   professor b: you know , that 's that 's the difference as far as the timing , right ? phd c: Yeah . professor b: and you could experiment with cutting various pieces of these back a bit , but  , we 're s we 're not we 're not in terrible shape . professor b: It 's it 's not like it 's adding up to four hundred milliseconds or something . phd a: Where where is this where is this fifty - seven point O two in in comparison to the last evaluation ? professor b: Well , it 's I think it 's better than anything , anybody got . Yeah , and r and phd c: It would phd d: Yeah , so this is this is like the first proposal .  , I 'm sure other people working on this are not sitting still either , but phd a: Yeah . professor b: but but , the important thing is that we learn how to do this better , and , you know . So , our ,  Yeah , you can see the kind of kind of numbers that we 're having , say , on SpeechDat - Car which is a hard task , cuz it 's really ,  I think it 's just sort of sort of reasonable numbers , starting to be . Yeah , even for a well - matched case it 's sixty percent error rate reduction , professor b: Yeah . So actually , this is in between what we had with the previous VAD and what Sunil did with an IDL VAD . Which gave sixty - two percent improvement , right ? phd d: Yeah , it 's almost that . phd a: What was that ? Say that last part again ? phd c: So , if you use , like , an IDL VAD , for dropping the frames , phd d: o o Or the best we can get . phd c: the best that we can get i That means that we estimate the silence probability on the clean version of the utterances . phd a: So that would be even That wouldn't change this number down here to sixty - two ? phd c: Yeah . So you you were get phd c: If you add a g good v very good VAD , that works as well as a VAD working on clean speech , phd a: Yeah . phd c: then you wou you would go phd a: So that 's sort of the best you could hope for . professor b: And ,  and sixty - two with the the , you know , quote , unquote , cheating VAD . phd c: yeah , the next thing is , I started to play Well , I don't want to worry too much about the delay , no . And the other stream is the output of a neural network , using as input , also , these , cleaned MFCC .  phd a: Those are th those are th what is going into the tandem net ? phd c: I don't have the comp Mmm ? phd a: Those two ? phd c: So there is just this feature stream , the fifteen MFCC plus delta and double - delta . phd a: Yeah ? phd c: so it 's makes forty - five features that are used as input to the HTK . professor b: cuz then it has one part that 's discriminative , phd c: Yeah . Right now it seems that i I just tested on SpeechDat - Car while the experiment are running on your on TI - digits . Well , it improves on the well - matched and the mismatched conditions , but it get worse on the highly mismatched .  , professor b: y phd c: like , on the well - match and medium mismatch , the gain is around five percent relative , but it goes down a lot more , like fifteen percent on the HM case . professor b: You 're just using the full ninety features ? phd c:  The professor b: Y you have ninety features ? phd c: i I have ,  From the networks , it 's twenty - eight . It 's i i i It 's because it 's what we did for the first proposal . phd c: But we have to for sure , we have to go down , because the limit is now sixty features .  phd a: So , it seems funny that I don't know , maybe I don't u quite understand everything , but that adding features I guess I guess if you 're keeping the back - end fixed . But I guess if you 're keeping the number of Gaussians fixed in the recognizer , then professor b: Well , yeah . professor b: But , just in general , adding information Suppose the information you added , well , was a really terrible feature and all it brought in was noise . professor b: Right ? So so ,  Or or suppose it wasn't completely terrible , but it was completely equivalent to another one feature that you had , except it was noisier . professor b: Right ? In that case you wouldn't necessarily expect it to be better at all . I 'm just surprised that you 're getting fifteen percent relative worse on the wel professor b:  - huh . professor b: So , " highly mismatched condition " means that in fact your training is a bad estimate of your test . professor b: So having having , a g a l a greater number of features , if they aren't maybe the right features that you use , certainly can e can easily , make things worse . If you have if you have , lots and lots of data , and you have and your your your training is representative of your test , then getting more sources of information should just help . professor b: So I wonder , Well , what 's your what 's your thought about what to do next with it ? phd c: I don't know . I 'm surprised , because I expected the neural net to help more when there is more mismatch , as it was the case for the professor b:   phd c: Yeah , it 's the same training set , so it 's TIMIT with the TI - digits ' , noises , added . phd c:  professor b: Well , we might  , we might have to experiment with ,  better training sets . professor b: I The other thing is , before you found that was the best configuration , but you might have to retest those things now that we have different The rest of it is different , right ? So , For instance , what 's the effect of just putting the neural net on without the o other other path ? phd c:   professor b: You don't necessarily know what phd a: What if you did the Would it make sense to do the KLT on the full set of combined features ? Instead of just on the phd c: Yeah . The reason I did it this ways is that in February , it we we tested different things like that , so , having two KLT , having just a KLT for a network , or having a global KLT . phd c: And phd a: So you tried the global KLT before phd c: Well phd a: and it didn't really phd c: Yeah . phd c: The differences between these configurations were not huge , but it was marginally better with this configuration . professor b: And I guess if the These are all so all of these seventy - three features are going into , the ,  the  . professor b: And is are i i are are any deltas being computed of tha of them ? phd c: Of the straight features , yeah . phd c: So , yeah , maybe we can add some context from these features also as Dan did in in his last work . i Yeah , but the other thing I was thinking was , now I lost track of what I was thinking . phd a: What is the You said there was a limit of sixty features or something ? phd c:   phd a: What 's the relation between that limit and the , forty - eight  , forty eight hundred bits per second ? professor b: Oh , I know what I was gonna say . phd a: So I I I don't understand , phd c: The f the forty - eight hundred bits is for transmission of some features . phd a: because i  , if you 're only using h phd c: And generally , i it s allows you to transmit like , fifteen , cepstrum . professor b: The issue was that , this is supposed to be a standard that 's then gonna be fed to somebody 's recognizer somewhere which might be , you know , it it might be a concern how many parameters are use u used and so forth . What I was going to say is that , maybe maybe with the noise removal , these things are now more correlated . So you have two sets of things that are kind of uncorrelated , within themselves , but they 're pretty correlated with one another . professor b: And , they 're being fed into these , variants , only Gaussians and so forth , and and , phd c:   professor b: so maybe it would be a better idea now than it was before to , have , one KLT over everything , to de - correlate it . phd d: What are the S N Rs in the training set , TIMIT ? phd c: It 's , ranging from zero to clean ? Yeah . So we found this this ,  this Macrophone data , and so forth , that we were using for these other experiments , to be pretty good . professor b: So that 's i after you explore these other alternatives , that might be another way to start looking , is is just improving the training set . professor b: we were getting , lots better recognition using that , than Of course , you do have the problem that , u i we are not able to increase the number of Gaussians , or anything to , to match anything . So we 're only improving the training of our feature set , but that 's still probably something . phd a: So you 're saying , add the Macrophone data to the training of the neural net ? The tandem net ? professor b: Yeah , that 's the only place that we can train . professor b: We can't train the other stuff with anything other than the standard amount , phd a: Right .  ,  phd a: What what was it trained on again ? The one that you used ? phd c: It 's TIMIT with noise . phd c: So , yeah , it 's rather a small professor b: How big is the net , by the way ? phd c: it 's , five hundred hidden units . And professor b: And again , you did experiments back then where you made it bigger and it and that was that was sort of the threshold point . @ @ ? phd d: So is it is it though the performance , big relation in the high ma high mismatch has something to do with the , cleaning up that you that is done on the TIMIT after adding noise ? phd c:  phd d: So it 's i All the noises are from the TI - digits , phd c: Yeah . phd d: right ? So you i phd c:  They k  phd d: Well , it it 's like the high mismatch of the SpeechDat - Car after cleaning up , maybe having more noise than the the training set of TIMIT after clean s after you do the noise clean - up . phd d: So it had like all these different conditions of S N Rs , actually in their training set of neural net . phd d: But after cleaning up you have now a different set of S N Rs , right ? phd c: Yeah . phd d: And is it something to do with the mismatch that that 's created after the cleaning up , like the high mismatch phd c: You mean the the most noisy occurrences on SpeechDat - Car might be a lot more noisy than phd d:   professor b: but , you 're saying Yeah , the noisier ones are still going to be , even after our noise compensation , are still gonna be pretty noisy . phd d: Yeah , so now the after - noise compensation the neural net is seeing a different set of S N Rs than that was originally there in the training set . phd d: And that SNR may not be , like , com covering the whole set of S N Rs that you 're getting in the SpeechDat - Car . professor b: Right , but the SpeechDat - Car data that you 're seeing is also reduced in noise by the noise compensation . phd c: Well , if the initial range of SNR is different , we the problem was already there before . phd c: Because Mmm professor b: Yeah , it depends on whether you believe that the noise compensation is equally reducing the noise on the test set and the training set .  professor b: Right ?  , you 're saying there 's a mismatch in noise that wasn't there before , phd d:  . professor b: but if they were both the same before , then if they were both reduic reduced equally , then , there would not be a mismatch . professor b: So , this may be Heaven forbid , this noise compensation process may be imperfect , but . I I just that could be seen from the TI - digits , testing condition because , the noises are from the TI - digits , right ? Noise phd c: Yeah . So phd d: So cleaning up the TI - digits and if the performance goes down in the TI - digits mismatch high mismatch like this phd c: Clean training , yeah . professor b: the Macrophone data , I think , you know , it was recorded over many different telephones . I don't think there 's anybody recording over a car from a car , but I think it 's it 's varied enough that if if doing this adjustments , and playing around with it doesn't , make it better , the most  , it seems like the most obvious thing to do is to improve the training set .   , what we were  the condition It it gave us an enormous amount of improvement in what we were doing with Meeting Recorder digits , even though there , again , these m Macrophone digits were very , very different from , what we were going on here . But it was just I think just having a a nice variation in acoustic conditions was just a good thing . phd c: Yeah , actually to s eh , what I observed in the HM case is that the number of deletion dramatically increases . phd c: When I added the num the neural network it doubles the number of deletions . Yeah , so I don't you know how to interpret that , but , mmm professor b: Yeah . phd c: t phd a: And and did an other numbers stay the same ? Insertion substitutions stay the same ? phd c: They p stayed the same , phd a: Roughly ? phd c: they maybe they are a little bit  , lower . But professor b: Did they increase the number of deletions even for the cases that got better ? phd c:   professor b: And it Remind me again , the " highly mismatched " means that the phd c: Clean training and professor b: sorry ? phd c: It 's clean training Well , close microphone training and distant microphone , high speed , I think . professor b: Close mike training phd c: Well The most noisy cases are the distant microphone for testing . phd c: The feature are the same except that professor b: that 's right , that 's right .  phd a: Well that that says that , you know , the ,  the models in in , the recognizer are really paying attention to the neural net features . professor b: But , yeah , actually the TIMIT noises are sort of a range of noises and they 're not so much the stationary driving kind of noises , right ? It 's it 's pretty different . " phd c: " Subway " , right ? and phd d: " Street " or " Airport " or something . phd c: So it 's mostly Well , " Car " is stationary , professor b:   phd c: " Babble " , it 's a stationary background plus some voices , professor b:   professor b: Well , I I think that if you run it Actually , you maybe you remember this . When you in in the old experiments when you ran with the neural net only , and didn't have this side path , with the the pure features as well , did it make things better to have the neural net ? phd c:   professor b: Was it about the same ?  , w i phd c: It was b a little bit worse . professor b: So , until you put the second path in with the pure features , the neural net wasn't helping at all . as soon as we added LDA on - line normalization , and all these things , then professor b: They were doing similar enough things . Well , I still think it would be k sort of interesting to see what would happen if you just had the neural net without the side thing . phd c: Yeah , professor b: And and the thing I I have in mind is , maybe you 'll see that the results are not just a little bit worse . You know ? And ,  But if on the ha other hand , it 's , say , somewhere in between what you 're seeing now and and and , what you 'd have with just the pure features , then maybe there is some problem of a of a , combination of these things , or correlation between them somehow . professor b: If it really is that the net is hurting you at the moment , then I think the issue is to focus on on , improving the the net . professor b: So what 's the overall effe  , you haven't done all the experiments but you said it was i somewhat better , say , five percent better , for the first two conditions , and fifteen percent worse for the other one ? But it 's but of course that one 's weighted lower , phd c: Y yeah , oh . That 's not that bad , but it was l like two percent relative worse on SpeechDat - Car . phd d: Well , it will overall it will be still better even if it is fifteen percent worse , because the fifteen percent worse is given like f w twenty - five point two five eight . So the so the worst it could be , if the others were exactly the same , is four , phd d: Is it like professor b: and and , in fact since the others are somewhat better phd d: Yeah , so it 's four . Is i So either it 'll get cancelled out , or you 'll get , like , almost the same . phd a: In the ,  a lot of the ,  the Hub - five systems , recently have been using LDA . So there 's the the LDA is is right there before the H M phd d: Yeah . phd a: So , you guys are using LDA but it seems like it 's pretty far back in the process . The LDA that you saying is , like , you take a block of features , like nine frames or something , and then do an LDA on it , phd a: Yeah . phd a: it 's you know , you 're just basically i phd d: Yeah , so this is like a two d two dimensional tile . And the LDA that we are f applying is only in time , not in frequency high cost frequency . So it 's like more like a filtering in time , rather than doing a r phd a: Ah . So what i what about ,  i u what i w  , I don't know if this is a good idea or not , but what if you put ran the other kind of LDA , on your features right before they go into the  ? phd d: it phd c:   phd a: But  , w but the other features that you have , th the non - tandem ones , phd c:  . Well , in the proposal , they were transformed u using PCA , but phd a:  - huh . professor b: The a the argument i is kind of i in and it 's not like we really know , but the argument anyway is that , we always have the prob  , discriminative things are good . professor b: they 're good because you you you learn to distinguish between these categories that you want to be good at distinguishing between . It PAC - PCA low - order PCA throws away pieces that are  , maybe not not gonna be helpful just because they 're small , basically . professor b: But , the problem is , training sets aren't perfect and testing sets are different . So you f you you face the potential problem with discriminative stuff , be it LDA or neural nets , that you are training to discriminate between categories in one space but what you 're really gonna be g getting is is something else . professor b: And so , Stephane 's idea was , let 's feed , both this discriminatively trained thing and something that 's not . So you have a good set of features that everybody 's worked really hard to make , phd a: Yeah . professor b: and then , you you discriminately train it , but you also take the path that that doesn't have that , phd a:  - huh . And that that seem So it 's kind of like a combination of the  , what , Dan has been calling , you know , a feature  , you know , a feature combination versus posterior combination or something . It 's it 's , you know , you have the posterior combination but then you get the features from that and use them as a feature combination with these these other things . And that seemed , at least in the last one , as he was just saying , he he when he only did discriminative stuff , i it actually was was it didn't help at all in this particular case . But by having them both there The fact is some of the time , the discriminative stuff is gonna help you . professor b: And some of the time it 's going to hurt you , phd a: Right . professor b: and by combining two information sources if , you know if if phd a: So you wouldn't necessarily then want to do LDA on the non - tandem features because now you 're doing something to them that professor b: That i i I think that 's counter to that idea . But if that 's the hypothesis , at least it would be counter to that hypothesis to do that . professor b: and in principle you would think that the neural net would do better at the discriminant part than LDA .  , we ,  we were getting ready to do the tandem , stuff for the Hub - five system , and , Andreas and I talked about it , and the idea w the thought was , " Well , yeah , that i you know th the neural net should be better , but we should at least have  , a number , you know , to show that we did try the LDA in place of the neural net , so that we can you know , show a clear path . phd a: You know , that you have it without it , then you have the LDA , then you have the neural net , and you can see , theoretically . professor b: or tha that 's a phd a: That 's what that 's what we 're gonna do next as soon as I finish this other thing . phd a: it everybody believes it , professor b: Oh , no it 's a g phd a: but you know , we just professor b: No , no , but it might not not even be true . professor b: it 's it 's it 's it 's it 's a great idea .  , one of the things that always disturbed me , in the the resurgence of neural nets that happened in the eighties was that , a lot of people Because neural nets were pretty easy to to use a lot of people were just using them for all sorts of things without , looking at all into the linear , versions of them . professor b: And , people were doing recurrent nets but not looking at IIR filters , and You know , so I think , yeah , it 's definitely a good idea to try it . phd a: Yeah , and everybody 's putting that on their systems now , and so , I that 's what made me wonder about this , professor b: Well , they 've been putting them in their systems off and on for ten years , phd a: but . professor b: but but but , phd a: Yeah , what  is it 's it 's like in the Hub - five evaluations , you know , and you read the system descriptions and everybody 's got , you know , LDA on their features . phd c: It 's the transformation they 're estimating on Well , they are trained on the same data as the final  are . Cuz they don't have these , you know , mismatches that that you guys have . phd a: So that 's why I was wondering if maybe it 's not even a good idea . professor b: part of why I I think part of why you were getting into the KLT Y you were describing to me at one point that you wanted to see if , you know , getting good orthogonal features was and combining the the different temporal ranges was the key thing that was happening or whether it was this discriminant thing , right ? So you were just trying I think you r  , this is it doesn't have the LDA aspect but th as far as the orthogonalizing transformation , you were trying that at one point , right ? phd c:   phd d: So , yeah , I 've been exploring a parallel VAD without neural network with , like , less latency using SNR and energy , after the cleaning up . So what I 'd been trying was ,  After the b after the noise compensation , n I was trying t to f find a f feature based on the ratio of the energies , that is , cl after clean and before clean . So that if if they are , like , pretty c close to one , which means it 's speech . And if it is n if it is close to zero , which is So it 's like a scale @ @ probability value . So I was trying , with full band and multiple bands , m ps  separating them to different frequency bands and deriving separate decisions on each bands , and trying to combine them .  , the advantage being like it doesn't have the latency of the neural net if it if it can professor b:   phd d: just like Which means that it 's it 's doing a slightly better job than the previous VAD , professor b:   phd d: so ,  professor b: But i d I 'm sorry , phd d: so u professor b: does it still have the median filter stuff ? phd d: It still has the median filter . professor b: So it still has most of the delay , phd d: So professor b: it just doesn't phd d: Yeah , so d with the delay , that 's gone is the input , which is the sixty millisecond . professor b: Well , w i phd d: At the input of the neural net you have this , f nine frames of context plus the delta . phd d: so the delay is only the forty millisecond of the noise cleaning , plus the hundred millisecond smoothing at the output . So the the di the biggest The problem f for me was to find a consistent threshold that works well across the different databases , because I t I try to make it work on tr SpeechDat - Car professor b:   phd d: and it fails on TI - digits , or if I try to make it work on that it 's just the Italian or something , it doesn't work on the Finnish . So there are there was , like , some problem in balancing the deletions and insertions when I try different thresholds . phd d: So The I 'm still trying to make it better by using some other features from the after the p clean up maybe , some , correlation auto - correlation or some s additional features of to mainly the improvement of the VAD . professor b: Now this this this , " before and after clean " , it sounds like you think that 's a good feature . That that , it you th think that the ,  the i it appears to be a good feature , right ? phd d:   phd c: Yeah , eventually we could could just phd d: Yeah , so Yeah , so that 's the Yeah . phd d: Because they did that itself phd c: Then you don't have to worry about the thresholds and phd d: There 's a threshold and Yeah . So if we if we can live with the latency or cut the latencies elsewhere , then then that would be a , good thing . professor b: anybody has anybody you guys or or Naren , somebody , tried the , second th second stream thing ?  . phd d: Oh , I just I just h put the second stream in place and ,  ran one experiment , but just like just to know that everything is fine . phd d: So it was like , forty - five cepstrum plus twenty - three mel log mel . phd d: And and , just , like , it gave me the baseline performance of the Aurora , which is like zero improvement . phd d: So I just tried it on Italian just to know that everything is But I I didn't export anything out of it because it was , like , a weird feature set . Well , what I think , you know , would be more what you 'd want to do is is is , put it into another neural net . phd d: The  , other thing I was wondering was , if the neural net , has any because of the different noise con unseen noise conditions for the neural net , where , like , you train it on those four noise conditions , while you are feeding it with , like , a additional some four plus some f few more conditions which it hasn't seen , actually , phd c:   phd d:  instead of just h having c  , those cleaned up t cepstrum , sh should we feed some additional information , like The the We have the VAD flag .  , should we f feed the VAD flag , also , at the input so that it it has some additional discriminating information at the input ? phd c:  -  !  professor b: Wh -  , the the VAD what ? phd d: We have the VAD information also available at the back - end . phd d: So if it is something the neural net is not able to discriminate the classes professor b: Yeah . phd d:  Because most of it is sil  , we have dropped some silence f We have dropped so silence frames ? professor b:   So , by having an additional , feature which says " this is speech and this is nonspeech " , it certainly helps in some unseen noise conditions for the neural net . phd a: What Do y do you have that feature available for the test data ? phd d: Well , we have we are transferring the VAD to the back - end feature to the back - end . phd d: Which is which is certainly giving a phd a: So you 're saying , feed that , also , into the neural net . The other thing you could do is just , p modify the , output probabilities of the of the , neural net , tandem neural net , based on the fact that you have a silence probability . professor b: So you have an independent estimator of what the silence probability is , and you could multiply the two things , and renormalize . professor b: But but ,  phd c: Yeah , so maybe , yeah , when phd a: But in principle wouldn't it be better to feed it in ? And let the net do that ? professor b: Well , u Not sure . " Or you can say , " It 's silence ! Go away ! "  , i Doesn't ? I think I think the second one sounds a lot more direct . So , what if you then ,  since you know this , what if you only use the neural net on the speech portions ? professor b: Well , phd c: That 's what phd a: Well , I guess that 's the same . professor b: Yeah , y you 'd have to actually run it continuously , phd a: But   , train the net only on professor b: but it 's @ @ Well , no , you want to train on on the nonspeech also , because that 's part of what you 're learning in it , to to to generate , that it 's it has to distinguish between . phd a: But  , if you 're gonna if you 're going to multiply the output of the net by this other decision , would then you don't care about whether the net makes that distinction , right ? professor b: Well , yeah . Now the only thing that that bothers me about all this is that I I I The the fact i i It 's sort of bothersome that you 're getting more deletions . But So I might maybe look at , is it due to the fact that  , the probability of the silence at the output of the network , is , professor b: Is too high . So maybe So phd c: If it 's the case , then multiplying it again by i by something ? phd d: It may not be it professor b: Yeah . phd d: Yeah , it it may be too it 's too high in a sense , like , everything is more like a , flat probability . phd d: So , like , it 's not really doing any distinction between speech and nonspeech phd c: yeah . phd a: Be interesting to look at the Yeah , for the I wonder if you could do this . But if you look at the , highly mism high mismat the output of the net on the high mismatch case and just look at , you know , the distribution versus the the other ones , do you do you see more peaks or something ? phd c: Yeah . phd c: or professor b: But I bu phd c: It it seems that the VAD network doesn't Well , it doesn't drop , too many frames because the dele the number of deletion is reasonable . But it 's just when we add the tandem , the final MLP , and then professor b: Yeah . Now the only problem is you don't want to ta I guess wait for the output of the VAD before you can put something into the other system , phd c: u professor b: cuz that 'll shoot up the latency a lot , right ? Am I missing something here ? phd c: But phd d:   But but I I guess phd a: But if you were gonna put it in as a feature it means you already have it by the time you get to the tandem net , right ? phd d: well . professor b: It 's kind of done in  , some of the things are , not in parallel , but certainly , it would be in parallel with the with a tandem net . So maybe , if that doesn't work ,  But it would be interesting to see if that was the problem , anyway . And and and then I guess another alternative would be to take the feature that you 're feeding into the VAD , and feeding it into the other one as well . professor b:  But that 's Yeah , that 's an interesting thing to try to see , if what 's going on is that in the highly mismatched condition , it 's , causing deletions by having this silence probability up up too high , phd c:   phd c: So , m professor b: Cuz Well , the V A if the VAD said since the VAD is is is right a lot ,  phd c: Yeah . Yeah , and the other thing Well , there are other issues maybe for the tandem , like , well , do we want to , w  n Do we want to work on the targets ? Or , like , instead of using phonemes , using more context dependent units ? phd a: For the tandem net you mean ? phd c: Well , I 'm Yeah . phd c: I 'm thinking , also , a w about Dan 's work where he he trained a network , not on phoneme targets but on the  state targets . professor b: Problem is , if you are going to run this on different m test sets , including large vocabulary , phd c: Yeah . I was just thinking maybe about , like , generalized diphones , and come up with a a reasonable , not too large , set of context dependent units , and and Yeah . But I d I d it it i it 's all worth looking at , phd c:   professor b: but it sounds to me like , looking at the relationship between this and the speech noise stuff is is is probably a key thing . phd a: So if ,  if the , high mismatch case had been more like the , the other two cases in terms of giving you just a better performance , how would this number have changed ? phd c:   If if i phd a: y Like sixty ? professor b: Well , we don't know what 's it 's gonna be the TI - digits yet . If you extrapolate the SpeechDat - Car well - matched and medium - mismatch , it 's around , yeah , maybe five . phd c: Well , it 's around five percent , because it 's s Right ? If everything is five percent . phd c: I d I d I just have the SpeechDat - Car right now , so phd a: Yeah . phd c: It 's running it shou we should have the results today during the afternoon , phd a:  . Well  So I won't be here for phd a: When When do you leave ? professor b: I 'm leaving next Wednesday .  , phd a: But you 're professor b: so I phd a: are you you 're not gonna be around this afternoon ? professor b: Yeah . professor b: This afternoon  Oh , right , for the Meeting meeting ? Yeah , that 's just cuz of something on campus . But , yeah , so next week I won't , and the week after I won't , cuz I 'll be in Finland .  , phd a: What 's September sixth ? professor b: and  , that 's during Eurospeech .  Right ? So it 'll be a few weeks , really , before we have a meeting of the same cast of characters . And and then  , we 'll start up again with Dave and Dave and Barry and Stephane and us on the , twentieth . Thirteenth ? About a month ? phd a: So , you 're gonna be gone for the next three weeks or something ? professor b: I 'm gone for two and a half weeks starting starting next Wed - late next Wednesday . Is that right ? professor b: I won't it 's probably four because of is it three ? Let 's see , twenty - third , thirtieth , sixth . And the the third one won't probably won't be a meeting , cuz cuz , Su - Sunil , Stephane , and I will all not be here . So it 's just , the next two where there will be there , you know , may as well be meetings , phd a: OK . And then starting up on the thirteenth , we 'll have meetings again but we 'll have to do without Sunil here somehow . phd a: When is the evaluation ? November , or something ? professor b: Yeah , it was supposed to be November fifteenth