phd e: So it 's well , it 's spectral subtraction or Wiener filtering , depending on if we put if we square the transfer function or not . phd e: And then with over - estimation of the noise , depending on the ,  the SNR , with smoothing along time , smoothing along frequency . phd e: And , the best result is when we apply this procedure on FFT bins , with a Wiener filter . phd e: So it 's good because it 's difficult when we have to add noise to to to find the right level . It 's the same , idea but it 's working on mel bands , and it 's a spectral subtraction instead of Wiener filter , and there is also a noise addition after , cleaning up the mel bins . professor b: if you look at databases , the , one that has the smallest smaller overall number is actually better on the Finnish and Spanish , but it is , worse on the , Aurora phd e: It 's worse on professor b:  on the , TI - TI - digits , phd e: on the multi - condition in TI - digits . But , when you say u  , unified do you mean , it 's one piece of software now , or ? phd e: So now we are , yeah , setting up the software .  , and we phd a: So what 's what 's happened ? I think I 've missed something . So a week ago maybe you weren't around when when when Hynek and Guenther and I ? phd c: Hynek was here .  And then if I summarize somebody can tell me if I 'm wrong , which will also be possibly helpful . phd e: p - p - p professor b: We ,  we looked at ,  anyway we after coming back from QualComm we had , you know , very strong feedback and , I think it was Hynek and Guenter 's and my opinion also that , you know , we sort of spread out to look at a number of different ways of doing noise suppression . professor b: we had a long discussion about how they were the same and how they were d  , completely different . professor b: And , fundamentally they 're the same sort of thing but the math is a little different so that there 's a a there 's an exponent difference in the index you know , what 's the ideal filtering , and depending on how you construct the problem . professor b: And , I guess it 's sort you know , after after that meeting it sort of made more sense to me because  , if you 're dealing with power spectra then how are you gonna choose your error ? And typically you 'll do choose something like a variance . Whereas when you 're when you 're doing the the , looking at it the other way , you 're gonna be dealing with signals phd c:   professor b: and you 're gonna end up looking at power  , noise power that you 're trying to reduce . And so , eh so there should be a difference of you know , conceptually of of , a factor of two in the exponent . professor b: But there 're so many different little factors that you adjust in terms of of , over - subtraction and and and and and so forth , that arguably , you 're c and and and the choice of do you do you operate on the mel bands or do you operate on the FFT beforehand . There 're so many other choices to make that are are almost well , if not independent , certainly in addition to the choice of whether you , do spectral subtraction or Wiener filtering , that , @ @ again we sort of felt the gang should just sort of figure out which it is they wanna do and then let 's pick it , go forward with it . And and , we said , take a week , go arm wrestle , you know , grad d: Oh . professor b: And and so they so instead they went to Yosemite and bonded , and and they came out with a single single piece of software . phd a: So so you guys have combined or you 're going to be combining the software ? professor b:  . phd c: Well , the piece of software has , like , plenty of options , phd e: Oh boy . professor b: Well , that 's fine , but the thing is the important thing is that there is a piece of software that you that we all will be using now . phd e: Well , if we want to , like , optimize different parameters of phd c: Parameters . But , still so , there will be a piece of software with , will give this system , the fifty - three point sixty - six , by default and professor b:   phd a: I I I don't have a sense of phd e: It 's just one percent off of the best proposal . professor b: So it so , it 's it it 's not using our full bal bag of tricks , if you will . professor b: And , and it it is , very close in performance to the best thing that was there before .  , but , you know , looking at it another way , maybe more importantly , we didn't have any explicit noise , handling stationary dealing with e e we didn't explicitly have anything to deal with stationary noise . phd a: So will the neural net operate on the output from either the Wiener filtering or the spectral subtraction ? Or will it operate on the original ? professor b: Well , so so so argu arguably , what we should do  , I gather you have it sounds like you have a few more days of of nailing things down with the software and so on . But and then but , arguably what we should do is , even though the software can do many things , we should for now pick a set of things , th these things I would guess , and not change that . And I think , you know , that our goal should be by next week , when Hynek comes back , to  , really just to have a firm path , for the you know , for the time he 's gone , of of , what things will be attacked . But I would I would I would thought think that what we would wanna do is not futz with this stuff for a while because what 'll happen is we 'll change many other things in the system , phd a:   professor b: and then we 'll probably wanna come back to this and possibly make some other choices . phd a: But just conceptually , where does the neural net go ? Do do you wanna h run it on the output of the spectrally subtracted ? phd e: Mmm . professor b: Well , depending on its size Well , one question is , is it on the , server side or is it on the terminal side ?  , if it 's on the server side , it you probably don't have to worry too much about size . So the issue is is , for instance , could we have a neural net that only looked at the past ? phd a: Right . professor b: what we 've done in  in the past is to use the neural net , to transform , all of the features that we use . This is essentially ,  I guess it 's it 's more or less like a spee a speech enhancement technique here phd a:   professor b: right ? where we 're just kind of creating new if not new speech at least new new FFT 's that that have you know , which could be turned into speech  , that that have some of the noise removed . professor b: after that we still do a mess of other things to to produce a bunch of features . And then the the way that we had it in our proposal - two before , we had the neural net transformed features and we had the untransformed features , which I guess you you actually did linearly transform with the KLT , phd e: Yeah . professor b: but but but  , to orthogonalize them but but they were not , processed through a neural net . And Stephane 's idea with that , as I recall , was that you 'd have one part of the feature vector that was very discriminant and another part that wasn't , phd a:   professor b: which would smooth things a bit for those occasions when , the testing set was quite different than what you 'd trained your discriminant features for .  , y you know , that 's still being debated by the by people in Europe but , no matter how they end up there , it 's not going to be unlimited amounts , phd a: Yeah . And I think those that we last time we agreed that those are the three things that have to get , focused on . phd a: What was the issue with the VAD ? professor b: Well , better ones are good . phd a: And so the w the default , boundaries that they provide are they 're OK , but they 're not all that great ? professor b: I guess they still allow two hundred milliseconds on either side or some ? Is that what the deal is ? phd e:   phd e: And all the speech pauses , which is Sometimes on the SpeechDat - Car you have pauses that are more than one or two seconds . And , yeah , it seems to us that this way of just dropping the beginning and end is not We cou we can do better , I think , phd a:   phd e: because , with this way of dropping the frames they improve over the baseline by fourteen percent and Sunil already showed that with our current VAD we can improve by more than twenty percent . phd e: So , our current VAD is is more than twenty percent , while their is fourteen . And another thing that we did also is that we have all this training data for let 's say , for SpeechDat - Car . And if we just take only the , VAD probabilities computed on the clean signal and apply them on the far - field , test utterances , then results are much better . phd e: So it means that there are stim still phd a: How how much latency does the ,  does our VAD add ? phd e: If if we can have a good VAD , well , it would be great . phd a: Is it significant , phd e: right now it 's , a neural net with nine frames . phd a: or ? phd e: So it 's forty milliseconds plus , the rank ordering , which , should be phd c: Like another ten frames . professor b: So what 's the ? If you ignore  , the VAD is sort of in in parallel , isn't i isn't it , with with the ?  , it isn't additive with the the , LDA and the Wiener filtering , and so forth . phd c: So we  , if so if we if so which is like if we reduce the delay of VA So , the f the final delay 's now ba is f determined by the delay of the VAD , because the LDA doesn't have any delay . So if we re if we reduce the delay of the VAD , it 's like effectively reducing the delay . phd a: How how much , delay was there on the LDA ? phd c: So the LDA and the VAD both had a hundred millisecond delay . So and they were in parallel , so which means you pick either one of them phd a: Mmm . professor b: And there didn't seem to be any , penalty for that ? There didn't seem to be any penalty for making it causal ? phd c: Pardon ? Oh , no . phd c: Or something like that professor b: And he says Wiener filter is is forty milliseconds delay . phd a: What amount of latency are you thinking about when you say that ? professor b:  . professor b: You know , they 're saying ,  one group is saying a hundred and thirty milliseconds and another group is saying two hundred and fifty milliseconds . phd a: Were you thinking of the two - fifty or the one - thirty when you said we should have enough for the neural net ? professor b: Well , it just it when we find that out it might change exactly how we do it , is all . professor b: how much effort do we put into making it causal ?  , I think the neural net will probably do better if it looks at a little bit of the future . And , you know , how how much time should we put into into that ? So it 'd be helpful if we find out from the the standards folks whether , you know , they 're gonna restrict that or not . But I think , you know , at this point our major concern is making the performance better and and , if , something has to take a little longer in latency in order to do it that 's you know , a secondary issue . professor b: But if we get told otherwise then , you know , we may have to c clamp down a bit more . phd c: So , the one one one difference is that was there is like we tried computing the delta and then doing the frame - dropping . phd c: The earlier system was do the frame - dropping and then compute the delta on the professor b:  - huh . So , yeah , what we do is we compute the silence probability , convert it to that binary flag , professor b:  - huh . phd c: and then in the end you c up upsample it to match the final features number of phd e:   phd a: Did that help then ? phd c: It seems to be helping on the well - matched condition . And it actually r reduced a little bit on the high mismatch , so in the final weightage it 's b b better because the well - matched is still weighted more than professor b: So , @ @  , you were doing a lot of changes . Did you happen to notice how much , the change was due to just this frame - dropping problem ? What about this ? phd c: y you had something on it . Sometime we we change two two things together and But it 's around maybe it 's less than one percent . But like we 're saying , if there 's four or five things like that then pretty sho soon you 're talking real improvement . Well , we can do the frame - dropping on the server side or we can just be careful at the terminal side to send a couple of more frames before and after , and So . phd a: You have ,  So when you  , maybe I don't quite understand how this works , but , couldn't you just send all of the frames , but mark the ones that are supposed to be dropped ? Cuz you have a bunch more bandwidth .  , it it always seemed to us that it would be kind of nice to in addition to , reducing insertions , actually use up less bandwidth . phd a: If the net 's on the server side then it could use all of the frames . It 's , like , you mean you just transferred everything and then finally drop the frames after the neural net . Right now we are  , ri Right now what wha what we did is , like , we just mark we just have this additional bit which goes around the features , saying it 's currently a it 's a speech or a nonspeech . phd c: So there is no frame - dropping till the final features , like , including the deltas are computed . phd c: And after the deltas are computed , you just pick up the ones that are marked silence and then drop them . professor b: So it would be more or less the same thing with the neural net , I guess , actually . Yeah , that 's what that 's what that 's what , this is doing right now . So , what 's ,  ? That 's that 's a good set of work that that ,  phd c: Just one more thing . Like , should we do something f more for the noise estimation , because we still ? professor b: Yeah . I tried just plugging the , Guenter noise estimation on this system , and it  , it got worse . professor b: it does seem like , you know , i i i i some compromise between always depending on the first fifteen frames and a a always depending on a a pause is is is a good idea .  , maybe you have to weight the estimate from the first - teen fifteen frames more heavily than than was done in your first attempt . No , do you have any way of assessing how well or how poorly the noise estimation is currently doing ? phd e: Mmm . phd e: We don't have nothing that phd c: Is there was there any experiment with ? Well , I I did The only experiment where I tried was I used the channel zero VAD for the noise estimation and frame - dropping . So I don't have a I don't have a split , like which one helped more . phd c: So , that 's the professor b: So that 's something you could do with , this final system . professor b: If it 's , you know , essentially not better , then it 's probably not worth phd e: Yeah . It 's , like , ev even even if I use a channel zero VAD , I 'm just averaging the the s power spectrum . But the Guenter 's argument is , like , if it is a non - stationary segment , then he doesn't update the noise spectrum . So , th the Guenter was arguing that , even if you have a very good VAD , averaging it , like , over the whole thing is not a good idea . phd c: Because you 're averaging the stationary and the non - stationary , and finally you end up getting something which is not really the s because , you anyway , you can't remove the stationary part fr  , non - stationary part from the signal . So , that 's so that 's still a slight difference from what Guenter is trying  professor b: Well , yeah . And and also there 's just the fact that , eh , although we 're trying to do very well on this evaluation , we actually would like to have something that worked well in general . And , relying on having fifteen frames at the front or something is is pretty phd c: Yeah , yeah .  , it 'd certainly be more robust to different kinds of input if you had at least some updates . What what do you ,  what do you guys see as as being what you would be doing in the next week , given wha what 's happened ? phd c: Cure the VAD ? phd e: Yeah . phd e: So , should we keep the same ? I think we might try to keep the same idea of having a neural network , but training it on more data and adding better features , I think , but because the current network is just PLP features . phd e: There 's no RASTA , no phd a: So , I I don't remember what you said the answer to my , question earlier . Will you will you train the net on after you 've done the spectral subtraction or the Wiener filtering ? professor b: This is a different net . phd c: So we have a VAD which is like neur that 's a neural net . phd c: So , right now we have , like ,  we have the cleaned - up features , so we can have a better VAD by training the net on the cleaned - up speech . So it 's , like , where do we want to put the VAD ?  , it 's like phd a: Can you use the same net to do both , or ? phd c: For phd a: Can you use the same net that you that I was talking about to do the VAD ? phd c:   phd c: So the net the final net  , which is the feature net so that actually comes after a chain of , like , LDA plus everything . And and you can actually do it for final frame - dropping , but not for the VA - f noise estimation . professor b: You see , the idea is that the , initial decision to that that you 're in silence or speech happens pretty quickly . phd a: Cuz that 's used by some of these other ? professor b: And that Yeah . And that 's sort of fed forward , and and you say " well , flush everything , it 's not speech anymore " . professor b: it is used ,  Yeah , it 's only used f Well , it 's used for frame - dropping . professor b: because , you know , there 's if you have more than five hundred milliseconds of of of nonspeech then you figure it 's end of utterance or something like that .  , keeping the same method but but , seeing if you cou but ,  noise estimation could be improved . And then , later on in the month I think we wanna start including the neural net at the end . phd a: So , Hynek is coming back next week , you said ? professor b: Yeah , that 's the plan . professor b: I guess the week after he 'll be , going back to Europe , and so we wanna phd a: Is he in Europe right now or is he up at ? professor b: No , no .  , the idea was that , we 'd we 'd sort out where we were going next with this with this work before he , left on this next trip .  , Barry , you just got through your quals , so I don't know if you have much to say . No , just , looking into some some of the things that , John Ohala and Hynek , gave as feedback , as as a starting point for the project . In in my proposal , I I was thinking about starting from a set of , phonological features , or a subset of them . grad d: He said , these these phonological features are are sort of figments of imagination also . grad d: Ye professor b: But we don't have too much trouble recognizing synthetic speech since we create it in the first place . grad d: just looking at the data and seeing what 's similar and what 's not similar . grad d: So , I 'm I 'm , taking a look at some of , Sangita 's work on on TRAPS . She did something where ,  w where the TRAPS learn She clustered the the temporal patterns of , certain certain phonemes in in m averaged over many , many contexts . grad d: And , so , those are interesting things to phd a: So you 're now you 're sort of looking to try to gather a set of these types of features ? grad d: Right . grad d: you know ? A a a set of small features and continue to iterate and find , a better set