professor c: And  Hans -  , Hans - Guenter will be here , I think by next next Tuesday or so . professor c: So he 's he 's going to be here for about three weeks , phd b: Oh ! That 's nice . professor c: So he 's gonna look in on everything we 're doing phd d:   grad e: Th - that 's his spectral subtraction group ? professor c: Yeah , grad e: Is that right ? professor c: yeah . He 's , very , very , easygoing , easy to talk to , and , very interested in everything . professor c: he 's he 's he 's he 's phd a: Wh - Back when I was a grad student he was here for a ,  a year or n six months .  , I went around and talked to everybody , and it seemed like they they had some new results but rather than them coming up and telling me I figured we should just wait a week and they can tell both you know , all of us . So , why don't we why don't we start with you , Dave , and then , we can go on . grad e: So , since we 're looking at putting this ,  mean log m magnitude spectral subtraction , into the SmartKom system , I I did a test seeing if , it would work using past only and plus the present to calculate the mean . So , I did a test , where I used twelve seconds from the past and the present frame to , calculate the mean . And phd a: Twelve seconds Twelve twelve seconds back from the current frame , is that what you mean ? grad e:  Twelve seconds , counting back from the end of the current frame , phd a: OK , OK . So it was , twen I think it was twenty - one frames and that worked out to about twelve seconds . grad e: And compared to , do using a twelve second centered window , I think there was a drop in performance but it was just a slight drop . grad e: Is is that right ? professor c: yeah , it was pretty it was pretty tiny . And , that that  , that 's encouraging for for the idea of using it in an interactive system like And , another issue I 'm I 'm thinking about is in the SmartKom system . So say twe twelve seconds in the earlier test seemed like a good length of time , but what happens if you have less than twelve seconds ? And ,  So I w bef before ,  Back in May , I did some experiments using , say , two seconds , or four seconds , or six seconds . In those I trained the models using mean subtraction with the means calculated over two seconds , or four seconds , or six seconds . And , here , I was curious , what if I trained the models using twelve seconds but I f I gave it a situation where the test set I was subtracted using two seconds , or four seconds , or six seconds . And , I th I think it was , four se I think I think it was , something like four seconds and , six seconds , and eight seconds . And it seems like it it it hurts compared to if you actually train the models using th that same length of time but it it doesn't hurt that much .  , u usually less than point five percent , although I think I did see one where it was a point eight percent or so rise in word error rate . But this is , w where , even if I train on the , model , and mean subtracted it with the same length of time as in the test , it the word error rate is around , ten percent or nine percent . professor c: But it but looking at it the other way , isn't it what you 're saying that it didn't help you to have the longer time for training , if you were going to have a short time for grad e: That that 's true .  , professor c: why would you do it , if you knew that you were going to have short windows in testing . grad e: Wa phd a: Yeah , it seems like for your  , in normal situations you would never get twelve seconds of speech , right ? I 'm not e u phd b: You need twelve seconds in the past to estimate , right ? Or l or you 're looking at six sec seconds in future and six in professor c: Yeah . phd a: Is this twelve seconds of  , regardless of speech or silence ? Or twelve seconds of speech ? grad e: Of of speech . professor c: The other thing , which maybe relates a little bit to something else we 've talked about in terms of windowing and so on is , that , I wonder if you trained with twelve seconds , and then when you were two seconds in you used two seconds , and when you were four seconds in , you used four seconds , and when you were six and you basically build up to the twelve seconds . grad e: But s so I g So I guess the que the question I was trying to get at with those experiments is , " does it matter what models you use ? Does it matter how much time y you use to calculate the mean when you were , tra doing the training data ? " professor c: Right . But  the other thing is that that 's  , the other way of looking at this , going back to , mean cepstral subtraction versus RASTA kind of things , is that you could look at mean cepstral subtraction , especially the way you 're doing it , as being a kind of filter . You know , basically you 're you 're you 're doing a high - pass filter or a band - pass filter of some sort and and just design a filter . And then , you know , a filter will have a certain behavior and you loo can look at the start up behavior when you start up with nothing . professor c: And and , you know , it will , if you have an IIR filter for instance , it will , not behave in the steady - state way that you would like it to behave until you get a long enough period , but , by just constraining yourself to have your filter be only a subtraction of the mean , you 're kind of , you know , tying your hands behind your back because there 's filters have all sorts of be temporal and spectral behaviors . professor c: And the only thing , you know , consistent that we know about is that you want to get rid of the very low frequency component . phd b: But do you really want to calculate the mean ? And you neglect all the silence regions or you just use everything that 's twelve seconds , and grad e: you do you mean in my tests so far ? phd b: Ye - yeah . grad e: th  , yeah , I guess it 's something I need to play with more to decide how to set that up for the SmartKom system . Like , may maybe if I trained on six seconds it would work better when I only had two seconds or four seconds , and professor c: Yeah . professor c: Yeah , and again , if you take this filtering perspective and if you essentially have it build up over time .  , if you computed means over two and then over four , and over six , essentially what you 're getting at is a kind of , ramp up of a filter anyway . But , if you do that , then , in practice somebody using the SmartKom system , one would think if they 're using it for a while , it means that their first utterance , instead of , you know , getting , a forty percent error rate reduction , they 'll get a  , over what , you 'd get without this , policy , you get thirty percent . And then the second utterance that you give , they get the full you know , full benefit of it if it 's this ongoing thing . phd a: Oh , so you you cache the utterances ? That 's how you get your ,  professor c: Well , I 'm saying in practice , yeah , grad e: M phd a: Ah . professor c: that 's If somebody 's using a system to ask for directions or something , phd a: OK . And and to begin with if it doesn't get them quite right , ma m maybe they 'll come back and say , " excuse me ? " phd a:   And so , there 's a higher probability of it making an error , in the first utterance . phd a: What would be really cool is if you could have  , this probably users would never like this but if you had could have a system where , before they began to use it they had to introduce themselves , verbally . professor c: Oh , the other thing I guess which which , I don't know much about as much as I should about the rest of the system but but , couldn't you , if you if you sort of did a first pass I don't know what kind of , capability we have at the moment for for doing second passes on on , some kind of little small lattice , or a graph , or confusion network , or something . But if you did first pass with , the with either without the mean sub subtraction or with a a very short time one , and then , once you , actually had the whole utterance in , if you did , the , longer time version then , based on everything that you had , and then at that point only used it to distinguish between , you know , top N , possible utterances or something , you you might it might not take very much time .  , I know in the large vocabulary stu  , systems , people were evaluating on in the past , some people really pushed everything in to make it in one pass but other people didn't and had multiple passes . And , the argument , against multiple passes was u u has often been " but we want to this to be r you know have a nice interactive response " . And the counterargument to that which , say , BBN I think had , was " yeah , but our second responses are second , passes and third passes are really , really fast " . grad e: S so , the the idea of the second pass would be waiting till you have more recorded speech ? Or ? professor c: Yeah , so if it turned out to be a problem , that you didn't have enough speech because you need a longer longer window to do this processing , then , one tactic is you know , looking at the larger system and not just at the front - end stuff is to take in , the speech with some simpler mechanism or shorter time mechanism , grad e:   professor c: do the best you can , and come up with some al possible alternates of what might have been said . And , either in the form of an N - best list or in the form of a lattice , or or confusion network , or whatever . professor c: And then the decoding of that is much , much faster or can be much , much faster if it isn't a big bushy network . And you can decode that now with speech that you 've actually processed using this longer time , subtraction . professor c: So  , it 's it 's common that people do this sort of thing where they do more things that are more complex or require looking over more time , whatever , in some kind of second pass . professor c: and again , if the second pass is really , really fast  , another one I 've heard of is is in in connected digit stuff , going back and l and through backtrace and finding regions that are considered to be a d a digit , but , which have very low energy . professor c: So , there 's lots of things you can do in second passes , at all sorts of levels . phd a: So is that ,  that it ? grad e: I guess that 's it .  , so , the last two weeks was , like So I 've been working on that Wiener filtering . And , found that , s single like , I just do a s normal Wiener filtering , like the standard method of Wiener filtering . And that doesn't actually give me any improvement over like  , b it actually improves over the baseline but it 's not like it doesn't meet something like fifty percent or something . So , I 've been playing with the v phd a: Improves over the base line MFCC system ? Yeah . professor c: Is that using in combination with something else ? phd b: No , just just one stage Wiener filter professor c: With with a phd b: which is a standard Wiener filter . professor c: No , no , but  in combination with our on - line normalization or with the LDA ? phd b: Yeah , yeah , yeah , yeah . phd b: So , I di i di professor c: So , does it g does that mean it gets worse ? Or ? phd b: No . Like I have an LDA f LDA plus on - line normalization , and then I plug in the Wiener filter in that , professor c: Yeah ? phd b: so it improves over not having the Wiener filter . So it improves but it it doesn't take it like be beyond like thirty percent over the baseline . So professor c: But that 's what I 'm confused about , cuz I think I thought that our system was more like forty percent without the Wiener filtering . phd a: Is this with the v new VAD ? phd b: well , these are not No , it 's the old VAD . So my baseline was , nine This is like w the baseline is ninety - five point six eight , and eighty - nine , and professor c: So  , if you can do all these in word errors it 's a lot a lot easier actually . phd b: What was that ? Sorry ? professor c: If you do all these in word error rates it 's a lot easier , right ? phd b: Oh , OK , OK , OK . phd d: The baseline is something similar to a w  , the t the the baseline that you are talking about is the MFCC baseline , right ? phd b: The t yeah , there are two baselines . So the baseline One baseline is MFCC baseline that When I said thirty percent improvement it 's like MFCC baseline . professor c: So so so what 's it start on ? The MFCC baseline is is what ? Is at what level ? phd b: It 's the it 's just the mel frequency and that 's it . professor c: No , what 's what 's the number ? phd b: so I I don't have that number here . What 's ten point seven ? phd b: It 's a medium misma OK , sorry . phd b: So professor c: OK , four point three , ten point seven , phd b: And forty forty . professor c: And what were you just describing ? phd b: Oh , the one is this one is just the baseline plus the , Wiener filter plugged into it . professor c: But where 's the , on - line normalization and so on ? phd b: Oh , OK . So , with the with the on - line normalization , the performance was , ten OK , so it 's like four point three . So the h well matched has like literally not changed by adding on - line or LDA on it . professor c: OK , and what kind of number an and what are we talking about here ? phd b: It 's the It - it 's Italian . professor c: Is this TI - digits phd b: I 'm talking about Italian , professor c: or Italian ? phd b: yeah . professor c: And what did So , what was the , corresponding number , say , for , the Alcatel system for instance ? phd b: Mmm .  professor c: Do you know ? phd d: Yeah , so it looks to be ,  phd b: You have it ? phd d: Yep , it 's three point four , eight point , seven , and , thirteen point seven . phd b: So , this is the single stage Wiener filter , with The noise estimation was based on first ten frames . phd b: Actually I started with using the VAD to estimate the noise and then I found that it works it doesn't work for Finnish and Spanish because the VAD endpoints are not good to estimate the noise because it cuts into the speech sometimes , so I end up overestimating the noise and getting a worse result . phd b: So ,  so this was ,  And so this was giving  , this this was like not improving a lot on this baseline of not having the Wiener filter on it . And , so , I ran this stuff with one more stage of Wiener filtering on it but the second time , what I did was I estimated the new Wiener filter based on the cleaned up speech , and did , smoothing in the frequency to to reduce the variance professor c:   phd b: I have I 've I 've observed there are , like , a lot of bumps in the frequency when I do this Wiener filtering which is more like a musical noise or something . And so by adding another stage of Wiener filtering , the results on the SpeechDat - Car was like ,  So , I still don't have the word error rate . And the rest is like the LDA plu and the on - line normalization all remaining the same .  , so this was , like , compared to ,  Fifty - seven is what you got by using the French Telecom system , right ? phd d: No , I don't think so . phd b: Y i phd d: Is it on Italian ? phd b: No , this is over the whole SpeechDat - Car . phd b: Yeah , so the new the new Wiener filtering schema is like some fifty - six point four six which is like one percent still less than what you got using the French Telecom system . But again , you 're you 're more or less doing what they were doing , right ? phd b: It 's it 's different in a sense like I 'm actually cleaning up the cleaned up spectrum which they 're not doing . They 're d what they 're doing is , they have two stage stages of estimating the Wiener filter , but the final filter , what they do is they they take it to their time domain by doing an inverse Fourier transform . phd b: And they filter the original signal using that fil filter , professor c:  - huh . phd b: which is like final filter is acting on the input noisy speech rather than on the cleaned up . So this is more like I 'm doing Wiener filter twice , but the only thing is that the second time I 'm actually smoothing the filter and then cleaning up the cleaned up spectrum first level . phd b: And actually I tried it on s the original clean  , the original spectrum where , like , I the second time I estimate the filter but actually clean up the noisy speech rather the c s first output of the first stage and that doesn't seems to be a giving , that much improvement . And and what I t what I tried was , by using the same thing but  , so we actually found that the VAD is very , like , crucial . phd b: by instead of using the current VAD , if you just take up the VAD output from the channel zero , when instead of using channel zero and channel one , because that was the p that was the reason why I was not getting a lot of improvement for estimating the noise . So I just used the channel zero VAD to estimate the noise so that it gives me some reliable mar markers for this noise estimation . professor c: What 's a channel zero VAD ? phd b: professor c: I 'm I 'm confused about that . phd b: so , it 's like phd d: So it 's the close - talking microphone . phd b: Yeah , the close - talking without professor c: Oh , oh , oh , oh . phd b: So because the channel zero and channel one are like the same speech , but only w  , the same endpoints . professor c:  phd b: But the only thing is that the speech is very noisy for channel one , so you can actually use the output of the channel zero for channel one for the VAD .  , so a are they going to pro What are they doing to do , do we know yet ? about as far as what they 're what the rules are going to be and what we can use ? phd d: Yeah , so actually I received a a new document , describing this . phd b: Yeah , that 's phd d: And what they did finally is to , mmm , not to align the utterances but to perform recognition , only on the close - talking microphone , phd b: Which is the channel zero . professor c: So it 's not like that 's being done in one place or one time . phd d: And professor c: That 's that 's just a rule and we 'd you you were permitted to do that . Is is that it ? phd d: I think they will send , files but we we don't Well , apparently professor c: Oh , so they will send files so everybody will have the same boundaries to work with ? phd d: Yeah . phd d: Oh , i Yeah , so what happened here is that , the overall improvement that they have with this method So Well , to be more precise , what they have is , they have these alignments and then they drop the beginning silence and and the end silence but they keep , two hundred milliseconds before speech and two hundred after speech .  , and the overall improvement over the MFCC baseline So , when they just , add this frame dropping in addition it 's r  , forty percent , right ? professor c:   phd b: Yeah , which is phd d: which is , t which is the overall improvement . phd b: It gives like negative Well , in in like some Italian and TI - digits , phd d: Yeah , some @ @ . So by using the endpointed speech , actually it 's worse than the baseline in some instances , which could be due to the word pattern . professor c: Yeah , phd d: And Yeah , the other thing also is that fourteen percent is less than what you obtain using a real VAD . phd d: So  So I think this shows that there is still work  , well , working on the VAD is still still important I think . professor c: Yeah , c phd d:  phd a: Can I ask just a a high level question ? Can you just say like one or two sentences about Wiener filtering and why why are people doing that ? phd b:  . phd a: What 's what 's the deal with that ? phd b: OK , so the Wiener filter , it 's it 's like it 's like you try to minimize  , so the basic principle of Wiener filter is like you try to minimize the , d  , difference between the noisy signal and the clean signal if you have two channels . Like let 's say you have a clean t signal and you have an additional channel where you know what is the noisy signal . And you get you can do that  , if if you have only a c noisy signal , at a level which you , you w try to estimate the noise from the w assuming that the first few frames are noise or if you have a w voice activity detector , you estimate the noise spectrum . phd b: And then you phd a: Do you assume the noise is the same ? phd b: Yeah . phd b: So but that 's not the case in , many many of our cases but it works reasonably well . And then you do this  , this is the transfer function of the Wiener filter , so " SF " is a clean speech spectrum , power spectrum phd a:   professor c: Right phd b: And , professor c: actually , I guess phd b: Yeah . phd b: So but the thing is that you have to estimate the SF from the noisy spectrum , what you have . So you estimate the NF from the initial noise portions and then you subtract that from the current noisy spectrum to get an estimate of the SF . phd a: So is this , basically s  , similar to just regular spectral subtraction ? phd b: It professor c: It 's all pretty related , phd b: Yeah . It 's it 's there 's a di there 's a whole class of techniques where you try in some sense to minimize the noise . phd a: Do people use the Wiener filtering in combination with the spectral subtraction typically , or is i are they sort of competing techniques ? phd b: Not seen . phd b: So it 's like I haven't seen anybody using s Wiener filter with spectral subtraction . professor c: but y but there you make different approximations , and in spectral subtraction , for instance , there 's a a an estimation factor . professor c: You sometimes will figure out what the noise is and you 'll multiply that noise spectrum times some constant and subtract that rather than and sometimes people even though this really should be in the power domain , sometimes people s work in the magnitude domain because it it it works better . phd a: So why did you choose , Wiener filtering over some other one of these other techniques ? phd b: the reason was , like , we had this choice of using spectral subtraction , Wiener filtering , and there was one more thing which I which I 'm trying , is this sub space approach . phd b: So I picked up phd a: So you 're sort of trying @ @ them all . phd b: Y Yeah , phd a: Ah , phd b: we just wanted to have a few noise production compensation techniques phd a: I see . professor c: I m  yeah , there 's Car - Carmen 's working on another , on the vector Taylor series . professor c: So they were just kind of trying to cover a bunch of different things with this task and see , you know , what are what are the issues for each of them . phd b: So so one of one of the things that I tried , like I said , was to remove those zeros in the fri filter by doing some smoothing of the filter . phd b: Like , you estimate the edge of square and then you do a f smoothing across the frequency so that those zeros get , like , flattened out . So what I did was like I p did this and then you I plugged in the one more the same thing but with the smoothed filter the second time . phd b: So that 's where I got like fifty - six point five percent improvement on SpeechDat - Car with that . And So the other thing what I tried was I used still the ten frames of noise estimate but I used this channel zero VAD to drop the frames . And that has taken the performance to like sixty - seven percent in SpeechDat - Car , which is which which like sort of shows that by using a proper VAD you can just take it to further , better levels . phd a: So that 's sort of like , you know , best - case performance ? phd b: Yeah , so far I 've seen sixty - seven  , no , I haven't seen s like sixty - seven percent . And , using the channel zero VAD to estimate the noise also seems to be improving but I don't have the results for all the cases with that . So I used channel zero VAD to estimate noise as a lesser 2 x frame , which is like , everywhere I use the channel zero VAD . And that seems to be the best combination , rather than using a few frames to estimate and then drop a channel . phd b: so we c so  , which which means , like , by using this technique what we improve just the VAD professor c: Yes . And , w  Yeah , but this all these things , I have to still try it on the TI - digits , which is like I 'm just running . And there seems to be not improving a a lot on the TI - digits , so I 'm like investigating that , why it 's not . So ,  so the other the other thing is like I 've been I 'm doing all this stuff on the power spectrum . So Tried this stuff on the mel as well mel and the magnitude , and mel magnitude , and all those things . So , one of one of reasons I thought like doing the averaging , after the filtering using the mel filter bank , that seems to be maybe helping rather than trying it on the mel filter ba filtered outputs . phd b: Yeah , th that 's that 's the only thing that I could think of why why it 's giving improvement on the mel . professor c: how about the subspace stuff ? phd b: Subspace , I 'm I 'm like that 's still in a little bit in the back burner because I 've been p putting a lot effort on this to make it work , on tuning things and other stuff .  , So to r to remind you a little bit of of what I did before , is just to apply some spectral subtraction with an overestimation factor also to get , an estimate of the noise , spectrum , and subtract this estimation of the noise spectrum from the , signal spectrum , but subtracting more when the SNR is is , low , which is a technique that it 's often used . phd a: " Subtracting more " , meaning ? phd d: So you overestimate the noise spectrum . phd d: So , above twenty DB , it 's one , so you just subtract the noise . phd d: And then it 's b Generally Well , I use , actually , a linear , function of the SNR , phd a:   phd d: which is bounded to , like , two or three , when the SNR is below zero DB . phd d: doing just this , either on the FFT bins or on the mel bands , t doesn't yield any improvement professor c: Oh !  , what are you doing with negative , powers ? phd d: o Yeah . So there is also a threshold , of course , because after subtraction you can have negative energies , phd a:   phd d: and So what I I just do is to put ,  to to add to put the threshold first and then to add a small amount of noise , which right now is speech - shaped .  phd a: Speech - shaped ? phd d: Yeah , so it 's a it has the overall overall energy ,  pow it has the overall power spectrum of speech . phd a: So when y when you talk about there being something less than zero after subtracting the noise , is that at a particular frequency bin ? phd d: i  - huh . phd a: And so when you say you 're adding something that has the overall shape of speech , is that in a in a particular frequency bin ? Or you 're adding something across all the frequencies when you get these negatives ? phd d: For each frequencies I a I 'm adding some , noise , but the a the amount of the amount of noise I add is not the same for all the frequency bins . Right now I don't think if it makes sense to add something that 's speech - shaped , because then you have silence portion that have some spectra similar to the sp the overall speech spectra . So this is something I can still work on , phd a: So what does that mean ? phd d: but  . phd a: I 'm trying to understand what it means when you do the spectral subtraction and you get a negative . It means that at that particular frequency range you subtracted more energy than there was actually phd d: That means that   So so yeah , you have an an estimation of the noise spectrum , but sometimes , of course , it 's as the noise is not perfectly stationary , sometimes this estimation can be , too small , so you don't subtract enough . phd a: So in in an ideal word i world if the noise were always the same , then , when you subtracted it the worst that i you would get would be a zero .  , the lowest you would get would be a zero , cuz i if there was no other energy there you 're just subtracting exactly the noise . phd d:  -  , professor c: Yep , there 's all there 's all sorts of , deviations from the ideal here . professor c: for instance , you 're you 're talking about the signal and noise , at a particular point . And even if something is sort of stationary in ster terms of statistics , there 's no guarantee that any particular instantiation or piece of it is exactly a particular number or bounded by a particular range . professor c: So , you 're figuring out from some chunk of of of the signal what you think the noise is . professor c: and there 's absolutely no reason to think that you 'd know that it wouldn't , be negative in some places . professor c: on the other hand that just means that in some sense you 've made a mistake because you certainly have stra subtracted a bigger number than is due to the noise . professor c:  Also , we speak the whole where all this stuff comes from is from an assumption that signal and noise are uncorrelated . And that certainly makes sense in s in in a statistical interpretation , that , you know , over , all possible realizations that they 're uncorrelated phd a:   professor c: or assuming , ergodicity that i that i  , across time , it 's uncorrelated . But if you just look at a quarter second , and you cross - multiply the two things , you could very well , end up with something that sums to something that 's not zero . But if down the road you 're making use of something as if it is a power spectrum , then it can be bad to have something negative . Now , the other thing I wonder about actually is , what if you left it negative ? What happens ? phd b: Is that the log ? professor c: because  , are you taking the log before you add them up to the mel ? phd b: After that . So the thing is , I wonder how if you put your thresholds after that , I wonder how often you would end up with ,  with negative values . phd b: But you will But you end up reducing some neighboring frequency bins @ @ in the average , right ? When you add the negative to the positive value which is the true estimate . But nonetheless , you know , these are it 's another f kind of smoothing , right ? that you 're doing . So , you 've done your best shot at figuring out what the noise should be , and now i then you 've subtracted it off . And then after that , instead of instead of , leaving it as is and adding things adding up some neighbors , you artificially push it up . professor c: Which is , you know , it 's there 's no particular reason that that 's the right thing to do either , right ? phd b: Yeah , yeah . professor c: So , i in fact , what you 'd be doing is saying , " well , we 're d we 're we 're going to definitely diminish the effect of this frequency in this little frequency bin in the in the overall mel summation " . I d I don't know if it would be phd a: Sort of the opposite of that would be if if you find out you 're going to get a negative number , you don't do the subtraction for that bin . professor c: although phd a: That would be almost the opposite , right ? Instead of leaving it negative , you don't do it . If your if your subtraction 's going to result in a negative number , you you don't do subtraction in that . professor c: Yeah , but that means that in a situation where you thought that that the bin was almost entirely noise , you left it . phd d: And , yeah , some people also if it 's a negative value they , re - compute it using inter interpolation from the edges and bins . professor c: People can also , reflect it back up and essentially do a full wave rectification instead of a instead of half wave . Well , actually I tried , something else based on this , is to to put some smoothing , because it seems to to help or it seems to help the Wiener filtering professor c:   Actually , when you do spectral subtraction you can , find this this equivalent in the s in the spectral domain . You can  compute , y you can say that d your spectral subtraction is a filter , and the gain of this filter is the , signal energy minus what you subtract , divided by the signal energy . And this is a gain that varies over time , and , you know , of course , depending on the s on the noise spectrum and on the speech spectrum . And what happen actually is that during low SNR values , the gain is close to zero but it varies a lot . Mmm , and this this is the cause of musical noise and all these the the fact you we go below zero one frame and then you can have an energy that 's above zero . But it 's the smoothing is nonlinear in the sense that I tried to not smooth if the gain is high , because in this case we know that , the estimate of the gain is correct because we we are not close to to to zero , and to do more smoothing if the gain is low . So , well , basically that 's this idea , and it seems to give pretty good results , although I 've just just tested on Italian and Finnish . And on Italian it seems my result seems to be a little bit better than the Wiener filtering , phd b:   phd d: I don't know if you have these improvement the detailed improvements for Italian , Finnish , and Spanish there phd b: Fff . No , I don't have , for each , phd d: or you have just have your own . professor c: So these numbers he was giving before with the four point three , and the ten point one , and so forth , those were Italian , right ? phd b: Yeah , yeah , yeah . phd d:  phd b: I actually didn't give you the number which is the final one , phd d: no , we 've phd b: which is , after two stages of Wiener filtering . phd d:  professor c: But do you have numbers in terms of word error rates on on Italian ? So just so you have some sense of reference ? phd d: Yeah . professor c: And this is , spectral subtraction plus what ? phd d: Plus plus nonlinear smoothing . Well , it 's the system it 's exactly the sys the same system as Sunil tried , professor c: On - line normalization and LDA ? phd d: but professor c: Yeah . But instead of double stage Wiener filtering , it 's it 's this smoothed spectral subtraction . phd a: for Do they use spectral subtraction , or Wiener filtering , or ? phd b: They use spectral subtraction , right . phd d: It it 's Wiener filtering , phd b: Oh , it 's it 's Wiener filtering . professor c: Yeah , plus , I guess they have some sort of cepstral normalization , as well . phd b: s They have like yeah , th the just noise compensation technique is a variant of Wiener filtering , phd d:   phd d: One in the time domain and one in the frequency domain by just taking the first , coefficients of the impulse response .  , what you did , it 's similar phd b: It 's similar in the smoothing and phd d: because you have also two two kind of smoothing . phd d: One in the time domain , and one in the frequency domain , phd b: Yeah . phd a: Does the smoothing in the time domain help phd d:  phd a: Well , do you get this musical noise stuff with Wiener filtering or is that only with , spectral subtraction ? phd b: No , you get it with Wiener filtering also . phd a: Does the smoothing in the time domain help with that ? Or some other smoothing ? phd b: Oh , no , you still end up with zeros in the s spectrum . phd b: the phd d: Well , I cannot you cannot hear beca well , actually what I d did not say is that this is not in the FFT bins .  So , it could be seen as a f a a smoothing in the frequency domain because I used , in ad mel bands in addition and then the other phase of smoothing in the time domain . But , when you look at the spectrogram , if you don't have an any smoothing , you clearly see , like in silence portions , and at the beginning and end of speech , you see spots of high energy randomly distributed over the the spectrogram . phd d:  phd a: That 's the musical noise ? phd d: Which is musical noise , phd a:   phd d: yeah , if if it If you listen to it  , if you do this in the FFT bins , then you have spots of energy randomly distributing . And if you f if you re - synthesize these spot sounds as , like , sounds , phd a:   phd d:  professor c: Well , none of these systems , by the way , have  , y you both are are working with , our system that does not have the neural net , phd d: And phd b: Yep . So one would hope , presumably , that the neural net part of it would would improve things further as as they did before .  Yeah , although if if we , look at the result from the proposals , one of the reason , the n system with the neural net was , more than well , around five percent better , is that it was much better on highly mismatched condition . I 'm thinking , for instance , on the TI - digits trained on clean speech and tested on noisy speech . phd d: And if we have no , spectral subtraction or Wiener filtering , i the system is  , we thought the neural neural network is much better than before , even in these cases of high mismatch . phd a: Could you train a neural net to do spectral subtraction ? professor c: Yeah , it could do a nonlinear spectral subtraction phd d:   professor c: but I don't know if it  , you have to figure out what your targets are . phd a: Yeah , I was thinking if you had a clean version of the signal and and a noisy version , and your targets were the M F -  , you know , whatever , frequency bins phd d:   professor c: Yeah , well , that 's not so much spectral subtraction then , phd d:   professor c: but but but it 's but at any rate , yeah , people ,  phd a: People do that ? professor c: y yeah , in fact , we had visitors here who did that I think when you were here ba way back when . professor c: The objection everyone always raises , which has some truth to it is that , it 's good for mapping from a particular noise to clean but then you get a different noise . professor c: And the experiments we saw that visitors did here showed that it there was at least some , gentleness to the degradation when you switched to different noises . phd a: How did it compare on  , for for good cases where it it  , stuff that it was trained on ? Did it do pretty well ? professor c: Oh , yeah , it did very well .  , we 're not doing exactly that , we 're not trying to generate good examples but by trying to do the best classifier you possibly can , for these little phonetic categories , phd a: it we 'll just have to see whether it helps more or less the same , but I would imagine it would help some . professor c: So in any event , all of this I was just confirming that all of this was with a simpler system .  , Yeah , so this is th the ,  Well , actually , this was kind of the first try with this spectral subtraction plus smoothing , professor c:   And it seems that the one that I chose for the first experiment was the optimal one , so  , professor c: It 's amazing how often that happens .  Yeah , another thing that I it 's important to mention is , that this has a this has some additional latency . Because when I do the smoothing , it 's a recursion that estimated the means , so of the g of the gain curve . So , instead o of using the current estimated mean to , subtract the current frame , it 's better to use an estimate that 's some somewhere in the future . phd b: You mean , the m the mean is computed o based on some frames in the future also ? professor c:   phd b: Or or no ? phd d: It 's the recursion , so it 's it 's the center recursion , right ? phd b:   professor c: One five ? phd d:  professor c: One five ? Five zero ? phd d: Five zero , professor c: Five zero . phd b: why why is that delay coming ? Like , you estimate the mean ? phd d: Yeah , the mean estimation has some delay , right ? phd b: Oh , yeah . phd b: It isn't OK , so it 's like it looks into the future also . professor c: What if you just look into the past ? phd d: It 's , not as good . professor c: How m by how much ? phd d: it helps a lot over the ba the baseline but , mmm professor c: By how much ? phd d: it It 's around three percent , relative . phd d: mmm So ,  professor c: It 's depending on how all this stuff comes out we may or may not be able to add any latency . Yeah , b but I don't think we have to worry too much on that right now while you kno . professor c: s Yeah , I think the only thing is that phd d: So professor c: I would worry about it a little . professor c: Because if we completely ignore latency , and then we discover that we really have to do something about it , we 're going to be find ourselves in a bind . professor c: Yeah , just , you know , just be be a little conservative phd d: Oh yes . professor c: because we may end up with this crunch where all of a sudden we have to cut the latency in half or something . So , yeah , there are other things in the , algorithm that I didn't , @ @ a lot yet , phd a: Oh ! phd d: which phd a: Sorry . If if there 's another part of the system that causes a latency of a hundred milliseconds , is this an additive thing ? Or c or is yours hidden in that ? phd d:   We can do something in parallel also , in some like some cases like , if you wanted to do voice activity detection . phd b: So you can make a decision on that voice activity detection and then you decide whether you want to filter or not . phd a: couldn't ,  I Couldn't you just also  , i if you know that the l the largest latency in the system is two hundred milliseconds , don't you couldn't you just buffer up that number of frames and then everything uses that buffer ? phd b: Yeah . phd a: And that way it 's not additive ? professor c: Well , in fact , everything is sent over in buffers cuz of isn't it the TCP buffer some ? phd b: You mean , the the data , the super frame or something ? phd d:   phd b: Yeah , but that has a variable latency because the last frame doesn't have any latency phd d:   phd b: So But the only thing is that the first frame in that twenty - four frame buffer has a twenty - four frame latency . phd b: Because it just goes as phd a: Yeah , I wasn't thinking of that one in particular phd b: Yeah . phd a: but more of , you know , if if there is some part of your system that has to buffer twenty frames , can't the other parts of the system draw out of that buffer and therefore not add to the latency ? professor c: Yeah . And and that 's sort of one of the all of that sort of stuff is things that they 're debating in their standards committee . Like , I played a little bit with this overestimation factor , but I still have to to look more at this , at the level of noise I add after .  , I know that adding noise helped , the system just using spectral subtraction without smoothing , but I don't know right now if it 's still important or not , and if the level I choose before is still the right one .  , and another thing is to Yeah , for this I just use as noise estimate the mean , spectrum of the first twenty frames of each utterance . I don't remember for this experiment what did you use for these two stage phd b: I used ten just ten frames . Yeah , because phd d: The ten frames ? phd b: the reason was like in TI - digits I don't have a lot . But , so what 's this result you told me about , the fact that if you use more than ten frames you can improve by t phd b: Well , that 's that 's using the channel zero . phd b: Which phd d: But this is ten frames plus plus phd b: Channel zero dropping . phd d: no , these results with two stage Wiener filtering is ten frames phd b: t Oh , this phd d: but possibly more . The second thing I was working on is to , try to look at noise estimation , mmm , and using some technique that doesn't need voice activity detection .  , and for this I u simply used some code that , I had from from Belgium , which is technique that , takes a bunch of frame , and for each frequency bands of this frame , takes a look at the minima of the energy . And then average these minima and take this as an an energy estimate of the noise for this particular frequency band . So , I compute an FFT based on the long , signal frame which is sixty - four millisecond phd a: So you have one minimum for each frequency ? phd d: What what I what I d  , I do actually , is to take a bunch of to take a tile on the spectrogram and this tile is five hundred milliseconds long and two hundred hertz wide . phd d: And this tile  , in this tile appears , like , the harmonics if you have a voiced sound , because it 's it 's the FTT bins . And when you take the m the minima of of these this tile , when you don't have speech , these minima will give you some noise level estimate , If you have voiced speech , these minima will still give you some noise estimate because the minima are between the harmonics . And If you have other other kind of speech sounds then it 's not the case , but if the time frame is long enough , like s five hundred milliseconds seems to be long enough , you still have portions which , are very close whi which minima are very close to the noise energy . You said five hundred milliseconds phd d: Mmm ? professor c: but you said sixty - four milliseconds . Which is which ? What ? phd d: Sixty - four milliseconds is to compute the FFT , bins . phd d: actually it 's better to use sixty - four milliseconds because , if you use thirty milliseconds , then , because of the this short windowing and at low pitch , sounds , the harmonics are not , wha  , correctly separated . phd d: So if you take these minima , it b they will overestimate the noise a lot . professor c: So you take sixty - four millisecond F F Ts and then you average them over five hundred ? Or ?  , what do you do over five hundred ? phd d: So I take to I take a bunch of these sixty - four millisecond frame to cover five hundred milliseconds , professor c: Ah . phd d: on the on on the bunch of  fifty frames , right ? professor c: I see . So the interest of this is that , as y with this technique you can estimate u some reasonable noise spectra with only five hundred milliseconds of of signal , so if the the n the noise varies a lot , you can track better track the noise , professor c:   The only requirement is that you must have , in these five hundred milliseconds segment , you must have voiced sound at least . So what I did is just to simply replace the VAD - based , noise estimate by this estimate , first on SpeechDat - Car Well , only on SpeechDat - Car actually . And it 's , slightly worse , like one percent relative compared to the VAD - based estimates .  , I think the reason why it 's not better , is that the SpeechDat - Car noises are all stationary . So , u y y there really is no need to have something that 's adaptive professor c:   But , I expect s maybe some improvement on TI - digits because , nnn , in this case the noises are all sometimes very variable . professor c: But are you comparing with something e I 'm I 'm p s a little confused again , i it  , when you compare it with the V A D - based , phd d:   professor c: VAD - Is this is this the ? phd d: It 's It 's the France - Telecom - based spectra , s  , Wiener filtering and VAD . professor c: Oh , you 're not doing this with our system ? phd d: In i I 'm not No , no . Actually , th the best system that we still have is , our system but with their noise compensation scheme , right ? professor c: Right . But phd d: So I 'm trying to improve on this , and by by replacing their noise estimate by , something that might be better . professor c: Couldn't you try this for that ? phd d: But I di professor c: Do you think it might help ? phd d: Not yet , because I did this in parallel , professor c: I see , phd d: and I was working on one and the other . phd b: So I 'm also using that n new noise estimate technique on this Wiener filtering what I 'm trying . phd b: So I I have , like , some experiments running , I don't have the results .  , the idea is just to  , flatten the log , spectrum , and to flatten it more if the the probability of silence is higher . So in this way , you can also reduce somewhat reduce the musical noise and you reduce the variability if you have different noise shapes , because the the spectrum becomes more flat in the silence portions . With this , no improvement , but there are a lot of parameters that we can play with and ,  Actually , this this could be seen as a soft version of the frame dropping because , you could just put the threshold and say that " below the threshold , I will flatten comp completely flatten the the spectrum " . So it would be like frame dropping , because during the silence portions which are below the threshold of voice activity probability , w you would have some kind of dummy frame which is a perfectly flat spectrum . And this , whitening is something that 's more soft because , you whiten you just , have a function the whitening is a function of the speech probability , so it 's not a hard decision . phd d: so I think maybe it can be used together with frame dropping and when we are not sure about if it 's speech or silence , well , maybe it has something do with this .  , you know , in in JRASTA we were essentially adding in , white  , white noise dependent on our estimate of the noise . professor c: You could imagine one that that that made use of where where the amount that you added in was , a function of the probability of it being s speech or noise . Yeah , w Yeah , right now it 's a constant that just depending on the the noise spectrum . professor c: Cuz that that brings in sort of powers of classifiers that we don't really have in , this other estimate . professor c: What what what point does the , system stop recording ? How much phd a: It 'll keep going till I guess when they run out of disk space , professor c: It went a little long ?  , disk phd a: but I think we 're OK .  Yeah , so there are with this technique there are some I just did something exactly the same as as the Ericsson proposal but , the probability of speech is not computed the same way . And I think , i for yeah , for a lot of things , actually a g a good speech probability is important . Like for frame dropping you improve , like you can improve from ten percent as Sunil showed , if you use the channel zero speech probabilities .  , so yeah , the next thing I started to do is to , try to develop a better voice activity detector . And ,  I d  yeah , for this I think we can maybe try to train the neural network for voice activity detection on all the data that we have , including all the SpeechDat - Car data .  , and the way I mi I do that is that I just use the HTK system but I train it only on the close - talking microphone .  It seems to be ,  i Actually what I observed is that for Italian it doesn't seem Th - there seems to be a problem . Because What ? phd b: you mean their d the frame dropping , right ? Yeah , it doesn't phd d: Yeah . phd d: so  , the c the current VAD that we have was trained on , t SPINE , right ? phd b: TI - digits . And actually we observed we listened to some of the utterances and sometimes for Finnish there is music in the recordings and strange things , right ? phd b: Yeah . phd d:  Yeah , so the idea was to train all the databases and obtain an alignment to train on these databases , and , also to , try different kind of features , as input to the VAD network . And we came up with a bunch of features that we want to try like , the spectral slope , the , the degree o degree of voicing with the features that , we started to develop with Carmen , e with , the correlation between bands and different kind of features , phd b: Yeah . Well , Hans - Guenter will be here next week so I think he 'll be interested in all all of these things