So ,  You can fill those out ,  after , actually , so So , I got ,  these results from , Stephane . I think s that , there were some other very good results that we 're gonna wanna compare to . phd a: I I 'm sorry ? I didn't professor d: I got this from you phd a: Yeah . professor d: and then I sent a note to Sunil about the cuz he has been running some other systems phd a:   professor d: You know most of the time , even  even though it 's true that the overall number for Danish we didn't improve it If you look at it individually , what it really says is that there 's ,  Looks like out of the six cases , between the different kinds of , matching conditions out of the six cases , there 's basically , a couple where it stays about the same , three where it gets better , and one where it gets worse . phd a: Y Actually , for the Danish , there 's still some kind of mystery because , when we use the straight features , we are not able to get these nice number with the ICSI OGI one ,  . We don't have this ninety - three seventy - eight , we have eight phd e: Eighty - nine forty - four . professor d: Oh , and and we have a little time on that and actually so phd a:  ? professor d: We have a little bit of time on that , actually . professor d: We have a day or so , so When when when do you folks leave ? phd a: Sunday . professor d: Sunday ? So So ,  Yeah , until Saturday midnight , or something , we have W we we have time , yeah .  , and , you know , i u when whenever anybody figures it out they should also , for sure , email Hynek because Hynek will be over there telling people what we did , so he should know .  , even with these results as they are , it 's it 's it 's really not that bad . But but ,  And it looks like the overall result as they are now , even without , you know , any any bugs being fixed is that , on the the other tasks , we had this average of , forty  nine percent , or so , improvement . And here we have somewhat better than that than the Danish , and somewhat worse than that on the German , but  , it sounds like , one way or another , the methods that we 're doing can reduce the error rate from from mel ceptrum down by , you know a fourth of them to , a half of them .  , I think that , one of the things that Hynek was talking about was understanding what was in the other really good proposals and and trying to see if what should ultimately be proposed is some , combination of things .  , if ,  Cuz there 's things that they are doing there that we certainly are not doing . phd c: How much how much better was the best system than ours ? professor d: So Well , we don't know yet . professor d: first place , there 's still this thing to to work out , and second place second thing is that the only results that we have so far from before were really development set results . And the best result of any system was about fifty - four , where these numbers are the , relative , reduction in , word error rate . professor d: And here the the worst system still reduced the error rate by thirty - three percent , or something , in development set . professor d: So so , you know , sort of everybody is doing things between , well , roughly a third of the errors , and half the errors being eliminated , and varying on different test sets and so forth . professor d: So I think  It 's probably a good time to look at what 's really going on and seeing if there 's a there 's a way to combine the best ideas while at the same time not blowing up the amount of , resources used , cuz that 's that 's critical for this this test . phd c: Do we know anything about who who 's was it that had the lowest on the dev set ? professor d: the , the there were two systems that were put forth by a combination of of , French Telecom and Alcatel . And ,  they they differed in some respects , but they e em one was called the French Telecom Alcatel System the other was called the Alcatel French Telecom System , which is the biggest difference , I think . professor d: you know ? So , my impression is they also did very well on on the the , evaluation set , but , I I we haven't seen you 've - you haven't seen any final results for that phd c: And they used the main thing that that they used was spectral subtraction ? professor d: yeah . And then then there was some p some modification of the cepstral parameters , where they phd a: Yeah , actually , something that 's close to cepstral mean subtraction . I 'm I 'm ,  So , basically , the mean is adapted during speech and not during silence . professor d: But some people have done exactly that sort of thing , of of and the  it 's not To to look in speech only , to try to m to measure these things during speech , phd a: Yeah , yeah . But i it it so it looks like they did some some , reasonable things , and they 're not things that we did , precisely . We did unreasonable things , which because we like to try strange things , and and , and our things worked too . professor d: And so , it 's possible that some combination of these different things that were done would be the best thing to do . But the only caveat to that is that everybody 's being real conscious of how much memory and how much CPU they 're using phd c:   professor d: because these , standards are supposed to go on cell phones with m moderate resources in both respects . phd c: Did anybody , do anything with the models as a an experiment ? Or professor d: they didn't report it , if they did .  , now , one of the things that 's nice about what we did is , we do have a a ,  a filtering , which leads to a a ,  a reduction in the bandwidth in the modulation spectrum , which allows us to downsample . It it said the same amount because for convenience sake in the particular way that this is being tested , they were repeating the packets . So it was they were s they they had twenty - four hundred bits per second , but they were literally creating forty - eight hundred bits per second , even though y it was just repeated . professor d: So , in practice phd c: So you could 've had a repeat count in there or something . professor d: Well , n  , this was just a ph phoney thing just to to fit into the the software that was testing the errors channel errors and so on . professor d: So so in reality , if you put this this system in into , the field , it would be twenty - four hundred bits per second , not forty - eight hundred . phd c: When is the development set  , the , test set results due ? Like the day before you leave or something ? professor d: probably the day after they leave , but we 'll have to we 'll have to stop it the day before we leave . So phd a: Yeah , probably , well professor d: I th I think I I think they are , phd a: Yeah , well professor d: yeah . So  , since we have a bit farther to travel than some of the others , we 'll have to get done a little quicker .  , just exactly this sort of thing of , you know , why why these features seem to be behaving differently , in California than in Oregon .  , we didn't we didn't have enough electrons here and  , but , I think , you know , the main reason for having  , it only takes w to run the the two test sets in just in computer time is just a day or so , right ? phd a: Yeah , professor d: So phd a: it 's very short interval . So , I think the who the whole reason for having as long as we have , which was like a week and a half , is is because of bugs like that . So Huh So , we 're gonna end up with these same kind of sheets that have the the percentages and so on just for the phd a: Yeah , so there are two more columns in the sheets , professor d: Oh , I guess it 's the same sheets , phd a: two . phd a: So , Hynek will try to push for trying to combine , different things ? Or  ? professor d: well that 's  yeah  , I think the question is " Is there is there some advantage ? "  , you could just take the best system and say that 's the standard . But the thing is that if different systems are getting at good things , a again within the constraint of the resources , if there 's something simple that you can do Now for instance , it 's , I think , very reasonable to have a standard for the terminal 's side and then for the server 's side say , " Here 's a number of things that could be done . " So , everything that we did could probably just be added on to what Alcatel did , and i it 'd probably work pretty well with them , too . And then on the terminal 's side , I don't know how much , memory and and CPU it takes , but it seems like the filtering  , the VAD stuff they both had , right ? And , so and they both had some kind of on - line normalization , right ? phd a: yeah . professor d: Of sorts , yeah ? So so , it seems like the main different there is the is the , filtering . And the filtering I think if you can shouldn't take a lot of memory to do that  , and I also wouldn't think the CPU , would be much either for that part . So , if you can if you can add those in  then , you can cut the data rate in half . professor d: So it seems like the right thing to do is to on the on the terminal 's side , take what they did , if it if it does seem to generalize well to German and Danish , take what they did add in a filter , and add in some stuff on the server 's side and and and that 's probably a reasonable standard .   phd a: They are working on this already ? Because yeah , Su - Sunil told me that he was trying already to put some kind of , filtering in the France Telecom . professor d: Yeah , so that 's that 's that 's what That would be ideal would be is that they could , you know , they could actually show that , in fact , a combination of some sort , would work even better than what what any of the systems had .  , you can always have another standard after that , but this process has gone on for a while on already and and people might just wanna pick something and say , " OK , this is it . It 's just that , if you disobey them , then you risk not being able to sell your product , or   And people often work on new standards while an old standard is in place and so on . The other hand , they might just say they just don't know enough yet to to declare a standard . So you you you will be you will become experts on this and know more far more than me about the tha this particular standards process once you you go to this meeting .  , you 're done in the sense that , you may be able to get some new features from Sunil , and we 'll re - run it .  , but other than that , you 're you 're basically done , right ? So , I 'm interested in hearing hearing your thoughts about where you think we should go from this . professor d: we tried a lot of things in a hurry , and , if we can back off from this now and sort of take our time with something , and not have doing things quickly be quite so much the constraint , what what you think would be the best thing to do . phd a: well  Well , first , to really have a look at at the speech from these databases because , well , we tried several thing , but we did not really look at what what 's happening , and where is the noise , and professor d: OK . Actually , there is one thing that well  , generally we we think that most of the errors are within phoneme classes , and so I think it could be interesting to to see if it I don't think it 's still true when we add noise , and so we have I I guess the confusion ma the confusion matrices are very different when when we have noise , and when it 's clean speech . phd a: And so ,  Yeah , so perhaps we could have a a large gain , eh , just by looking at improving the , recognition , not of phonemes , but of phoneme classes , simply . phd a: And which is a s a s a simpler problem , perhaps , but which is perhaps important for noisy speech . professor d: The other thing that strikes me , just looking at these numbers is , just taking the best cases , some of these , of course , even with all of our our wonderful processing , still are horrible kinds of numbers . But just take the best case , the well - matched  , German case after er well - matched Danish after we phd a:   professor d: the kind of numbers we 're getting are about eight or nine  p percent error per digit . professor d: if you have ten digits for a phone number  , every now and then you 'll get it right .  , it 's it 's ,  So , the other thing is that ,  And and a and and also ,  part of what 's nice about this is that this is ,  a realistic almost realistic database .  , it 's still not people who are really trying to accomplish something , but but , within the artificial setup , it isn't noise artificially added , you know , simulated , additive noise . And , the the training the training , I guess , is always done on the close talking phd a: No , actually actually the well - matched condition is still quite di still quite difficult . professor d: No ? phd a: it 's they have all these data from the close mike and from the distant mike , from different driving condition , open window , closed window , professor d: Yeah . phd a: and they take all of this and they take seventy percent , I think , for training and thirty percent for testing . phd a: So , training is done on different conditions and different microphones , and testing also is done on different microphone and conditions . So , probably if we only take the close microphones , I guess the results should be much much better than this . Wha - what about i in so the the phd a: Yeah , so there is this , the mismatched is ,  the same kind of thing , professor d: go ahead . phd a: but the driving conditions ,  the speed and the kind of road , is different for training and testing , is that right ? phd e: Yeah . So , yeah , so the high so the right so the highly mismatched case is in some sense a good model for what we 've been , you know , typically talking about when we talk about additive noise in And so and i i k it does correspond to a realistic situation in the sense that , people might really be trying to , call out telephone numbers or some or something like that , in in their cars phd a: Yeah . professor d:  phd a: Actually , yeah , it 's very close to clean speech training because , well , because the close microphone and noisy speech testing , professor d: Yeah . And the well - matched condition is what you might imagine that you might be able to approach , if you know that this is the application . And then , when y you sell it to somebody , they will be a different person with a different car , and so on . So it 's this is a an optim somewhat optimistic view on it , so , you know , the real thing is somewhere in between the two . professor d: but phd a: But the  , the th th professor d: Even the optimistic one is phd a: it doesn't work . So , in a way , that 's , you know , that 's sort of the dominant thing is that even , say on the development set stuff that we saw , the , the numbers that , that Alcatel was getting when choosing out the best single numbers , it was just you know , it wasn't good enough for for a a for a real system . professor d: and ,  I don't know So , looking at the data , where , you know what 's the what 's what 's th what 's characteristic i e yeah , I think that 's that 's a good thing . Does a any you have any thoughts about what else y you 're thinking that you didn't get to that you would like to do if you had more time ?  phd e: Oh , f a lot of thing . professor d: Yeah , cuz a lot of time it 's true , there were a lot of times when we 've tried something and it didn't work right away , even though we had an intuition that there should be something there .  And , one of the things I don't remember the details on , but I remember at some point , when you were working with a second stream , and you tried a low - pass filtering to cepstrum , in some case you got phd e: MSG Yeah . professor d: Well , but it was an MSG - like thing , but it wasn't MSG , right ?  , you y I think in some case you got some little improvement , but it was , you know , sort of a small improvement , and it was a a big added complication , so you dropped it . But , that was just sort of one try , right ? You just took one filter , threw it there , phd a: Yeah , professor d: right ? And it seems to me that , if that is an important idea , which , you know , might be , that one could work at it for a while , as you 're saying . professor d: And , and you had , you know , you had the multi - band things also , and , you know , there was issue of that . phd a: Yeah , professor d: Barry 's going to be , continuing working on multi - band things as well . Kind of inspired by the stuff by Larry Saul with the , learning articulatory feature in I think , in the case of his paper with sonorance based on , multi - band information where you have a a combination of gradient learning an and , EM . professor d: and  , so , I think that , you know , this is a ,  this is a neat data set .  , and then , as we mentioned before , we also have the the new , digit set coming up from recordings in this room .  and ,  what I like about it , in a way , is that , the results are still so terrible . But even after that it 's still , you know , so poor that that , no one could really use it . So ,  I think that 's great that because and y also because again , it 's not something sometimes we 've gotten terrible results by taking some data , and artificially , you know , convolving it with some room response , or something we take a very  , at one point , Brian and I went downstairs into the the basement where it was it was in a hallway where it was very reverberant and we we made some recordings there . And then we we , made a simulation of the of the room acoustics there and and applied it to other things , phd a:   professor d: and  But it was all pretty artificial , and and , you know , how often would you really try to have your most crucial conversations in this very reverberant hallway ?  So ,  This is what 's nice about the Aurora data and the data here , is that is that it 's sort of a realistic room situation  , acoustics acoustic situation , both terms in noise and reflections , and so on and n n And , with something that 's still relatively realistic , it 's still very very hard to do very well . phd a: Yeah , so d well Actually , this is tha that 's why we well , it 's a different kind of data . So this would be the first thing , and then , of course , try to well , kind of debug what was wrong , eh , when we do Aurora test on the MSG particularly , and on the multi - band . What do you think ? phd c: About professor d: Anything phd c: About other experiments ?  , now , I 'm interested in ,  looking at the experiments where you use , data from multiple languages to train the neural net . And I don't know how far , or if you guys even had a chance to try that , but that would be some it 'd be interesting to me . phd a: Yeah , but professor d: S b phd a: Again , it 's the kind of of thing that , we were thin thinking thinking that it would work , but it didn't work . And , eh , so there is kind of of not a bug , but something wrong in what we are doing , perhaps . phd a: something wrong , perhaps in the just in the the fact that the labels are phd c: Right . So one form of different data was is from different languages , but the other Well , i in fact , m in those experiments it wasn't so much combining multiple nets , it was a single net that had different phd a: Yeah . professor d: So , first thing is would it be better if they were multiple nets , for some reason ? Second thing is , never mind the different languages , just having acoustic conditions rather than training them all up in one , would it be helpful to have different ones ? So ,  That was a question that was kind of raised by Mike Shire 's thesis , and on in that case in terms of reverberation . Whe - when are you two getting back ? phd e: I 'm phd a: You on Friday or S on Saturday or ? phd e: Sunday phd a: S oh yeah , Sunday , yeah . You 've gotta S have a Saturday overnight , right ? phd a: I 'll be back on Tuesday . phd c: Where where 's the meeting ? professor d: Amsterdam , I think , yeah ? phd a: Yeah , Amsterdam . professor d: so Yeah , well , if we don't have an anything else to discuss , we should , turn off the machine and then say the real nasty things