sort of thing I see Mmm know who was who was since we weren't in on it uh do you know who was in from OGI Was was was Hynek involved or was it Sunil no or idea Oh Mmm I OK Yeah Alright Um yeah So the points were the the weights how to weight the different error rates that are obtained from different language and and conditions Um not clear that they will keep the same kind of weighting Right now a weighting on on improvement Some people are arguing that it would be better to have weights on uh well to to combine error rates before computing improvement Uh and the fact is that for right now for the English they have weights they they combine error rates but for the other languages they combine improvement So not very consistent Um Yeah The um Yeah And so Well this is a point And right now actually there is a thing also uh that happens with the current weight is that a very improvement on the case result in huge differences in in the final number Hmm so perhaps they will change the weights to Yeah How should that be done I mean it it seems like a simple way Uh this seems like an obvious mistake or something Th Well I mean the fact that inconsistent is an obvious mistake But the but In um the other thing I don't know I haven't thought it through but one one would think that each It like if you say the the best way to do an average an arithmetic average or a geometric average It depends what you wanna show Each each one is gonna have a different characteristic Yeah So Well it seems like they should do like the percentage improvement or something rather than the absolute improvement Tha Well they are what doing they that do Yeah No that is relative But the question is do you average the relative improvements Yeah or do you average the error rates and Yeah take the relative improvement maybe of that And the thing is not just a pure average because there are these Oh weightings a weighted average Um Yeah And so when you average the the relative improvement it tends to to give a lot of of um importance to the case because the baseline is already very good and um Why i look at improvements but just look at your av your scores You know figure out how to combine the scores with a weight or whatever and then give you a score your score And then they can do the same thing for the baseline system and its score And then you can look at Well what seeing as one of the things they could do Yeah just when you when you get all done I think that they pro I m I I wasn't there but I think they started off this process with the notion that you should be significantly better than the previous standard And um so they said how much is significantly better what do you And and so they said well you know you should have half Hmm something that you had before Yeah So uh Hmm But it does seem like i i it does seem like more logical to combine them first and then do the Combine error rates Yeah then Yeah Yeah Well But there is this this is this still this problem of weights When when you combine error rate it tends to give more importance to the difficult cases Oh yeah and some people think that well they have different um opinions about this Some people think that more important to look at to have ten percent imp relative improvement on case than to have fifty percent on the m mismatched and other people think that more important to improve a lot on the mismatch and It sounds So like bu don't l really de have a good idea about what the final application is gonna be Mmm Well you Yeah know Mmm thing is that if you look at the numbers on the on the more difficult cases um if you really believe that was gonna be the predominant use none of this would be good enough Yeah Nothing whereas you sort of with some reasonable error recovery could imagine in the better cases that these these systems working So um I think the hope would be that it would uh it would work well for the good cases and uh it would have reasonable reas soft degradation as you got to worse and worse conditions Um I I guess what I mean I I was thinking about it in terms of if I were building the final product and I was gonna test to see which I wanted to use I would try to weight things depending on the exact environment that I was gonna be using the system in If I But but No Well no well no I mean it isn't the operating theater I mean they don they they Yeah don't they don't really know I think I mean So th if if they don't know doesn't that suggest the way for them to go Uh you assume equal I mean y y I mean you Well I mean I I think one thing to do is to just not rely on a single number to maybe have two Yeah three numbers Right know and and and say how much you uh you improve the uh the the relatively clean case and or or case and how how much you uh So not So not try to combine them Yeah Uh Yeah actually true Uh I had forgotten this uh but uh is not actually clean What it is is just that The training u and the testing training and testing are similar Mmm So I guess what you would do in practice is try to get as many uh examples of similar sort of stuff as you could Yeah and then uh So the argument for that being the the the more important thing is that gonna try and do that but you wanna see how badly it deviates from that when when when the uh So a little different Um weight those other conditions v very you know really small But No I mean a more a of an information kind of thing an ar Well an argument for it but let me give you the opposite argument The opposite argument is never really gonna have a good sample of all these different things I mean are you gonna have w uh uh examples with the windows open half open full open Going seventy sixty fifty forty miles an hour On what kind of roads With what passing you With uh I mean I I I think that you could make the opposite argument that the case is a fantasy