What depends on the definition of fairness? Surely not whether the queries are the same. The queries are only the same if one factor determines the other. This is why you immediately retreat to a 99%, which still is not practically the same, and still doesn't matter if the correct features are being captured.

If I am looking to predict who can run a 4 minute mile, and it turns out my model returns no people over 60, is that unfair? Hardly. The record for an over 60 person is 4:49 so most good models would not return anyone over that age. However, if my model is based on various physical features, not including age, if a 61 year old shows up with those physical features, "age bias" will not stop the model from predicting them as capable.

You are trying to shift the discussion to a 99% case, which is quite different from most correlations in the first place. This is clearly not the case for predictive variables like education. If you have a graduation rate of 90% for one group and 80% for another group, your algorithm is favoring some of each group, and disfavoring some of each group based on education, not group membership.

> In this definition, the setting I described is unfair, since the correlation on education will bias the model to predict in a way that is correlated with race.

You must be leaving something out of the explanation or I missed it. Say p(insurance_claim) = f(education). If this relationship holds across groups, the model is not biased. The model would be biased only if education were predictive for some subgroups and not others, which would point to education not actually being the cause, and another latent variable driving the relationship (say conscientiousness). You are suggesting that even if the relationship holds across groups, that education is predictive for all groups, that the model is still unfair, because education is not equal across all groups. I think this is more of a case of education perhaps not being fair or life not being fair than the model not being fair? If evidence of education, let's a say passing a driver safety course, is strongly, negatively correlated with insurance claims, but you find that some people cannot afford a driver safety course, it is not the model which is unfair, but perhaps the economic circumstances that led to someone not being able to afford the course. Let's not burden models of insurance risk or expected cost by making them less correct in order to pretend that unfairness in society does not exist. Having a model that is correlated with race is not evidence of bias in the model, but may be evidence of bias in society.

> The only practical way to satisfy such a strong definition is to do a correction on the data, which is a pretty straightforward optimization problem if you formalize the constraints. And yeah, usually you lose predictive power in doing so.

If, as in your example, you had a feature that was 99% determined by group membership, you would lose 99% of the value of that feature. This is why the "strong" definition is not helpful.

> For the record, the strong definition makes sense often times (to me). You wouldn’t want health insurance using your DNA as a predictor, likely, even if genetics are very predictive in that domain.

This is a completely different argument, that a more predictive model might not benefit you. However, it does get me to understand your viewpoint a little bit. You seem to want a world where the luck of what DNA you are born with does not affect you. However, if I were to use a model to predict my own healthcare costs using DNA and decide whether to buy long term care insurance, I certainly wouldn't want a model that was modified to remove "bias" and gave me a less accurate prediction of the real costs.