> This means that your entirely imaginary (and therefore entirely unconvincing) scenario will remain entirely imaginary (and unconvincing).

Well, that very much depends on how you look at it. It might be imaginary in so far as regulation in the US possibly prevents those consequences, which is great. One might also see it as evidence that collection of personal data has risks--and that regulation might be one way to deal with those risks, at least in some cases. After all, this kind of regulation is in effect a prohibition on collecting certain kinds of personal data, even if the collection in itself is permissible, as companies won't collect data when they can't use it for anything anyway.

> Personally, I would prefer if car insurers could price discriminate more based on data. I think this would lower rates for me personally, both in the short term because I try to cultivate safe driving habits, and in the long term because it would create an incentive for everybody to try to drive more safely.

Are you sure that it would? Remember that for the insurer, it doesn't matter whether they calculate your risk (and thus your premium) correctly, what matters for them is that they aren't worse at calculating risks than the competition (i.e., the competition can't outcompete them on price or cause them to be left with a non-representative sample of their risk model), and that on average, their criteria match reality (i.e., they don't take on risks that are actually larger than they can pay for). Even if you in fact do drive more safely than the average driver (as in: at the end of your live, you will have had fewer/less severe accidents), it might happen that their predictive models group you in a different category, because characteristics of your driving behaviour that they use to categorize you are correlated with high-risk drivers. If insurers don't know how to (economically) measure why your driving behaviour is safe, it doesn't matter whether it actually is.

Also, the incentive can actually be a problem, exactly because risk models employed by ensurers tend to not be an exact representation of reality. If you have an incentive structure that does not align with reality, the incentive can end up promoting harmful behaviour. For example, one obvious proxy for safe driving habits could be lack of sudden decelaration. It's easy to measure, and generally, if you pay attention to traffic and drive with foresight, you usually will not need to brake suddenly as much as a reckless driver. So, it's probably true that both, incentivising people to not brake suddenly would have as one consequence people driving with more foresight, which should reduce accidents, and also that people who don't brake suddenly generally are a lower risk for the ensurer than those who do. However, this proxy can not distinguish whether you brake suddenly because you didn't pay attention--or because someone else didn't pay attention and surprised you. In the latter case, though, the thing to do to avoid an accident might be to brake as hard as you can. But that will be seen by your insurance as risky driving behaviour (which it most of the time is) that comes with a higher premium, so you have created an incentive for the driver to let an avoidable accident happen. Note that the driver in question won't think about this for an hour before deciding what to do, it's a gut reaction that might well be influenced by having internalized "braking hard costs money".

And also: What if you actually are a really good driver but you enjoy braking hard? What if you brake hard just for the fun of it, in situations where it's completely harmless. Is it fair if you have to pay higher premiums for that? Such incentives that work via proxy measurements of the actual risks tend to force adherence to a standard of behaviour. I find the idea frightening that insurance companies might get to dictate "safe behaviour", where the specific behaviour is not actually necessary for safety, it just happens to be easily distinguishable from risky behaviour, so behaving differently costs you money, simply because it's difficult to figure out that your behaviour is not actually risky.