> I have a hard time connecting the dots in your argument?

Unfortunately, the root of the issue is that the line is very thin and hard to see. Which is why it gets crossed so often and only realized post hoc.

I apologize for the lengthiness that is to follow.

For a desk, it's something that seems nowhere near nefarious, even if it leaks to the internet, right? Well we could imagine a world where your boss can see this data because they set up such a system inside the office. Probably with some good intent too like raising all the desks at night to help janitors clean under them or something like that. But a new manager comes in and uses that data as a means of determining how much you are working. Might even implement policies like having to stand up and sit down every so often because they are reading blog posts on worker productivity and sitting too long. It can be "in the best interest" but can quickly become abusive. The concept of "turnkey tyranny" isn't limited to governments and it's important to remember tyrants don't view themselves as evil. More often they just view themselves as "better", "know what's best", or "ends justify the means." Generally it is an extreme version of Main Character Syndrome (which MCS is not too uncommon in our communities...)

As a clearer example, where we have the advantage of hindsight, I think social media can fit in here. I very much don't think social media started with bad intentions and honestly, I don't think people today working on it have bad intentions. It's more the problem with "the road to Hell is paved with good intentions." Social media does good, it brings people together across the world, and allows us to have competing (in a healthy way or unhealthy) views. We're doing it this very instant! But one issue at the core of the problem is "engagement." We know these algorithms optimize for is, but what's it mean? We can see how it is based on activity. Things like post interactions, shares, commenting, likes, and so on. But think about this problem hard. How do you __actually__ measure engagement? How do you __actually__ differentiate "good" engagement from "bad" engagement? Can you find natural language words for this? I bet you're going to have a really hard time. And then an even harder time expressing this in code. So what happened? Well we got an overly simplified version (obviously, given that this concept is essentially intractable and highly dynamic), and got metric hacked.

These people are trying to do the right things, but just have a limited scope. It's worse if they don't realize their scope is limited (see MCS). People who don't understand the metrics just understand that they are metrics and so optimize for them. Managers can dismiss engineer complaints because these people can't speak the same language and are working at different levels of abstractions. It's a whole complicated mess where the main issues come down to the chaotic interaction. It is almost never someone seeking out to perform evil, but people who can't see beyond themselves (which is hard!). And even when we can see beyond ourselves it is hard to see very far. Be that into the future, into other "tribes", or whatever. The best defense we have against these things is to stay defensive, communicate, acknowledge concerns (even if moving forward. Just means tread carefully), and to balance short term rewards and long term (short term is easy to measure, but can easily lead you away from huge future payouts). We live in such a complex world that to maximize your objectives you often need to maximize other peoples objectives in certain situations[0], because if we treat everything as "zero-sum", "single round", or similar naive approximations, we will push ourselves away from our goals. This is the tyranny of complexity, how trying to do what's best is always far harder than it seems. And worst of all, it's incredibly hard to measure the impact of your decisions through these chains of interaction.

So if all this was hard to understand, I get it, because that's also the problem. It's the little things adding up, not any one or even a few big thing/s.

[0] https://www.youtube.com/watch?v=goQ4ii-zBMw