This is very naive and misguided. I disagree completely.

To address the main points. Historically people use things in ways that they can, not how they ought to, this has happened every time no matter what people say or do beforehand. If the incentive for can is greater than ought; that is how it is used. This can and will inevitably replace humans. When that happens, people starve because the productivity gains (replacement) exceed market demand for jobs, and unrest is not long to follow when the velocity of money through markets stalls (factor->product->factor).

It is trivial to use for distortion of reflected appraisal, coercion and other intolerable acts, and political warfare. When you see or sense a lack of agency in a system you are interacting with, most of the time you are seeing the effects of coercion within an engineered system, someone at some point chose for you, and you weren't given a choice or had important things disclosed. Its coercive.

There's no incentive to fix failures once you reach a certain marketshare because it gives the illusion of following an 80/20 rule when its probably closer to 50/50.

Worse, these things lie. Usually not outright, but by omission, or structure. Relying on them as a magic bullet is just stupid. They are often wrong, but are right enough to have some illusion of credibility. They may even structure towards your psychological blindspots, for example, how you might compare two things.

Here's a neat little psychology experiment you can do about comparisons. Get 3 cups of water, a room temperature cup of water ,a separate hot (not scalding/too hot), and a cold glass of ice water, and put one hands finger (30s) in the cold, the other one in the hot , and then remove them and place both immediately in the same room temperature cup, what do you feel.

You know the room temperature cup is the same but it certainly doesn't feel that way. There are plenty of little biases like this. AI doesn't need to know about them to pick them up because its just weights, it can be embedded unintentionally.

There are many other such biases that are very well documented. If you've ever seen a master shady used car salesman at work, imagine an AI that can do what they do; where you can't walk away because it follows you everywhere.

The more plausible something may be, the more effort and time you need to spend validating it, and not just validating what's given (because that's the easy part); but what's not given, which is much harder and requires domain knowledge which you likely don't have.

The idea that AI/Chat-GPT creates productivity gains is just flawed. It simply makes those costs hidden costs, and gives companies a plausible excuse for the coercion of their customers, under the argument: it can't possibly be intentional if an AI does it... yes it can, because they designed and approved it as a suitable replacement.

The rate of change in development of these things also makes this now a runaway train because we can no longer react quick enough to address any issues. It will keep going at full speed until we all go over the cliff, and the societal systems we depend upon for survival fail. The stupidity of profit that ended the world.