I'm a physicist and I have published research in neuroscience. I don't feel like you understand the material you are talking about. Let's consider the question of formal verification. It is entirely possible for a computer to generate a proof for any proposition for which a person can generate a proof if by no other means than randomly printing out characters that represent the proof and then checking if the proof is correct. A proof is always a finite number of characters and the verification of a proof is always a finite number of steps (otherwise we wouldn't be able to write it down and to convince ourselves that the proof is true).

There are certainly problems which are hard for computers which are easy for people (and vice versa) but I don't know of any formal version of the claim that people can perform actions which are in principal impossible for human beings. You seem to be alluding to the claims of Penrose that humans can somehow do things machines cannot, but this isn't even what Penrose is claiming in the book. Instead, Penrose claims humans _somehow perceive_ the truth of propositions which cannot be proven in a given formal system, but this is a very vague claim which few scholars take seriously. He also thinks quantum mechanics is involved somehow, but this is also entirely vague. Even if one were to demonstrate that quantum mechanics were somehow involved in cognition there isn't any account that I know of that would describe how, per se, merely having a quantum computer involved would somehow introduce computations that classical computers could not do. I mean for one thing one can always simulate any quantum process on a classical computer. I'm not an expert in complexity theory, but as far as I know there are no known problems for which it is proven that there are no non-deterministic classical algorithms that perform as well as quantum algorithms. So we cannot even demonstrate with certainty that quantum computers are in fact better than classical computers with randomness. We only have situations where quantum algorithms are theoretically better than any known classical algorithm. It is also worth noting that most neuroscientists are dubious about the importance of quantum mechanical effects in cognition.

All that said, some people believe that there is some property of people that allows them to do things that computers cannot do (at the very least generate or have consciousness). But I don't know of any way of supporting that idea that doesn't involve dualism of some kind. That is, that asserts that there is something more than material that somehow constitutes a person. Like I said, a lot of people believe that there is. Most people. But a plurality of scientists do not believe this. I do not believe it. While I wouldn't say I'm a materialist, I would say I am a monist and that belief basically makes the assertion that a machine can do what a person does trivially true.

All that aside, I agree with you that language models cannot reason, almost certainly have no consciousness, and are have the propensity to be bad for people "spiritually" (speaking loosely of course). I'm not sure things are quite as dire as you assert. The moral rot of working in a call center, indeed, the existence of call centers, is surely worse than the imposition of an AI agent between the caller and the worker.