I don't think the parent's point is a stylistic criticism of the linked post.

Rather, I'd take the parent as saying that the linked post might cite the properties of current LLM systems but it ultimately isn't using them in it's arguments. Rather, it's same old argument - "start adding capabilities and boom suddenly you have an agent that take over the world".

The understanding we have of current LLMs is that they're very capable as text synthesizers but are quite random with any accuracy concerning the world. We've gone from GPT-2 to ChatGPT and the systems have gotten many times better (as smooth text synthesizers) but haven't gotten many times reliable in the particular descriptions of the world they give. They still say clearly wrong thing without prompting regularly - like every paragraph regularly for anything slightly obscure.

The main thing is that the rise of deep learning has actually highlighted goal accomplishment as a far more difficult task than classification, information retrieval and text/image synthesis. Self-cars keep getting mentioned and that's justified imo 'cause a huge amount of resources have been put into getting modern system to accomplish a fairly defined and limited "real world task" and all those efforts have mostly failed.

The key distinction of goal accomplishment is a system has engaging a very loop of making small judgements, each of which has be correct to an extraordinarily high degree. The linked text elides the difference between these tasks by talking of a vague breakthrough that makes the program "insightful". But we have to consider what's actually needed for what simple one wishes to do. Our present machine, ChatGPT, might, for example, give correct instructions for some complex auto repair (combining the patterns of several simple repair, maybe). But it couldn't "walk you through the process of doing the task" since it would continue it's tendency to say wrong things and such wrong things actually cause damage (like wrong turns in the self driving cars).

Your point and the linked text's point is that these things are insidious and can "get more complex" without one realizing it. But no one has made a reliable goal getting device out of just hooking a Turing machine to a neural network. The point is neural network today aren't "nerfed" in any way in the spectrum of possibilities, they're best people can do and they're making great progress in some measures but still by many fairly clear measures, failing to go beyond their limitations.

I'm not saying that it's impossible that a great advance happens involving making neural network reliable enough to accomplish goals (and to respond robustly the changes in the world, etc,etc). It's not impossible that it would happen at random but it seems more likely than any other advance happening at random. And also, if such an advance happens purposefully, there's no reason to think it will happen in a "you give it the ability to be much accurate and to seek goals but you know nothing about the goals it seeks" way.

And I'm familiar with the Lesswrong community. That group seems very tied to it's initial assumptions and it seems to fail to note the processes involved in current and potential future AIs. This is a long post so I'll just say one consistent error they make is assigning probabilities to fundamental unknowns. That's an abuse of the assumptions of any theory of probability and mostly results in a belief in whatever thing having some chance of appearing without explanation.