> If I understand correctly, that's what the OP, and others, are claiming, that it's possible to figure out how to represent meaning by putting words next to each other, if you look for long enough, at many enough words sitting next to other words. So far this remains a matter of debate.

The fact that we can have conversations with GPT where these models are able to correctly e.g. write working code and symbolically evaluate code is sufficient proof of that assertion that it is possible to figure this out that I don't think there is any reasonable basis for debating this any more. Claiming it's not possible is just demonstrably wrong.

> And, as Young et al (1976) have demonstrated, it is possible to move words out of their expected context for great fun and profit, but it's not clear that a word can always be moved in any context, and still make sense. I would even go so far as to say that can probably not be done, at least not without a drastic reconfiguration of all of English (where "ball" is used).

I don't see what this has to do with anything. Yes, we can shift around, alter their meanings, use them in unusual contexts and still make sense of it, and yes of course they won't always make sense in every context. How is that relevant?

> The objection I though you were making, with the "ball" etc, is that all these words that sit next to other words according to what the words mean, must mean something in isolation. For example, even though "ball" can be used in different contexts, it can only mean so many things, and it would never mean, say, "a cooking utensil made of aluminum where I fry my eggs each morning" no matter what other words you put it next to.

If I use it to mean "a cooking utensil", then it means a cooking utensil when I communicate with others that have that shared context. It has no meaning separate from context.

> So the question is not only "what other words do we find the word 'ball' close to" but also "why do we find 'ball' next to those other words?". The latter question can't be answered just by looking at what words hang out with what other words, unless we already know what words mean on their own.

"What words mean on their own" makes no sense. No word has a meaning separate from context. Without context there's nothing to assign them meaning. And that context is interactions. For GPT purely words. For us, words and other sensory input. In either case, without context assigning attributes to the word "ball" it is nothing more than a meaningless a sequence of letters. What does "JMw3rfd" mean? What's its intrinsic meaning? Until I tell you that this is my new word for "ball" it has none. Once I do, it has an extrinsic meaning. It's one that isn't widespread and will soon be forgotten, but the meaning exists the moment we label it, and only once we label it, and it is purely extrinsic.

> And that is, at least for me, the objection about LLMs "meaning" and "understanding" anything. No matter what text an LLM is generating, the entity that is decoding the meaning of the generating text is always a human. The LLM can't do that on its own. Because there is no mechanism that it is equipped with that could ever do that.

We don't know enough about human reasoning, and so how close how LLMs are to how human reasoning works to be able to even begin to determine whether this is true or false.