> 1. I'm thinking of it in terms of two different kinds of knowledge that we're going after when we try to (A) understand a proposition vs (B) make a probabilistic guess at the right answer from among a discrete range of possibilities. However our understanding of language works, it's not the same as flagging a maximized probability. If someone says "the bank is on Fifth Street next to the fire hydrant", you immediately know all kinds of things when you take this in - that you're getting directions, that you will soon be looking for a fire hydrant, the address of the bank, that the speaker knows the area pretty well... this big "bundle of truth" emerges from the proposition immediately when we take it in. I just don't see it as analogous to flagging the highest softmax probability score, which at best provides a "most likely" translation into SQL code or a vector, or something like that.

I certainly agree that it doesn't feel analogous, but i'm not sure that it's as clearly distinct as it intuitively seems. Certainly for most utterances you and I can imagine several plausible alternative interpretations. We are just very good at discerning the right one quickly and accurately, which means that we experience that determination as certainty, rather than probability. But that doesn't necessarily mean that we are perfectly collapsing it to a single meaning right away. In fact for many sentences, the meaning may change completely by adding/removing a word from the end, e.g.:

https://en.wikipedia.org/wiki/Garden-path_sentence

These are obviously not typical of all language, but I don't think it's unreasonable to say that all language in some sense behaves this way, and these types of sentences just make it unusually apparent to us.

> 2. If we're introducing concepts like insertion into the model, aren't we then returning to a logical (as opposed to a probabilistic/statistical) language model? Or said differently, aren't we merely introducing our own understanding of language into the model before it's even trained?

Sort of. This gets into some tricky territory about what concepts are, and what language actually is. But I think if you play around with word vectors a bit, you'll get a sense of what I mean here. Here's a good article summarizing some of their properties:

https://kawine.github.io/blog/nlp/2019/06/21/word-analogies....

You can see that in some sense conceptual essence is being captured statistically here. You can represent concepts as embeddings in a statistical space, where the angles within that space capture relationships between them. It gets kind of difficult to reason about exactly what this "is", but, such is the difficulty in using language to analyze and dissect language.

There can sometimes be a bit of a Homunculus fallacy that underlies the way people reason about language, concepts, and thinking, and I think the original article here is guilty of that. That is, people assert the presence of some immaterial essence to human reasoning or cognition in an implicit way, but don't really justify or define it. Like, what is the concept of 'insertion' really? Intuitively it feels like it has some essence that exists outside of its relations to other concepts. But the only thing we can truly directly talk about is its relationship to other concepts. In this sense, we can say that any given concept is defined by its relation to every other concept. Reifying this abstract notion in the language of mathematics, we can define this as angles in Hilbert space (though it's important to note, that this is not an assertion of essence, merely a descriptive and useful formalism). When we do things like that it tends to feel like these statistical/mathematical descriptions are not "really" capturing meaning, and while I don't want to positively assert that they are necessarily, I do want to challenge the idea that this is obviously true on its face. I think the evidence for this claim is actually pretty weak, when you really analyze it.

I found this article to be very helpful in thinking about a lot of these issues:

https://www.lesswrong.com/posts/9iA87EfNKnREgdTJN/conceptual...