> A lot of what we think we “understand” is actually memorized.

For sure. But these definitions are not well defined, despite thousands of years of research. One point I think everyone in that chain would agree upon is that with respect to understanding, a thing would not explain its answer by giving evidence to the contrary. If someone did so you rightfully would say "You don't know [understand] what you're talking about."

> Empirically, understanding can be tested with a novel set of challenge questions.

I have to stop you right there. Empirical measures are always signals, not proofs. This too has been long established for hundreds of years and is the reason many engineers and experimental physicists fight. Why experimental physicists and theoretical physicists fight. Empirical evidence is always limited to its context window and we have to be VERY clear at what those limitations are. This is the lesson of Goodhart's Law, not that people will exploit the measure (though that is also important). Every intro ML course shows RL agents metric hacking, and this is always a result of a limited context.

So I want to be clear, that empirical data doesn't test a hypothesis but rather tests the null-hypothesis. That's important because the set of possible hypotheses is just decreased through empirical testing, rather than reducing the set of possible results a single element set. It rejects hypotheses, not proves them.

We must also understand that testing understanding is similarly an unsolved problem. One too that has been questioned for millennia.

> Anything further, any attempt to peer inside the box (other than to identify repeatable flaws), is not a productive use of time.

This is strange to me. The parenthetical statement is unbounded and justifies what you are arguing against. Second, just because we can't prove a specific hypothesis (in most cases) doesn't mean it isn't useful. Rejection is how the vast majority of science works and this has been highly successful. Limiting potential is clearly a helpful tool as it increases your odds of a correct answer (which is why it is a common strategy for multiple choice testing).

> Further, we should keep in mind that LLMs aren’t usually actually regurgitating training data verbatim.

Yes, this is the stochastic part of the term stochastic parrot. No reasonable researcher is suggesting that LLMs only recite. Every one of us recognizes that they can generate things that were not handed to it. In fact, a hallucination is an explicit example of this. The failure to respond to my answer correctly is an example of it generating new data. (It is also an example of it following a statistical pattern).

But we do also need to be careful about our distinctions of generalization vs overfitting. There are clearly certain areas that are overfit (as I demonstrated). This is extra difficult in models that have been trained on datasets which you are not privy to. But we can also see good examples of how the code LLMs are overfit. I have written a number of comments on this site about exactly this, which you're welcome to search my history for (see "HumanEval" with my name). Here's one such comment id=35806152