> I don't know how relevant this argument even is anymore considering I was wrong about the move rate. Is 1 in ~1650 really underperforming a 1800 Elo Human ? How much so ?

Yes, I would still be surprised if an 1800 elo human makes 1/1650 illegal moves during formal matches.

> If "first principles" were what we thought they were, we wouldn't be using GPT or artificial neural networks for nearly all computer intelligence tasks. We would be using symbolic systems.

This would only be true if you believe in a singular definition of intelligence where you believe there is only one right way to get an answer. There are lots of reasons we don't train AI from first principles, one of the biggest being that neural networks are just flat-out easier to train. Training from first principles is annoying and time consuming and requires either an understanding of first principles or some simulation or methodology to consistently demonstrate those principles to the AI. That is hard to do. But first principles and predicate logic haven't failed as a way to accomplishing tasks or as a way to reason about problems. First principles are ill-suited for an extremely specific field.

And now people are acting like however AIs work that has to necessarily match how humans learn? That's not the case, what I am getting at when I talk about diversity of intelligence is that you don't need to map to human reasoning in order to build tools that are useful and that generally solve certain categories of tasks, sometimes even better than humans do.

It's incredibly limiting to define what kinds of intelligence do and don't exist in the world based purely on what is useful for training an AI.

----

Humans observably do not learn the same way as GPT does, we do not start out by learning language and then logic, we do the opposite. Any study of any infant will show you that humans develop logical capabilities before we develop language skills. However that process is happening, it factually, observably happens in the opposite order than what we see occurring in LLMs.

We do not need to know exactly how that process works to know that it is different from how GPT learns. And we certainly don't need to look at a poorly understood process and assume that it necessarily must match whatever the current state-of-the-art methodology we have to simulate it. As you accurately say, we do not know how to define intelligence broadly -- so it's wildly reductive to say "but we do know how to build neural networks, and when we tried to replicate first principles or composite logic that didn't work well, so probably that doesn't exist" -- I just don't view that as a serious take.

I still feel like you're falling into the trap of getting really invested into showing that GPT is meeting some human standard of reasoning, and I don't understand why it matters whether or not GPT is meeting that standard. Why is it a problem to say, "we don't understand how humans work, but we're seeing surprising differences between GPT's capabilities and the capabilities of comparable humans, and that suggests there is something very different going on under the hood."?

Does that actually make GPT any less interesting or useful? I don't think so.