Here goes: LLMs don't behave, they exhibit behavior.

Who's behavior? Whoever wrote the training corpus and the prompt. So far, that definitely means one or more humans.

The problem is that nearly everything you have read about LLMs has personified them. The character of an LLM as a person is invented, then conclusions about LLMs are drawn from that character.

Nearly every thing that is interesting about LLMs is not actually an LLM feature: it's a language feature. Language is an incredibly powerful tool, and an incredibly versatile data format.

What is the data? Human behavior.

When a human writes text, they don't just spit out characters at random: humans explicitly choose the characters they write, and implicitly choose the characters they don't write.

In the most literal sense, text contains the entropy of a human making a string of choices for which character to write next. In a literal sense, that's a 1-dimensional ordered list; just like the string of characters is. Text gets a lot more interesting when we introduce language...

The reason a human chooses one character over another is not 1-dimensional. There are patterns of entropy driving that decision, and we explicitly know about some of them.

We have defined words, punctuation, grammar, idioms, etc. that we structure text with. We call these patterns language. These are the explicit patterns that allow us to hack the writing process: instead of encoding a 1-dimensional list of decisions, we encode a recursive many-dimensional structure of symbols and grammar.

Now for the interesting part: when humans write language into text, our intentions become explicit patterns, but they aren't the only patterns that end up there.

Every arbitrary decision about writing style gets implicitly encoded as a pattern of negative space. The reasons why we write one concept instead of another get implicitly encoded, too. It's a bit lossy, but most of these patterns end up in the text.

---

So what does an LLM do? It implicitly models patterns, and presents them.

That's it. That's the only behavior. A "continuation" is made by modeling the prompt, and showing what's "next" in the model.

An LLM explicitly knows nothing at all; but implicitly, it knows everything. It models every pattern that exists in the text, but it doesn't have a clue what or why it is modeling. A pattern is a pattern.

The whole system ends up generating valid language, because most of the implicit patterns an LLM models "align to" the explicit patterns humans (language) encoded into the text.

The whole system ends up "exhibiting behavior", because it also modeled the implicit patterns of human behavior (narrative) that were are also encoded into the text. Patterns of narrative determine which part of the model a prompt explores, and which pattern is considered "next".

---

Next time you read about the "features" and the "limitations" of an LLM (including GPT-4), know that you are really reading about the "features" of language itself, and the "features" of the narratives that were written in the training corpus and prompts.

The behavior that an LLM exhibits has much less to do with how well its model "aligns with" language grammar, and much more to do with how well the text itself (and the narrative it contains) will behave.