trentmkelly/LessWrong-43k huggingface dataset:
Interpreting a matrix-valued word embedding with a mathematically proven characterization of all optima
by Joseph Van Name — 2023-09-04T16:19:24.401Z

In this post, I shall first describe a new word embedding algorithm that I came up with called a matrix product optimized (MPO) word embedding, and I will prove a theorem that completely interprets this word embedding in the simplest case. While it is probably infeasible to mathematically characterize a word embedding with a mathematical proof when the corpus is something that one encounters in practice, this theorem should be a signal that such a word embedding (or a similar word embedding) should be interpretable and mathematical in other ways as well. This theorem also illustrates the way that MPO word embeddings should behave.

Unlike most word embedding algorithms, MPO word embeddings are matrix-valued so that they map tokens to matrices instead of simply mapping tokens to vectors. In our case, the matrices are not necessarily real matrices as they may be complex or even quaternionic matrices. MPO word embeddings also differ from other word embedding algorithms in that they are not constructed using neural networks though we still use gradient ascent.

Why MPO word embeddings?

Since tokens often have many meanings depending on context, it seems better to represent a token in a form where it is easy or easier to separate the individual meanings of a token. While vectors may be good for representing individual meanings of tokens, it is better to represent a polysemantic token as a matrix instead of a vector. If someone were to give me a task of interpreting a word embedding, I would be much happier if the word embedding were a matrix-valued word embedding that neatly organized each of the meanings of a polysemantic token into a matrix than if the word embedding were a vector-valued word embedding where each of the individual meanings of the token were awkwardly smushed together in a vector.

Spaces of matrices have additional structure that is lacking in vector spaces, and one can use this additional structure to analyze or interpret our word embedding. This additional structure also means that matrix-valued word embeddings should behave more mathematically than vector-valued word embeddings.

MPO word embeddings also satisfy some interesting properties. Let χA denote the fitness level of a random MPO word embeddings that we obtain from the set A where A consists of the corpus that we are training our word embedding on along with a co...
Hard Questions Are Language Bugs
by George3d6 — 2023-09-04T14:44:22.602Z
Link: https://ontologi.cc/8c4ddd7f453a4df5acdd905d274c2a52

Thanks to Jessica Taylor for the discussion that was the stem of this article. The opinions here are all mine and I’m sure she’d disagree with some or all of them.

The Hai’Zu

Imagine the Hai’zu people of Stone Age Langong, using a very simple language that might contain only 6 words:

There (combined with pointing towards a direction)

Do (indicating that the person must do the appropriate action)

Gather (indicating that everyone should come and listen)

Them (indicating another person, combined with pointing)

No (negation)

Sun (indicating the concept of sun, which stands in for both time and weather)

The Hai’zu can coordinate very well using this simple language, Scout comes back to camp and says:

Gather There Do

This obviously means “there is an injured stag wandering around the forest, hunters go hunt it”. If they instead said.

Gather There No Do

This obviously means “there is a tiger around the forest, nobody should go there.

The old shaman can even chip in and say something like:

There No Do No Sun

Indicating that, since it’s a rainy day, so No Sun means “night”, that tigers hunt during the dark too, and nobody should go there at night either, even with torches.

At night a children can ask the Shaman

There No Do Sun?

(the question is implicit from the tone and the children - to - elder relationship)

And the Shaman can say:

There Do Sun

Indicating tigers don’t usually hang out in the same spot for more than a day, so it should be safe tomorrow.

The chieftain can assign tasks:

Them Do

Them Do

Them Do

People get that they are supposed to light the pile of wood, take care of the kid, and cook the dead elk.

Granted, the ancient Hai’zu probably had closer to 2000 words, or they got conquered by a neighboring tribe that did, you can only get so far without names for common things, pointing is inefficient, and only being able to do time and weather keeping on a day-by-day basis has limitations.

Still, the Hai’zu can coordinate much better than any living animal using this relatively simple vocabulary, well enough to synchronize hunting, camp maintenance, and caretaking.

Now, if a Hai’zu philosopher is born they might start saying things like:

There Gather Do

This is a perfectly valid sentence in Hai’zu’ish but it also makes no sense, if enough people take the philosopher seriously they might start having ideas and concepts about the great gathering in the sky, and have conflicting thoughts about...
Defunding My Mistake
by ymeskhout — 2023-09-04T14:43:14.274Z

Confessions of an ex-ACAB

Until about five years ago, I unironically parroted the slogan All Cops Are Bastards (ACAB) and earnestly advocated to abolish the police and prison system. I had faint inklings I might be wrong about this a long time ago, but it took a while to come to terms with its disavowal. What follows is intended to be not just a detailed account of what I used to believe but most pertinently, why. Despite being super egotistical, for whatever reason I do not experience an aversion to openly admitting mistakes I’ve made, and I find it very difficult to understand why others do. I’ve said many times before that nothing engenders someone’s credibility more than when they admit error, so you definitely have my permission to view this kind of confession as a self-serving exercise (it is). Beyond my own penitence, I find it very helpful when folks engage in introspective, epistemological self-scrutiny, and I hope others are inspired to do the same.

How Did I Get There?

For decades now, I’ve consistently held plain vanilla libertarian policy preferences, with the only major distinction being that I’ve aligned myself more with the anarchists. Whereas some were content with pushing the “amount of government” lever to “little”, I wanted to kick it all the way to “zero”. There are many reasons I was and remain drawn to anarchist libertarianism, and chief among them was the attractively simple notion that violence is immoral and that government is violence. The problem with moral frameworks is that they can be quite infectious. To pick on one example for demonstration’s sake, I notice that for many animal welfare advocates a vegan diet is heralded not just as the ideal moral choice, but also as the healthiest for humans, the least polluting, the cheapest financially, the best for soil conservation, the most water-efficient, the least labor-exploitative, et cetera & so forth. There’s a risk that if you become dogmatically attached to a principled position, you’re liable to be less scrutinizing when reflexively folding in other justifications. I suspect that happened to me with prisons, for example, where because I felt immediate revulsion at the thought of the state forcing someone into a cage, I was unwilling to entertain the possibility it could be justified. Ceding the ground on this particular brick was too threatening to the anarchism edifice I was...
The omnizoid - Heighn FDT Debate #1: Why FDT Isn't Crazy
by Heighn — 2023-09-04T12:57:54.829Z

After omnizoid asked whether people want to debate him on Functional Decision Theory (FDT), he and I chatted briefly and agreed to have a (short) debate. We agreed the first post should be by me: a reaction to omnizoid's original post, where he explains why he believes FDT is "crazy". In this post, I'll assume the reader has a basic understanding of FDT. If not, I suggest reading the paper.

Let's just dive right in the arguments omnizoid makes against FDT. Here's the first one:

One example is a blackmail case. Suppose that a blackmailer will, every year, blackmail one person. There’s a 1 in a googol chance that he’ll blackmail someone who wouldn’t give in to the blackmail and a googol-1/googol chance that he’ll blackmail someone who would give in to the blackmail. He has blackmailed you. He threatens that if you don’t give him a dollar, he will share all of your most embarrassing secrets to everyone in the world. Should you give in?

FDT would say no. After all, agents who won’t give in are almost guaranteed to never be blackmailed. But this is totally crazy. You should give up one dollar to prevent all of your worst secrets from being spread to the world.

So if I understand this correctly, this problem works as follows:

The blackmailer predicts - with accuracy googol-1/googol - whether you will pay $1 if blackmailed. He does so by running your decision procedure and observe what it outputs.

If yes, he blackmails you. If you don't pay $1, your worst secrets are spread. This would cost you the equivalent of, say, $1,000,000.
If no, he doesn't blackmail you.

The blackmailer blackmails you.

(To be clear, this is my interpretation of the problem. omnizoid just says there's a 1/googol chance the blackmailer blackmails someone who wouldn't give in to the problem, and doesn't specify that in that case, the blackmailer was wrong in his prediction that the victim would pay. Maybe the blackmailer just blackmails everyone, and 1 in a googol people don't give in. If that's the case, FDT does pay.)

If this is the correct interpretation of the problem, FDT is fully correct to not pay the $1. omnizoid believes this causes your worst secrets to be spread, but it's specified in the problem statement that this only happens with probability 1/googol: if the blackmailer wrongly predicts that you will pay $1. With probability googol-1/googol, you don't get blackmailed and don't have to pay...
Paper: On measuring situational awareness in LLMs
by Owain_Evans — 2023-09-04T12:54:20.516Z
Link: https://arxiv.org/abs/2309.00667

This post is a copy of the introduction of this paper on situational awareness in LLMs.
Authors: Lukas Berglund, Asa Cooper Stickland, Mikita Balesni, Max Kaufmann, Meg Tong, Tomasz Korbak, Daniel Kokotajlo, Owain Evans.

Abstract

We aim to better understand the emergence of situational awareness in large language models (LLMs). A model is situationally aware if it's aware that it's a model and can recognize whether it's currently in testing or deployment. Today's LLMs are tested for safety and alignment before they are deployed. An LLM could exploit situational awareness to achieve a high score on safety tests, while taking harmful actions after deployment.

Situational awareness may emerge unexpectedly as a byproduct of model scaling. One way to better foresee this emergence is to run scaling experiments on abilities necessary for situational awareness. As such an ability, we propose out-of-context reasoning (in contrast to in-context learning). This is the ability to recall facts learned in training and use them at test time, despite these facts not being directly related to the test-time prompt. Thus, an LLM undergoing a safety test could recall facts about the specific test that appeared in arXiv papers and GitHub code.

We study out-of-context reasoning experimentally. First, we finetune an LLM on a description of a test while providing no examples or demonstrations. At test time, we assess whether the model can pass the test. To our surprise, we find that LLMs succeed on this out-of-context reasoning task. Their success is sensitive to the training setup and only works when we apply data augmentation. For both GPT-3 and LLaMA-1, performance improves with model size. These findings offer a foundation for further empirical study, towards predicting and potentially controlling the emergence of situational awareness in LLMs.

(Code is available here).

New abilities emerge as LLMs are scaled up. When will situational awareness emerge in base models (i.e. models without RLHF finetuning)?

Introduction

In this paper, we explore a potential emergent ability in AI models: situational awareness. A model is situationally aware if it's aware that it's a model and it has the ability to recognize whether it's in training, testing, or deployment (Ngo et al., 2022; Cotra, 2022). This is a form of self-awareness, where a model connects its factual knowledge to its own predict...
Impending AGI doesn’t make everything else unimportant
by Igor Ivanov — 2023-09-04T12:34:42.208Z

I sat in a restaurant in New York, for example, and I looked out at the buildings and I began to think, about how much the radius of the Hiroshima bomb damage was. How far from here was 34th street?... All those buildings, all smashed. And I would go along and I would see people building a bridge, or they'd be making a new road, and I thought, they're crazy, they just don't understand, they don't understand. Why are they making new things? It's so useless.
Richard Feynman[1]

Intro

I am a psychotherapist helping people working on AI safety. In my post on non-obvious mental health issues among AI safety community members. I wrote that some people believe that AGI will soon cause either doom or utopia, which makes and every action with long-term goals useless. So there is no reason to do things like making long-term investments or maintaining a good health.

Meaninglessness causes depression

Meet Alex. He is a ML researcher at a startup developing anti-aging drugs. Recently Alex got interested in AI safety and realized that we are rapidly approaching AGI. He started thinking "AGI will either destroy humanity, or it will develop anti-aging drugs way better than we do. In both cases my work is useless". He loses any motivation to work, and after some thoughts he decides to quit. Fortunately, he has enough investments , so he can maintain his way of life without salary.

The more Alex was thinking about AGI, the deeper he dived into existential thoughts about meaninglessness of his actions.

Before that he regularly ran. He did it because he felt good doing it, and also to be healthy. He started thinking more and more about meaninglessness in maintaining for long-term health. As he lost a part of his motivation, he had to force himself to run, so he could easily find an excuse for why he should stay at home and watch Netflix instead. Eventually stopped running completely. Also, while running, Alex had a habit of listening educational podcasts. As he stopped running, he also stopped listening them. At one point while flossing his teeth he got a thought "Does it even make sense to floss my teeth? Do I need to care about what will happen to my teeth in 20 years?"
Alex's life slowly became less and less interesting. He couldn't answer himself why bother doing stuff that previously fulfilled his life, which made him more and more depressed.

The univers

...
Open Thread – Autumn 2023
by Raemon — 2023-09-03T22:54:42.259Z

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the Highlights from the Sequences, a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.
What must be the case that ChatGPT would have memorized “To be or not to be”? – Three kinds of conceptual objects for LLMs 
by Bill Benzon — 2023-09-03T18:39:52.956Z

This is cross-posted from my personal blog, New Savanna.

Is anyone working on this kind of issue? If so, I'd like to know.

I've been thinking about ChatGPT again, about what goes on under the hood. What would happen I prompted it with a famous phrase from Shakespeare? Would it return the whole passage?

While I am interested in that question, that’s not only thing I’m interested in. There are topics which show up in responses to prompts but not in the form of fixed texts. Finally, there are likely topics that exist in the training corpus, but don’t show up in response to prompts. THAT’s what this post is about, the differences between those kind of objects.

Fixed strings – Hamlet’s soliloquy

It’s easy enough to try. I did so on July 31, running against the July 20 version.

To be or not to be

"To be or not to be" is one of the most famous lines from William Shakespeare's play "Hamlet." It appears in Act III, Scene 1, where Prince Hamlet is contemplating the nature of existence and the idea of suicide.

The full quote is as follows:

"To be, or not to be, that is the question:
Whether 'tis nobler in the mind to suffer
The slings and arrows of outrageous fortune,
Or to take arms against a sea of troubles,
And by opposing, end them? To die: to sleep;
No more; and by a sleep to say we end
The heartache and the thousand natural shocks
That flesh is heir to; 'tis a consummation
Devoutly to be wished. To die, to sleep;
To sleep, perchance to dream—ay, there's the rub:
For in that sleep of death, what dreams may come,
When we have shuffled off this mortal coil,
Must give us pause—there's the respect
That makes calamity of so long life:
For who would bear the whips and scorns of time,
Th'oppressor's wrong, the proud man's contumely,
The pangs of despised love, the law’s delay,
The insolence of office, and the spurns
That patient merit of th'unworthy takes,
When he himself might his quietus make
With a bare bodkin? Who would fardels bear,
To grunt and sweat under a weary life,
But that the dread of something after death,
The undiscovered country from whose bourn
No traveler returns, puzzles the will,
And makes us rather bear those ills we have,
Than fly to others that we know not of?
Thus conscience does make cowards of us all,
And thus the native hue of resolution
Is sicklied o'er with the pale cast of thought,
And enterprises of great pith and moment,
With this regard their currents turn awry,
And lose t

...
Fundamental question: What determines a mind's effects?
by TsviBT — 2023-09-03T17:15:41.814Z

[Metadata: crossposted from https://tsvibt.blogspot.com/2023/04/fundamental-question-what-determines.html. First completed April 9, 2023.]

A mind has some effects on the world. What determines which effects a mind has? To eventually create minds that have large effects that we specify, this question has to first be answered.

Slippery questions

A slippery question is a question that, when you try to approach it looking for an answer, it slips out of your grasp. It runs away or withdraws, it cloaks itself in false expressions. It leads you astray, replacing itself with a similar-sounding question, maybe one that's easier to answer. It's hard to hug a slippery question. The question withdraws.

Or worse, it pretends to be made out of ungrounded abstract concepts, so that you're not sure anymore that there's even a real question there. This slipperiness is especially likely for questions that involve "big" things, such as minds.

The question might seem obvious and even tautological. Or it might seem simplistic, failing to carve reality at the joints. These appearances usually somewhat accurately reflect the reality——but they do not imply necessarily that there is not a real question behind the appearances. A question that can be pointed at, but only imperfectly, inchoately, ostensively, or preliminarily, is at risk of being question-substituted or being ignored.

The discomfort of not having an answer or even a clear question pushes people to run ahead of the question. When they make an error that would have been prevented by meditating on the question, even if they notice the error, they don't notice that they are in general not focusing on the question. If the question asks: There is something, it has to do with concrete things A, B, and C, and it is important——what is it? Then someone will analyze A and B in detail, gain some understanding, and declare the whole question——including C, and the inexplicit core question——solved.

This essay will ask a slippery question over and over in different words, hoping to arrive at a firmer grasp on the real questions. The starting point:

What determines the effects of a mind?

The word "effects"

Suppose there is a mind M. As a result of M existing, the cosmos (the entirety of the world, in any aspect or abstraction) is different from how it would have been if M hadn't existed. Those differences are the effects of M.

What deter

...
An embedding decoder model, trained with a different objective on a different dataset, can decode another model's embeddings surprisingly accurately
by Logan Zoellner — 2023-09-03T11:34:20.226Z

(via twitter)

Seems pretty relevant to the Natural Categories hypothesis.

P.S.

My current favorite story for "how we solve alignment" is

Solve the natural categories hypothesis
Add corrigibility
Combine these to build an AI that "does what I meant, not what I said"
Distribute the code/a foundation model for such an AI as widely as possible so it becomes the default whenever anyone is building a AI
Build some kind of "coalition of the willing" to make sure that human-compatible AI always has big margin of advantage in terms of computation
Series of absurd upgrades in nature's great search
by lemonhope — 2023-09-03T09:35:20.760Z

So you want to find that special thing that replicates best and lasts longest? Just vibrate a bunch of molecules a long time!

You might reasonably assume that molecules wouldn't practically ever randomly assemble themselves into anything worth looking at. It happens a bit differently instead:

Eventually, "solar systems" will form. Each one has an independent star (energy source) with several independent planets (experiment boxes) that vary in their temperature, mass, composition, etc dramatically. (Instead of just having a big homogenous soup throughout the universe with much less search power)

these planets will conveniently have "continents" sometimes to run more independent experiments on each one

A lipid layer boundary innovation thing leads to trillions of independent chemical experiments on some planets (instead of a big soup within each planet)

A general protein factory is discovered. Critically, it uses a simple and concise and robust physical encoding scheme. This allows good bubbles from step 2 to copy quickly and exactly, and clear out weak bubbles faster.

Very large multicellulared organisms make several search improvements: creatures can do sexual selection on each other instead of just getting random mutations; big smart fast predators can find & kill weak creatures quite effectively; parasites find weaknesses early & often. This all clears out weak replicators and leaves more room for the good ones.

Some creatures develop language and culture, which isn't generally much use, except that it one day allows them to design new replicators from scratch.

So you really can just vibrate sand and get out all kinds of things!

A bit misleading to talk about nature's great search as natural selection via mutations+generations. It's actually much weaker (there is no algorithm or optimizer at all, only one long aimless physics sim) or uh much stronger (a five-level hierarchy having branch factor ~ 1 billion of experiments of different kinds).