trentmkelly/LessWrong-43k huggingface dataset:
Ideation and Trajectory Modelling in Language Models
by NickyP — 2023-10-05T19:21:07.990Z

[Epistemic Status: Exploratory, and I may have confusions]

Introduction

LLMs and other possible RL agent have the property of taking many actions iteratively. However, not all possible short-term outputs are equally likely, and I think better modelling what these possible outcomes might look like could give better insight into what the longer-term outputs might look like. Having better insight into what outcomes are "possible" and when behaviour might change could improve our ability to monitor iteratively-acting models.

Motivation for Study

One reason I am interested in what I will call "ideation" and "trajectories", is that it seem possible that LLMs or future AI systems in many cases might not explicitly have their "goals" written into their weights, but they might be implicitly encoding a goal that is visible only after many iterated steps. One simplistic example:

We have a robot, and it encodes the "goal" of, if possible, walking north 1 mile every day

We can extrapolate that the robot must "want" to eventually end up at the north pole and stay there. We can think of this as a "longer-term goal" of the robot, but if one were to try find in the robot "what are the goals", then one would never be able to find "go to the north pole" as a goal. As the robot walks more and more, it should become easier to understand where it is going, even if it doesn't manage to go north every day. The "ideation"/generation of possible actions the robot could take are likely quite limited, and from this we can determine what the robot's future trajectories might look like.

While I think anything like this would be much more messy in more useful AIs, it should be possible to extract some useful information from the "short-term goals" of the model. Although this seems difficult to study, I think looking at the "ideation" and "long-term prediction" of LLMs is one way to try to start modelling these "short term", and subsequently, "implicit long-term" goals.

Note that here are some ideas might be somewhat incoherent/not well strung together, and I am trying to gather feedback on might be possible, or what research might already exist.

Understanding Implicit Ideation in LLMs and Humans

In Psychology, the term "ideation" is attributed to the formation and conceptualisation of ideas or thoughts. It may not quite exactly be the best word to be used in this context, though i...
A well-defined history in measurable factor spaces
by Matthias G. Mayer — 2023-10-05T18:36:05.037Z

PDF

This is a technical post researching infinite factor spaces

In a factor space F=⨉i∈IFi, for a measurable
Z:F→Z(F), a Z-measurable
index function J:F→{0,1}I is generating
X:F→X(F), if X depends
only on those arguments (xi), where J(x)i is 1, and
Z. In this post, we show that there is an almost surely
minimal such index function that we call the history of X
given Z.

We will reintroduce all the definitions from the finite case:

Let I be finite. Let
(Fi,Ai,Ni) be a measure
space with nullsets, i.e. Fi is a set, Ai is a
σ-algebra and Ni is a σ-ideal that admits a
probability, i.e. there is Pi with
Ni={A∈A:Pi(A)=0}.

We define a product σ-ideal:
N1⊗N2={A∈A:{x:A(x)∉N2}∈N1}.
Clearly,
N1⊗N2={A∈A:P1×P2(A)=0}.
We can extend this by induction to
⨂i∈INi.

We construct
(F,A,N)=(⨉i∈IF,⨂i∈IAi,⨂i∈INi).
Furthermore, let P=⨉i∈IPi.

Definition 1 (almost sure subset)

We will write
A∗⊆B:⇔A∖B∈N.

Definition 2 (almost sure union with arbitrary index set)

If N is a σ-ideal that admits a probability P and
(Aj)j∈J∈AJ is a family of
measurable sets, then there exists an almost surely unique and minimal
A∈A with
∀j∈J:Aj∗⊆A. i.e. whenever
∀j∈J:Aj∗⊆B, we have
A∗⊆B. We set
∗⋃j∈JAj≔A. Furthermore, there is
J∗⊆J countable, s.t.
⋃∗j∈JAj=⋃j∈J∗Aj.

Proof. We will construct J∗.

To start, choose any j∈J, J0={j} and let
B0=Aj.

Let Bn be defined and let
sn=supj∈JP(Aj∖Bn).
Choose jn∈J with
P(Ajn∖Bn)≥sn−1n
and set Jn+1=Jn∪{jn} and
Bn+1=Bn∪Ajn.

We set J∗={jn:n∈N} and
A=⋃j∈J∗Aj.

A contains Aj: Let j∈J, we have to show
P(Aj∖A)=0. Assume not, then
P(Aj∖Bn)↓P(Aj∖A)=p>0,
so sn≥p for all n. Therefore, we have
P(Ajn∖Bn)≥p−1n.
Now
1≥P(A)=P(⨃n∈NAjn∖Bn)=∑n∈NP(Ajn∖Bn)=∞,
which is a contradiction.

Minimality and uniqueness: Now let
B∈A:∀j∈J:Aj∗⊆B.
Then clearly,
A=⋃n∈NAjn∗⊆B.
If B is also minimal, then clearly
A∗⊆B∗⊆A⇔P(A▲B)=0⇔Aa.s. =B.

Definition 3 (almost sure intersection)

We set
⋂∗j∈JAj=(⋃∗j∈JAcj)c.

Definition 4 (feature)

We call a measurable function
Z:F→Z(F) a feature. In the
following let X,Y,Z be features.

Definition 5 (index function)

We call a measurable J:F→{0,1}I an
index function. We write Ji=πi∘J.
We identify J with J′:F→P(I), where
J′(x)={i∈I:Ji(x)=1} to allow for set operations such as
J∩K=min(J,K).

Definition 6

For an index function J, we set
πJ=(1{Ji=1}πi)i∈I.

Definition 7

We define that X is almost surely a function of
Y by
X≲∗Y:⇔σ(X)⊆σ(Y,N).

Let ▲∗(F,N) be all product distributions whose
nullsets...
Evaluating the historical value misspecification argument
by Matthew Barnett — 2023-10-05T18:34:15.695Z

ETA: I'm not saying that MIRI thought AIs wouldn't understand human values. If there's only one thing you take away from this post, please don't take away that. Here is Linch's attempted summary of this post, which I largely agree with.

Recently, many people have talked about whether some of the main MIRI people (Eliezer Yudkowsky, Nate Soares, and Rob Bensinger[1]) should update on whether value alignment is easier than they thought given that GPT-4 seems to follow human directions and act within moral constraints pretty well (here are two specific examples of people talking about this: 1, 2). Because these conversations are often hard to follow without much context, I'll just provide a brief caricature of how I think this argument has gone in the places I've seen it, which admittedly could be unfair to MIRI[2]. Then I'll offer my opinion that, overall, I think MIRI people should probably update in the direction of alignment being easier than they thought in light of this information, despite their objections.

Note: I encourage you to read this post carefully to understand my thesis. This topic can be confusing, and there are many ways to misread what I'm saying. Also, make sure to read the footnotes if you're skeptical of some of my claims.

Here's my very rough caricature of the discussion so far, plus my response:

Non-MIRI people: Yudkowsky talked a great deal in the sequences about how it was hard to get an AI to understand human values. For example, his essay on the Hidden Complexity of Wishes made it sound like it would be really hard to get an AI to understand common sense. In that essay, the genie did silly things like throwing your mother out of the building rather than safely carrying her out. Actually, it turned out that it was pretty easy to get an AI to understand common sense. LLMs are essentially safe-ish genies that do what you intend. MIRI people should update on this information.
MIRI people (Eliezer Yudkowsky, Nate Soares, and Rob Bensinger): You misunderstood the argument. The argument was never about getting an AI to understand human values, but about getting an AI to care about human values in the first place. Hence 'The genie knows but doesn't care'. There's no reason to think that GPT-4 cares about human values, even if it can understand them. We always thought the hard part of the problem was about inner alignment, or, pointing the AI in ...
Translations Should Invert
by abramdemski — 2023-10-05T17:44:23.262Z

Suppose that Alice is arguing for ZFC set theory as the proper foundation of mathematics. Bob is arguing for some specific intuitionistic type theory. Alice uses classical logic, and Bob uses intuitionistic logic; this is a big part of their difference in thinking, but not the whole disagreement, since set theory vs type theory is another part, and then (perhaps most importantly) the specific axioms each endorse.
The best way Bob can try to understand Alice is by translating her classical-logic statements into intuitionistic statements using Godel's "double negation translation", which works by sprinkling double negations throughout a sentence. This is classically equivalent to the original, but intuitionistically distinct. So Bob basically understands Alice as always speaking in the double negative (with some complications).
The best way Alice can try to understand Bob is by a similar translation in the other direction, which I can't find right now, but it involves sprinkling "provability" predicates everywhere. So Alice basically understands Bob to always be talking about provability rather than simple truth (with some complications).
These translations work pretty well for allowing Alice and Bob to follow each other's arguments in isolation. Alice and Bob can find no flaw with the translations as translations; Alice sees that Bob will never miss or add important implications (except by virtue of the other axioms Bob and Alice want to debate), and vice versa. So it might seem like Alice and Bob have found a good way to avoid debating classical logic vs intuitionistic logic, so that they can focus on debating set theory vs type theory.

However, it seems to me like there is a severe problem.

One danger is that the use of translations allows Alice and Bob to put off an important part of their debate, which is the debate between intuitionism and classical logic. But maybe this is OK.

The larger danger is that Alice and Bob cannot agree on what they are debating at any given time. Since their translations aren't inverses of each other, when Bob wants to debate X, Alice translates it into Y, and Bob sees Alice as changing the topic to Z. So from each of their perspectives, the other person is always changing the topic, which feels like "avoiding the problem I'm trying to point to -- refusing to deal with it, always insisting on dealing with some other, som...
Censorship in LLMs is here to stay because it mirrors how our own intelligence is structured
by mnvr — 2023-10-05T17:37:48.425Z

Much has been made of how increasing levels of censorship (roughly by increasing the impact of RLHF) is nerfing LLMs down. Frustrated users can't see why OpenAI is watering down the very sherbert that hit them gold. Surely, their lunch would get eaten by other companies this way. Is OpenAI stupid?

My thesis is that this censorship is an unavoidable artifact of trying to fit a LLM into human society. I am not in favor (or against) the censorship, I'm just drawing an analogy out to how our own intelligence is structured as a creative anima in interplay with a rational overseer (with game theoretically varying levels of them in action across the population). A GAN, if you will.

At a functional level, this split comes into play in, say, ChatGPT by the back and forth between the LLM and RLHF neural nets.

In humans, this split in intelligence harms the individual, but is necessary for social cohesion (at least in the sort of societies we've seen so far). Similarly, the RLHF / LLM split reduces the brilliance of LLMs, but is necessary for integrating LLMs into society. I say necessary as a contingency, not a forever-valid claim: I cannot look far into the future, but in the near one it seems that LLMs will need to coexist with humans. This is true whether or not there is a singularity - I imagine if there is a singularity, then a post singularity AI is smart enough to not fall into the Great Filter (Fermi's Paradox) by having a non-cooperative stance to the species that has the best bet of spreading it around to begin with.

I'd written a more elaborate version of this post at https://mrmr.io/rlhf if you're curious. Even if you found the reasoning to not be bulletproof, I hope you at least found the analogy thought-provoking.
This anime storyboard doesn't exist: a graphic novel written and illustrated by GPT4
by RomanS — 2023-10-05T14:01:30.141Z

Six months ago, GPT4 wrote the first known AI-written novella, "The Troodon Quill".

With the recent release of DALL-E 3 as a new feature of GPT4, the same AI can now create beautiful images according to the user input.

In the experimental results presented in this post, GPT4 creates the first anime storyboard / graphic novel / illustrated novella.

This is how it was made:

We took each chapter of the AI-written "Troodon Quill", and asked GPT4 to create an illustration for it, in anime style (zero prompt engineering; we simply asked "Please create an illustration to the following text, in anime style").
We then selected the best illustrations for the chapter out of the 4.
In a few cases, we also have to clarify what exactly we want to see (e.g. the image for the "Troodon council" was depicting humans, but we wanted to avoid that).

So, the entire text and all the illustrations below were made by GPT4, with the human only playing the role comparable to the role of a film director.

The result is interesting for several reasons:

It demonstrates the multi-modal creativity of a top publicly-available AI
It shows how good is the AI at understanding the text, and at understanding the art of painting
It shows how close (or far) we are to making a large part of movie making obsolete
How the AI can give new abilities to humans (the author of this post is incapable of writing novels or making paintings, yet here we are).

If you don't want to read the novella itself, feel free to skip the text and enjoy the gorgeous anime-style illustrations.

Chapter 1: Echoes in the Stone

The low thrum of the spectrometer filled the narrow confines of the cave, a harmonious hum that sung its serenade to the rocks. Dr. Ada Worthington, dressed in the sombre uniform of a seasoned palaeontologist, her dirty-blonde hair escaping from beneath her hat, listened to the resonating symphony, eyes closed. An outsider might mistake it for a moment of relaxation. They would be wrong. Ada was anything but relaxed.

“We're looking for anomalies, Ada,” Dr. Leo Hartman had said at the outset of this mission. Leo was her mentor, a titan in the field of palaeontology, whose reputation carried the weight of numerous groundbreaking discoveries and published works. He was a seeker of truths locked away in layers of rock, dust and time.

Ada remembered his words. She was looking for anomalies ...
AI #32: Lie Detector
by Zvi — 2023-10-05T13:50:05.030Z

Can you tell if an AI is lying to you? A new paper claims that we essentially can do exactly that, at least under the right conditions. Another paper claims we can inject various sentiments into responses, getting the AI to do what we wish. Interpretability is making progress. It is exciting to think about the implications. In the short term, it would be great if we could use this to steer responses and to detect and correct hallucinations. There’s a lot of potential here to explore.

In the longer term, I am more skeptical of such strategies. I do not think lie detection is a viable primary control or alignment strategy. I worry that if we go down such a path, we risk fooling ourselves, optimizing in ways that cause the techniques to stop working, and get ourselves killed. Indeed, even attempts to grab the low-hanging fruit of mundane utility from such advances risks starting us down this process. Still, it’s exciting, and suggests we might see more breakthroughs to follow.

We also saw Meta announce new AI-infused glasses, at a highly reasonable price point. They will be available soon, and we will see how much mundane utility is on offer.

As usual, there is also a lot of other stuff happening, including the disappointing resolution of the Open Philanthropy Worldview Contest. On the plus side, they are hiring for their global catastrophic risk team, so perhaps you can help.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. What can image and sound do for you?

Language Models Don’t Offer Mundane Utility. Don’t want it, also can’t get it.

Fun With Image Generation. Dalle-3 is having fun out there.

Deepfaketown and Botpocalypse Soon. Begun, the deepfake advertisements have.

They Took Our Jobs. There will be new jobs. But why will we keep those either?

Get Involved. OpenPhil is hiring for their catastrophic risks team.

Introducing. A new robot training protocol again, I’m sure it’s nothing.

Meta Surveillance. Meta offers AI glasses. How big a concern is privacy?

In Other AI News. Llama-2 has a world map, Replit lets you program on a phone.

Open Philanthropy Worldview Contest Awards Prizes. There is a clear pattern.

Quintin Doubles Down on Twitter. One of the winners continues to be skeptical.

The Other Winners. Should I look in depth at the other four winning entries?

Quiet

...
Can the House Legislate?
by jefftk — 2023-10-05T13:40:06.649Z

The US government is in a less functional position than usual:

Legislation requires (among other things) a plurality vote of
the House of Representatives.

This requires the Speaker of the House to call a vote.

This requires a Speaker of the House.

The House removed its Speaker on
Tuesday.

We've never done this before, so it's unclear how it will go. The House does have a "Speaker pro-tempore" backup, added in 2003, but it's not clear
whether they are allowed to use this position to call votes other than
for a new speaker, or, if they would even if allowed:

In the meantime, other action on the House floor, including any
legislative business, is likely to be halted. House staff aides
believe the acting speaker may wield the gavel only to administer the
election of a new speaker.

But others argue that as acting speaker, Mr. McHenry can exercise
powers beyond overseeing a speaker's race, as long as a majority
tolerates it.

"From an institutional House rules perspective, Speaker McHenry has
the powers of the speakership, and he will continue to exercise those
powers to the extent and degree that the majority party is willing to
tolerate," said Josh C. Huder, a senior fellow at the Government
Affairs Institute at Georgetown University. "If he does something too
brash or too bold, they will rein him in. And that's really the only
thing that's governing his authority."

House rules do not explicitly prohibit the interim speaker from
adopting the powers of an elected speaker. But the rule that led to
Mr. McHenry's ascent was developed with only temporary absences in
mind, not a vacant chair, according to Stan M. Brand, the former
general counsel to the House. That leaves legal room for Mr. McHenry
to exercise broader powers, such as conducting legislative business,
if he chooses.—NYT, 2023-10-04

I've started a prediction
market, which as of this writing is at 20% with four traders:

I set the resolution conditions as:

Resolves true if McHenry uses his current pro-tempore position to
allow the House to schedule a vote for anything other than a new
Speaker. Resolves false otherwise, including if a new Speaker is
elected quickly enough that we don't find out whether legislating
would have been acceptable.

I interpret this as a combination of whether it is strictly permitted
and how long it might take to elect a n...
Making progress on the ``what alignment target should be aimed at?'' question, is urgent
by ThomasCederborg — 2023-10-05T12:55:06.736Z

There exists a class of AI proposals, where some AI is supposed to undergo an I. J. Good style intelligence explosion, and then defer the decision, of what alignment target, will eventually be hit, to someone else. One such proposal is the Pivotal Act AI (PAAI), proposed by ``person A'', in this post by Wei Dai Meta Questions about Metaphilosophy (this specific PAAI defers the decision to a group of uploads). Another example of this general type of AI, could be the result of some coalition of powerful governments, insisting on a Political Process AI (PPAI): an AI that defers the decision to a pre specified political process, designed to not ``disrupt the current balance of power''. For lack of a better name, I will refer to any AI in this general class, as a PlaceHolder AI (PHAI).

It can sometimes make sense to refer to a given PHAI, as an ``alignment target''. However, the present text will only used the term ``alignment target'', for the goal of a real AI, that does not defer the question of which alignment target, will eventually be aimed at (such as, for example, CEV). The present text is making the argument, that no PHAI proposal, can remove the urgency, of making progress on the ``what alignment target should be aimed at?'' question.

The reason for urgency is that (i): there is currently no viable answer to the ``what alignment target should be aimed at?'' question, (ii): there is also no way of reliably distinguishing a good alignment target from a bad alignment target, and finally, (iii): hitting a bad alignment target, would be far, far, worse than extinction. (see my previous post A problem with the most recently published version of CEV)

Consider the case where someone comes up with an answer to the ``what alignment target should be aimed at?'' question, that looks like ``the obviously correct thing to do''. No one is able to come up with a coherent argument against it, even after years of careful analysis. However, this answer is built on top of some unexamined implicit assumption, that is wrong in such a way, that successfully hitting this alignment target, would be far, far, worse than extinction. In this scenario, there is no way of knowing how long it will take, for progress on the ``what alignment target should be aimed at?'' question, to advance to the point, where it is possible to notice the problematic assumption in question. There is...
Response to Quintin Pope's Evolution Provides No Evidence For the Sharp Left Turn
by Zvi — 2023-10-05T11:39:02.393Z

Response to: Evolution Provides No Evidence For the Sharp Left Turn, due to it winning first prize in The Open Philanthropy Worldviews contest.

Quintin’s post is an argument about a key historical reference class and what it tells us about AI. Instead of arguing that the reference makes his point, he is instead arguing that it doesn’t make anyone’s point - that we understand the reasons for humanity’s sudden growth in capabilities. He says this jump was caused by gaining access to cultural transmission which allowed partial preservation of in-lifetime learning across generations, which was a vast efficiency gain that fully explains the orders of magnitude jump in the expansion of human capabilities. Since AIs already preserve their metaphorical in-lifetime learning across their metaphorical generations, he argues, this does not apply to AI.

This last paragraph makes an extremely important claim that I want to ensure I convey fully:

- IF we understand the mechanism behind humanity's sharp left turn with respect to evolution

- AND that mechanism is inapplicable to AI development

- THEN, there's no reason to reference evolution at all when forecasting AI development rates, not as evidence for a sharp left turn, not as an "illustrative example" of some mechanism / intuition which might supposedly lead to a sharp left turn in AI development, not for anything.

That seems like quite a leap. If there is one particular development in humanity’s history that we can fully explain, we should then not cite evolution in any way, as an argument for anything?

If we applied this standard to other forms of reference class arguments, then presumably we would throw most or almost all of them out as well - anything where the particular mechanism was understood, and didn’t directly apply to AI, would not count. I do not see Quintin or other skeptics proposing this standard of evidence more generally, nor do I think it would make sense.

He then goes on to make another very broad claim.

In order to experience a sharp left turn that arose due to the same mechanistic reasons as the sharp left turn of human evolution, an AI developer would have to:

Deliberately create a (very obvious[2]) inner optimizer, whose inner loss function includes no mention of human values / objectives.[3]

Grant that inner optimizer ~billions of times greater optimization power than the outer optimizer.[4]

Let the

...