trentmkelly/LessWrong-43k huggingface dataset:
What I Was Thinking About Before Alignment
by johnswentworth — 2022-04-06T16:08:32.309Z

People frequently ask me about my backstory - how I got into alignment/agency research, what I did before, that sort of thing. One of the main things I emphasize is that I was thinking about analogous problems in economics and especially in biology, and I think the view from that angle made it much more obvious where AI alignment was going to run into the same major barriers.

Below is an essay I wrote in summer 2017, arguing that understanding foundational problems of agency is the primary bottleneck to progress in a wide variety of scientific fields. Hopefully this will give some idea of where my views on alignment/agency research stem from.

The Scientific Bottleneck

Imagine you’re in a sci-fi universe in the style of StarTrek or Stargate or the like. You’ve bumped into a new alien species, drama ensued, and now you’re on their ship and need to hack into their computer system. Actually, to simplify the discussion, let’s say you’re the aliens, and you’re hacking into the humans’ computer system.

Let’s review just how difficult this problem is.

You’re looking at billions of tiny electronic wires and switches and capacitors. You have a rough idea of the high-level behavior they produce - controlling the ship, navigating via the stars, routing communications, etc. But you need to figure out how that behavior is built up out of wires and switches and electronic pulses and whatnot. As a first step, you’ll probably scan the whole CPU and produce a giant map of all the switches and wires and maybe even run a simulation of the system. But this doesn’t really get you any closer to understanding the system or, more to the point, any closer to hacking it.

So how can we really understand the computer system? Well, you’ll probably notice pretty quickly that there’s regular patterns on the CPU. At the low level, there’s things like wires and switches. You might also measure the voltages in those wires and switches, and notice that the exact voltage level doesn’t matter much; there’s high voltages and low voltages, and the exact details don’t seem to matter once you know whether it’s high or low. Then you might notice some higher-level structures, patterns of wires and switches which form other standard elements, like memory elements and logic gates. But eventually, you’re going to exhaust the “hardware” properties, and you’ll need to start mapping “software”. That problem will ...
[Link] A minimal viable product for alignment
by janleike — 2022-04-06T15:38:29.646Z

This is a link post for https://aligned.substack.com/p/alignment-mvp

I'm writing a sequence of posts on the approach to alignment I'm currently most excited about. This second post argues that instead of trying to solve the alignment problem once and for all, we can succeed with something less ambitious: building a system that allows us to bootstrap better alignment techniques.
[Link] Why I’m excited about AI-assisted human feedback
by janleike — 2022-04-06T15:37:58.322Z

This is a link post for https://aligned.substack.com/p/ai-assisted-human-feedback

I'm writing a sequence of posts on the approach to alignment I'm currently most excited about. This first post argues for recursive reward modeling and the problem it's meant to address (scaling RLHF to tasks that are hard to evaluate).
Strategic Considerations Regarding Autistic/Literal AI
by Chris_Leong — 2022-04-06T14:57:11.494Z

Epistemic Status: Take with a grain of salt. This post was written relatively quickly and it heavily relies on making an analogy between AI and human behaviour. This post kind of just takes a sledgehammer to these concerns and tries to reason it out using these analogies anyway. I'd encourage others to consider whether I've fallen into the trap of mistakenly anthropomorphising AI.

The less "autistic"[1] we make AI the more powerful it is in terms of capabilities:
Suppose you tell an autistic AI to steal a diamond. Maybe it's mostly been trained on only a few ways of stealing diamonds, so innovative methods, such as teleportation, score poorly as the algorithm is uncertain as to whether they are in scope.
Powerful autistic AIs can be used for attacks despite not being completely reliable. Suppose you tell the AI to take down an adversaries network. It might do want you want and inflict long-lasting damage, such as by erasing all their machines. Or it might not do what you want, such as it if just turned their machines off and such that they would be back online again five minutes later. It may even hurt you as well, such as if it released a virus that took down everyone's networks. So while there are good reasons why you might not want to use it for an attack, and indeed why this is in fact dangerous, they still can be used for attacks if you're willing to bear the risk. This is worrying as even if you have ten enemies and nine of them think it would be too dangerous to use such an AI against you, you could still suffer attacks from your most risk-taking adversary.
Autistic AI is very difficult to use for defence. Maybe you tell it to defend your network and it assumes that you mean to physically defend it only, so it doesn't even try to stop hackers. Maybe it locks you out of using the network so no-one can accidentally bring in a virus. Sure without the AI, you'd be vulnerable to an adversary's AI, but with it you could be taken out even if no one ever attacks you.
For a constant level of power, making AI more autistic favours attackers over defenders, so we want it to be as little autistic as possible. However, reducing autism increases capabilities, so pushing hard on this may be net-negative if it sufficiently reduces the timeline to recursive self-improving AI. This is concerning because it suggests that the optimal policy will involve maintaining a fra

...
[Intro to brain-like-AGI safety] 11. Safety ≠ alignment (but they’re close!)
by Steven Byrnes — 2022-04-06T13:39:42.104Z

(Last revised: July 2024. See changelog at the bottom.)

(If you’re already an AGI safety expert, you can probably skip this short post—I don’t think anything here is new, or too specific to brain-like AGIs.)

11.1 Post summary / Table of contents

Part of the “Intro to brain-like-AGI safety” post series.

In the previous post, I talked about “the alignment problem” for brain-like AGIs. Two points are worth emphasizing: (1) the alignment problem for brain-like AGIs is currently unsolved (just like the alignment problem for any other type of AGI), and (2) solving it would be a giant leap towards AGI safety.

That said, “solving AGI alignment” is not exactly the same as “solving AGI safety”. This post is about how the two may come apart, at least in principle.

As a reminder, here’s the terminology:

“AGI alignment” (previous post) means that an AGI is trying to do things that the AGI designer had intended for it to be trying to do.[1] This notion only makes sense for algorithms that are “trying” to do something in the first place. What does “trying” mean in general? Hoo boy, that’s

a whole can of worms

. Is a sorting algorithm “trying” to sort numbers? Or is it merely sorting them?? I don’t want to go there. For this series, it’s easy. The “brain-like AGIs” that I’m talking about can definitely “try” to do things, in exactly the same common-sense way that a human can “try” to get out of debt.
“AGI safety” (

Post #1

) is about what the AGI actually does, not what it’s trying to do. AGI safety means that the AGI’s actual behavior does not lead to a “catastrophic accident”, as judged by the AGI’s own designers.[2]

Thus, these are two different things. And my goal in this post is to describe how they may come apart:

Section 11.2 is “alignment without safety”. A possible story would be: “I wanted my AGI to mop the floor, and my AGI did in fact try to mop my floor, but, well, it’s a bit clumsy, and it seems to have accidentally vaporized the entire universe into

pure nothingness

.”
Section 11.3 is “safety without alignment”. A possible story would be: “I don’t really know what my AGI is trying to do, but it is constrained, such that it can’t do anything catastrophically dangerous even if it wanted to.” I’ll go through four special cases of safety-without-alignment: “boxing”, “data curation”, “impact limits”, and “non

...
Research agenda: Can transformers do system 2 thinking?
by p.b. — 2022-04-06T13:31:39.327Z

This is one of the posts that detail my research agenda, which tries to marry Deep Learning and chess to gain insight into current AI technologies. One of my main motivations for this research is trying to identify gaps aka unsolved problems in AI, because these would uniquely inform AGI timelines.

There are many critics of Deep Learning, however, they uniformly disappoint when it comes to making concrete verifiable statements about what it is that Deep Learning is missing and what fundamental gaps in capabilities will remain as long as that is the case.

One candidate for a gap is system 2 thinking. Deliberation. Solving complex problems by thinking them through. There is reason to think that this is a gap in current Deep Learning. As Andrew Ng has stated: Deep Learning can do anything a human can do in under a second.

In sucessful systems where that is not enough, deliberation is often replaced by search (AlphaGo) or by roll-outs with selection (AlphaCode). These are tailored to the problem in question and not uniformly applicable.

Large language models show signs of deliberation. Especially the possibility to elicit a form of inner monologue to better solve problems that demand deliberation points into that direction.

However, the evaluation of large language models has the persistent problem that they exhibit "cached intelligence". I.e. for all seemingly intelligent behavior the distance to existing training examples is unclear.

In chess system 1 thinking and system 2 thinking have straightforward analogues. Positional intuition guides the initial choice of move candidates. Calculation is the effortful and slow way to search through possible moves and counter-moves.

When I trained a transformer to predict human moves from positions, it came apparent that although the benefit of scaling had maxed out the accuracy, the model was incapable of avoiding even tactical mistakes that are immediately punished.

After making many just so very sensible moves the model hangs a rook with Re8-e2.

When playing against the model this is especially striking because all its moves, including the mistakes, are just so very plausible. Playing against search based engines often feels like playing against somebody who doesn't know any chess, but beats you anyway. Playing against a transformer feels like playing against a strong player, who nevertheles...
PaLM in "Extrapolating GPT-N performance"
by Lukas Finnveden — 2022-04-06T13:05:12.803Z

A bit more than a year ago, I wrote

Extrapolating GPT-N performance

, trying to predict how fast scaled-up models would improve on a few benchmarks. Google Research just released

a paper

reporting benchmark performance of PaLM: a 540B parameter model trained on 780B tokens. This post contains an updated version of one of the old graphs, where I've added PaLM's performance.

(Edit: I've made a further update here.)

You can read the original post for the full details, but as a quick explainer of how to read the graph:

Each dot represents a particular model’s performance on a particular benchmark (taken from the GPT-3 paper). Color represents benchmark; y-position represents benchmark performance (normalized between random and my guess of maximum possible performance); and the x-position represents loss on GPT-3’s validation set.

The x-axis is also annotated with the required size+data that you’d need to achieve that loss (if you trained to convergence) according to the

original scaling laws paper

.
(After the point at which OpenAI’s scaling-laws predicts that you’d only have to train on each data point once, it is also annotated with the amount of FLOP you’d need to train on each data point once.)

The crosses represent Google’s new language model, PaLM. Since they do not report loss, I infer what position it should have from the size and amount of data it was trained on. (The relationship between parameters and data is very similar to what OpenAI's scaling laws recommended.)
The sigmoid lines are only fit to the GPT-3 dots, not the PaLM crosses.

Some reflections:

SuperGLUE is above trend (and happens to appear on the Cloze & completion trendline — this is totally accidental). ANLI sees impressive gains, though nothing too surprising given ~sigmoidal scaling.
Common sense reasoning + Reading tasks are right on trend.
Cloze & completion, Winograd, and Q&A are below trend.
The average is amusingly right-on-trend, though I wouldn’t put a lot of weight on that, given that the weighting of the different benchmarks is totally arbitrary.

(The current set-up gives equal weight to everything — despite e.g. SuperGLUE being a much more robust benchmark than Winograd.)

And a few caveats:

The GPT-3 paper was published 2 years ago. I would’ve expected some algorithmic progress by now — and the PaLM authors claim to have made some improvements. Accounting for that, this looks more

...
Solving the Brazilian Children's Game of 007
by Epirito — 2022-04-06T13:03:01.454Z

There are two versions of zero-zero-sete or 007. In the first one, there are two players. Each player has three options: shooting, reloading or blocking. They start with zero bullets. Each turn, both play at the same time in regular intervals (they achieve this by saying the name of the game in unison and making their move by handsigns as they say "sete", kind of like rock-paper-scissors). If you shoot as the other reloads, you win. If both shoot, it's a draw. Everytime you shoot you spend one bullet. Now, this version is really boring, since you can (and should) block forever without being punished.

The actually interesting version of the game is just like the first one, but if you hoard a set number of bullets, you get a rocket launcher that can break the shield, so from that point your opponent is playing for a draw.

I tried to solve the interesting version of the game using game theory. Specifically, the version where the rocket launcher costs only two bullets, because it's the simplest.

This is the pay-off matrix for player one when both players have one bullet:

S	B	R
S	0	-u(1,0)	1
B	u(1,0)	0	-u(2,1)
R	-1	u(2,1)	0

S means shoot. B means block. R means reload. u(1,0) means the expected utility of going to the state where you have one bullet and the oponent has zero, and u(2,1), the utility of going to the state where you have two bullets and he has one.

Now, let's calculate u(2,1).

This is the matrix for the state where I have two bullets and my opponent has one:

S	R
S	0	1
B	1	0

It's smaller because, since I already have the rocket launcher, I have no reason to reload, and my opponent has no reason to block. And blocking also wins me the game because after that I have the rocket launcher and the opponent has no bullets, and then he can't threaten a draw, so I shoot him in the next turn. The nash equilibrium is for us to shoot 50% of the time, which leaves me with the expected utility of .5. So, back to the main matrix, we have:

S	B	R
S	0	-u(1,0)	1
B	u(1,0)	0	-.5
R	-1	.5	0

Now, here's where I'm stuck. What's the utility of going to the state where I have one bullet and the opponent has none?

If I try to do the matrix for that, I get this:

B	R
S	0	1
R	1	.5

Here, the opponent can't shoot because he has no bullets and thus I have no reason to block. Missing a shot gets me back to where I started and reloading either wins me the game the next turn or gets me in the situation where I either ...
Accelerating personal growth: a mashup of models
by  — 2022-04-06T12:10:52.194Z

I wrote this in 2018, as part of a rationalist co-housing project with lofty goals (some of you might remember the Accelerator). It's been sitting on Arbital since then, but I've recently been reducing the amount of websites that have my name on it, so I'm republishing it here.
Epistemic status: literally a shower thought. Please read with caution, and add criticism.

Accelerating personal growth: a mashup of models

I have multiple related models of this, and none of them seem to fully map conditionspace. Therefore I have decided to include them all, to provide a list of recommendations that is as complete as possible. If you have a model that seems completely different, please add it to the list.

Model 1: it all boils down to (perceived) physical safety

With “perceived” I mean that your system 1 has to agree. If you have a sense that your physical body will be well cared for in the foreseeable future, you will have an easy time focusing on long-term non-physical things. Things that bring one to this state are:

Being able to unconditionally take care of yourself, or being fully accepted by those that take care of you.
Having a sense of high status in your community, so that more resources will be allocated to you.
Having a good record of being physically well (remember s1 cares mostly about data). This means things like pulling all-nighters and skipping breakfast have subtle negative externalities.
Keeping a good stack of currency (i.e. money, energy, karma) on hand

The list can go on forever, but I hope this is enough for you to be able to generate it yourself.

A few practical recommendations based on this model:

Residents should be self-sufficient
Have temporary residents pay ahead before coming to the accelerator so they are covered for the length of their stay, but give them a linear slice of their money back if they leave early so they don’t have to incur financial loss for leaving. They don't get stuck.
Hold operations (taking care of food & shelter, cleaning, etc) to high standards
Give residents at least small responsibilities so they can have a sense of being needed
Establish a culture of self-care

Again, this list is not exhaustive.

Model 2: reach self-actualization

You must know about Maslow’s hierarchy, and you might be aware of the positive effects of climbing that pyramid. Maslow called the first 4 needs ‘deficiency needs’, indicating that not meeting these ne...
Don't die with dignity; instead play to your outs
by Jeffrey Ladish — 2022-04-06T07:53:05.172Z

In

MIRI announces new "Death With Dignity" strategy

, Eliezer makes a number of claims. As I understood them, they are:

Humanity is very likely doomed, because AI alignment is extremely hard, and no one has a strategy that could plausibly work (or is on track to find one)
The right thing to do is maximize the log odds of humanity’s survival
The way to do this to adopt a mindset Eliezer calls “dying with dignity”

Assuming 1) and 2) are true, I’m not convinced the right mindset to adopt is “dying with dignity”. I don’t like it as a framing of the problem. Personally, it sounds demotivating and depressing, and not the thing that will help me maximize the log odds of success. I recognize that not everyone has the same mental setup as me, and some people might find the dying with dignity framing helpful. If it works for you, great, but I think there is a better problem framing that will work for a wider variety of mental setups, and I call it playing to your outs.

This problem framing arose out of a discussion about the dying with dignity post with Eli Rose and Seraphina Nix. Eli pointed out that in Magic the Gathering, there is a useful strategy to adopt when you’re likely to lose, called

playing to your outs

. Basically, there are times when you’re losing and it’s tempting to try to minimize your losses and make what gains you can. But if you’re sufficiently behind, this will almost always fail. Instead you need to think ahead to what cards you can draw, or what cards your opponent might fail to draw, that will afford you a chance to win. Then plan for one of those scenarios happening, even if it’s quite unlikely.
If 1) is true, then we are losing badly in our game of AI existential risk. It’s hard to stay motivated when things are dire, and I claim our stance should be like that of a good Magic player. Playing carefully, choosing strategies that have a shot at winning, keeping our eye on the prize. Being ready to win if the opportunity presents itself. I don’t think this is a different strategy than the one Eliezer is proposing; I think the strategy is identical. But I think there’s an important mindset difference, even if it’s just in the name, and that mindset difference matters.
I am not saying to condition upon some particular unlikely clever scheme working. The actual world is not a Magic game; it’s much more complex. So you can’t just wait fo...
Distilling and approaches to the determinant
by AprilSR — 2022-04-06T06:34:09.553Z

There's two steps to deeply understanding a complicated problem:

Get a handle on what is going on
Distill the idea down to the very core fundamentals

We're going to talk about step (2) here. Having a deep, elegant understanding of a concept makes it much easier to effectively use without needing to memorize tons of details. It also often makes it easier to guide others towards an understanding of the topic, since you can focus on conveying the central ideas.

I'm currently taking a linear algebra class, so I decided to analyze several different approaches to describing the concept of the determinant and see what I could learn about distillation. If you aren't already familiar with linear algebra, probably skim the math details.

Approaches

My college linear algebra lectures

When introducing the determinant, my linear algebra class simply provided a definition without any justification: a determinant is a function from square matrices to the reals which is:

Multilinear on rows
Alternating on rows
Normalized

We then discussed what exactly each of these conditions means and proved things that follow from them. If you don't already understand determinants this looks like you're just pulling criteria out of a hat.

We then went on to derive, from that definition, the properties that Jim Hefferon uncovers. The initial introduction to a concept being entirely arbitrary doesn't mean you can't learn things about the concept. We also eventually showed that the function which gives the volume of the image of a hypercube under a linear transformation satisfies the conditions for being a determinant function.

Linear Algebra by Jim Hefferon

After explaining how to find the inverse of a matrix, Hefferon gives an exercise: show that (abcd)−1=1ad−bc(d−b−ca), and thus that a 2x2 matrix is invertible if and only if ad−bc≠0. The ad−bc term is identified as significant; it would be useful to have a generalization to higher dimension matrices.

A good strategy for solving problems is to explore which properties the solution must have, and then search for something with those properties. So we shall start by asking what properties we’d like the determinant formulas to have.

It's been established earlier in the textbook that a matrix is invertible iff it does not have any zeroes in the diagonal after performing Gaussian elimination. So the determination might be a function th...