trentmkelly/LessWrong-43k huggingface dataset:
The case for more Alignment Target Analysis (ATA)
by Chi Nguyen — 2024-09-20T01:14:41.411Z

Summary

We don’t have good proposals for alignment targets: The most recently published version of Coherent Extrapolated Volition (CEV), a fairly prominent alignment target, is Parliamentarian CEV (PCEV). PCEV gives a lot of extra influence to anyone who intrinsically values hurting other individuals (search the

CEV arbital page

for ADDED 2023 for Yudkowsky’s description of the issue). This feature went unnoticed for many years and would make a successfully implemented PCEV

very dangerous

.
Bad alignment target proposals are dangerous: There is no particular reason to think that discovery of this problem was inevitable. It went undetected for many years. There are also plausible paths along which PCEV (or a proposal with a similar issue) might have ended up being implemented. In other words: PCEV posed a serious risk. That risk has probably been mostly removed by the arbital update. (It seems unlikely that someone would implement a proposed alignment target without at least reading the basic texts describing the proposal). PCEV is however not the only dangerous alignment target, and risks from scenarios where someone successfully hits some other bad alignment target remains.
Alignment Target Analysis (ATA) can reduce these risks. We will argue that more ATA is needed and urgent. ATA can informally be described as analyzing and critiquing Sovereign AI proposals, for example along the lines of CEV. By Sovereign AI we mean a clever and powerful AI that will act autonomously in the world (as opposed to tool AIs or a pivotal act AI of the type that follows human orders and that can be used to shut down competing AI projects). ATA asks what would happen if a Sovereign AI project were to succeed at aligning their AI to a given alignment target.
ATA is urgent. The majority of this post will focus on arguing that ATA cannot be deferred. A potential Pivotal Act AI (PAAI) might fail to buy enough calendar time for ATA since it seems plausible that a PAAI wouldn’t be able to sufficiently reduce internal time pressure. Augmented humans and AI assistants might fail to buy enough subjective time for ATA. Augmented humans that are good at hitting alignment targets will not necessarily be good at analyzing alignment targets. Creating sufficiently helpful AI assistants might be hard or impossible without already accidentally locking in an alignment ta

...
Piling bounded arguments
by momom2 — 2024-09-19T22:27:41.534Z

TL;DR: A series of valid bounded arguments all arguing the same proposition can only provide as much evidence as that proposition at best, so even if it looks like they're piling up in favor of a higher point, they're only as good as the most central of them.

Epistemic status: Armchair epistemology. Something I noticed + anecdotal evidence.

Introduction

During the LWCW last WE, I played an activity called Steelmanning the devil, where you had to pick a position you disagreed strongly with and steelman it. Participants were scored on their number of valid arguments and fallacies: 1 point for a valid argument, 0.5 point for an unconvincing argument, -1 point for a fallacy (with a lot of room for human judgment so score reflects how convincing the participants were).
I won using arguments which did not prove my point, which I will call piling bounded arguments, and I want to point them out because I think it's easy to be confused online because of them, and they're apparently not common knowledge enough that I wasn't called out during a rationalist competition about arguing.[1]

Boundedly arguing for the devil

What it looks like

The activity went like thus: you named an opinion A you disagreed with (if possible, on a tribal level); you were paired with someone who'd attack A and you'd do your best to steelman it.

You disagree with A. You are informed on the subject, you are confident in your opinion because you've looked at all arguments and found them, on balance, to tend towards ¬A and that arguments for A are generally subtly (or non-subtly) flawed.

This does not mean that there is no valid argument for A. They might just be too weak or too few compared to the arguments against.

Suppose you know one good argument for A (call it P). You share it. Your partner says
"But what about X? That would make P invalid, so you don't have evidence in favor of A."
"Actually, Q=>¬X. This is valid, central, and final. Nothing left to argue about X, it was just plain wrong."
"Right. What about Y? Idem."
"Well, R=>¬Y."
"What about Z? Idem."
"Nice one, but it's a non-central counter-argument; the central part of P is still valid."

And so on; after half an hour of arguing like that, your partner might be quite impressed that you can defend all counter-arguments and figure out that in the end, A must be true (or at least that there is a lot of strong evidence for it). But you only ever...
We Don't Know Our Own Values, but Reward Bridges The Is-Ought Gap
by johnswentworth — 2024-09-19T22:22:05.307Z

Background: “Learning” vs “Learning About”

Adaptive systems, reinforcement “learners”, etc, “learn” in the sense that their behavior adapts to their environment.

Bayesian reasoners, human scientists, etc, “learn” in the sense that they have some symbolic representation of the environment, and they update those symbols over time to (hopefully) better match the environment (i.e. make the map better match the territory).

These two kinds of “learning” are not synonymous[1]. Adaptive systems “learn” things, but they don’t necessarily “learn about” things; they don’t necessarily have an internal map of the external territory. (Yes, the active inference folks will bullshit about how any adaptive system must have a map of the territory, but their math does not substantively support that interpretation.) The internal heuristics or behaviors “learned” by an adaptive system are not necessarily “about” any particular external thing, and don’t necessarily represent any particular external thing[2].

We Humans Learn About Our Values

“I thought I wanted X, but then I tried it and it was pretty meh.”

“For a long time I pursued Y, but now I think that was more a social script than my own values.”

“As a teenager, I endorsed the view that Z is the highest objective of human existence. … Yeah, it’s a bit embarrassing in hindsight.”

The ubiquity of these sorts of sentiments is the simplest evidence that we do not typically know our own values[3]. Rather, people often (but not always) have some explicit best guess at their own values, and that guess updates over time - i.e. we can learn about our own values.

Note the wording here: we’re not just saying that human values are “learned” in the more general sense of reinforcement learning. We’re saying that we humans have some internal representation of our own values, a “map” of our values, and we update that map in response to evidence. Look again at the examples at the beginning of this section:

“I thought I wanted X, but then I tried it and it was pretty meh.”
“For a long time I pursued Y, but now I think that was more a social script than my own values.”
“As a teenager, I endorsed the view that Z is the highest objective of human existence. … Yeah, it’s a bit embarrassing in hindsight.”

Notice that the wording of each example involves beliefs about values. They’re not just saying “I used to feel urge X, but now I...
Interested in Cognitive Bootcamp?
by Raemon — 2024-09-19T22:12:13.348Z

I'm running more 4-day "Cognitive Bootcamps" over the next couple months (during Lighthaven Eternal September season). DM me if you're potentially interested (either as an individual, or as a team).

The workshop is most valuable to people who:

control their decisionmaking process (i.e. you decide what projects you or a team work on, rather than working at a day-job on someone else's vision)
are either a) confused about planmaking / have a vague sense that they aren't as strategically ambitious as they could be.
and/or, b) are at a place where it's natural to spend a few days thinking big-picture thoughts before deciding on their next project.

There's a secondary[1] focus on "practice solving confusing problems", which IMO is time well spent, but requires more followup practice to pay off.

I wrote about the previous workshop here. Participants said on average they'd have been willing to pay $850 for it, and would have paid $5000 for the ideal, perfectly-tailored-for-them version. My plan is to charge $500/person for the next workshop, and then $1000 for the next one.

I'm most excited to run this for teams, who can develop a shared skillset and accompanying culture. I plan to tailor the workshops for the needs of whichever people show up.

The dates are not scheduled yet (depends somewhat on when a critical mass of participants are available). DM me if you are interested.

The skills being taught will be similar to the sort of thing listed in Skills from a year of Purposeful Rationality Practice and the Feedbackloop-first Rationality sequence. My default curriculum is aiming to teach several interrelated related skills you can practice over four days, that build into a coherent metaskill of "ambitious planning, at multiple timescales."

I'm likely updating the curriculum from last time, but for reference and a rough idea of what to expect, here was the structure at the previous workshop:

Beforehand:

People sent me a short writeup of their current plans for the next 1-2 weeks, and broader plans for the next 1-6 months.

Day 1: Practice skills on quick-feedback exercises

Everyone installs the fatebook chrome/firefox extension
Solve a puzzle with Dots and a Grid with an unspecified goal
Solve a GPQA question with 95% confidence
Try to one-shot a Baba is You puzzle
For both of those puzzles (Baba and GPQA), ask "How could I have thought that faster?"
Play a videogame like Luc

...
Just How Good Are Modern Chess Computers?
by nem — 2024-09-19T18:57:21.254Z

When you play chess against a grandmaster, you know you're going to lose. But do you really understand how badly? You might think to yourself, “There's only like 20 possible moves per turn. Even a novice like me can quickly rule out half of those.”

In this post, I'd like to help build an intuition for just HOW good the best chess players are, while picking from “just” the best 10 moves.
I’ll also add some intuition for how much better chess computers are than humans. As most people don’t play chess, I’ll develop this intuition using something most people are inherently familiar with: Physical ability.

Specifically, in my metaphor, chess is a 1-on-1 soccer game. First player to score a goal wins.

In this metaphor, most people are toddlers. At 2.5 years old, you could be taught to play soccer, and you might get the ball in the goal with some effort, but at the end of the day, you are mostly just struggling with basic coordination. Your ELO is around 800.

Me on the other hand, I’m basically an expert (sarcasm tag here). At an ELO of 1300, I am a 5 year old. I can easily beat you, you dumb toddler. In fact, I'm going to beat you every time! Well almost every time. I am a 5 year old. I could get distracted by a cloud, or stung by a bee, and you might slip in a goal.

As you can probably tell, a 5 year old is... not actually very good in the scheme of things. A chess hustler on the street with a rating of 1800 will pick me apart.
Imagine me, a 5 year old, playing an older kid. Say, an 8 year old; and one who has prepped for the match in advance! Can I win? If I cheat or get very lucky, maybe I can trip the 8 year old and make them cry. In any normal game, I'm just going to lose.

But a chess hustler still isn't a pro.

To be a chess Master, you need a rating of 2200 or better. This chess master would laugh at the hustler in the park. The master is a 12 year old kid who actually plays on the soccer team. They almost didn’t make the cut, but still. They aren’t even slightly concerned about the 8 year old. It's such a mismatch, that there's no real point in playing.

The ladder doesn't stop there. The best chess players in the world stand head and shoulders above mere "masters". Take Magnus Carlson, best chess player in all of human history.
In our metaphor, he is a 16 year old in a world where t...
RLHF is the worst possible thing done when facing the alignment problem
by tailcalled — 2024-09-19T18:56:27.676Z

Epistemic status: The title might not be literally true in the sense that e.g. if the inventors of RLHF hadn't come up with it then someone else probably would, so the counterfactual effect is small, or e.g. that the worst possible thing you could do would be "invent RLHF and then do some other things that makes the alignment problem worse", but it's "spiritually true" in the sense that it's hard to name one singular thing that's worse for our chances than the existence of RLHF, so I wouldn't call the title hyperbole per se.

Post TL;DR: Adversarial conflict requires coherence which implies unbounded utility maximization which is bad because we don't know an acceptable utility function. RLHF does not solve the alignment problem because humans can't provide good-enough feedback fast-enough. RLHF makes the alignment problem worse because it advances AI and covers up misalignment. Solving the alignment problem is about developing technology that prefers good things to bad things.

While some forms of AI optimism (or at least opposition to some forms of AI pessimism) seem justified to me, there's a strand of AI optimism that goes "RLHF has shown that alignment is quite tractable". That strand is completely wrong.

I think the intuition goes that neural networks have a personality trait which we call "alignment", caused by the correspondence between their values and our values. This alignment trait is supposed to be visible (at least in low-capability models) in whether the neural network takes actions humans like or actions humans dislike, and so by changing the neural network to take more actions humans like and fewer actions humans dislike, we are raising the level of the alignment trait. RLHF'ers acknowledge that this is not a perfect system, but they think the goal for solving the alignment problem is to increase the alignment trait faster than the capabilities trait.

The main problem with this model is that it's the completely wrong way to think about the alignment problem. Here's the correct way:

The alignment problem

Section TL;DR: adversarial conflict requires coherence which implies unbounded utility maximization which is bad because we don't know an acceptable utility function.

Humans are dependent on all sorts of structures - e.g. farmers to feed us, police to give us property rights, plants and environmental regulations to give us air to breathe, and computers...
AISafety.info: What are Inductive Biases?
by Algon — 2024-09-19T17:26:24.581Z
Link: https://aisafety.info/questions/NK46/What-are-inductive-biases

AISafety.info writes AI safety intro content. We'd appreciate any feedback.

The

inductive bias

of a learning process is its tendency to learn particular kinds of patterns from data. It would help with AI alignment to know which learning processes have inductive biases toward patterns used in human cognition, and in particular human values.

In a learning process L, you have some class of models in which you look for an approximation M to some finite set of data D. This data is generated by some function f, which usually you don’t know. You want M to replicate the target function f as closely as possible. Since you can almost never get enough data to fully infer f, you have to hope that your learning process L will find an approximation M that is close to f even far from the training data. If there’s a class of target functions that L can approximate well, we say that L has an inductive bias toward such functions.

Example: Let’s say you want to predict how energetic you will feel over time. That’s your target function. So for a couple of days, whenever you get the chance, you log how energetic you feel (that’s your data D). And because you’re lazy, you decide to just draw straight lines between the data points. At the edges, you just keep drawing the outermost lines you got from D. That’s your learning process L, which will spit out some piecewise linear curve M. This is a poor learning process, for reasons we’ll get to in a bit.

You run the experiment, and then plot the results in Figure 1. The resulting gray line isn’t that bad an approximation — at least, not near existing data points. And if we have dense enough data within some range, the approximation therein will be arbitrarily good.

Far from our data points, our approximation M will fall indefinitely. This does not match our everyday experience, in which our energy fluctuates. This is a fundamental issue with this choice of L: it will never globally approximate a periodic function well[1], no matter how much data you give it. Almost all the functions that L can learn will keep growing or falling far outside the training data. That is an inductive bias of this learning process.[2]

Why does any of this matter for AI or AI alignment? Because we want our AI to...
Why good things often don’t lead to better outcomes
by DMMF — 2024-09-19T16:37:07.778Z
Link: https://danfrank.ca/why-good-things-often-dont-lead-to-good-outcomes/

Many people believe that when something good happens, positive outcomes naturally follow — but I'm here to explain why that's often not the case. In complex systems, improvements or advancements frequently set off a chain of reactions that can undermine the original benefits. This creates a paradox where progress in one domain often doesn’t lead to better results.

Consider the case of restaurants awarded Michelin stars. Surely this must be good for the restaurant, right? However, a recent academic paper showed that when restaurants receive Michelin stars, ostensibly a good thing, they paradoxically become more likely to go out of business than similar restaurants without the star. Staff seek higher salaries, leveraging their elevated status for better opportunities. Property owners and suppliers feel justified in demanding more money. Customer expectations rise, as does the composition of the restaurant’s clientele. Ultimately, despite the substantial benefits the Michelin star brings, the ripple effects throughout the value chain often make it difficult for restaurants to capture the value. The star, intended as a blessing, instead triggers a destructive spiral.

Let’s consider another example. Israel currently has the most advanced missile defence system in the world, capable of intercepting and neutralizing over 99% of incoming rockets. In the past year alone, Hezbollah has launched more than 8,000 missiles at Israel, resulting in numerous deaths, significant property damage, and the displacement of an estimated 100,000 people. Now, imagine how much worse the devastation would be without such a defence system.

Yet, paradoxically, the prevailing view in Israel is that the situation wouldn't necessarily be worse without it. When Israel first developed this missile defence system, it fundamentally changed the strategic calculus for both Israel and Hezbollah. Before, Hezbollah could inflict substantial harm with only a few rockets, carefully managing the damage to avoid provoking an overwhelming Israeli response. Now, with the defence system in place, Israel can absorb a much higher volume of rockets, so Hezbollah simply fires more missiles. Individually, each missile has a reduced impact, but collectively, they sustain the same overall level of destruction — just enough to stay below the threshold that would trigger a larger Israeli retaliation. The equilib...
To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning
by Bogdan Ionut Cirstea — 2024-09-19T16:13:55.835Z
Link: https://arxiv.org/abs/2409.12183

Authors: Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, Greg Durrett.

Abstract:

Chain-of-thought (CoT) via prompting is the de facto method for eliciting reasoning capabilities from large language models (LLMs). But for what kinds of tasks is this extra ``thinking'' really helpful? To analyze this, we conducted a quantitative meta-analysis covering over 100 papers using CoT and ran our own evaluations of 20 datasets across 14 models. Our results show that CoT gives strong performance benefits primarily on tasks involving math or logic, with much smaller gains on other types of tasks. On MMLU, directly generating the answer without CoT leads to almost identical accuracy as CoT unless the question or model's response contains an equals sign, indicating symbolic operations and reasoning. Following this finding, we analyze the behavior of CoT on these problems by separating planning and execution and comparing against tool-augmented LLMs. Much of CoT's gain comes from improving symbolic execution, but it underperforms relative to using a symbolic solver. Our results indicate that CoT can be applied selectively, maintaining performance while saving inference costs. Furthermore, they suggest a need to move beyond prompt-based CoT to new paradigms that better leverage intermediate computation across the whole range of LLM applications.

X(/twitter) thread: https://x.com/ZayneSprague/status/1836784332704215519.
Laziness death spirals
by PatrickDFarley — 2024-09-19T15:58:30.252Z

I’ve claimed that

Willpower compounds

and that small wins in the present make it easier to get bigger wins in the future. Unfortunately, procrastination and laziness compound, too.

You’re stressed out for some reason, so you take the evening off for a YouTube binge. You end up staying awake a little later than usual and sleeping poorly. So the next morning you feel especially tired; you snooze a few extra times. In your rushed morning routine you don’t have time to prepare for the work meeting as much as you’d planned to. So you have little to contribute during the meeting. You feel bad about your performance. You escape from the bad feelings with a Twitter break. But Twitter is freaking out. Elon Musk said what? Everyone is weighing in. This is going to occupy you intermittently for the rest of the day. And so on.

Laziness has a kind of independent momentum to it. When you’re having a day like the above, even if you consciously commit to getting back on track, the rut tends to find its way back to you within a couple of hours. Keep this up for a few days and your sleep is utterly messed up, and you walk around in a fog. Keep it up for a week or two and you’re fully off your workout routine. In a month or two, you might have noticeably fallen behind on work; you might be absent from your social life; you might’ve visibly gained fat or lost muscle; you can no longer feel excited about your personal goals because they’re behind a pile of mundane tasks you need to catch up on first. And so on.

How do we stop the vicious circle?

I’m spiraling! I’m spiraling!

When you’re in a laziness death spiral, it’s hard to do anything deliberate. The first and most important step, which does take some willpower but not a lot, is to acknowledge, “I’m in a laziness death spiral today.”

If you don’t acknowledge it, here’s what happens: You vaguely notice you you’ve been wasting time today; you feel a twinge of guilt, so you quickly decide, “I’m going to turn the rest of the day around, starting right now.” And does that work?

Often it doesn’t! Sure, after a small lapse you can just get back on track, but if enough laziness momentum has built up, a momentary reaction doesn’t cut it. Deciding things quickly, in response to negative emotions, is exactly how you got into this situation! You’re going to turn it around on a whim? You’ll have a different whim in the ...