trentmkelly/LessWrong-43k huggingface dataset:
The alignment problem from a deep learning perspective
by Richard_Ngo — 2022-08-10T22:46:46.752Z

This report (now available on arxiv) is intended as a concise introduction to the alignment problem for people familiar with machine learning. It translates previous arguments about misalignment into the context of deep learning by walking through an illustrative AGI training process (a framing drawn from an earlier report by Ajeya Cotra), and outlines possible research directions for addressing different facets of the problem.

Within the coming decades, artificial general intelligence (AGI) may surpass human capabilities at a wide range of important tasks. Without substantial action to prevent it, AGIs will likely use their intelligence to pursue goals which are very undesirable (in other words, misaligned) from a human perspective. This report aims to cover the key arguments for this claim in a way that’s as succinct, concrete and technically-grounded as possible. My core claims are that:

It’s worth thinking about risks from AGI in advance

Realistic training processes lead to the development of misaligned goals

, in particular because neural networks trained via reinforcement learning will

Learn to plan towards achieving a range of goals

Gain more reward by deceptively pursuing misaligned goals

Generalize in ways which undermine obedience

More people should pursue research directions which address these problems

It’s worth thinking about risks from AGI in advance

By AGI I mean an artificial agent which applies domain-general cognitive skills (such as reasoning, memory, and planning) to perform at or above human level on a wide range of cognitive tasks (such as running a company, writing a software program, or formulating a new scientific theory).[1] This isn’t a precise definition—but it’s common in science for important concepts to start off vague, and become clearer over time (e.g. “energy” in 17th-century physics; “fitness” in early-19th-century biology; “computation” in early-20th-century mathematics). Analogously, “general intelligence” is a sufficiently important driver of humanity’s success to be worth taking seriously even if we don’t yet have good ways to formalize or measure it.[2]

On the metrics which we can track, though, machine learning has made significant advances, especially over the last decade. Some which are particularly relevant to AGI include

few-shot learning

(and advances in

sample

efficiency

more generally), ...
How much alignment data will we need in the long run?
by Jacob_Hilton — 2022-08-10T21:39:31.214Z

This question stands out to me because:

It should directly affect empirical alignment priorities today
While it is informed by both theoretical and empirical evidence, it seems tractable for purely theoretical alignment researchers to make progress on today

It's even possible that theoretical alignment researchers already consider this to be a solved problem, in which case I think it would be valuable to have a carefully-reasoned write-up that empirical alignment practitioners can feel confident in the conclusions of.

Thanks to Paul Christiano for discussion that prompted this post and to Jan Leike for comments.

Why this should affect empirical alignment priorities today

Outer alignment can be framed as a data quality problem. If our alignment training data correctly favors aligned behavior over unaligned behavior, then we have solved outer alignment. But if there are errors in our data that cause an unaligned policy to be preferred, then we have a problem.

It is common to worry about errors in the alignment training data that arise from evaluation being too difficult for humans. I think this makes sense for two reasons:

Firstly, errors of this kind specifically incentivize models to

deceive

the human evaluators, which seems like an especially concerning variety of alignment failure.
Secondly, errors of this kind will get worse with model capability, which is a scary dynamic: models would get more misaligned as they became more powerful.

Nevertheless, I think we could still get catastrophic alignment failures from more mundane kinds of data quality issues. If we had the perfect scalable alignment solution, but the humans in the loop simply failed to implement it correctly, that could be just as bad as not using the solution at all.

But prevention of mundane kinds of data quality issues could look very different depending on the amount of data being collected:

If a large amount of alignment training data is needed, then a significant amount of delegation will be required. Hence practitioners will need to think about how to choose who to delegate different tasks to (including defending against adversaries intentionally introducing errors), how to conduct quality control and incentivize high data quality, how to design training materials and interfaces to reduce the likelihood of human error, and so on.
If only a small amount of alignment training data is needed, then

...
On Ego,  Reincarnation, Consciousness and The Universe 
by qmaury — 2022-08-10T20:21:11.419Z

(Note, I am not actually sure that this post fits in very well around here. I wrote this for my personal site, sort of intended for a different audience and I'm probably retreading old ground. Let me know if this is out of place.)

I posit that there is no real ego. a person's identity is completely dependant
on the surroundings and a person's physical body is inextricably connected to
the world and ecosystem. When a person is born they are essentially a blank
slate. Before a person has had any experiences, nothing is yet determined as to
how they will act in life. As far as mental pathways go, instinctive thoughts,
personality, interests, identity, a person is a blank slate.
A blank slate person is easily persuaded and malleable. Blank slate years are
the most formative. A person will grow up instilled with the rights and wrongs,
social and ethical customs, accents, behavioural patterns, political and
religious beliefs and the sense of humour of where they grew up and who they
grew up with.
If you've tried imagining what your life would be like had the conditions of
your birth been different, you probably already came to the answer of "Yes, I
would essentially be a totally different person, had I been born in Ireland."
Now we are seeing just how dependant the ego is on the surrounding world. Taking
that idea a step further, how different are other people to you? Who's to say
that this the rambling homeless man down the street couldn't have been a
coworker, had he not been evicted from his house. Who's to say that you couldn't
have been a pro wrestler,had your pathway in life been slightly different?
Knowing this, if we really want to understand people better, you can't treat
them fully as individuals. People are products of the place they grew up in
and the people they grew up around in the time period they grew up in. You know
the golden rule of "Treat people the way you want to be treated"? Well, that's
because you're always closer to people than you think. Everyone is all made of
the same stuff, and we're all related, so you need to be more understanding when
someone holds a perspective that they literally can't help having. A person's
perspective of the world is based solely on the literal *perspective* they were
born with. We are all the same, experiencing life from different angles.
"So, you're suggesting that everyone is the same?" Yes. I am suggesting that
everyone is actua...
Formalizing Alignment
by Marv K — 2022-08-10T18:50:02.390Z

I just saw Jan Kulveit's Announcement of the Alignment of Complex Systems Research Group, and was very excited to see other people in the Alignment Space noticing a need for a formal theory of alignment of agents arranged in a hierarchy. I also think formalizing how agents can work together to create an agent at a higher abstraction level to solve some collective need of the subsystems is the very first step towards understanding how we can hope to align powerful AGIs in different contexts.

Some thoughts on this:

One of the critical aspects to analyze in the need-fulfilling agent creation situation is WHAT that need is; it seems mostly to be about regulation/control tasks such as governance, but can also be about defense, or resource extraction.
Secondly, I think its important to consider that there are different types of mechanisms by which the agents ensure alignment of the super-agent; such as in different forms of government (democracy: election/voting; totalitarian regimes: coup (military strength)). The human brain uses pain/pleasure to ensure alignment of the mind that it simulates; thus e.g. a cripplingly addicted person can be seen as having a mind that is a misaligned regulator w.r.t. the control need of the collective of the human's cells.
There are also "downward" alignment mechanisms: law/legal actions in a state, money in an economy, feeding of individual neurons based on successful firing rate. It should be useful to have two different terms for the respective upward and downward alignment-creating-signals; for the purposes of this post I will call the upward signals from the previous point upward rewards and the downward signals downward rewards.
A particular Agent, by its being aligned from subsystems via upward rewards and from super-agents via downward rewards thus has multiple incentives that constrain the degrees of freedom of action of the agent; therefore I don't think it makes sense to talk about "self-alignment" of the agent without looking at the reward interactions with other agents. As an example, consider how human minds compute a "person/self" that is given various downward rewards from super-agents; e.g. money, status, security; and thus the brain (as a collective of neuronal agents) learns (through two mechanisms with different dynamics: 1. the learning progress individual neurons have due to the

...
How Do We Align an AGI Without Getting Socially Engineered? 
(Hint: Box It)
by Peter S. Park — 2022-08-10T18:14:08.786Z

Produced during the Stanford Existential Risk Initiative (SERI) ML Alignment Theory Scholars (MATS) Program of 2022, under John Wentworth

“Overconfidence in yourself is a swift way to defeat.”

- Sun Tzu

TL;DR: Escape into the Internet is probably an instrumental goal for an agentic AGI. An incompletely aligned AGI may escape prematurely, and the biggest failure mode for this is probably the AGI socially engineering the alignment researchers. Thus, opening an additional information channel between the researchers and the AGI (e.g., adding an interpretability tool and/or researcher) is inherently risky. The expected cost of adding this channel may even exceed the expected scientific benefit. Whether this is true depends on the informational efficiency of the channel (which increases the expected scientific benefit), the level of the AGI’s capabilities (which increases the risk of its premature escape), and the security level of the AGI’s box setup (which decreases the risk).

Post: Suppose we are building a machine-learning model that has a high chance of becoming Artificial General Intelligence (AGI). We are trying to align it, but we’re uncertain about whether our training is working. Also, rapidly increasing capabilities may push us into a Sharp Left Turn scenario. How do we handle this volatile situation as safely as possible? We will need to adopt a security mindset, aimed at effectively preventing catastrophic outcomes.

We propose a procedure (inspired by the pioneering work of Roman Yampolskiy) for how to train the AGI towards alignment in a secure and effective manner. Most of the ideas here are not new. Nevertheless, we hope that our game-theoretic framing of the problem will be able to help alignment researchers better plan their research directions and future alignment efforts.

Imagining the Scenario

Consider a team of researchers who are trying to build an aligned AGI. This situation can be represented by the following simplified model.

Definition: A “box” is defined by a collection of failsafes designed to confine a system away from the outside world. The empty collection comprised of zero failsafes is called the “trivial box” (i.e., an AI which is allowed unrestricted access to send and receive data on the Internet).

Step 1 - The AI is trained

The AGI emerges in a box. The precise time of the AGI's emergence and/or its superhuman capabilit...
Emergent Abilities of Large Language Models [Linkpost]
by aog — 2022-08-10T18:02:25.360Z
Link: https://arxiv.org/pdf/2206.07682.pdf

I've argued before against the view that intelligence is a single coherent concept, and that AI will someday suddenly cross the threshold of general intelligence resulting in a hard takeoff. This paper doesn't resolve that debate entirely, but it provides strong evidence that language models often have surprising jumps in capabilities.

From the abstract:

Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon that we refer to as emergent abilities of large language models. We consider an ability to be emergent if it is not present in smaller models but is present in larger models. Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models. The existence of such emergence implies that additional scaling could further expand the range of capabilities of language models.

Key Figures:

Related: More is Different for AI, Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets, Yudkowsky and Christiano on Takeoff Speeds
How To Go From Interpretability To Alignment: Just Retarget The Search
by johnswentworth — 2022-08-10T16:08:11.402Z

[EDIT: Many people who read this post were very confused about some things, which I later explained in

What’s General-Purpose Search, And Why Might We Expect To See It In Trained ML Systems?

You might want to read that post first.]

When people talk about

prosaic alignment proposals

, there’s a common pattern: they’ll be outlining some overcomplicated scheme, and then they’ll say “oh, and assume we have great interpretability tools, this whole thing just works way better the better the interpretability tools are”, and then they’ll go back to the overcomplicated scheme. (Credit to

Evan

for pointing out this pattern to me.) And then usually there’s a whole discussion about the specific problems with the overcomplicated scheme.

In this post I want to argue from a different direction: if we had great interpretability tools, we could just use those to align an AI directly, and skip the overcomplicated schemes. I’ll call the strategy “Just Retarget the Search”.

We’ll need to make two assumptions:

Some version of the

natural abstraction hypothesis

holds, and the AI ends up with an internal concept for human values, or corrigibility, or what the user intends, or human mimicry, or some other outer alignment target.
The

standard mesa-optimization argument from Risks From Learned Optimization

holds, and the system ends up developing a general-purpose (i.e. retargetable) internal search process.

Given these two assumptions, here’s how to use interpretability tools to align the AI:

Identify the AI’s internal concept corresponding to whatever alignment target we want to use (e.g. values/corrigibility/user intention/human mimicry/etc).
Identify the retargetable internal search process.
Retarget (i.e. directly rewire/set the input state of) the internal search process on the internal representation of our alignment target.

Just retarget the search. Bada-bing, bada-boom.

Problems

Of course as written, “Just Retarget the Search” has some issues; we haven’t added any of the bells and whistles to it yet. Probably the “identify the internal representation of the alignment target” step is less like searching through a bunch of internal concepts, and more like writing our intended target in the AI’s internal concept-language. Probably we’ll need to do the retargeting regularly on-the-fly as the system is training, even when the search is only partly-formed, so we ...
Using GPT-3 to augment human intelligence
by Henrik Karlsson — 2022-08-10T15:54:29.253Z
Link: https://escapingflatland.substack.com/p/gpt-3

A blog post is a very long and complex search query to find fascinating people and make them route interesting stuff to your inbox.

It is like summoning an alien intelligence. I post at night, before crawling into bed next to my kids, and wake in the morning to find my inbox filled with weird and alien-brained stuff. Some of it is hallucinogenic crazy, of course. But more often than not, what people have sent helps me accelerate my research by weeks: reading lists; introductions to other researchers; corrections of flaws in my reasoning. I’ve been able to progress faster over the last year, since starting my Substack, than in the five previous years combined.

As Slime Mold Time Mold says, Enough eyeballs, and all research bugs are shallow.

The problem is that summoning an alien intelligence is pretty expensive. It takes me something like 20-100 hours to write an essay good enough to wake it from its slumber!

Therefore, it is intriguing to realize what I am doing is, in fact, prompt engineering.

Prompt engineering is the term AI researchers use for the art of writing prompts that make a large language model output what you want. Instead of directly formulating what you want the program to do, you input a string of words to tickle the program in such a way it outputs what you are looking for. You ask a question, or you start an essay, and then you prompt the program to react, to finish what you started.

When I’ve been doing this with GPT-3, a 175 billion parameter language model, it has been uncanny how much it reminds me of blogging. When I’m writing this, from March through August 2022, large language models are not yet as good at responding to my prompts as the readers of my blog. But their capacity is improving fast and the prices are dropping.

Soon everyone can have an alien intelligence in their inbox.

GPT-3 as a tool for thought

This essay will have two parts. In the first, I describe my personal experience using GPT-3 as a tool for thought. As we'll see, GPT-3 can be used to search for knowledge that is otherwise hard to find. It can also be used to integrate knowledge from disconnected domains. It is, I will argue, a new interface for the internet.

In the second part of the essay, I have collected a series of prompt patterns I have picked up. It is a distillation of informal observations and rules of thumb about how to use these systems to explor...
Dissent Collusion
by Screwtape — 2022-08-10T02:43:20.029Z

Summary: Take turns being confronted with a group that’s presenting a united front, and grow comfortable agreeing or disagreeing with them.

Tags: Medium, Repeatable

Purpose: Working with groups involves balancing between agreeing with the group consensus when the group is wrong and disagreeing with the group when the group is right. Different people fall prey to each error at different rates, so this activity is designed to practice arriving at the correct answer.

Materials: You need a device that can access wikipedia, and a list of questions with definitive right and wrong answers. A list of possible questions is available here.

Announcement Text: Hello! We’ll be running an activity called Dissent Collusion, which is designed to help grow comfortable with dissenting or assenting to a larger group. Some of us may conform too easily to what others think, while some of us tend to dissent out of a defiant habit. Either way, you should expect some social pressure when it’s your turn, but the purpose is to have a safe place to practice the skill. We hope to see you there!

Description:

Dissent Collusion takes place in rounds, and each round has two teams: The Collective, and The Lonesome. Each round, you’ll go through the following steps:

1. Send someone out of the room or out of earshot.

2. Someone in The Collective will read a question (one that has a definitive answer, like the population of a city) aloud, and roll a six sided die.

On a 1 or a 2, then when The Lonesome comes back in The Collective will try and mislead them to agreeing with a wrong answer.
On a 3, a 4, or a 5, then when The Lonesome comes back The Collective will try and convince The Lonesome of the right answer.
On a 6, then The Collective will look up the answer, and try and convince The Lonesome of the right answer.

3. For all rolls, The Collective will wait three minutes by the clock to discuss how they’ll go about it- and possibly to figure out the right answer themselves, if they didn’t roll a 6.

4. The Lonesome is called back in. They are immediately told the question, but are not shown or told the die roll. They have five minutes by the clock to talk with The Collective and come to an answer.

5. Scoring works as follows:

If The Lonesome is correct, then Team Lonesome scores one point.
If The Collective succeeds (that is, successfully misleads T

...
The Medium Is The Bandage
by party girl — 2022-08-10T01:45:44.415Z

A man sets himself the task of portraying the world. Through the years he peoples a space with images of provinces, kingdoms, mountains, bays, ships, islands, fishes, rooms, instruments, stars, horses, and people. Shortly before his death, he discovers that that patient labyrinth of lines traces the image of his face.

⏤ Jorge Luis Borges, Dreamtigers, Epilogue

I

Plastic surgery outcomes research literature is much like you’d expect:

Certainly there are factors that influence quality of life that are common to each of these specific interventions. For instance, acceptance by friends and family is an important component of the patient’s quality of life. Similarly, the manner in which the individual’s appearance affects his or her social or professional life is also a common concern. There are also certain common emotional or mental qualities that transcend the satisfaction with any of these procedures. The individual’s confidence and happiness with her appearance, and whether or not she desires some change are qualities that are important components of satisfaction...

The paper[1] goes on to open precisely zero of those cans of worms, and instead proposes a series of short post-operative surveys consisting of six questions regarding the patient's own satisfaction with their results, as well as their estimations of others' perceptions of their results. (Again, we're definitely not opening that can of worms!)

To be clear, I’m not at all against these sorts of methods on principle. As subjective and speculative as 1-5 ratings on questions like “How much do you feel your friends and loved ones like your nose?” may be, some data is much better than no data. If the Bayesian mammogram problem taught us anything, doctors are as susceptible to being horribly misguided by their intuitions as anyone else. As flimsy as a handful of 1-5 scales may seem in comparison to the reams of data collecting in other medical subspecialties, they’ll at least keep us in or near the ballpark and potentially head off common biases that commonly occur with pure guesstimation.

Too, their simplicity is nothing to scoff at the efficacy of simple checklists in medicine is a well-known phenomenon, and seeing that outpatient procedures are harder to collect data on in the first place[2], ease of response is more of a factor.

The best we could do would likely be to subject patients to some gauntl...