trentmkelly/LessWrong-43k huggingface dataset:
Vingean Agency
by abramdemski — 2022-08-24T20:08:53.237Z

I've been involved with several discussions about different notions of agency (and their importance/relationships) lately, especially with the PIBBSS group including myself, Daniel, Josiah, and Ramana; see here.

There's one notion of agency (not necessarily "The" notion of agency, but a coherent and significant notion) which vanishes if you examine it too closely.

Imagine that Alice is "smarter than Bob in every way" -- that is, Bob believes that Alice knows everything Bob knows, and possibly more. Bob doesn't necessarily agree with Alice's goals, but Bob expects Alice to pursue them effectively. In particular, Bob expects Alice's actions to be at least as effective as the best plan Bob can think of.

Because Bob can't predict what Alice will do, the only way Bob can further constrain his expectations is to figure out what's good/bad for Alice's objectives. In some sense this seems like a best-case for Bob modeling Alice as an agent: Bob understands Alice purely by understanding her as a goal-seeking force.

I'll call this Vingean agency, since Vinge talked about the difficulty of predicting agents who are smarter than you. and since this usage is consistent with other uses of the term "Vingean" in relation to decision theory.

However Vingean agency might seem hard to reconcile with other notions of agency. We typically think of "modeling X as an agent" as involving attribution of beliefs to X, not just goals. Agents have probabilities and utilities.

Bob has minimal use for attributing beliefs to Alice, because Bob doesn't think Alice is mistaken about anything -- the best he can do is to use his own beliefs as a proxy, and try to figure out what Alice will do based on that.[1]

When I say Vingean agency "disappears when we look at it too closely", I mean that if Bob becomes smarter than Alice (understands more about the world, or has a greater ability to calculate the consequences of his beliefs), Alice's Vingean agency will vanish.

We can imagine a spectrum. At one extreme is an Alice who knows everything Bob knows and more, like we've been considering so far. At the other extreme is an Alice whose behavior is so simple that Bob can predict it completely. In between these two extremes are Alices who know some things that Bob doesn't know, while also lacking some information which Bob has.

(Arguably, Eliezer's notion of optimization power is one forma...
OpenAI's Alignment Plans
by dkirmani — 2022-08-24T19:39:04.620Z
Link: https://openai.com/blog/our-approach-to-alignment-research/

Our alignment research aims to make artificial general intelligence (AGI) aligned with human values and follow human intent. We take an iterative, empirical approach: by attempting to align highly capable AI systems, we can learn what works and what doesn’t, thus refining our ability to make AI systems safer and more aligned. Using scientific experiments, we study how alignment techniques scale and where they will break.

We tackle alignment problems both in our most capable AI systems as well as alignment problems that we expect to encounter on our path to AGI. Our main goal is to push current alignment ideas as far as possible, and to understand and document precisely how they can succeed or why they will fail. We believe that even without fundamentally new alignment ideas, we can likely build sufficiently aligned AI systems to substantially advance alignment research itself.

Unaligned AGI could pose substantial risks to humanity and solving the AGI alignment problem could be so difficult that it will require all of humanity to work together. Therefore we are committed to openly sharing our alignment research when it’s safe to do so: We want to be transparent about how well our alignment techniques actually work in practice and we want every AGI developer to use the world’s best alignment techniques.

At a high-level, our approach to alignment research focuses on engineering a scalable training signal for very smart AI systems that is aligned with human intent. It has three main pillars:

Training AI systems using human feedback

Training AI systems to assist human evaluation

Training AI systems to do alignment research

Aligning AI systems with human values also poses a range of other significant sociotechnical challenges, such as deciding to whom these systems should be aligned. Solving these problems is important to achieving our mission, but we do not discuss them in this post.

Training AI systems using human feedback

RL from human feedback is our main technique for aligning our deployed language models today. We train a class of models called InstructGPT derived from pretrained language models such as GPT-3. These models are trained to follow human intent: both explicit intent given by an instruction as well as implicit intent such as truthfulness, fairness, and safety.

Our results show that there is a lot of low-hanging fruit on alignment-focused fine-tunin...
Beliefs and Disagreements about Automating Alignment Research
by Ian McKenzie — 2022-08-24T18:37:00.419Z

Epistemic status: Mostly organizing and summarizing the views of others.

Thanks to those whose views I summarized in this post, and to Tamera Lanham, Nicholas Kees Dupuis, Daniel Kokotajlo, Peter Barnett, Eli Lifland, and Logan Smith for reviewing a draft.

Introduction

In my current view of the alignment problem, there are two paths that we could try to take:

Come up with an alignment strategy that allows us to both build aligned AGI and to keep that AGI (or its successors) aligned as they improve towards superintelligence
Come up with an alignment strategy that allows us to build AI systems that are powerful (but not so powerful as to be themselves dangerous) and use that to execute some kind of ‘

pivotal act

’ that means that misaligned

ASI

is not built

For the purposes of this post, I am going to assume that we are unable to do (1) – maybe the problem is too difficult, or we don’t have time – and focus on (2).

Within the category of ‘pivotal act’, I see two main types:

Preventative pivotal acts: acts that makes it impossible for anyone to build AGI for a long period of time
Constructive pivotal acts: acts that makes it possible to build aligned ASI

People disagree about

whether preventative pivotal acts are possible

or even if they were possible,

if they’d be a good idea

. Again, for the purposes of this post, I am going to assume we can’t or don’t want to execute a preventative pivotal act, and focus on constructive pivotal acts. In particular: can we use AI to automate alignment research safely?

What does ‘automating alignment research’ even mean?

I see three overlapping categories that one could mean when referring to ‘automating alignment research’, ordered in terms of decreasing human involvement:

Level 1: AIs help humans work faster

Examples include brainstorming, intelligent autocomplete, and automated summarization/explanation.

Level 2: AIs produce original contributions

This could be key insights into the nature of intelligence, additional problems that were overlooked, or entire alignment proposals.

Level 3: AIs build aligned successors

Here, we have an aligned AGI that we entrust with building a successor. At this point, the current aligned AGI has to do all the alignment research required to ensure that its successor is aligned.

Mostly I have been thinking about Levels 1 and 2, although some people I spoke to (e.g. Richard Ngo) we...
Interspecies diplomacy as a potentially productive lens on AGI alignment
by Shariq Hashme — 2022-08-24T17:59:42.084Z

You wake up tomorrow, and find out that super-smart and powerful aliens are arriving here in 10 years. We don't know anything about them other than the fact that they're super smart and powerful: e.g, their minds run on giant computers at THz clock speeds, they can integrate planets' worth of sensor data at once, they can do direct mass to energy conversion in their spaceships, they have weapons capable of destroying our planet in an instant, etc etc.

What do you, personally, do?

What should humanity overall do?

I think the answers to these questions shine a very important light on overlooked AGI alignment strategies. Rather than seeing the arrival of AGI as a technological change, we can see it as the arrival of a potentially competing / potentially collaborating species. Approximate intuition pump: humans :: AGI : animals :: humans -- if you were an emu (about to be hunted to extinction), how would you negotiate with the humans? If you were a monkey, how would you negotiate with the humans?

What are the frameworks we would use to negotiate with supersmart powerful aliens, or which animals might use to negotiate with people?

creating an ultimate ground rules moral framework: if we can create a moral framework which every species could agree to to improve the stable equilibrium of our interspecies game (where the lowest stable equilibrium is violence), then it might be convincing for an AGI coming out of our human values to maintain that ultimate ground rules moral framework. my best idea for that framework is some version of "information is sacred": I have no idea how exactly to measure information, but I think it all sort of ends up working out.

preservation of information: we must try to maintain as much information as possible. most of the universe could be compressed quite a bit, but not humans: our incompressability is what gives us moral worth. the same could be said of animals. how do you determine what can actually be compressed well? it's not physically possible to do perfectly, but you can have heuristics. this is also a good reason to keep animals in special wildlife conservation areas. unfortunately it might accord quite a high moral valence to hard drives: that's fine if humans still make it out in the end
continuation of computation: storing the mind state of a person isn't enough. you want their mind state to continue forwards. so somehow, we

...
The Case Against Digital Sentience: A Summary
by lucas11 — 2022-08-24T17:59:18.415Z

Note: This is meant as an accessible introduction to an interesting, novel, and not widely-known idea about digital sentience primarily due to Andrés Gómez Emilsson and David Pearce. See links at the bottom for the original arguments on which this summary is based. I do my best to portray them as accurately as possible.

Note 2: We use terms like consciousness, sentience, and experience interchangeably. These each refer to intrinsic “what-it's-like”-ness.

Part 1: Reimagining the Problem

tl;dr: The debate about digital sentience is often entirely framed in terms of the algorithms that a computer executes, implicitly assuming that consciousness is computational. If we instead assume that consciousness is the inherent nature of the physical, then we can rule out systems that cannot create unified, bound experiences from being sentient without having to make reference to the algorithms they execute.

Introduction

It seems like people who think about whether modern digital computers can have subjective experience often run into the two following contradictory ideas:

First, that computers can’t be conscious, since they are simply wires and logic gates sending 1s and 0s according to the user’s instruction. It feels like there is something about biological brains that is fundamentally different, as demonstrated by how strange it sounds prima facie to say that a computer can experience pain or pleasure.

But secondly, most people think that consciousness must come from the brain, and the brain is entirely controlled by physical laws. Therefore we should be able to specify how the brain processes information, and emulate its algorithms on a sufficiently powerful computer. Then, in theory, such a computer would also emulate the brain’s conscious experience.

Of course, these are just intuitions, not rigorous arguments. Yet they both feel quite compelling. Given that these conclusions are contradictory, at least one of the arguments must be incorrect. In order to understand more precisely how one of these arguments can be invalid, it is useful to formulate and label their assumptions more formally.

The first intuition relies on two assumptions: that computers are simply wires and logic gates (A1), and that wires and logic gates cannot have subjective experience (A2). The second relies on the three assumptions: that human consciousness comes from the bra...
Why are some problems Super Hard?
by Gabriel Alfour — 2022-08-24T17:58:41.031Z

The problem

I have been wondering a lot about a problem lately, that I believe has far reaching consequences.
Some context: There are many theoretical problems that are considered to be obviously far far outside our problem solving ability. Two examples of those would be the Collatz Conjecture and P vs NP.
My reasoning: If that's the case, for each of those problems, there should be a strong knock-out argument for *why* they are intrinsically hard to solve.
My problem: I don't know of any such argument.

If anyone knows any such knock-down argument for why those problems are so hard, I'd be very interested.

Preemption #1

"Many smart researchers have tried solving them, and failed, and made no progress. This is why it is obvious that those problems are very hard."

I understand this, but this does not help. It does not lead to a causal model for why those problems are hard. Let alone causal, it is not even a straightforward predictive model: to which extent people having failed at it inform that they will keep failing? You need some kind of model or informed prior to talk about that.

Even worse, it might entirely be tautological, where the definition of "hard" is something like "the amount to which we have tried to solve the problem and failed".

It also fails to address the source of my confusion. When I try to describe it more precisely, it looks like "Given that so many geniuses have tried and failed, why don't we have a solid explanation for why it doesn't work?"
The shape of what I am looking for is something like a deep principle, a fundamental reason, or just a simple razor that explains how those problems fall outside the scope of what we know to solve

Phrased differently: it should be possible to define an over-approximation of what we can prove right now, and then show that those problems are not within this.
The more "far outside our problem solving ability" they are, the easier it should be to define this over-approximation and a razor to separate the problem from it.

Preemption #2

"There are typical barriers to P vs NP described here (and many approaches in Scott Aaronson's survey) or to the Collatz's Conjecture"

I understand this, but after reading them, I am still confused. They are more along the line of "We have tried specific proof techniques and failed", more than "they are super intractable".
If you don't make progress on a problem, a natural next best ...
Adversarial epistemology
by jchan — 2022-08-24T16:57:03.165Z

This is the first article in my Bah-Humbug Sequence a.k.a. "Everything I Don't Like Is A Defect/Defect Equilibrium". Epistemic status: strong opinion weakly held, somewhat exaggerated for dramatic effect; I'm posting this here so that the ensuing discussion might help me clarify my position. Anyway, the time has now come for me to explain my overbearing attitude of cynicism towards all aspects of life. Why now, of all times? I hope to make that clear by the end.

You are asking me to believe a certain claim. There is a simple and easy thing you can do to prove its trustworthiness, and yet you have not done that. I am therefore entitled to [Weak Adversarial Argument] disregard your claim as of no evidentiary value / [Strong Adversarial Argument] believe the negation of your claim purely out of spite.

What's going on here? Are these valid arguments?

It may help to give some examples:

The Hearsay Objection - In a court of law, if a witness X tries to testify that some other person Y said Z, in trying to establish the truth of Z, the opposing side may object. This objection takes the form: "The opposition has brought in X to prove Z by way of the fact that Y said Z. But X is not the most reliable witness they could have called, because they could have summoned Y instead. If they were genuinely seeking the truth as to Z, they would have done so; and yet we see that they did not. Therefore I insist that X's testimony be stricken from the record."

The Cynical Cryptographer - My company's HR department emails me a link to an employee satisfaction survey. The email is quick to say "Your responses are anonymous", and yet I notice that the survey link contains a bunch of gobbledegook like ?id=2815ec7e931410a5fb358588ee70ad8b. I think to myself: If this actually is anonymous, and not a sham to see which employees have attitude problems and should be laid off first, the HR department could have set up a Chaumian blind signature protocol to provably guarantee that my response cannot be linked to my name. But they didn't, and so I conclude that this survey is a sham, and I won't fill it out.

So, again, are these valid arguments? From a Bayesian perspective, not really:

X saying that Y said Z is not literally zero evidence of Z. If there is any chance >0 that X and Y are honest, then I must update at least somewhat towards the truth of Z upon hearing X's testimo

...
Ethan Perez on the Inverse Scaling Prize, Language Feedback and Red Teaming
by Michaël Trazzi — 2022-08-24T16:35:43.086Z
Link: https://theinsideview.ai/perez

I talked to Ethan Perez about the Inverse Scaling Prize (deadline August 27!), Training Language Models with Language Feedback and Red-teaming Language Models with Language models.

Below are some highlighted quotes from our conversation (available on Youtube, Spotify, Google Podcast, Apple Podcast). For the full context for each of these quotes, you can find the accompanying transcript.

Inverse Scaling Prize

"We want to understand what, in language model pre-training, objectives and data, is causing models to actively learn things that we don’t want them to learn. Some examples might be that large language models are picking up on more biases or stereotypes about different demographic groups. They might be learning to generate more toxic content, more plausible misinformation because that’s the kind of data that’s out there on the internet.

It’s also relevant in the long run because we want to have a very good understanding of how can we find where our training objectives are training models pick up the wrong behavior, because the training objective in combination with the data defines what exactly it is we’re optimizing the models very aggressively to maximize... this is a first step toward that larger goal. Let’s first figure out how can we systematically find where language models are being trained to act in ways that are misaligned with our preferences. And then hopefully, with those insights, we can take them to understand where other learning algorithms are also failing or maybe how we can improve language models with alternative objectives that have less of the limitations that they have now."

Training Language Models With Language Feedback

Why Language Feedback Instead of Comparisons

"The way that RL from human feedback typically works is we just compare two different generations or outputs from a model. And that gives very little information to the model about why, for example, a particular output was better than another. Basically, it's one bit of information or even less than one bit of information to the model that's doing the generation or output about how it should improve. And there are so many ways an output could be wrong or good or bad, and it's hard to do that attribution of which of the hundred words I generated is good or bad... that was kind of the motivation for us to look for other sources of feedback that are much more informatio

...
Schelling Points in Thing-Space
by Richard Henage — 2022-08-24T16:19:05.248Z

A group of frogs secedes from the Swamp Union and starts their own frog country in a neighboring pond. After some political turmoil and the beheading of any frogs that wished to go back to the swamp, Ugly Frog gathers all the remaining frogs and says they need to make a list of rules. First on the list is "no associating with foreigners." Squeaky Frog asks, "what exactly constitutes a foreigner?" After beheading him for speaking out of turn, the rest of the frogs contemplate this question and decide that a foreigner is anyone who lives outside the new pond. The ghost of Squeaky Frog then remarks that if they send frogs to infiltrate the Swamp Union as sleeper agents, these agents would count as foreigners, and if the Swamp Union sends agents to the pond, those agents would not count as foreigners. The frogs exorcize the ground to destroy all remnants of Squeaky Frog and then begin a furious debate over the definition of "foreigner."

After hours of physical and verbal fighting, two dominant sides emerge, with a few smaller groups. The first side holds that all frogs currently outside the pond should be permanently dubbed "foreign," and all frogs currently in the pond should be permanently dubbed "non-foreign." The second side thinks frogs should count as "foreign" or "non-foreign" based on where their current allegiance lies, and not on any geographical position. They are both determined to shame and ostracize all foreigners, and not listen to their ideas or opinions.

Both of these sides share some common goals. They both want Funny Frog to be a "non-foreigner" because he is beloved by nearly all frogs and brings life to every frog party you can imagine. And both sides also want Hairy Frog to be a "foreigner" because his hair is so un-amphibian that it disgraces their entire species. In fact, both sides are so adamant that these two frogs stay in their respective categories that any definition which did not satisfy this requirement would be seriously frowned upon, and the proponents would be beaten to death in ditches. For example, one of the smaller factions claims that all frogs in the valley should count as "non-foreign," and that only the frogs of the mountains should be "foreigners." But this definition allows Hairy Frog to be a "non-foreigner," so it's rejected by most of the frogs. Another small faction is telling everyone that only the original proponen...