trentmkelly/LessWrong-43k huggingface dataset:
“Endgame safety” for AGI
by Steven Byrnes — 2023-01-24T14:15:32.783Z

(Status: no pretense to originality, but a couple people said they found this terminology useful, so I’m sharing it more widely.)

There’s a category of AGI safety work that we might call “Endgame Safety”, where we’re trying to do all the AGI safety work that we couldn’t or didn’t do ahead of time, in the very last moments before (or even after) people are actually playing around with powerful AGI algorithms of the type that could get irreversibly out of control and cause catastrophe.

I think everyone agrees that Endgame Safety is important and unavoidable. If nothing else, for every last line of AGI source code, we can do an analysis of what happens if that line of code has a bug, or if a cosmic ray flips a bit, and how do we write good unit tests, etc. But we’re obviously not going to have AGI source code until the endgame. That was an especially straightforward example, but I imagine that there will be many other things that also fall into the Endgame Safety bucket,[1] i.e. bigger-picture important things to know about AGI that we only realize when we’re in the thick of building it.

So I am not an “Endgame Safety denialist”; I don’t think anyone is. But I find that people are sometimes misled by thinking about Endgame Safety, in the following two ways:

Bad argument 1:

“Endgame Safety is really important. So let’s try to make the endgame happen ASAP, so that we can get to work on Endgame Safety!”

This is a bad argument because, what’s the rush? There’s going to be an endgame sooner or later, and we can do Endgame Safety Research then! Bringing the endgame sooner is basically equivalent to having all the AI alignment and strategy researchers hibernate for some number N years, and then wake up and get back to work. And that, in turn, is strictly worse than having all the AI alignment and strategy researchers do what they can during the next N years, and also continue doing work after those N years have elapsed.

I claim that there are plenty of open problems in AGI safety / alignment that we can do right now, that people are in fact working on right now, that seem robustly useful, and that are not in the category of “Endgame Safety”, e.g. my list of 7 projects, these 200 interpretability projects, this list, ELK, everything on Alignment Forum, etc.

For example, sometimes I’ll have this discussion:

ME: “I don’t want to talk about (blah) aspect of how I

...
Thoughts on hardware / compute requirements for AGI
by Steven Byrnes — 2023-01-24T14:03:39.190Z

[NOTE: I have some updates / corrigenda at the bottom. ]

Let’s say I know how to build / train a human-level (more specifically, John von Neumann level) AGI. And let’s say that we (and/or the AGI itself) have already spent a few years[1] on making the algorithm work better and more efficiently.

Question: How much compute will it take to run this AGI?

(NB: I said "running" an AGI, not training / programming an AGI. I'll talk a bit about “training compute” at the very end.)

Answer: I don’t know. But that doesn’t seem to be stopping me from writing this post. ¯\_(ツ)_/¯ My current feeling—which I can easily imagine changing after discussion (which is a major reason I'm writing this!)—seems to be:

75%: One current (Jan 2023) high-end retail gaming PC (with an Nvidia GeForce RTX 4090 GPU) will be enough (or more than enough) for human-level human-speed AGI,
85%: One future high-end retail gaming PC, that will on sale in a decade (2033)[2], will be enough for human-level AGI, at ≥20% human speed.

This post will explain why I currently feel this way.

Table of Contents / TL;DR

In the prologue (Section 1), I’ll give four reasons that I care about this question: one related to our long-term prospects of globally monitoring and regulating human-level AGI; one related to whether an early AGI could be “self-sufficient” after wiping out humanity; one related to whether AGI is even feasible in the first place; and one related to “compute overhangs”. I’ll also respond to two counterarguments (i.e. arguments that I shouldn’t care about this question), namely: “More-scaled-up AGIs will always be smarter than less-scaled-up AGIs; that relative comparison is what we care about, not the absolute intelligence level that’s possible, on, say, a single GPU”, and “The very earliest human-level AGIs will be just barely human-level on the world’s biggest compute clusters, and that’s the thing that we should mainly care about, not how efficient they wind up later on”.
In Section 2, I’ll touch on a bit of prior discussion that I found interesting or thought-provoking, including a claim by Eliezer Yudkowsky that human-level human-speed AGI requires ridiculously little compute, and conversely a Metaculus forecast expecting that it requires orders of magnitude more compute than what I'm claiming here.
In Section 3, I’ll argue that the amount of computation used b

...
Parameter Scaling Comes for RL, Maybe
by 1a3orn — 2023-01-24T13:55:46.324Z

TLDR

Unlike language models or image classifiers, past reinforcement learning models did not reliably get better as they got bigger. Two DeepMind RL papers published in January 2023 nevertheless show that with the right techniques, scaling up RL model parameters can increase both total reward and sample-efficiency of RL agents -- and by a lot. Return-to-scale has been key for rendering language models powerful and economically valuable; it might also be key for RL, although many important questions remain unanswered.

Intro

Reinforcement learning models often have very few parameters compared to language and image models.

The Vision Transformer has 2 billion parameters. GPT-3 has 175 billion. The slimmer Chinchilla, trained in accord with scaling laws emphasizing bigger datasets, has 70 billion.

By contrast, until a month ago, the largest mostly-RL models I knew of were the agents for Starcraft and Dota2, AlphaStar and OpenAI5, which had 139 million and 158 million parameters. And most RL models are far smaller, coming in well under 50 million parameters.

The reason RL hasn't scaled up the size of its models is simple -- doing so generally hasn't made them better.

Increasing model size in RL can even hurt performance. MuZero Reanalyze gets worse on some tasks as you scale network size. So does a vanilla SAC agent.

There has been good evidence for scaling model size in somewhat... non-central examples of RL. For instance, offline RL agents trained from expert examples, such as DeepMind's 1.2-billion parameter Gato or Multi-Game Decision Transformers, clearly get better with scale. Similarly, RL from human feedback on language models generally shows that larger LM's are better. Hybrid systems such as PaLM SayCan benefit from larger language models. But all these cases sidestep problems central to RL -- they have no need to balance exploration and exploitation in seeking reward.

In the typical RL setting -- there has generally been little scaling and little evidence for the efficacy of scaling. (Although there has not been no evidence.)

None of the above means that the compute spent on RL models is small or that compute scaling does nothing for them. AlphaStar used only a little less compute than GPT-3, and AlphaGo Zero used more, because both of them trained on an enormous number of games. Additional compute predictably improves performance of RL agents. But, rather t...
How to find cool things in a new place
by Sam F. Brown — 2023-01-24T11:20:34.345Z

A friend asked me for advice on how to find stuff that's going on in a place. I feel a bit self-conscious positioning myself as some kind of expert here, but I do think I've done a good job of this in Oxford. None of this is galaxy-brain stuff, just advice that I hope might be helpful for people.

The main trick is to talk to people. This is true, and bears repeating: talk to people. It helps a lot to cultivate a relentless curiousity, and to be comfortable being out of place (or at least, to let the discomfort wash over you). It's people who make these things happen, publicity etc is secondary.

Notice what you're interested in, what you enjoy, and do it even if it's kinda lame. Mention the mildly-embarrassing stuff you're doing, early in conversation. This will help people know what to recommend to you.

There's a saying, that if you wanna source drugs in a new place, ask your sketchiest friend who their sketchiest friend is, meet them, and repeat. This is true in general: if you wanna find a scene, talk to the people you know who are most in that direction, and ask them about their friends.

If you're at a place and it's good, ask the people there how they found out about it. Often, it's word-of-mouth, but sometimes there's a hub. In Oxford, the answer might be something like "a flier at the Magic Cafe" or "Instagram" or "Daily Info". Check these places, see what's there.

I don't use dating apps any more, but some of my most important let's-say-platonic friendships have come from that. Being bi is helpful, but whatever your situation I recommend matching/meeting people explicitly for friendship.

Perhaps the strongest move you can make is to host something. What do you most wish existed in your area, that doesn't seem to? If you can make this happen, maybe starting with a very small group, then good people will start to come to you. Hosting Oxford Rationalish has brought me in contact with so many people who I'd otherwise never have found.

Oh! And it's more important to take lots of shots than to have a perfect hit rate. I really like ending up in random places that don't make sense and that I probably wouldn't revisit, which exposes me to more upside.

Hoping to be helpful <3
[Crosspost] ACX 2022 Prediction Contest Results
by Scott Alexander — 2023-01-24T06:56:33.101Z

Original here.

Submission statement/relevance to Less Wrong: This forecasting contest confirmed some things we already believed, like that superforecasters can consistently outperform others, or the "wisdom of crowds" effect. It also found a surprising benefit of prediction markets over other aggregation methods, which might or might not be spurious.

Several members of the EA and rationalist community scored highly, including one professional AI forecaster. But Less Wrongers didn't consistently outperform members of the general (ACX-reading, forecasting-competition-entering) population.

Last year saw surging inflation, a Russian invasion of Ukraine, and a surprise victory for Democrats in the US Senate. Pundits, politicians, and economists were caught flat-footed by these developments. Did anyone get them right?

In a very technical sense, the single person who predicted 2022 most accurately was a 20-something data scientist at Amazon’s forecasting division.

I know this because last January, along with amateur statisticians Sam Marks and Eric Neyman, I solicited predictions from 508 people. This wasn’t a very creative or free-form exercise - contest participants assigned percentage chances to 71 yes-or-no questions, like “Will Russia invade Ukraine?” or “Will the Dow end the year above 35000?” The whole thing was a bit hokey and constrained - Nassim Taleb wouldn’t be amused - but it had the great advantage of allowing objective scoring.

Sample questions.

Our goal wasn’t just to identify good predictors. It was to replicate previous findings about the nature of prediction. Are some people really “superforecasters” who do better than everyone else? Is there a “wisdom of crowds”? Does the Efficient Markets Hypothesis mean that prediction markets should beat individuals? Armed with 508 people’s predictions, can we do math to them until we know more about the future (probabilistically, of course) than any ordinary mortal?

After 2022 ended, Sam and Eric used a technique called log-loss scoring to grade everyone’s probability estimates. Lower scores are better. The details are hard to explain, but for our contest, guessing 50% for everything would give a score of 40.21[1], and complete omniscience would give a perfect score of 0.

Here’s how the contest went:

Note truncated vertical axis

As mentioned above: guessing 50% corresponds to a score of 40.2. This would have put you in...
The Human-AI Reflective Equilibrium
by Allison Duettmann — 2023-01-24T01:32:44.906Z

A pluralist perspective on AI Alignment

Introduction

AI progress is accelerating, leading to some pretty crazy predictions about “

why the next decades might be wild

.” I won’t debate those here but, following

Ajeya Cotra

, I will just assume we’ll hit some sort of human-level Artificial General Intelligence (AGI) in the next few decades. This is cause for both excitement at AI-enabled breakthroughs and for concern about whether this future is human-compatible.

Fortunately, work on aligning AGI with human values is also picking up. To be successful, it must address at least two really hard problems:

Alignment: How do we communicate our values to a non-human AGI that lacks our biological history, cultural context, and physical hardware?
Values: What are our values anyways?

The first problem is the technical domain of alignment, and receives most of the attention in the AI safety community. The second problem is the normative domain of ethics, which requires asking some hard questions that have escaped philosophical agreement for thousands of years.

I think the second problem is somewhat understudied in relation to AI because it is less tractable than technical alignment work and feels a bit airy fairy. But since we have to address it at some point, here I outline some lessons from human ethics that are relevant for AGI.

In particular, my goal is to do three things:

Caution against hopes that universal moral discovery, convergence, or agreement is possible or even useful for AGI alignment. Instead, we could adapt the “Reflective Equilibrium” from human ethics for AI ethics, and encourage especially those training AI systems to explore it in the hope that they align AIs with a more coherent version of their individual morality.
Lend support to multi agent alignment techniques such as “Debate” because they enable Reflective Equilibrium-esque deliberation styles.
Point to a scenario that is defined by a diversity of AGIs, separated by boundaries, watching and cooperating with each other and with humans.

If you know your undergraduate moral philosophy 101, please skip to the section entitled “Solving” Human Ethics: The Reflective Equilibrium

If you know the Reflective Equilibrium, please skip to the section entitled “Solving” AI Ethics: A Diversity of AIs Trained On Reflective Equilibria

In general, my aspiration is t...
"Status" can be corrosive; here's how I handle it
by Orpheus16 — 2023-01-24T01:25:04.539Z

Do you ever think about your social status? I do, and sometimes I find it

corrosive

.

Examples of thoughts that might come up:

Should I share X doc/idea with Alice? Her feedback might be useful, but it might make her think worse of me.
Why did my friend Bob get invited to X event and I didn’t?
I wonder if Carol has a negative impression of me.
David responds to my friend but not me. Why?

In many situations, these thoughts are helpful. For example, it might be good to occasionally examine if “Carol” has a negative impression of me, especially if Carol is someone I respect, because it might help me find areas where I can grow.

But sometimes, these thoughts are unproductive, limit my thinking, or distract me from more important tasks at hand.

In the rest of this post, I’ll describe some techniques/frames that I’ve found helpful when I notice status coming up in corrosive ways.

Note: If you haven’t read “PR” is corrosive; “reputation" is not

,

I suggest reading that first.

My Rob Bensinger shoulder model

My Rob Bensinger shoulder represents integrity, honesty, and transparency. He tells me things like:

What do you actually believe? How can you communicate it accurately?
What kind of community do you want to foster? Do you want to be in a community where people are afraid to voice their true opinions?
Sharing our beliefs and ideas openly is how we get closer to the truth.

“Be the sort of person internally who you would find most admirable and virtuous. Be brave, be thoughtful, be discerning, be honest, be honorable, be fair, be compassionate, be trustworthy; and insofar as you’re not those things, be honest about it.” --

Rob

When I talk to my Rob Bensinger shoulder model, he reminds me that the options are not merely “think about how others perceive you” vs. “don’t pay attention at all to your self-presentation.” There is a nice third option which looks something like “imagine a person who is scoring well on a bunch of virtuous axes, and aim to be that person. Be honest, share your views, and aim for integrity rather than likability.”

Habryka’s comment about identifying as an EA

When it comes to status concerns, my Habryka shoulder model is based on one comment he wrote in 2021. He reminds me to notice if my attachment to the community is harming my ability to think and act in the world. And he reminds me that identifying strongly as part of the community...
What area of the digital domain seems safe from AI in the next 5-10 years?
by Adrien Chauvet — 2023-01-24T01:16:22.010Z

I see a lot of digital services being built when they will quickly be automated by artificial intelligence. It's as if there's a disconnect between people who are aware of the rapid advances in AI and the average person. I see the same thing in education in my country, France. Students are preparing to go to university for a degree and skills that will be completely obsolete in 5 years. Even in computer science, everyone is promoting the idea that you have to learn to code to become a code worker, while automation tools are advancing at a rapid pace.
As for content creators, the on-demand generation of text, video, and music could quickly make them irrelevant because most people copy other people. It's as if only people with real creativity will survive. I suspect that AI will tell us within 30 seconds that our innovative idea already exists on the Internet.
I read on Twitter that the age of hackers is over and the age of people with ideas is beginning.
I get lost every time I think about what will not soon be automated by AI in the digital domain.
Some of my disagreements with List of Lethalities
by TurnTrout — 2023-01-24T00:25:28.075Z

This was an appendix of Inner and outer alignment decompose one hard problem into two extremely hard problems. However, I think the material is self-contained and worth sharing separately, especially since AGI Ruin: A List of Lethalities has become so influential. (I agree with most of the points made in AGI Ruin, but I'm going to focus on disagreements in this essay.) (Stricken on 1/9/24)

Here are some quotes with which I disagree, in light of points I made in Inner and outer alignment decompose one hard problem into two extremely hard problems (consult its TL;DR and detailed summary for a refresher, if need be).

List of Lethalities

“Humans don't explicitly pursue inclusive genetic fitness; outer optimization even on a very exact, very simple loss function doesn't produce inner optimization in that direction. This happens in practice in real life, it is what happened in the only case we know about, and it seems to me that there are deep theoretical reasons to expect it to happen again”

(Evolution) → (human values) is not the only case of inner alignment failure which we know about. I have argued that human values themselves are inner alignment failures on the human reward system. This has happened billions of times in slightly different learning setups.

Strictly separately, it seems to me that people draw rather strong inferences from a rather loose analogy with evolution. I think that

(evolution) → (human values) is far less informative for alignment than (human reward circuitry) → (human values)

. I don’t agree with a strong focus on the former, given the latter is available as a source of information.

We want to draw inferences about the mapping from (AI reward circuitry) → (AI values), which is an iterative training process using reinforcement learning and self-supervised learning. Therefore, we should consider existing evidence about the (human reward circuitry) → (human values) setup, which (AFAICT) also takes place using an iterative, local update process using reinforcement learning and self-supervised learning.

Brain architecture and training is not AI architecture and training, so the evidence is going to be weakened. But for nearly every way in which (human reward circuitry) → (human values) is disanalogous to (AI reward circuitry) → (AI values), (evolution) → (human values) is even more di...
Rounding Someone Off
by David Udell — 2023-01-24T00:03:48.682Z

Epistemic status: Exploratory and not profoundly confident. This is a quickly written-up presentation of an idea I've picked up.

Differing Models of the Same Person

To the left we see Brown's intentional interpretation of Ella; to the right, Jones's interpretation. Since these are intentional interpretations, the pixels or data points represent... personal cognitive idiosyncracies (e.g., "She thinks she should get her queen out early")…

Notice that here the disagreements can be substantial -- at least before the fact: when Brown and Jones make a series of predictive bets, they will not always make the same bet. They may often disagree on what, according to their chosen pattern, will happen next. To take a dramatic case, Brown may predict that Ella will decide to kill herself; Jones may disagree. This is not a trivial disagreement of prediction, and in principle this momentous difference may emerge in spite of the overall consonance of the two interpretations.

Suppose, then, that Brown and Jones make a series of predictions of Ella's behavior, based on their rival interpretations. Consider the different categories that compose their track records. First, there are the occasions where they agree and are right. Both systems look good from the vantage point of these successes. Second, there are the occasions where they agree and are wrong. Both chalk it up to noise, take their budgeted loss and move on to the next case. But there will also be the occasions where they disagree, where their systems make different predictions, and in these cases sometimes (but not always) one will win and the other lose. (In the real world, predictions are not always from among binary alternatives, so in many cases they will disagree and both be wrong.) When one wins and the other loses, it will look to the myopic observer as if one "theory" has scored a serious point against the other, but when one recognizes the possibility that both may chalk up such victories, and that there may be no pattern in the victories which permits either one to improve his theory by making adjustments, one sees that local triumphs may be insufficient to provide any ground in reality for declaring one account a closer approximation of the truth.

Now, some might think this situation is always unstable; eventually one interpretation is bound to ramify better to new cases, or be deducible from some larger schem

...