trentmkelly/LessWrong-43k huggingface dataset:
Oversight Leagues: The Training Game as a Feature
by Paul Bricman — 2022-09-09T10:08:03.266Z

This post is part of my hypothesis subspace sequence, a living collection of proposals I'm exploring at Refine. Followed by ideological inference engines.

Thanks Adam Shimi for advice on putting more legible content out there. Thanks Eric Winsor, Leo Grinsztajn, Linda Linsefors, Lucas Texeira, Tammy Leake, Ze Shen for discussions which inspired this post.

TL;DR: An oversight league is a training scheme which incentivizes an agent and an evaluator to constantly try to game each other, leading to synchronized increases in capability for the two players. However, the evaluator is being offered a host of additional learning signals to help it maintain a consistent (and potentially provable) lead over the agent. Oversight leagues heavily draw on ideas from capability literature, including: league training in AlphaStar, game theory in GANs, adversarial robustness, etc.

Intro

The whole project of oversight leagues relies on the following non-exhaustive list of assumptions:

Assumption 1, "AGI Hard, Human Values Harder": We are unlikely to formulate the True Name of human values in closed-form before deploying transformative AI. The best we are likely to do before takeoff is model human values approximately and implement an imperfect evaluator.

Assumption 2, "Linear Capability Ordering": Any fixed evaluator (e.g. a reward model) can be gamed by an agent above a certain threshold of capability. More generally, an agent whose capability improves consistently faster than the capability of an evaluator will eventually be able to game said evaluator. By "evaluator capability," I'm referring to its ability to prevent being gamed.

Assumption 3, "Humans Are Not True Gamers": Human oversight is impractical because our capabilities as evaluators can't improve at an arbitrary large rate. Save for cyborgian schemes for human augmentation, human oversight would eventually be gamed by an agent of sufficient capability.

Assumption 4, "Zone of Proximal Development": There is a relatively small interval of evaluator lead which allows the agent to improve from challenging it, even if the evaluator is slightly ahead. This assumption is motivated by the fact that GANs are actually working, despite an inevitable imbalance between the two components involved at any given time.

Assumption 5, "Point of No Return": Even a minor lead in capability for the agent relative to the evaluator is catastrophi...
Understanding and avoiding value drift
by TurnTrout — 2022-09-09T04:16:48.404Z

I use the shard theory of human values to clarify what value drift is, how it happens, and how it might be avoided by a highly intelligent agent—even if that agent doesn't have any control over its future experiences. Along the way, I give a shard theory account of rationalization.

Defining "value drift"

Recapitulating part of shard theory. Reward is that which reinforces. Considering the case of reinforcement learning in humans, reward causes your brain’s credit assignment algorithms[1] to reinforce the actions and thoughts which led to that reward, making those actions and thoughts more likely to be selected in the future.

For example, suppose you recognize a lollipop, and move to pick it up, and then lick the lollipop. Since the lollipop produces reward, these thoughts will be reinforced and you will be more likely to act similarly in such situations in the future. You become more of the kind of person who will move to pick up a lollipop when you recognize lollipops, and who will navigate to lollipop-containing locations to begin with.

With that in mind, I think that shard theory offers a straightforward definition of "value drift":

Definition. Value drift occurs when reinforcement events substantially change the internal "balance of power" among the shards activated in everyday situations.

For example, consider the classic "example" of taking a pill which makes you enjoy killing people. Under shard theory, this change would be implemented as a murder-shard that activates in a wide range of contexts in order to steer planning towards murder, and therefore starts steering your decision-making substantially differently.

But it's better to try to explain phenomena which, you know, are known to actually happen in real life. Another simple example of value drift is when someone snorts cocaine. At a (substantial) gloss, the huge hit of reward extremely strongly upweights the decision to do cocaine; the strength of the reward leads to an unusually strong cocaine-shard which activates in an unusually wide range of situations.

Here's a more complicated example of value drift. I'll give one possible mechanistic story for the "value drift" which occurs to an atheist (Alice) dating a religious person (Rick), and why that situation might predictably lead to Alice converting or Rick deconverting. I'll consider a scenario where Alice converts.

F...
Samotsvety's AI risk forecasts
by elifland — 2022-09-09T04:01:18.958Z

Crossposted to EA Forum and Foxy Scout

Introduction

In

my review of What We Owe The Future

(WWOTF), I wrote:

Finally, I’ve updated some based on my experience with

Samotsvety forecasters

when discussing AI risk… When we discussed the report on power-seeking AI, I expected tons of skepticism but in fact almost all forecasters seemed to give >=5% to disempowerment by power-seeking AI by 2070, with many giving >=10%.

In the comments,

Peter Wildeford asked

:

It looks like Samotsvety also forecasted AI timelines and AI takeover risk - are you willing and able to provide those numbers as well?

We separately received a request from the

FTX Foundation

to forecast on 3 questions about AGI timelines and risk.

I sent out surveys to get Samotsvety’s up-to-date views on all 5 of these questions, and thought it would be valuable to share the forecasts publicly.

A few of the headline aggregate forecasts are:

25% chance of misaligned AI takeover by 2100, barring pre-

APS-AI

catastrophe
81% chance of

Transformative AI

(TAI) by 2100, barring pre-TAI catastrophe
32% chance of AGI being developed in the next 20 years

Forecasts

In each case I aggregated forecasts by removing the single most extreme forecast on each end, then taking the

geometric mean of odds

.

To reduce concerns of in-group bias to some extent, I calculated a separate aggregate for those who weren’t highly-engaged EAs (HEAs) before joining Samotsvety. In most cases, these forecasters hadn’t engaged with EA much at all; in one case the forecaster was aligned but not involved with the community. Several have gotten more involved with EA since joining Samotsvety.

Unfortunately I’m unable to provide forecast rationales in this post due to forecaster time constraints, though I might in a future post. I provided my personal reasoning for relatively similar forecasts (35% AI takeover by 2100, 80% TAI by 2100) in

my WWOTF review

.

WWOTF questions

Aggregate (n=11)

Aggregate, non-pre-Samotsvety-HEAs (n=5)

Range

What's your probability of misaligned AI takeover by 2100, barring pre-

APS-AI

catastrophe?

25%

14%

3-91.5%

What's your probability of

Transformative AI

(TAI) by 2100, barring pre-TAI catastrophe?

81%

86%

45-99.5%

FTX Foundation questions

For the purposes of these questions, FTX Foundation defined AGI as roughly “AI systems that power a comparably profound transformation (in economic terms or otherwise...
Most People Start With The Same Few Bad Ideas
by johnswentworth — 2022-09-09T00:29:12.740Z

Epistemic status: lots of highly subjective and tentative personal impressions.

Occasionally people say “hey, alignment research has lots of money behind it now, why not fund basically everyone who wants to try it?”. Often this involves an analogy to venture capital: alignment funding is hits-based (i.e. the best few people are much more productive than everyone else combined), funders aren’t actually that good at distinguishing the future hits, so what we want is a whole bunch of uncorrelated bets.

The main place where this fails, in practice, is the “uncorrelated” part. It turns out that most newcomers to alignment have the same few Bad Ideas.

The most common, these days, is some variant of “train an AI to help with aligning AI”. Sometimes it’s “train an AI to interpret the internals of another AI”, sometimes it’s “train an AI to point out problems in another AI’s plan”, sometimes it’s “train an AI to help you design aligned AI”, etc. I would guess about 75% of newcomers from ML suggest some such variant as their first idea.

People who are less aware of standard alignment arguments tend to start with “train an AI on human feedback” or “iterate until the problems go away”. In the old days, pre-sequences, people started from even worse ideas; at least the

waterline

has risen somewhat.

People with more of a theory bent or an old-school AI background tend to reinvent IRL or CIRL variants. (A CIRL variant was my own starting Bad Idea - you can read about it in

this post

from 2020, although the notes from which that post was written were from about 2016-2017.)

My impression (based on very limited data) is that it takes most newcomers ~5 years to go from their initial Bad Idea to actually working on something plausibly useful. For lack of a better name, let’s call that process the Path of Alignment Maturity.

My impression is that progress along the Path of Alignment Maturity can be accelerated dramatically by actively looking for problems with your own plans - e.g. the builder/breaker framework from the

Eliciting Latent Knowledge

doc, or some version of the

Alignment Game Tree exercise

, or having a group of people who argue and poke holes in each others’ plans. (Of course these all first require not being too emotionally attached to your own plan; it helps a lot if you can come up with a second or third line of attack, thereby building confidenc...
Monitoring for deceptive alignment
by evhub — 2022-09-08T23:07:03.327Z

This post is a follow-up to “AI coordination needs clear wins.” Thanks to Ethan Perez, Kate Woolverton, Richard Ngo, Anne le Roux, Sam McCandlish, Adam Jermyn, and Danny Hernandez for useful conversations, comments, and feedback.

In this post, I want to propose a clear, concrete coordination task that I think might be achievable soon given the current landscape, would generate a clear coordination win, and that I think would be highly useful in and of itself. Specifically:

I want DeepMind, OpenAI, and Anthropic to commit to actively monitor and look for evidence of deceptive alignment in their models—as well as run experiments to try to predict when and where deceptive alignment might occur before it does.

Notably, I am specifically referring only to the narrow case of deceptive alignment here, not just any situation where models say false things. Deceptive alignment is specifically a situation where the reason the model looks aligned is because it is actively trying to game the training signal for the purpose of achieving some ulterior goal.[1]

I think this is a pretty minimal ask that would nevertheless be a clear win. Among all AI failure modes, deceptive alignment is one of the most unambiguously bad, which means, though I expect lots of disagreement on its likelihood, there should be little disagreement regarding its severity. Furthermore, things that I’m not asking for:

that they make such a commitment highly public or legally binding,

that they commit to any specific way of monitoring for or addressing the problem,

that any organization has to share anything they find with any other organization, or

that they commit to anything other than testing and monitoring.

Nevertheless, as I stated, I think this would still be quite useful.

First, I think deceptive alignment is clearly one of the most concerning problems in AI alignment. Though they don’t always use the framing of deceptive alignment specifically, most presentations of AI risk feature the phenomenon of an AI that wants something in the world deceiving our oversight procedures to get that something. See: treacherous turns, schemers, influence-seeking behavior, power-seeking AI. Furthermore, I think there is a strong case for why deceptive alignment is reasonably likely to appear by default in advanced machine learning systems.

Second, I think that it is plausible that, if deceptive alignment is event...
[An email with a bunch of links I sent an experienced ML researcher interested in learning about Alignment / x-safety.]
by David Scott Krueger (formerly: capybaralet) — 2022-09-08T22:28:54.534Z

[[FYI: I'm just copying this in and removing a few bits; apologies for formatting; I don't intend to post the attachments]]
EtA: I (embarrasingly and unfortunately) understated Richard Ngo's technical background in email; I left what I originally wrote in strikethrough.
EtA(2): I thought a bit more context would be useful here:
- This email was designed for a particular person after several in person conversations.
- I'm not proposing this as anything like "The best things to send to an arbitrary 'experienced ML researcher interested in learning about Alignment / x-safety'".
- I didn't put a ton of effort into this.
- I aimed to present a somewhat diverse and representative sampling of x-safety stuff.
EtA (3): Suggestions / Feedback welcome!
OK I figure perfect is the enemy of the good and this is already a doozy of an email, so I'm just going to send it :)
I think it would be great to get a better sense of what sort of materials are a good introduction for someone in your situation, so please let me know what you find most/least useful/etc.!
A few top recommendations...

- read about DeepMind's "safety, robustness, and assurance" breakdown of alignment
- sign up for the Alignment and ML Safety newsletters, and skim through the archives

- Look at this syllabus I mocked up for my UofT application (attached). I tried to focus on ML and include a lot of the most important ML papers, although there's a lot missing.

- Read about RL from human preferences if you haven't already. I imagine you might've seen the "backflipping noodle" blog post. I helped author a research agenda based on such approaches Scalable agent alignment via reward modeling. The research I talked about in my talk is part of a project on understanding reward model hacking; we wrote a short grant proposal for that (attached). I'm in the midst of rethinking how large a role I expect reward modeling (or RL more generally) to play in future AI systems, but I previously considered this one of the highest priority directions to work on. Others (e.g. Jan, first author of the agenda) are working on showing what you can do with reward modeling; I'm more concerned with figuring out if it is a promising approach at all or not (I suspect not, because of power-seeking / instrumental convergence).

- Look at the "7 Alternatives for agent alignment" in the agenda for a brief overview of alte...
Progress links & tweets, 2022-09-08
by jasoncrawford — 2022-09-08T20:43:03.121Z
Link: https://rootsofprogress.org/links-and-tweets-2022-09-08

Opportunities

Nat Friedman and Daniel Gross are investing $10M in AI-first products, apply here (via @natfriedman)
Congressman Jake Auschincloss is hiring a Chief of Staff, @MarkLutter encourages the progress community to apply

Announcements

Superabundance: The Story of Population Growth, Innovation, and Human Flourishing on an Infinitely Bountiful Planet is a new book out from Tupy & Pooley. Jeremy Horpedahl has a brief review.. Arnold Kling is slightly critical; I found his comments interesting but not fully convincing
Prediction market Kalshi is seeking approval to create election contracts, add your comment to the CFTC (thread from CEO @mansourtarek_)
Aletheia is a new platform for communicating peer-evaluated research (via @namalhotra)

Links

I don’t think we can assign quantitative probabilities to all our beliefs, but this essay argues that if we make coherent choices, then we have them at least implicitly
“Not even death can exempt you from TSA screening”

Queries

Is there a case in the last century where a new technology didn’t become cheap enough for everyone? (@davidmanheim)
Who should Derek Thompson talk to about why the British economy is such a mess? (@DKThomp)
Is there a nice comparable table of rocket turbopumps by flow rate and pressure head? (@anderssandberg)
Did Queen Elizabeth I really inhibit the development of the stocking frame?

Quotes

Gutenberg had to solve many intricate problems to make the printing press
Bacon anticipating Lakatos
Jefferson anticipating Paul Romer
Sawmills outnumbered even gristmills in 19th-century America
“Life in the big city,” London, 1850s

Tweets & retweets

Norway’s tunnel: < $7M/km; Seattle’s tunnel: $1,167M/km
Diablo Canyon will continue operations until at least 2030 (@ANS_org)
The more money an organization gives out, the less likely it is to fund unusual people (@SamoBurja)
Beware the deepity (@JeremiahDJohns)

Charts

The shift to WFH is saving ~200M hours and 6B miles of commuting/week in the US alone (@I_Am_NickBloom)
Postmortem: Trying out for Manifold Markets
by Milli | Martin — 2022-09-08T17:54:09.890Z

Intro

I tried out for Manifold Markets in May and June 2022. It didn't work out, welcome to the postmortem.

Here's an article by the founders about Manifold, some of this post won't make sense if you don't know anything about prediction markets.

Timeline

I found out in May that Manifold Markets was hiring. A long shot, or so I thought. But since sending an email with my resume is low effort, I applied.

I don't know if my email was read, but Austin quickly replied to the market I created (above) and my chances "skyrocketed" from 4% to 25%. According to the market.

The video call was scheduled: Normal working hours in the US, 9:30 pm for me. When Austin showed up a few minutes late, I was relieved that he hadn't stood me up. The call was very productive, and afterwards I had a feature to implement as a work trial.

Over the next few weeks, I worked on at the feature in my spare time, also fixing some bugs and making other minor changes to Manifold. On the social side, I chatted with the core team and other users on Discord. It was a generally a good time.

But it became clear that it wasn't going to work out.

After another call with Austin we decided to part ways. Someone from the core team took over the feature, but it was scrapped after a week.

The Feature

The feature was chosen semi-spontaneously during the call. The goal was to allow markets to be resolved by mechanisms than the market creator – an API, a rule, or by other person(s). In its entirety, this was way too much to get into the project. But it worked as a direction.

It became clear (to me) that the most pressing problem was abandoned markets: Markets where the "money" was stuck because the market creators had (apparently) left the platform.

The exact specifications went through a few iterations, and this is what I settled on in the end:

Automatic resolution means that unresolved markets get resolved as 'MKT' (paying out investors according to the current odds) at a specified date. By default that date is 1 week after the market closes but it can be modified by the market creator.

GitHub: https://github.com/manifoldmarkets/manifold/pull/506

What went wrong

First things first: This might not have worked out if I had done everything perfectly. I wasn't the perfect person for the job.
But I think there was a better than 50% chance that it would have worked out.

My first mistake was the very firs...
Thoughts on AGI consciousness / sentience
by Steven Byrnes — 2022-09-08T16:40:34.354Z

Summary

I’ll walk through my take on how consciousness works (basically endorsing “illusionism”) and why I think computer programs can be conscious, and how that relates to moral and ethical obligations towards conscious entities in general and AIs in particular.

Up to here, everything I say will be pretty popular (albeit far from universal) opinions within the rationality / EA communities, from what I gather.

…But then the last section is my (I think) unusual-in-this-community take that we’re probably going to get conscious AGIs, even if we don’t want them and are trying to avoid them. I will argue that human-like consciousness is pretty closely entangled with the powerful capabilities that we need for a system to do things like autonomously invent technologies and solve hard problems.

Suggested audience: People who are curious about what I think about this topic, and/or want to talk about it with me, for whatever reason, despite the fact that I’m really not up on the literature compared to other people.

(If you’re just generally interested in AGI consciousness / sentience, a better starting point would probably be any of the articles

here

.)

Quick points before starting

I think the Blake Lemoine thing was very stupid. (I’ll explain why below.) I hope that people will not permanently associate “thinking about AGI sentience” with “that stupid Blake Lemoine thing”.
I feel sorely tempted to say “Don’t we have our hands full thinking about the impact of AGI on humans?” But on reflection, I actually don’t want to say that. AGI sentience is an important topic and we should be thinking about it.
This post will NOT discuss what to do about AGI sentience right now (if anything). That’s a complicated question, tied up with every other aspect of AGI deployment. For example, maybe the AGIs will wind up in charge, in which case we hardly need to be advocating for their well-being—they can take care of themselves! Anyway, I’ll talk about that kind of stuff in a different upcoming post, hopefully.

My take on the philosophy of consciousness / sentience, in a nutshell

(Side note: I haven’t read much philosophy of consciousness, sorry. I’m open to discussion.)

As background: when I say (sincerely) “I’m looking at my wristwatch, and it says it’s 8:00”, then somewhere in the chain of causation that led to me saying that, is my actual physical wristwatch (which reflects photo...