trentmkelly/LessWrong-43k huggingface dataset:
Strategy For Conditioning Generative Models
by james.lucassen — 2022-09-01T04:34:17.484Z

This post was written under the mentorship of Evan Hubinger, and assisted by discussions with Adam Jermyn and Johannes Treutlein. See also

their

previous

posts

on this project.

Summary

Conditioning Generative Models (CGM) is a strategy to accelerate alignment research using powerful semi-aligned future language models, which seems potentially promising but constrained by several tradeoffs.
When we have to succeed despite multiple independent failure modes, we should favor trade-offs that move risk away from the more probable failure modes towards less probable ones.
The risk of a manipulative CGM simulation is currently very low. We can use CGM to make beneficial tradeoffs against more dangerous failure modes, such as running out of time or downside risks from a pivotal act gone wrong.
The factors that make a CGM strategy good are mostly the same as the factors that make a non-CGM strategy good, but there are a few important ways they come apart.
Depending on how likely our strategy is to succeed and what the generative model’s prior for manipulative AGI is, it might be better to simulate our plan all in one step, or batch it into multiple separate simulations, or ignore simulations and just execute the plan in the real world.
Lots of open questions!

Motivation

To quickly recap the central puzzle laid out in previous posts on conditioning generative models: we would like to use our generative model to do alignment research. We could do this by using a conditional generative model to simulate an alternate world, conditioned on some facts which would lead to good alignment research if they were true. We can then plagiarize this research and win! However, to get results significantly better than what our world seems on track to produce anyway, we have to start asking for unlikely scenarios. If we ask for scenarios that are too unlikely, the model may start to disbelieve that our scenario happened “naturally”. In particular, the model may assume that there exists some AI agent behind the scenes manipulating events into this highly unlikely outcome. There are many possible motivations for an AI agent to do this: believing it may be in a simulation (even if the world around it seems perfectly realistic), or

evidential cooperation in large worlds

, or a

Counterfactual Mugging

. This manipulative AGI could then control the output of our sim...
Safety Committee Resources
by jefftk — 2022-09-01T02:30:01.389Z

A friend recently asked me for some advice about starting a safety
committee for their contra dance. This got me pulling together some
resources, so I thought I'd share them here as well.

Overall, I think safety committees can be positive, but there are also
a lot of ways you can accidentally make things worse. For example, in
BIDA's case, when we first started
we did not do a very good job laying out what people could expect from
us. A group of amateur volunteers is, for example, not generally
going to be able to get to the bottom of disputes, and by leading
people to expect service beyond what we could provide we let them
down. Which is quite bad: sharing traumatic details is generally
painful, and you really don't want to leave someone wishing they had
never come to you.

I think the current version of our "How We Can Help"
page does pretty well here, but earlier versions didn't, and we also
didn't do a good job in conversations with people reporting issues. I wrote some about this re-evaluation
at the time.

Other resources, for people who are thinking of starting doing this
kind of work:

Julia Wise's The
community health team's work on interpersonal harm in the
community. She does this work professionally in
the EA community, and this post both lays out some of how her team
thinks about this work and gives a long list of situations they've
handled. While the EA community is not a dance community, there's
still a lot that's relevant.

My Taking Someone
Aside and Taking a Safety
Report, getting into some detail about two specific components of
the work.

Dennis Merritt's
Tales
of a Dance Monarch. He walks through eleven different situations
he was involved in over the years, and I think the post is very
helpful for getting a sense of what sorts of situations come up in a
dance context.

Tales
From the Front Line of Ombuds Work: Handling Sexual Harassment
Cases. Twelve anonymized cases in a university context. Somewhat
less applicable, since it's such a different setting and framework,
but it's still helpful to read example cases.

If anyone else has resources to recommend, please share!

Comment via: facebook
Progress links and tweets, 2022-08-31
by jasoncrawford — 2022-08-31T21:54:57.424Z
Link: https://rootsofprogress.org/links-and-tweets-2022-08-31

Links

The Breakthrough Institute is hiring directors to run its new DC office (via @atrembath)
Stable Diffusion: an AI image generator you can download and run (via @elidourado)

Queries

Why didn’t clockwork technology get applied to other practical purposes for hundreds of years?
Who are the best economic historians that focus a lot on technology? (@_brianpotter)
What questions in economics have been “settled” since 1945? (@dkedrosky)

Quotes

We should all have a little more awe and gratitude for the amazing world we live in
In 1809, books or carpets in the home were considered luxuries
Voltaire on the religious tolerance engendered by capitalism

Tweets

Let’s retire the phrase “lifted out of poverty”
Negativity should be productive
Enantiodromia
by ChristianKl — 2022-08-31T21:13:35.496Z

Oxford Languages defines enantiodromia as “the tendency of things to change into their opposites, especially as a supposed governing principle of natural cycles and of psychological development”. While Plato’s works are so influential that the project of Western philosophy is sometimes described as footnotes to Plato, his discussion of enantiodromia is not part of our standard Western worldview. To the extent that the principle of enantiodromia exists, it matters individually, it matters for choosing the right public policy and it matters for AI safety.

Enantiodromia in Jewish college admissions

One practical example of enantiodromia is a public policy with the goal of creating racial equality. In the 20th century, elite universities found that they admitted a higher proportion of Jewish people than people of other backgrounds. They were admitting more Jewish people because Jewish people scored on average higher than non-Jewish people. To cut down the number of Jewish admissions they thought up a clever way where they added subjective character judgments as an admission criterion which allowed them to cut down the percentage of Jewish admission.

In a world where the performance of a lawyer or doctor correlates with their SAT score if the SAT score is the only thing that matters for admissions, whether or not a lawyer or doctor is Jewish is not going to correlate strongly with their SAT scores. Once you have a policy, where Jewish people need a higher SAT score to be accepted, the Jewish lawyers and doctors will on average have a higher SAT score and thus higher competency.

This competency difference in turn reinforces stereotypes. In a world, where the admission criteria result in Jewish lawyers being on average more competent than non-Jewish lawyers the stereotype of Jewish lawyers being more competent is justified. A smart person who seeks a competent lawyer has an easier time determining whether the last name of the lawyer sounds Jewish than determining the SAT score of the lawyer.

In the end, the policy to create equality between Jewish people and non-Jewish created stereotypes about how Jewish people are unequal. This is enantiodromia.

Given that the dynamic of enantiodromia does not exist in the public consciousness it’s very hard to talk about enantiodromia plays out with modern social justice policies, so it’s left as an exercise fo...
New 80,000 Hours problem profile on existential risks from AI
by Benjamin Hilton — 2022-08-31T17:36:13.022Z
Link: https://80000hours.org/problem-profiles/artificial-intelligence

80,000 Hours has just released an updated version of our problem profile on reducing existential risks from AI.

This post includes some context, the summary from the article, the table of contents, and the acknowledgements. There is a lot of formatting in the article so it seemed easier not to copy paste the whole thing.

(You can also see this post on the EA forum here).

Context

Like most 80,000 Hours content, this profile is aimed at an audience that has probably spent a bit of time on the 80,000 Hours website but is otherwise unfamiliar with the ideas of EA and AI risk -- so it's meant to be introductory.

The profile primarily represents Benjamin Hilton (the author)'s views, though it was edited by Arden Koehler (website director) and reviewed by Howie Lempel (CEO), who both broadly agree with the takeaways.

Note from Benjamin:
I tried to do a few things with this profile to make it as useful as possible for people new to the issue:

I focus on what I see as the biggest issue: risks of power-seeking AI from strategically aware planning systems with advanced capabilities, as set out by Joe Carlsmith.
I try to make things feel more concrete, and have released a whole separate article on what an AI-caused catastrophe could actually look like. (This owes a lot to Carlsmith's report, as well as Christiano's What failure looks like and Bostrom's Superintelligence.)
I give (again, what I see as) important background information, such as the results of surveys of ML experts on AI risk, an overview of recent advances in AI and scaling laws
I try to honestly explain the strongest reasons why the argument I present might be wrong
I include a long FAQ of common objections to working on AI risk to which I think there are strong responses

Why we're posting it here

We're aiming for this to be a high quality, totally-introductory explainer on why we're so worried about AI risk, which might be useful for LessWrong users either to read themselves or to send to people who are new to the field.
If there are mistakes in the article, we want to know about them! (There's also a feedback form if you want to give feedback and prefer that to posting publicly.)

Summary

We expect that there will be substantial progress in AI in the next few decades, potentially even to the point where machines come to outperform humans in many, if not all, tasks. This could have enormous benefits, helping to sol...
Grand Theft Education
by Zvi — 2022-08-31T11:50:01.072Z

A story, in three parts.

First, what they’ve done going backwards, and what it lays the rhetorical groundwork to do in the future to help make things worse.

Second, what they’re doing going forwards to actively make things worse.

Third, a bird’s eye view of how much worse things were made.

Part 1: Loan Forgiveness Present

Those defending student loan cancellation are aggressively attacking anyone who disagrees with them, usually (but far from always) by pointing to their involvement in PPP – ‘This you?’

The implications are not great, including this obvious one.

This is indeed how I am interpreting the claim – that because the government once gave out a bunch of money to people, no one can ever object to giving other bunches of money to other people.

The argument flips freely between implying PPP was just and implying PPP was theft.

The White House is taking point on this.

I appreciate the brutal honesty here and also here:

A lot more than 51% of Twitter, and Democratic circles generally, approves of this.

The sky is blue. Also, the sky is blue.

It is far worse than this, because PPP loans were not functionally loans at all. They were grants for disaster relief, to be repaid if not spent, and when the government is handing out trillions it is rather expensive to simply turn down your allocated share. Drawing a parallel here implies that student loans were also ‘loans’ without intention of repayment. And any future loans are the same. There will be a scramble to take on and keep as much forgivable debt as possible, and things like tuition will adjust accordingly. The sky is the limit, the debtors’ revolt is complete, the treasury doors are open.

The response to this objection seems to be alternating between ‘the word loan is right there, checkmate (classical) liberals’ and ‘the outgroup’s words don’t have meaning so it is completely unfair to point out that ours also do not have meaning.’

If this general maneuver succeeds, it is also strong evidence that loan forgiveness, even fake ‘loan forgiveness’, leads to such spirals.

(Freddie DeBoer says the question of whether PPP was loans is irrelevant, what matters is the spending, except the whole point of the attacks is to say PPP ‘loans’ were loans and this justifies forgiving other loans. Back at the motte, he says this is a question of where the federal government spends money. He s...
How much impact can any one man have?
by GregorDeVillain — 2022-08-31T10:26:39.027Z

Because in truth, had there been no Gutenberg, the printing press would have been discovered in (or imported to) Europe - eventually. Would it have been five years later? 10 years later? 100 years later? Impossible to know. But, we should think, eventually, it would have been.

Yet there are interesting cases from history. We know of five different modes of human stone tool-making that pre-date the Neolithic (the New Stone Age). They are:

The Oldowan Industry. Simple core form rocks, used for things like chopping. Oldowan tools emerged 2.6 million years ago, devised by either Homo habilis or Homo erectus in Northern Africa.

The Acheulean Industry. Biface rock tools, with more complex design than Oldowan tools. The axe is the most notable example of this type of tool. Acheulean tools first date to 1.76 million years ago, devised by Homo erectus in Western and Southern Africa.

The Mousterian Industry. Fine-pointed rock tools that may have relied on greater grip strength to create. Mousterian tools date to 315,000 years ago, devised by Neanderthal man in Europe.

The Aurignacian Industry. Fine bladed stone tools, along with worked bone and antler points, struck from prepared cores rather than crude flakes. Aurignacian tools have been found throughout Europe and the Levant, and are believed to have emerged in the Levant around 43,000 years ago.

The Microlithic Industry. Microliths were small stones used in composite tools, fastened to a haft. They were devised in Europe and the Levant about 35,000 years ago.

Now here's something to think about.

We know how invention works. It is rarely (or never) collaborative across an entire society; an entire society does not invent something. Instead, one man comes up with an idea, and either builds it himself, or sets off a chain of competitors who race to be the first to build the thing (in which case you have the collaboration of the guy who proposes the idea, a few guys who try to build it, and one or more who actually succeeds).

Essentially though, you can boil the invention down to a handful of people. The guy who thinks it up, and maybe a few other guys who devise it and refine it (if the original idea guy doesn't do all that himself - and he may).

So, now. At some point, 2.6 million years ago, some human started to make simple rock tools. He taught it to his tribemates, and perhaps some of them helped to refine his designs.

No on...
How might we make better use of AI capabilities research for alignment purposes?
by Jemal Young — 2022-08-31T04:19:33.239Z

When I check ArXiv for new AI alignment research papers, I see mostly capabilities research papers, presumably because most researchers are working on capabilities. I wonder if there’s alignment-related value to be extracted from all that capabilities research, and how we might get at it. Is anyone working on this, or does anyone have any good ideas?
AI Box Experiment: Are people still interested?
by Double — 2022-08-31T03:04:38.518Z

I have a novel plan for the AI player that I believe will work against most gatekeeper players. Even knowing how it works, it would work against me! It isn't quite ready, but I'd like to know if there is still interest around the topic before finishing work on the preparation.

There's enough incentive for both players to play well that I don't think that we need to involve money. (I'm also very reluctant to send money to strangers on the internet!)

We would be playing under Tuxedage's rules: https://tuxedage.wordpress.com/2013/09/04/the-tuxedage-ai-box-experiment-ruleset/.

Edit: I have finished preparation, and I'm now scheduling the time with the Gatekeeper.

Edit 2: The game was a Gatekeeper victory (i.e. I, the AI Player, lost). Read about it here: Gatekeeper Victory: AI Box Reflection.
Survey of NLP Researchers: NLP is contributing to AGI progress; major catastrophe plausible
by Sam Bowman — 2022-08-31T01:39:54.533Z

I was part of a group that ran a PhilPapers-style survey and metasurvey targeting NLP researchers who publish at venues like ACL. Results are here (Tweet-thread version). It didn't target AGI timelines, but had some other questions that could be of interest to people here:

NLP is on a path to AGI: 58% agreed that Understanding the potential development of artificial general intelligence (AGI) and the benefits/risks associated with it should be a significant priority for NLP researchers.

Related: 57% agreed that Recent developments in large-scale ML modeling (such as in language modeling and reinforcement learning) are significant steps toward the development of AGI.

AGI could be revolutionary: 73% agreed that In this century, labor automation caused by advances in AI/ML could plausibly lead to economic restructuring and societal changes on at least the scale of the Industrial Revolution.
AGI could be catastrophic: 36% agreed that It is plausible that decisions made by AI or machine learning systems could cause a catastrophe this century that is at least as bad as an all-out nuclear war.

46% of women and 53% of URM respondents agreed.
The comments suggested that people took a pretty wide range of interpretations to this, including things like OOD robustness failures leading to weapons launches.

Few scaling maximalists: 17% agreed that Given resources (i.e., compute and data) that could come to exist this century, scaled-up implementations of established existing techniques will be sufficient to practically solve any important real-world problem or application in NLP.

The metasurvey responses predicted that 47% would agree to this, so there are fewer scaling maximalists than people expected there to be.

Optimism about ideas from cognitive science: 61% agreed that It is likely that at least one of the five most-cited systems in 2030 will take clear inspiration from specific, non-trivial results from the last 50 years of research into linguistics or cognitive science.

This strikes me as very optimistic, since it's pretty clearly false about the most cited systems today.

Optimism about the field: 87% agreed that On net, NLP research continuing into the future will have a positive impact on the world.

32% of respondents who agreed that NLP will have a positive future impact on society also agreed that there is a plausible risk of global catastrophe.

Most NLP research is

...
Worlds Where Iterative Design Fails
by johnswentworth — 2022-08-30T20:48:29.025Z

In most technical fields, we try designs, see what goes wrong, and iterate until it works. That’s the core iterative design loop. Humans are good at iterative design, and it works well in most fields in practice.

In worlds where AI alignment can be handled by iterative design, we probably survive. So long as we can see the problems and iterate on them, we can probably fix them, or at least avoid making them worse.

By the same reasoning: worlds where AI kills us are generally worlds where, for one reason or another, the iterative design loop fails. So, if we want to reduce X-risk, we generally need to focus on worlds where the iterative design loop fails for some reason; in worlds where it doesn’t fail, we probably don’t die anyway.

Why might the iterative design loop fail? Most readers probably know of two widely-discussed reasons:

Fast takeoff: there will be a sudden phase shift in capabilities, and the design of whatever system first undergoes that phase shift needs to be right on the first try.
Deceptive inner misalignment: an inner agent behaves well in order to deceive us, so we can’t tell there’s a problem just by trying stuff and looking at the system’s behavior.

… but these certainly aren’t the only reasons the iterative design loop potentially fails. This post will mostly talk about some particularly simple and robust failure modes, but I’d encourage you to think on your own about others. These are the things which kill us; they’re worth thinking about.

Basics: Hiding Problems

Example/Analogy: The Software Executive

Imagine that a software company executive, concerned about the many errors coming from the software, creates a new incentive scheme: software developers get a monetary reward for changes which decrease the rate of error messages showing up on the manager’s dashboard, and get docked for changes which increase the rate of error messages.

As Tyler Cowen would say: “solve for the equilibrium”. Obvious equilibrium here: the developers stop throwing error messages when they detect a problem, and instead the software just fails silently. The customer’s experience remains the same, but the manager’s dashboard shows fewer error messages. Over time, the customer’s experience probably degrades, as more and more problems go undetected.

In the short run, the strategy may eliminate some problems, but in the long run it breaks the iterative design loop: problems ...