trentmkelly/LessWrong-43k huggingface dataset:
How "Discovering Latent Knowledge in Language Models Without Supervision" Fits Into a Broader Alignment Scheme
by Collin — 2022-12-15T18:22:40.109Z

Introduction

A few collaborators and I recently released a new paper:

Discovering Latent Knowledge in Language Models Without Supervision

. For a quick summary of our paper, you can check out this Twitter thread.

In this post I will describe how I think the results and methods in our paper fit into a broader scalable alignment agenda. Unlike the paper, this post is explicitly aimed at an alignment audience and is mainly conceptual rather than empirical.

Tl;dr: unsupervised methods are more scalable than supervised methods, deep learning has special structure that we can exploit for alignment, and we may be able to recover superhuman beliefs from deep learning representations in a totally unsupervised way.

Disclaimers: I have tried to make this post concise, at the cost of not making the full arguments for many of my claims; you should treat this as more of a rough sketch of my views rather than anything comprehensive. I also frequently change my mind – I’m usually more consistently excited about some of the broad intuitions but much less wedded to the details – and this of course just represents my current thinking on the topic.

Problem

I would feel pretty optimistic about alignment if – loosely speaking – we can get models to be robustly “honest” in a way that scales even to superhuman systems.[1] Moreover, I think a natural sub-problem that captures much or most of the difficulty here is: how can we make a language model like GPT-n “truthful” or “honest” in a way that is scalable? (For my purposes here I am also happy to make the assumption that GPT-n is not actively deceptive, in the sense that it does not actively try to obscure its representations.)

For example, imagine we train GPT-n to predict news articles conditioned on their dates of publication, and suppose the model ended up being able to predict future news articles very well. Or suppose we train GPT-n to predict the outcomes of particular actions in particular situations, all described (imperfectly by humans) in text. Then I would expect GPT-n would eventually (for large enough n) have a superhuman world model in an important sense. However, we don’t currently know how to recover the “beliefs” or “knowledge” of such a model even in principle.

A naive baseline for trying to make GPT-n truthful is to train it using human feedback to output text that human evaluators believe to ...
High-level hopes for AI alignment
by HoldenKarnofsky — 2022-12-15T18:00:15.625Z

Click lower right to download or find on Apple Podcasts, Spotify, Stitcher, etc.

In previous pieces, I argued that there's a real and large risk of AI systems' aiming to defeat all of humanity combined - and succeeding.

I first argued that this sort of catastrophe would be likely without specific countermeasures to prevent it. I then argued that countermeasures could be challenging, due to some key difficulties of AI safety research.

But while I think misalignment risk is serious and presents major challenges, I don’t agree with sentiments1 along the lines of “We haven’t figured out how to align an AI, so if transformative AI comes soon, we’re doomed.” Here I’m going to talk about some of my high-level hopes for how we might end up avoiding this risk.

I’ll first recap the challenge, using Ajeya Cotra’s young businessperson analogy to give a sense of some of the core difficulties. In a nutshell, once AI systems get capable enough, it could be hard to test whether they’re safe, because they might be able to deceive and manipulate us into getting the wrong read. Thus, trying to determine whether they’re safe might be something like “being an eight-year-old trying to decide between adult job candidates (some of whom are manipulative).”

I’ll then go through what I see as three key possibilities for navigating this situation:

Digital neuroscience: perhaps we’ll be able to read (and/or even rewrite) the “digital brains” of AI systems, so that we can know (and change) what they’re “aiming” to do directly - rather than having to infer it from their behavior. (Perhaps the eight-year-old is a mind-reader, or even a young Professor X.)

Limited AI: perhaps we can make AI systems safe by making them limited in various ways - e.g., by leaving certain kinds of information out of their training, designing them to be “myopic” (focused on short-run as opposed to long-run goals), or something along those lines. Maybe we can make “limited AI” that is nonetheless able to carry out particular helpful tasks - such as doing lots more research on how to achieve safety without the limitations. (Perhaps the eight-year-old can limit the authority or knowledge of their hire, and still get the company run successfully.)

AI checks and balances: perhaps we’ll be able to employ some AI systems to critique, supervise, and even rewrite others. Even if no single AI system would be sa

...
Two Dogmas of LessWrong 
by Bentham's Bulldog — 2022-12-15T17:56:06.462Z

Lesswrongers seem to largely be in agreement in rejecting robust moral realism and accepting physicalism about consciousness. This is a shame, I think, because both of these views are incorrect. The thing I find most frustrating about this is they tend to be supremely confident on topics when a hefty percentage of philosophers disagree with them. 'Don't believe things that are widespread in your ingroup with super high confidence when a large percentage of philosophers disagree with you' seems to be a pretty good heuristic -- yet LessWrongers seem not to adhere to it, at least, in evaluating these views.

I'd like to say at the outset a few things. First, this clearly doesn't apply to all people on LessWrong. Second, I agree with LessWrongers on a huge number of things -- AI risks, for example, as well as the desirability of effective altruism. I'm broadly on board with the project of being less wrong. Thus, my criticism of LW is less of the idea behind it, and more of the particular sets of beliefs that actual LessWrongers tend to have. Third, most of this will be a crosspost of things I've written elsewhere on my blog -- I have no desire to reinvent the wheel when it comes to arguments for moral realism.

1 Moral Realism

0 An Introduction to moral realism

There are vast numbers of superficially clever arguments one can generate for crazy, skeptical conclusions; conclusions like that the external world doesn’t exist, we can’t know anything, memory isn’t reliable, and so on. These arguments, while interesting and no doubt useful if one ever comes across a real honest-to-god skeptic — a rather rare breed — don’t have much significance; skepticism exists as little more than a curiosity in the mind of the modern philosopher, something which takes real thought to refute, yet is not worth taking seriously as a serious set of views.

Yet there’s one

1

form of extreme skepticism with actually existing trenchant advocates — real advocates who fill philosophy departments, rather than, like the external world or memory skeptic, merely being hypothetical advocates for the devil in philosophy papers. This skeptic is one who doubts that there are objective moral truths — moral facts made true not by the beliefs of any person.

Moral realism is the claim that there are true moral facts — ones that are not made...
Covid 12/15/22: China’s Wave Begins
by Zvi — 2022-12-15T16:20:00.746Z

Cases increased this week, as expected. Deaths also increased, which I did not expect so soon, indicating a bigger and faster winter wave. That was not the big event in the West this week with respect to Covid.

The big event this week in the West’s Covid experience was a memory of events past. Patrick McKenzie wrote a very long and very worthwhile magnum opus about his efforts in VaccinateCA to get vaccine shots into arms, and what he learned about our government and public health systems along the way. If you read one thing this week, read his story. You can also read my key facts and takeaways post, and this thread with another person’s remembrances (here are Patrick’s takes on that). Patrick also offers his thoughts on implications for charitable efforts at the EA Forum, which includes this gem:

If I had to point at any one artifact in particular, Inadequate Equilibria is a good articulation of a larger memeplex that made me comfortable with “If the evidence of a system’s operation contradicts what the Efficient Market Hypothesis counsels is the probable functioning of the system, trust the evidence. Thousands of lives savable by one dedicated team of non-specialists is actually not all that low probability.”

Yes.

What else happened recently?

China stopped attempting to contain Covid-19. They have accepted that the costs of continuing to do this are unacceptably high. Instead, they have decided they are content to contain official case numbers while they brace for the impact of the Covid surge they have successfully put off for three years. It is their turn to have trouble staffing hospitals, to have panic buying and shortages and voluntarily deserted streets. I wish them the best of luck in getting through this as best they can.

Executive Summary

Patrick McKenzie wrote about VaccineCA. You should read it, and remember.

China is finally experiencing the Covid wave they have postponed for three years.

Cases and deaths are on the rise, the winter wave is looking a bit larger.

Let’s run the numbers.

The Numbers

Predictions

Predictions from Last Week: 371k cases (+5%) and 2,240 deaths (-6%).

Results: 391k cases (+10%) and 2,610 deaths (+9%).

Predictions for Next Week: 435k cases (+10%) and 2,900 deaths (+10%).

(Note: North Dakota dumped a lot of deaths on us after reporting no deaths at all for months, so I reduced that num...
The next decades might be wild
by Marius Hobbhahn — 2022-12-15T16:10:04.750Z

I’d like to thank Simon Grimm and Tamay Besiroglu for feedback and discussions.

Update (early April 2023): I now think the timelines in this post are too long and expect the world to get crazy faster than described here. For example, I expect many of the things suggested for 2030-2040 to already happen before 2030. Concretely, in my median world the CEO of a large multinational company like Google is an AI. This might not be the case legally but effectively an AI makes most major decisions.

This post is inspired by

What 2026 looks like

and an AI vignette workshop guided by Tamay Besiroglu. I think of this post as “what would I expect the world to look like if

these timelines

(median compute for transformative AI ~2036) were true” or “what short-to-medium timelines feel like” since I find it hard to translate a statement like “median TAI year is 20XX” into a coherent imaginable world.

I expect some readers to think that the post sounds wild and crazy but that doesn’t mean its content couldn’t be true. If you had told someone in 1990 or 2000 that there would be more smartphones and computers than humans in 2020, that probably would have sounded wild to them. The same could be true for AIs, i.e. that in 2050 there are more human-level AIs than humans. The fact that this sounds as ridiculous as ubiquitous smartphones sounded to the 1990/2000 person, might just mean that we are bad at predicting exponential growth and disruptive technology.

Update: titotal points out in the comments that the correct timeframe for computers is probably 1980 to 2020. So the correct time span is probably 40 years instead of 30. For mobile phones, it's probably 1993 to 2020 if you can trust this statistic.

I’m obviously not confident (see confidence and takeaways section) in this particular prediction but many of the things I describe seem like relatively direct consequences of more and more powerful and ubiquitous AI mixed with basic social dynamics and incentives.

Taking stock of the past

Some claims about the past

Tech can be very disruptive: The steam engine, phones, computers, the internet, smartphones, etc; all of these have changed the world in a relatively quick fashion (e.g. often just a few decades between introduction and widespread use). Furthermore, newer technologies disrupt faster (e.g. smartphones were adopted faster than cars) due to faster su

...
Basic building blocks of dependent type theory
by Thomas Kehrenberg — 2022-12-15T14:54:30.043Z

This post is the first in a sequence of posts meant to be an introduction to pragmatic type theory. “Type theory” here refers to Martin-Löf dependent type theory (MLTT), as opposed to theories like the simply-typed lambda calculus; the main difference being that in MLTT, functions can also act on types and not just values, which makes it a more powerful theory.

I was initially motivated to study type theory because of a remark in the Alignment Research Field Guide about how defining terms and adding type annotations to them can make the topic of alignment research discussions more precise, as an intermediate step between vague idea and fully formalized proof.

You might also be aware that type theory can serve as an alternative foundation for mathematics, in place of set theory. And some people think type theory is the more elegant choice of the two; a point which this article is for example arguing for. Another reason to learn type theory is that it serves as the basis of most theorem provers.

I’m not aware of any short-ish but still pretty comprehensive introductions to MLTT, so that's why I wrote this sequence of posts. The content is mostly based on the book Homotopy Type Theory, with the main difference that the book tries to convince you to do constructive mathematics, whereas I won’t. (I also won’t talk about any homotopies at all.) This means though that if you’re interested in constructive mathematics (or homotopies), you should probably just read the book instead.

I will sometimes compare type concepts to concepts found in programming languages, but if you don’t have any programming experience, you can just ignore that. What you should have though, is a good understanding of set theory, because I will reference it frequently.

With that said, let’s get into it!

Type judgments

A simple type annotation looks like this:

x:T

where T is a type and x is an element of this type. This looks very similar to the statement “x∈T” where T is a set, and it is quite similar, but there are some important differences.

For one, types are usually stricter than sets when it comes to membership. In set theory, an object can easily be a member of multiple sets. In type theory, an object usually only belongs to a single type (though maybe that's not quite the case when union types are involved).

The other difference is that “x∈T” can be used as a p...
AI Neorealism: a threat model & success criterion for existential safety
by davidad — 2022-12-15T13:42:11.072Z

Threat Model

There are many ways for AI systems to cause a catastrophe from which Earth-originating life could never recover. All of the following seem plausible to me:

Misuse: An AI system could help a human or group of humans to destroy or to permanently take over (and lock their values into) the world. The AI could be:

An oracle AI (e.g. a question-answering LLM)
An LLM simulating an intent-aligned agent and taking real-world actions via APIs
An intent-aligned RL agent
An interaction of multiple systems

Power-Seeking: An AI system could destroy or permanently take over the world on its own account, by leveraging advanced instruments of force projection. The AI could be:

An LLM simulating a misaligned agent
"Specification gaming": An RL agent that is aligned to a formal objective and Goodharts to catastrophe
"Goal misgeneralization": A surprise mesa-optimiser (most likely in model-free RL, but could conceivably arise through evolutionary processes in any iterative algorithm which has or learns sufficiently reality-like structure)
An interaction of multiple systems, participating in coordination mechanisms that exclude humans

Economic Squeeze: an AI system could acquire nearly all means of production through a gradual process of individually innocent economic transactions, thereby squeezing humanity out of resource allocation decisions and removing most human influence over the future.

This would most likely be an "interaction of multiple systems".
A single RL agent, or a unipolar tree of agents, might also do this, especially if they are successfully aligned to avoid use of force against humans.

Superpersuasion: an AI system could generate stimuli which reliably cause humans to adopt its arbitrary goals. The AI could be:

An LLM merely extrapolating from persuasive human text
An RL agent trained on human approval
A surprise mesa-optimiser
Some mixture of the above
Many AIs, collectively shaping a new human culture with an alien ideology

Security Dilemma: If AI-enabled technological advancements turn out to be offence-dominant, and if partial alignment success leads AIs to be unable to make credible commitments to each other (e.g. due to corrigibility), the equilibrium strategy for AI-enabled militaries may involve high-risk preemptive strikes and increasingly escalated retaliation to a point of existential catastrophe.

This would almost surely be a multipolar failure mode.

But, i...
Who should write the definitive post on Ziz?
by Nicholas / Heather Kross — 2022-12-15T06:37:16.150Z

CW: Possible infohazards, definitely discussion of infohazards.

EDIT 25 Feb 2023: This post has more details and corrects some things this post got wrong. Consider it as superseding this post.

Backstory

I was writing a comment on this post, and I was going to end it with this... but then decided to turn it into a full Question Post.

The post basically says something like "being in intense emotional doomer mode w.r.t. AI safety is counterproductive and addictive", which is broadly agreeable. However, the post also has a framing (my kinda uncharitable but also defensible reading) that's something like "this is caused by mind-virus stuff and you're trapped in the matrix and you need to go cold-turkey on caring emotionally about it, also if you disagree you are mindkilled. Go take a deep breath, then come back and agree with me".

So I was gonna write a comment like:

"Also like Richard_Ngo said, this framing seems kinda... bad. I don't think this is a direct inspiration for you, but [word phrase chunks to somehow relate the central themes/vibes to] posts by Ziz, and Scott Alexander's response to somebody else talking about CFAR and Michael Vassar (and Levarage?).

Someday I hope to write an extremely long thorough post (and possibly accompanying video essay?) about the whole Ziz/memetic-virus/hemispheres/alleged-conspiracies thing. It will both chronicle The Gigantic Drama and also respond to... Ziz's... entire worldview? Something like that. However, this project by nature would be extremely labor-intensive, and it's not my top priority. (Perhaps someone else wants to pick it up?)"

Then I realized it should be a post.

What are you talking about? (The Actual Backstory)

CFAR. Conspiracy theories. Mental illness. Using different hemispheres of your brain as if they were different people. Tumblr blogs. mumble mumble Nick Land mumble meme-virus. Michael Vassar. Possibly multiple suicides, including maybe Ziz.

"That's not a post idea! That's a list of keywords, some with emotional connotations!"

There are ideas in here, but they are tied to a gigantic ideology, known to very few. There are events in here, but they are jumbled out of order, and accounts conflict. There are people in here, and some of them are even important in the rationality community.

Maybe it's masochistic epistemology, maybe it's the social brain flaring up... but I kinda care about ideas that would ...
«Boundaries», Part 3b: Alignment problems in terms of boundaries 
by Andrew_Critch — 2022-12-14T22:34:41.443Z

AI alignment is a notoriously murky problem area, which I think can be elucidated by rethinking its foundations in terms of boundaries between systems, including soft boundaries and directional boundaries. As anticipated previously in this «Boundaries» sequence, I'm doing that now, for the following problem areas:

Preference plasticity & corrigibility
Mesa-optimizers
AI boxing / containment
(Unscoped) consequentialism
Mild optimization & impact regularization
Counterfactuals in decision theory

Each of these problem areas probably warrants a separate post, from the perspective of making research progress within already-existing framings on the problem of aligning AI technology with humanity. However, with this post, my goal is mainly just to continue conveying the «boundaries» concept, by illustrating how to start using it in a bunch of different problem areas at once. An interest score for each problem area was determined by voting on an earlier LessWrong comment, here, and the voting snapshot is here.

You many notice that throughout this post that I've avoided saying things like "the humans prefer that {some boundary} be respected". That's because my goal is to treat boundaries as more fundamental than preferences, rather than as merely a feature of them. In other words, I think boundaries are probably better able to carve reality at the joints than either preferences or utility functions, for the purpose of creating a good working relationship between humanity and AI technology.

Alright, let's begin by analyzing:

Preference Plasticity & Corrigibility

Preference plasticity is the possibility of changes to the preferences of human preferences over time, and the challenge of defining alignment in light of time-varying preferences (Russell, 2019, p.263).

Interest score: 12/12

Corrigibility is the problem of constructing a mind that will cooperate with what its creators regard as a corrective intervention (Soares et al, 2015).

Interest score: 3/12

I think these two problems are best discussed together, because they are somewhat dual to each other: corrigibility has to do with the desirability of humans making changes to AI systems, and preference plasticity has to do with the undesirabiltiy — or least confusingness — of AI make changes to humans, or sometimes humans making changes to each other or themselves.

Preference plasticity

When ...