trentmkelly/LessWrong-43k huggingface dataset:
Key Papers in Language Model Safety
by aog — 2022-06-20T15:00:59.858Z

Introduction

One of the fastest growing subfields of AI safety focuses on language models (LMs) such as OpenAI’s GPT-3. This article provides summaries of a few dozen key papers in several important subfields of language model safety, namely:

Persuasion
Truthfulness
Honesty
Robustness
Law
Machine Ethics

This is not a comprehensive introduction to the field of language model safety. Specifically, this overview does not discuss important topics such as

bias

,

interpretability

, and

decision-making agents

. It also disproportionately focuses on arguments and ideas advanced by authors within the effective altruism community, as I’ve had the most exposure to their work. Over time, the field of AI safety should seek to engage more with the broader field of machine learning (see

PAIS

for more). Instead, this overview aims to summarize a useful set of papers from which one could begin researching language model safety.

Why study language model safety?

There are several reasons to believe that language model safety could present an unusually strong opportunity to reduce catastrophic AI risks:

Rapid recent progress.

GPT-3

,

LaMDA

,

BERT

, and other large language models have blown away the RNNs, LSTMs, and statistical methods of years past with the modern

Transformer architecture

. Capabilities such as following instructions, doing math, and answering questions truthfully have

emerged suddenly

in larger models, while their loss function performance has

scaled consistently

with growth in training data, model parameters, and computational power.
Potential to scale to AGI. Language is a powerful medium for understanding and interacting with the world, perhaps more so than vision or robotics. If language model performance continues to scale as we build models that approach and exceed the size of the human brain (see

BioAnchors

), LMs could be an important component of advanced AI. Some argue that this is

possible

, while others

disagree

. Language models seem particularly important in

short timelines scenarios

.
Alignment failures today. Language models are trained on the

self-supervised objective

of imitating their training data. This allows LMs to use large text datasets scraped from the internet without the expensive and time-consuming step of human annotation. But it introduces an important failure mode: Language models will mimic their training data to a fault, often resulting in

hate

...
Relationship Advice Repository
by Ruby — 2022-06-20T14:39:36.548Z

Over the years, LessWrong has hosted some pretty great advice threads, including Best Textbooks on Every Subject, Boring Advice Repository, Useful Concepts Repository, and Best Software For Every Need.

Since relationships are extremely core to people's happiness and well-being, seems worth an attempt at collecting good advice on this topic too. I've been unsure how to structure this thread. What follows is my guest guess - we'll see how it goes.

How to contribute

Format

This is a thread of advice specifically about being in a romantic/sexual relationship. Probably a bunch of it applies to platonic relationships too, but a narrow focus seems better. Also, the thread is focused on BEING in the relationship, rather than seeking out a relationship.

Please contribute content in one of the following forms if you want it added to the main post.

An argument, idea, model, or concept.
A verified story containing actions and outcome, e.g. "I held off breaking up with my girlfriend for six months and this made things 10x awkward and worse". Longer stories seem good too but probably won't be added to the core post text.
A link to or quotes from an external resource like a book, blog, or lecture; optionally with a review and/or summary.
A combination of the above.

Content

I've seeded this post with a mix of advice, experience, and resources from myself and a few friends, plus various good content I found on LessWrong through the Relationships tag. The starting content is probably not the very best content possible (if it was, why make this a thread?), but I wanted to launch with something. Don't anchor too hard on what I thought to include! Likely as better stuff is submitted, I'll move some stuff out of the post text and into comments to keep the post from becoming absurdly long.

I think relationship advice is hard, and also that it's likely some of the best advice will be controversial. And also, of course, "consider reversing all advice you hear". I think the "final" version of this page should contain diverse and conflicting advice.

Please disagree with advice you think is wrong! (It probably makes sense to add notes/links about differing views next to advice items in the main text, so worth the effort to call out stuff you disagree with.)

Ultimately, readers should apply their own judgment about what they ought to be trying.

How to submit content

You ca

...
Adaptation Executors and the Telos Margin
by Plinthist — 2022-06-20T13:06:29.519Z

Thank you to Justis Mills for feedback on a draft of this post.

You'll often hear this bit of wisdom: "Humans are not utility optimizers, but rather adaptation executors." At first glance, it seems to be pretty self-explanatory. Humans are not effectively described by the optimization of some particular utility function - to the contrary, human behavior is the product of a slew of hot-fix adaptations, most easily understood in terms of how they function.

On a second look, though, there's a little more here. What's the difference between these two representations? For any given pattern of behavior, a utility function can be selected that values precise adherence to any given behavioral pattern. On some level, then, an adaptation executor is a utility maximizer - at minimum, we can retrospectively say that its utility function was based on how well it did what the adaptations drove it toward. That's not very satisfying, though, as there does seem to be a substantial difference. Looking for what sets them apart, the only option is the operative qualifier: that behavior is effectively described.

To effectively describe some behavior, it's necessary to describe that behavior as simply and directly as possible. Notice how convoluted that boilerplate utility function is. "Actions that would be produced by behavioral model X" fits the form of a utility function, but it's certainly not the form that comes to mind when thinking about utility functions in general. Taking this route to transform a pattern of behavior into a utility function is always going to increase the complexity. So while you can theoretically construct a utility function to explain human choices, there very well may not be a utility function which expresses human decisions with substantially less complexity than the behavior produced by a taxonomy of adaptations.

What I propose to call an agent's telos is the difference between how complex that agent's behavior is and how complex the simplest utility function corresponding to that behavior is. It's the degree to which an agent is better expressed by purpose than procedural action. I'm going to formalize that in the next few sections, but if you just want to see how it looks in practice, you can skip to "Back to Reality."

The Behavior Form

That's a good start, but it's not worth much to wave around algorithmic complexity without having phrased ...
Are we there yet?
by theflowerpot — 2022-06-20T11:19:56.253Z

How good of an idea do the people on this website have in regards to predicting an AGI apocalypse?

I get the impression that a lot of people seem to be very certain that it will happen relatively soon and I think I've figured out a way to test it. The spreadsheet for the data would look something like this. This questionnaire would be asked each year and the average percentage for each question would be used.

The interesting thing is that I think I know what it will look like over time

The smallest red box would be your prediction for the first year. It would probably be very low. The largest box (in this case) would be the probability that it happens within 25 years. As time passes by and nothing happens the graph would simply shift left.

What do you think about this approach?

Regards, theflowerpot
Causal confusion as an argument against the scaling hypothesis
by RobertKirk — 2022-06-20T10:54:05.623Z

Abstract

We discuss the possibility that causal confusion will be a significant alignment and/or capabilities limitation for current approaches based on "the scaling paradigm": unsupervised offline training of increasingly large neural nets with empirical risk minimization on a large diverse dataset. In particular, this approach may produce a model which uses unreliable (“spurious”) correlations to make predictions, and so fails on “out-of-distribution” data taken from situations where these correlations don’t exist or are reversed. We argue that such failures are particularly likely to be problematic for alignment and/or safety in the case when a system trained to do prediction is then applied in a control or decision-making setting.

We discuss:

Arguments for this position
Counterarguments
Possible approaches to solving the problem
Key Cruxes for this position and possible fixes
Practical implications for capability and alignment
Relevant research directions

We believe this topic is important because many researchers seem to view scaling as a path toward AI systems that 1) are highly competent (e.g. human-level or superhuman), 2) understand human concepts, and 3) reason with human concepts.
We believe the issues we present here are likely to prevent (3), somewhat less likely to prevent (2), and even less likely to prevent (1) (but still likely enough to be worth considering). Note that (1) and (2) have to do with systems’ capabilities, and (3) with their alignment; thus this issue seems likely to be differentially bad from an alignment point of view.

Our goal in writing this document is to clearly elaborate our thoughts, attempt to correct what we believe may be common misunderstandings, and surface disagreements and topics for further discussion and research.

A DALL-E 2 generation for "a green stop sign in a field of red flowers".
Current foundation models still fail on examples that seem simple for humans, and causal confusion and spurious correlations may be among the culprits causing such failures. Examples like these show that DALL-E 2 makes systematic deviations from the way humans interpret text, possibly (in this case) because stop signs are almost always red in the training dataset, especially if the word red appears in the caption.

Introduction and Framing

GPT-3 and Scaling Laws (among other works) have made the case that scaling will be a key ...
An AI defense-offense symmetry thesis
by Chris van Merwijk — 2022-06-20T10:01:18.968Z

Epistemic status: Not well argued for, haven’t spent much time on it, and it’s not very worked out. This thesis is not new at all, and is implicit in a lot (most?) of the discussion about x-risk from AI. The intended contribution of this post is to state the thesis explicitly in a way that can help make the alignment problem clearer. I can imagine the thesis is too strong as stated.

I sometimes hear implicitly or explicitly in people’s skepticism of AI risk that we can just not build dangerous AI. I want to formulate a thesis that does part of the work of countering this idea.

If we want to be safe against misaligned AI, we need to defend the vulnerabilities that (a) misaligned AI system(s) could exploit that would pose an existential threat. We need aligned AI that defends against misaligned AI, in order to make sure such misaligned AI poses no threat, (either by ensuring they are not created, or never are able to accumulate the resources needed to be a threat).

In game theory terms: If we successfully build aligned AI systems, then we need to play an asymmetric game between humans with (aligned) defensive AI, versus potential (misaligned) offensive AI. It is not obvious that in an asymmetric game both sides need to have the same capabilities. The thesis I’m making is that the defensive AI needs to have at least all the capabilities that the offensive AI would need to pose an existential threat: The defensive AI needs to be basically capable of starting from a set of resources X, and doing all the damage that the offensive AI could do using resources X.

The/an AI offense-defense symmetry thesis: In order for aligned AI systems to defend indefinitely against existential threat from misaligned AI without themselves harming humanity’s potential, that defensive AI needs to have broadly speaking all the capabilities of the misaligned AI systems that it defends against.

Note that the defender may need to have a higher (or lower) level of those capabilities than the attacker, and it may need additional kinds of capabilities as well that the attackers don’t have.

I don’t think this thesis is exactly right, but more or less right. I won’t really make a solid argument for it, but give some analogies and mechanisms:

Some analogies from offense-defense in human domains:

State security services who defend against terrorists: ...
Let's See You Write That Corrigibility Tag
by Eliezer Yudkowsky — 2022-06-19T21:11:03.505Z

The top-rated comment on "AGI Ruin: A List of Lethalities" claims that many other people could've written a list like that.

"Why didn't you challenge anybody else to write up a list like that, if you wanted to make a point of nobody else being able to write it?" I was asked.

Because I don't actually think it does any good, or persuades anyone of anything, people don't like tests like that, and I don't really believe in them myself either. I couldn't pass a test somebody else invented around something they found easy to do, for many such possible tests.

But people asked, so, fine, let's actually try it this time. Maybe I'm wrong about how bad things are, and will be pleasantly surprised. If I'm never pleasantly surprised then I'm obviously not being pessimistic enough yet.

So: As part of my current fiction-writing project, I'm currently writing a list of some principles that dath ilan's Basement-of-the-World project has invented for describing AGI corrigibility - the sort of principles you'd build into a Bounded Thing meant to carry out some single task or task-class and not destroy the world by doing it.

So far as I know, every principle of this kind, except for Jessica Taylor's "quantilization", and "myopia" (not sure who correctly named this as a corrigibility principle), was invented by myself; eg "low impact", "shutdownability". (Though I don't particularly think it hopeful if you claim that somebody else has publication priority on "low impact" or whatevs, in some stretched or even nonstretched way; ideas on the level of "low impact" have always seemed cheap to me to propose, harder to solve before the world ends.)

Some of the items on dath ilan's upcoming list out of my personal glowfic writing have already been written up more seriously by me. Some haven't.

I'm writing this in one afternoon as one tag in my cowritten online novel about a dath ilani who landed in a D&D country run by Hell. One and a half thousand words or so, maybe. (2169 words.)

How about you try to do better than the tag overall, before I publish it, upon the topic of corrigibility principles on the level of "myopia" for AGI? It'll get published in a day or so, possibly later, but I'm not going to be spending more than an hour or two polishing it.
Half-baked alignment idea: training to generalize
by Aaron Bergman — 2022-06-19T20:16:43.735Z

Note: I think there's a decent chance that the idea I describe is misguided, redundant, naive. My apologies if that is the case, and please feel free to point me towards existing, related writing or research.

Thanks to Peter Barnett and Justis Mills for feedback on a draft of this post. It was inspired by Eliezer's Lethalities post and Zvi's response.

Central idea: can we train AI to generalize out of distribution?

I'm thinking, for example, of an algorithm like the following:

Train a GPT-like ML system to predict the next word given a string of text only using, say, grade school-level writing (this being one instance of the object level)

Assign the system a meta-level award based on how well it performs (without any additional training) at generalizing; in this case, that is, predicting the next word from more advanced, complex writing (perhaps using many independent tests of this task without updating/learning between each test, and allowing parameters to update only after the meta-level aggregate score is provided)

Note: the easy→hard generalization is not a necessary feature. Generalization could be from fiction→nonfiction writing or internet→native print text, for instance.

After all these independent samples are taken, provide the AI its aggregate or average score as feedback

(Maybe?) repeat all of step I on a whole new set of training and testing texts (e.g., using text from a different natural language like Mandarin)

Repeat this step an arbitrary number of times

For example, using French text, then Korean, then Arabic, etc.

Each time a “how well did you generalize” score is provided (which is given once per natural language in this example), the system should improve at the general task of generalizing from simple human writing to more complex human writing, (hopefully) to the point of being able to perform well at generalizing from simple Hindi (or whatever) text to advanced Hindi prediction even if it had never seen advanced Hindi text before.
^Steps 1-3 constitute the second meta-level of training an AI to generalize, but we can easily treat this process as a single training instance (e.g., rating how well the AI generalizes to Hindi advanced text after having been trained on doing this in 30 other languages) and iterate over and over again. I think this would look like:

Running the analogs of steps 1-4 on

...
Where I agree and disagree with Eliezer
by paulfchristiano — 2022-06-19T19:15:55.698Z

(Partially in response to

AGI Ruin: A list of Lethalities

. Written in the same rambling style. Not exhaustive.)

Agreements

Powerful AI systems have a good chance of deliberately and irreversibly disempowering humanity. This is a much more likely failure mode than humanity killing ourselves with destructive physical technologies.
Catastrophically risky AI systems could plausibly exist soon, and there likely won’t be a strong consensus about this fact until such systems pose a meaningful existential risk per year. There is not necessarily any “fire alarm.”
Even if there were consensus about a risk from powerful AI systems, there is a good chance that the world would respond in a totally unproductive way. It’s wishful thinking to look at possible stories of doom and say “we wouldn’t let that happen;” humanity is fully capable of messing up even very basic challenges, especially if they are novel.
I think that many of the projects intended to help with AI alignment don't make progress on key difficulties and won’t significantly reduce the risk of catastrophic outcomes. This is related to people gravitating to whatever research is most tractable and not being too picky about what problems it helps with, and related to a low level of concern with the long-term future in particular. Overall, there are relatively few researchers who are effectively focused on the technical problems most relevant to existential risk from alignment failures.
There are strong social and political pressures to spend much more of our time talking about how AI shapes existing conflicts and shifts power. This pressure is already playing out and it doesn’t seem too likely to get better.
Even when thinking about accident risk, people’s minds seem to go to what they think of as “more realistic and less sci fi” risks that are much less likely to be existential (and sometimes I think less plausible). It’s very possible this dynamic won’t change until after actually existing AI systems pose an existential risk.
There is a good chance that an AI catastrophe looks like an abrupt “coup” where AI systems permanently disempower humans with little opportunity for resistance. People seem to consistently round this risk down to more boring stories that fit better with their narratives about the world. It is quite possible that an AI coup will be sped up by humans letting AI systems control killer robo

...
AI misalignment risk from GPT-like systems?
by fiso64 — 2022-06-19T17:35:41.095Z

Right now, it seems that the most likely way we're gonna get an (intellectually) universal AI is by scaling models such as GPT. That is, models trained by self-supervised learning on massive piles of data, perhaps with a similar architecture to the transformer.

I do not see any risk due to misalignment here.

One failure mode I've seen discussed is that of manipulative answers, as seen in Predict-O-Matic. Maybe those AIs will learn that manipulating users to do actions with low entropy outcomes decreases the overall prediction error?
But why should a GPT-like ever output manipulative answers? I am not denying the possibility that a GPT successor develops human level intelligence. When it learns to predict the next word, it may genuinely go through an intellectual process which was created as it was forced to compress its predictions due to the ever increasing amounts of data it had to go through.
However, nowhere in the process of constructing a valid response does there seem to be an incentive to produce responses which manipulate the environment, be it to make it easier to predict, or to make it more in-line with the AI's predictions. After all, it wasn't trained in a responsive environment as an agent, but on a static dataset. And when it is in use, it's just a frozen model, so there is obviously no utility function.

Am I wrong here? Are there any other failure modes I did not think of?
[Link-post] On Deference and Yudkowsky's AI Risk Estimates
by bmg — 2022-06-19T17:25:14.537Z

This is a link-post to a piece I just posted to the EA Forum, discussing negative aspects of Eliezer Yudkowsky's forecasting track record. In case it also receives significant discussion here, you may also want to look at the comments on the Forum (e.g. Gwern just posted a useful critical comment there).