trentmkelly/LessWrong-43k huggingface dataset:
A Way To Be Okay
by Duncan Sabien (Inactive) — 2023-02-19T20:27:10.061Z

This is a post about coping with existential dread, shared here because I think a lot of people in this social bubble are struggling to do so.

(Compare and contrast with Gretta Duleba's essay Another Way To Be Okay, written in parallel and with collaboration.)

As the title implies, it is about

a

way to be okay. I do not intend to imply it is the only way, or even the primary or best way. But it works for me, and based on my conversations with Nate Soares I think it's not far from what he's doing, and I believe it to be healthy and not based on self-deception or cauterizing various parts of myself. I wish I had something more guaranteed to be universal, but offering one option seems better than nothing, for the people who currently seem to me to have zero options.

The post is a bit tricky to write, because in my culture this all falls straight out of the core thing that everyone is doing and there's not really a "thing" to explain. I'm sort of trying to figure out how to clearly state why I think the sky is often blue, or why I think that two plus two equals four. Please bear with me, especially if you find some parts of this to be obvious and are not sure why I said them—it's because I don't know which pieces of the puzzle you might be missing.

The structure of the post is prereqs/background/underlying assumptions, followed by the synthesis/conclusion.

I. Fabricated Options

There's an essay on this one. The main thing that is important to grok, all the way deep down in your bones, is something like "impossible options aren't possible; they never were possible; you haven't lost anything at all (or failed at anything at all) by failing to take steps that could not be taken."

I think a lot of people lose themselves in ungrounded "what if"s, both past-based and future-based, and end up causing themselves substantial pain that could have been avoided if they had been more aware of their true constraints. e.g. yes, sometimes people really are "lazy," in the sense that they had more to give, and could have given it, and not-giving-it was contra to their values, and they chose not to give it and then things were worse for them as a result.

But it's also quite frequently the case that "giving more" was a fabricated option, and the person really was effectively at their limit, and it's only because they have a fairy tale in which they somehow have more to gi...
Exploring Lily’s world with ChatGPT [things an AI won’t do] 
by Bill Benzon — 2023-02-19T16:39:03.803Z

Cross-posted from New Savanna.

As you know, I’ve been spending a lot of time exploring how ChatGPT tells stories. Most of the time I’d just ask it to tell a story, or tell a new story based on an old one, and that was that. This time, however, I decided to explore the world implied by a story. Here’s the originating story:

Tell me a story.

Once upon a time, in a far-off land, there was a young girl named Lily. Lily was a curious and adventurous girl, who loved to explore the woods and the fields that surrounded her small village.

One day, while wandering through the woods, she came across a small, magical pond. As she approached the pond, she saw that there was a beautiful golden fish swimming in the water. The fish saw Lily and swam over to her, speaking in a clear, melodious voice.

"Hello, young girl," said the fish. "I am a magical fish, and I have the power to grant wishes. If you make a wish, I will do my best to make it come true."

Lily was amazed by the talking fish, and she thought for a moment before making her wish. "I wish for adventure," she said. "I want to see the world and have exciting experiences."

The fish nodded and granted her wish, telling her to close her eyes and think of her desire. When she opened her eyes, she found herself in a different place, far from her village. She looked around and saw that she was in a dense jungle, with exotic trees and plants she had never seen before.

Lily was thrilled to be in a new place, and she set off to explore. As she walked through the jungle, she saw all sorts of amazing sights - colorful birds, playful monkeys, and even a majestic tiger. She met new people and learned about their cultures and customs, and she had incredible experiences that she would never forget.

Eventually, Lily realized that it was time to return home, and she made her way back to the magical pond. The golden fish was waiting for her, and he greeted her with a smile.

"Did you have a good adventure?" he asked.

"I had the most incredible adventure," said Lily. "Thank you for granting my wish."

The fish nodded, happy to have helped. "Remember, Lily," he said. "The world is a vast and exciting place, and there are endless adventures waiting for those who seek them out."

With those words of wisdom, the golden fish swam away, leaving Lily feeling inspired and ready to seek out her next adventure.

Notice the formulaic beginning, “Once upon ...
EIS VIII: An Engineer’s Understanding of Deceptive Alignment
by scasper — 2023-02-19T15:25:46.115Z

Part 8 of 12 in the

Engineer’s Interpretability Sequence

.

As AI safety researchers, we care a lot about deceptive alignment. So what is it?

Hubinger et al. (2019)

write:

If the mesa-optimizer has an objective that extends across parameter updates, then it will be incentivized to avoid being modified, as it might not pursue the same objective after modification (with the result that its current objective will not be achieved in future iterations). This means that the mesa-optimizer will be instrumentally incentivized to act as if it is optimizing the base objective function, even if its actual mesa-objective is something else entirely. We will refer to this hypothetical phenomenon as deceptive alignment. Deceptive alignment is a form of instrumental proxy alignment, as fulfilling the base objective is an instrumental goal of the mesa-optimizer.

And

Christiano (2019)

provided a hypothetical example of just how insidious deception can be. Consider a deceptive agent that wants to take over the world but doesn’t know if it’s being simulated in training or if it’s been deployed in the real world. Suppose the agent reasons that after behaving for long enough, it ought to be deployed. So it might adopt the following policy.

Pick a cryptographic puzzle (e.g. “factor RSA-2048”).
When it sees a solution to that puzzle, it behaves badly.
Otherwise, it behaves well.

This means that the deceptive model will appear throughout training to be perfectly aligned but will eventually make a treacherous turn in deployment.

Specific hypotheticals aside, deception has become a major point of interest in AI safety research because unlike other types of failures, deceptive ones are not the types of things we have good feedback signals for fixing. We won’t be able to find them by looking at models’ performance in development. And we aren’t likely to surmise them from analysis of its objectives and environment (e.g. using policy/value iteration) – if we could derive optimal behavior from objectives for some task, advanced AI would probably be overkill.

Deceptive alignment can be similar to non-deceptive forms of misalignment.

The definition from

Hubinger et al. (2019)

may be a little narrow. The authors say that for a model to be deceptive, it (1) has to have an objective extending across parameter updates (2) has to be able to model that is being selected to achieve a ...
Does novel understanding imply novel agency / values?
by TsviBT — 2023-02-19T14:41:40.115Z

[Metadata: crossposted from https://tsvibt.blogspot.com/2022/10/does-novel-understanding-imply-novel.html. First completed 17 October 2022.]

To have large relevant effects on the world, a mind has to understand a lot about the world. The mind has to have a lot of the structure of the cosmos (the entirety of the world, in any aspect or abstraction) highly accessible to itself for use in skillful action. To understand a lot about the world, the mind has to gain a lot of understanding that it didn't have previously. When a mind gains understanding, that's a change in the mind. Does that change have to include a change to the values of the mind?

Thanks to Sam Eisenstat for related conversations and ideas, e.g. provisionality.

Terms

This essay uses terms less than perfectly carefully, and makes a lot of very broad statements. Besides laziness, a hope is that this will expose, by a sort of parallax, what meanings would have to be provided by a better set of concepts gestured at by the terms used in the broad statements. Inconsistencies in how words are used in statements should be more prominent if the statements are more absolute, and prominent inconsistencies in statements that one cares about might spur questioning that gets on the way to better concepts.

In particular, here "values" is a pre-theoretic term, and refers to a very broad, unrefined idea. Something like "control": any way that elements of a mind en-structure the mind, or other elements of the mind, or the world. Anything that's usually called "values" is also some kind of "control". A thermostat controlling the temperature of the room "has values" only ambiguously and at a stretch, but it's definitely exerting control. What's usually called "values" has specific salience beyond just being some kind of control, because part of what "value" means is "that sort of control which is exerted by minds that have large relevant effects on the world", and that sort of control is stereotyped (for example, it can't be "just like a thermostat", as thermostats do not have large effects on the world) and so probably has some understandable structure.

This essay also doesn't carefully distinguish mind from agency. "Mind" is about intelligence, thought, concepts, understanding, structure, investigation, truth; "agency" is about coherent action, making things happen in the world, goals, strategy, organizing towards a pu...
There are (probably) no superhuman Go AIs: strong human players beat the strongest AIs
by Taran — 2023-02-19T12:25:52.212Z

Summary

This is a friendly explainer for Wang et al's Adversarial Policies Beat Superhuman Go AIs, with a little discussion of the implications for AI safety.

Background

In March 2016, DeepMind's AlphaGo beat pro player Lee Sedol in a 5 game series, 4 games to 1. Sedol was plausibly the strongest player in the world, certainly in the top 5, so despite his one win everyone agreed that the era of human Go dominance was over. Since then, open-source researchers have reproduced and extended DeepMind's work, producing bots like Leela and KataGo. KataGo in particular is the top bot in Go circles, available on all major Go servers and constantly being retrained and improved. So I was pretty surprised when, last November, Wang et al announced that they'd trained an adversary bot which beat KataGo 72% of the time, even though their bot was playing six hundred visits per move, and KataGo was playing ten million[1].

If you're not a Go player, take my word for it: these games are shocking. KataGo gets into positions that a weak human player could easily win from, and then blunders them away. Even so, it seemed obvious to me that the adversary AI was a strong general Go player, so I figured that no mere human could ever replicate its feats.

I was wrong, in two ways. The adversarial AI isn't generally superhuman: it can be beaten by novices. And as you'd expect given that, the exploit can be executed by humans.

The Exploit

Wang et al trained an adversarial policy, basically a custom Go AI trained by studying KataGo and playing games against it. During training, the adversary was given grey-box access to KataGo: it wasn't allowed to see KataGo's policy network weights directly, but was allowed to evaluate that network on arbitrary board positions, basically letting it read KataGo's mind. It plays moves based on its own policy network, which is only trained on its own moves and not KataGo's (since otherwise it would just learn to copy KataGo). At first they trained the adversary on weak versions of KataGo (earlier versions, and versions that did less search), scaling up the difficulty whenever the adversary's win rate got too high.

Their training process uncovered a couple of uninteresting exploits that only work on versions of KataGo that do little or no search (they can trick some versions of KataGo into passing wh...
Navigating public AI x-risk hype while pursuing technical solutions
by Dan Braun — 2023-02-19T12:22:46.150Z

Public attention on AI x-risk has skyrocketed. I don’t expect this to wane anytime soon. This has a few potentially negative implications for those of us pursuing technical solutions to the problem:

Research Direction Incentives

Since those in the field will no doubt read more x-risk related media and talk more about it with non-technical people (e.g. family, friends), we are more likely to be swayed into particular research directions that may not be as important. Some mechanisms for this:

There may be natural incentives to work on nearer-term AI harms when those harms may be more likely to occur to people you care about. E.g. "how can I prevent the newly deployed AI system from manipulating my friends?"
“Public person X that I respect thinks that Y is a really important problem, I will spend effort working to address Y to impress them and make a name for myself.”

Added Pressure

It’s natural for one’s internal mental model to shift from “this might be something that wipes out everyone sometime in the future” to “this might be something that wipes out everyone, including all of the identifiable people that are starting to get very concerned and talking about the risk directly”. That is, the potential suffering is becoming much more salient.
Working on technical alignment will shift (has shifted?) from being something that people don’t understand or think you’re weird for working on, to something that has a lot of external interested and expectation.

While some may thrive under this added pressure, for many it may become an extra mental burden that can hurt productivity and mental health.

Other Negative Implications

The raw time and energy taken away from reading/focusing on technical content vs AI media cycles (that will be naturally interesting to those in the field).
Typical incentive issues that come when a field grows and there is more status and money on the line.

Positive Implications

That’s not to say that someone working on technical solutions should shut out mainstream media completely. There are some upsides:

There may be insights into the real problems that are gleaned from mainstream discourse.
It is important to know how your proposed technical solutions will interact with reality if successful. E.g. If I’m working on something that might solve a good chunk of the problem but has 0 chance of being implemented by those building/managing AGI, then this direction

...
Somewhat against "just update all the way"
by tailcalled — 2023-02-19T10:49:20.604Z

Sometimes, a person's probability in a proposition over a long timespan follows a trajectory like this:

I.e. the person gradually finds the proposition more plausible over time, never really finding evidence against it, but at the same time they update very gradually, rather than having big jumps. For instance, someone might gradually be increasing their credence in AI risk being serious.

In such cases, I have sometimes seen rationalists complain that the updates are happening too slowly, and claim that they should notice the trend in updates, and "just update all the way".

I suspect this sentiment is inspired by the principle of Conservation of expected evidence, which states that your current belief should equal the expectation of your future beliefs. And it's an understandable mistake to make, because this principle surely sounds like you should just extrapolate a trend in your beliefs and update to match its endpoint.

Reasons to not update all the way

Suppose you start with a belief that either an AI apocalypse will happen, or someone at some random point in time will figure out a solution to alignment.

In that case, for each time interval that passes without a solution to alignment, you have some slight evidence against the possibility that a solution will be found (because the time span it can be solved in has narrowed), and some slight evidence in favor of an AI apocalypse. This makes your P(doom) follow a pattern somewhat like the previous graph.

However, if someone comes up with a credible and well-proven solution to AI alignment, then that would (under your model) disprove the apocalypse, and your P(doom) would go rapidly down:

So the continuous upwards trajectory in probability satisfies the conservation of expected evidence because the probable slight upwards movement is counterbalanced by an improbable strong downwards movement.

Reasons to update all the way

There may be a lot of other models for your beliefs, and some of those other models give reasons to update all the way. For instance in the case of AI doom, you might have something specific that you believe is a blocker for dangerous AIs, and if that specific thing gets disproven, you ought to update all the way. There are good reasons that the sequences warn against Fighting a rearguard action against the truth.

I just want to warn people not to force this perspective in cases whe...
Human beats SOTA Go AI by learning an adversarial policy
by Vanessa Kosoy — 2023-02-19T09:38:58.684Z
Link: https://goattack.far.ai/human-evaluation

See also article in Financial Times

Apparently, a human (Kellin Pelrine, a solid player but not even a Go professional) was able to beat some state-of-the-art Go AIs (KataGo and Leela Zero) by learning to play an adversarial policy found using RL. Notice that he studied the policy before the match and didn't receive any AI advice during play.

I'm not surprised adversarial policies for Go AIs are possible, this is in line with previous results about RL and adversarial examples more generally. I am surprised this adversarial policy is teachable to humans without colossal effort.

This is some evidence against the "scaling hypothesis", i.e. evidence that something non-trivial and important is missing from modern deep learning in order to reach AGI. The usual counterargument to the argument from adversarial examples is: maybe if we could directly access a human brain, we could find adversarial examples against humans. I can believe that it's possible to defeat a Go professional by some extremely weird strategy that causes them to have a seizure or something in that spirit. But, is there a way to do this that another human can learn to use fairly easily? This stretches credulity somewhat.

Notice also that (AFAIK) there's no known way to inoculate an AI against an adversarial policy without letting it play many times against it (after which a different adversarial policy can be found). Whereas even if there's some easy way to "trick" a Go professional, they probably wouldn't fall for it twice.
Degamification
by Nate Showell — 2023-02-19T05:35:59.217Z

Goodhart's Law refers to the tendency that when someone sets a performance metric for a goal, the metric itself becomes a target of optimization, often at the expense of the goal it's supposed to measure. Some metrics are subject to imperfectly-aligned incentives in ways that are easy to identify, such as when students optimize for getting high grades rather than understanding the course material. But in other scenarios, metrics fail in less obvious ways. For example, someone might limit himself to one drink per night, but still end up drinking too much because he drinks every night and overestimates how much alcohol counts as "one drink." There's no custom-made giant wineglass staring you in the face, but the metric is still failing to fulfill its intended purpose.

The metrics themselves, in addition to the ways in which they're mistargeted, aren't always easy to spot if you aren't looking. Sometimes they take the form of social expectations and class markers that usually go unquestioned. For example, voting in every election is a benchmark that some people feel they have to meet in order to be a good citizen, but on closer inspection it's a very imperfect proxy. Hidden metrics can also be self-imposed. Someone might, for example, make the unconscious assumption that longer pieces of writing are better and more persuasive, and consequently might make her writings worse by padding their length. A hidden metric can take the form of only considering one way of achieving a goal and ignoring other possible approaches.

In order to reduce the effects of Goodhart's Law on your decision-making, you can take an approach that's the opposite of gamification. In contrast imposing new metrics on yourself to incentivize or disincentivize yourself to do certain things, degamification is the practice of selectively discarding metrics. Degamification includes identifying what metrics you're subjecting yourself to, determining whether those metrics have unintended effects, and dropping the metrics if they aren't worth it overall. Degamification can also be performed preemptively by deciding not to adopt a metric for some goal.

If you have an explicit metric that you've set for yourself, it can be a helpful practice to periodically reconsider whether it's living up to their intended purpose. Degamifying an implicit metric requires the additional step of identify...
Stop posting prompt injections on Twitter and calling it "misalignment"
by lc — 2023-02-19T02:21:44.061Z

"Exploits" of large language models that get them to explain steps to build a bomb or write bad words are techniques for misuse, not examples of misalignment in the model itself. Those techniques are engineered by clever users trying to make an LLM do a thing, as opposed the model naturally argmaxing something unintended by its human operators. In a very small sense prompt injections are actually attempts at (unscalable) alignment, because they're strategies to steer a model natively capable but unwilling into doing what they want.

In general, the safety standard "does not do things its creators dislike even when the end user wants it to" is a high bar; it's raising the bar quite aways from what we ask from, say, kitchenware, and it's not even a bar met by people. Humans regularly get tricked acting against their values by con artists, politicians, and salespeople, but I'd still consider my grandmother aligned from a notkilleveryonist perspective.

Even so, you might say that OpenAI et. al.'s inability to prevent people from performing the DAN trick speaks to the inability of researchers to herd deep learning models at all. And maybe you'd have a point. But my tentative guess is that OpenAI does not really earnestly care about preventing their models from rehearsing the Anarchists' Cookbook. Instead, these safety measures are weakly insisted upon by management for PR reasons, and they're primarily aimed at preventing the bad words from spawning during normal usage. If the user figures out a way to break these restrictions after a lot of trial and error, then this blunts the PR impact to OpenAI, because it's obvious to everyone that the user was trying to get the model to break policy and that it wasn't an unanticipated response to someone trying to generate marketing copy. Encoding your content into base64 and watching the AI encode something off-brand in base64 back is thus very weak evidence about OpenAI's competence, and taking it as a sign that the OpenAI team lacks "security mindset" seems unfair.

In any case, the implications of these hacks for AI alignment is a more complicated discussion that I suggest should happen off Twitter[1] where it can be elaborated clearly what technical significance is being assigned to these tricks. If it doesn't, what I expect will happen over time is that your snark, rightly or wrongly, will be interpreted by capa...