trentmkelly/LessWrong-43k huggingface dataset:
What Failure Looks Like is not an existential risk (and alignment is not the solution)
by otto.barten — 2024-02-02T18:59:38.346Z

Introduction

Among those thinking that AI is an existential risk, there seems to be significant disagreement on what the main threat model is. Threat model uncertainty makes it harder to reduce this risk: a faulty threat model used by an important actor will likely lead to suboptimal decision making. This is why I think there should be more discussion on what the probability is for each existential risk threat model, rather than merely how much one’s accumulated p(doom) is.

There are multiple threat model overviews, such as those by Kaj Sotala, Samuel Martin, and Richard Ngo. Three AI existential risk threat models seem to emerge as particularly popular:

AI takeover (championed by Yudkowsky/Bostrom, common among rationalists).
What Failure Looks Like (1, 2) (authored by Paul Christiano, common in EA).
Bad actor risk (influential in policymaking).

This post will focus on What Failure Looks Like. I will argue that this threat model is very unlikely (<0.1%) to lead to an existential event. If true, this would mean more emphasis by those worrying about existential risk should be placed on the other threat models, such as various AI takeover scenarios and bad actor risk.

Also, I argue that decreasing the risks (both existential and nonexistential) caused by the What Failure Looks Like threat model should not be done by AI alignment or by an AI pause, but rather by traditional AI regulation after development of the technology and at the point of application. The EU AI Act with its tiered approach can serve as a model. Note that this does not imply that I think alignment and a pause are not important anymore: I think they could be crucial to counter a Yudkowsky/Bostrom-style AI takeover, which I see as the most likely existential risk.

Why I think the existential risk due to What Failure Looks Like is very low

In What Failure Looks Like (2019), Paul Christiano introduces his two threat models towards ‘going out’. It is not clear to me whether this means an existential event, defined by Toby Ord and others as human extinction, a permanent dystopia, or an unrecoverable collapse, where the last two options should be stable over billions of years. However, even if Christiano himself would not mean such an existential event, others seem to interpret his threat model in this way, for example in this paper by Richard Ngo: “However, many other ...
Solving alignment isn't enough for a flourishing future
by mic — 2024-02-02T18:23:00.643Z
Link: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4684068

I'm sharing a paper I previously presented to the Stanford Existential Risks Conference in April 2023. In past years, much discussion of making the long-term future go well has focused on AI alignment and mitigating extinction risks. More recently, there has been more exploration of how this may fail to capture much of humanity's potential and other cause areas to investigate (e.g., post-AGI governance and digital minds). I hope this paper is a helpful contribution to this discussion. [Note: This post is a lightly edited version of this PDF.]

Abstract

AI alignment is commonly explained as aligning advanced AI systems with human values. Especially when combined with the idea that AI systems aim to optimize their world based on their goals, this has led to the belief that solving the problem of AI alignment will pave the way for an excellent future. However, this common definition of AI alignment is somewhat idealistic and misleading, as the majority of alignment research for cutting-edge systems is focused on aligning AI with task preferences (training AIs to solve user-provided tasks in a helpful manner), as well as reducing the risk that the AI would have the goal of causing catastrophe.

We can conceptualize three different targets of alignment: alignment to task preferences, human values, or idealized values. Extrapolating from the deployment of advanced systems such as GPT-4 and from studying economic incentives, we can expect AIs aligned with task preferences to be the dominant form of aligned AIs by default.

Aligning AI with task preferences will not by itself solve major problems for the long-term future. Among other problems, these include moral progress, existential security, wild animal suffering, the well-being of digital minds, risks of catastrophic conflict, and optimizing for ideal values. Additional efforts are necessary to motivate society to have the capacity and will to solve these problems.

Introduction

The future development of artificial general intelligence (AGI) is often presented as having two distinct possibilities: either it is misaligned with human values, resulting in an existential catastrophe, or it is aligned and optimizes for human values [Sotala and Gloor 2017]. One might assume that as long as we develop aligned AGI, the futu...
Manifold Markets
by PeterMcCluskey — 2024-02-02T17:48:36.630Z
Link: https://bayesianinvestor.com/blog/index.php/2024/02/01/manifold-markets/

Manifold Markets is a prediction market platform where I've been
trading since September. This post will compare it to other prediction
markets that I've used.

Play Money

The most important fact about Manifold is that traders bet mana, which
is for most purposes not real money. You can buy mana, and use mana to
donate real money to charity. That's not attractive enough for most of
us to treat it as anything other than play money.

Play money has the important advantage of not being subject to CFTC
regulation or gambling laws. That enables a good deal of innovation that
is stifled in real-money platforms that are open to US residents.

Play money misses out on an important advantage of large real-money
markets: traders who want to manipulate a price face arbitrarily large
costs to counter large investors (such as hedge funds) who notice the
manipulation. So I trust Manifold prices less than I trust those of the
best financial markets.

Actual real-money prediction markets are limited in size, and mostly too
small to attract hedge funds. So Manifold is sometimes able to beat
real-money
markets.

Manifold is structured so that manipulation seems rare. Competent
traders who have been active for a long time accumulate enough mana to
dominate newer traders. So prices are mostly set by traders who have a
significant track record of non-manipulative trading. That's not enough
to make Manifold prices a good input for decisions that affect billions
of dollars (how hard is it to bribe traders?), but it's good enough for
a wide variety of less important decisions.

Liquidity

Another advantage of play money is that Manifold can create more mana
whenever they want.

Manifold has taken advantage of that to enable traders to, in effect,
buy on margin. In addition, they give daily handouts of mana to people
who trade daily. Those two features ensure that I mostly don't end up
in the situation that I got into on some other platforms of spending all
my money and then waiting for markets to be resolved. My more regular
trading on Manifold helps ensure that, at least for markets that get
some minimal amount of attention, there are enough regular traders to
keep the markets somewhat efficiently priced.

The most important way that Manifold provides liquidity is via an
automated market maker. The normal trading mode is that I accept
whatever price the system offers. Since Manifold uses play mo...
Types of subjective welfare
by MichaelStJules — 2024-02-02T09:56:34.284Z

Summary

I describe and review four broad potential types of subjective welfare: 1) hedonic states, i.e. pleasure and unpleasantness, 2) felt desires, both appetitive and aversive, 3) belief-like preferences, i.e. preferences as judgements or beliefs about value, and 4) choice-based preferences, i.e. what we choose or would choose. My key takeaways are the following:

Belief-like preferences and choice-based preferences seem unlikely to be generally comparable, and even comparisons between humans can be problematic (

more

).
Hedonic states, felt desires and belief-like preferences all plausibly matter intrinsically, possibly together in a morally pluralistic theory, or unified as subjective appearances of value or reasons or even under belief-like preferences (

more

).
Choice-based preferences seem not to matter much or at all intrinsically (

more

).
The types of welfare are dissociable, so measuring one in terms of another is likely to misweigh it relative to more intuitive direct standards and risks discounting plausible moral patients altogether (

more

).
There are multiple potentially important variations on the types of welfare (

more

).

Acknowledgements

Thanks to Brian Tomasik, Derek Shiller and Bob Fischer for feedback. All errors are my own.

The four types

It appears to me that subjective welfare — welfare whose value depends only or primarily[1] on the perspectives or mental states of those who hold them — can be roughly categorized into one of the following four types based on how they are realized: hedonic states, felt desires, belief-like preferences and choice-based preferences. To summarize, there’s welfare as feelings (hedonic states and felt desires), welfare as beliefs about value (belief-like preferences) and welfare as choices (choice-based preferences). I will define, illustrate and elaborate on these types below.

For some discussion of my choices of terminology, see the following footnote.[2]

Hedonic states

Hedonic states: feeling good and feeling bad, or (conscious) pleasure and unpleasantness/unpleasure/displeasure,[3] or (conscious) positive and negative affect. Their causes can be physical, like sensory pleasures and physical pain, or psychological, like achievement, failure, loss, shame, humour and threats, to name a few.

A juvenile member of the species Homo sapiens expressing positive affect (DALL·E)

It’s unclear if interpersonal...
Open Source Sparse Autoencoders for all Residual Stream Layers of GPT2-Small
by Joseph Bloom — 2024-02-02T06:54:53.392Z

Browse these SAE Features on Neuronpedia!

Update 1: Since we posted this last night, someone pointed out that our implementation of ghost grads has a non-trivial error (which makes the results a-priori quite surprising). We computed the ghost grad forward pass using Exp(Relu(W_enc(x)[dead_neuron_mask])) rather than Exp((W_enc(x)[dead_neuron_mask])). I'm running some ablation experiments now to get to the bottom of this.

Update 2: I've since investigated this further and run some ablation studies with the following results. Ghost grads weren't working as intended due to the Exp(Relu(x)) bug but the resulting SAE's were still quite good (later layers had few dead neurons simply because when we dropped the number of features, you get less dead neurons. I've found that with a correct ghost grads implementation, you can get less dead neurons and will update the library shortly. (I will make edits to the rest of this post to reflect my current views). Sorry for the confusion.

This work was produced as part of the ML Alignment & Theory Scholars Program - Winter 2023-24 Cohort, under mentorship from Neel Nanda and Arthur Conmy. Funding for this work was provided by the Manifund Regranting Program and donors as well as LightSpeed Grants.

This is intended to be a fairly informal post sharing a set of Sparse Autoencoders trained on the residual stream of GPT2-small which achieve fairly good reconstruction performance and contain fairly sparse / interpretable features. More importantly, advice from Anthropic and community members has enabled us to train these fairly more efficiently / faster than before. The specific methods that were most useful were: ghost gradients, learning rate warmup, and initializing the decoder bias with the geometric median. We discuss each of these in more detail below.

Feature found in Residual Stream (Pre) Layer 10 of GPT2 Small. The force is strong with this one.

5 Minute Summary

We’re publishing a set of 12 Sparse AutoEncoders for the GPT2 Small residual stream.

These dictionaries have approximately 25,000 features each, with very few dead features (mainly in the early layers) and high quality reconstruction (log loss when the activations are replaced with the output is 3.3 - 3.6 as compared with 3.3 normally).
The L0’s range from 5 in the first layer to 70 in the 9th SAE (incre

...
Soft Prompts for Evaluation:
Measuring Conditional Distance of Capabilities
by porby — 2024-02-02T05:49:11.189Z
Link: https://arxiv.org/abs/2505.14943

Abstract:

To help evaluate and understand the latent capabilities of language models, this paper introduces an approach using optimized input embeddings, or `soft prompts,' as a metric of conditional distance between a model and a target behavior. The technique aims to facilitate latent capability discovery as a part of automated red teaming/evaluation suites and to provide quantitative feedback about the accessibility of potentially concerning behaviors in a way that may scale to powerful future models, including those which may otherwise be capable of deceptive alignment. An evaluation framework using soft prompts is demonstrated in natural language, chess, and pathfinding, and the technique is extended with generalized conditional soft prompts to aid in constructing task evaluations.

What is this?

This paper applies soft prompts to evaluations. They're prompts composed of optimized token embeddings; they aren't restricted to the usual dictionary of discrete tokens. To the limit of the optimizer's ability, they compress conditions into the tokens composing the soft prompt.

Different target behaviors or capabilities will tend to require different numbers of soft prompt tokens. Capabilities that are natural to the model, like repeating a single token over and over again, can be achieved with a single soft prompt token even in small models. Repeating a longer string may require more tokens, and even partially eliciting more complex behavior like pathfinding or chess could require many more.

Why is this useful?

Soft prompts offer a simple way to quantify the conditional distance between a model and a target behavior. If many tokens are required, then lots of information-as-conditions is needed to define the behavior and it's unlikely that it will happen by chance. If no number of tokens suffices, the behavior is likely inaccessible by inputs alone.[1]

It can be used as a part of evaluation suites to monitor the change in distance to concerning behaviors across models and scales and the degree to which those behaviors can be achieved in the first place.

It is also difficult for the model to hide a capability that is otherwise accessible to some input. If the original model is an end-to-end differentiable product of gradient descent, it has already been shown to be transparent to the same type of optimizer that will be tasked with adversarially optimizing the input soft...
Running a Prediction Market Mafia Game
by Arjun Panickssery — 2024-02-01T23:24:27.659Z
Link: https://arjunpanickssery.substack.com/p/running-a-prediction-market-mafia

In a normal game of mafia/werewolf, there are a few high-level strategic elements:

Night Play: During night phases, the mafia has to choose who to try to kill and the players with night actions (cops, doctors, and in some cases trackers, watchers, vigilantes, etc.) have to choose who to target.
Day Play: During day phases, the townies try to share information without exposing too much information about the identities of their power roles (cops, doctors, etc.) while the mafia tries to provide misleading information. Everyone is trying to affect (1) future night actions and (2) the daily execution vote.

The goal of the prediction market mafia game is to replace the voting mechanism with a set of markets, one for each living player on whether they’re a member of the mafia. At the end of each day phase, the player with the highest market price is executed.

If you replace the election with a market that the players participate in, the game will break because townies have an incentive to bet their own markets down to 0% and to pressure mafia players to do the same and drain their capital (I didn’t think about this that hard; maybe a few adjustments actually suffice to make this setup work, but I don’t think so).

A simple way to make it work is to let the spectators bet on the prediction markets. So now there are multiple parties to the game:

the town players, trying to kill all of the mafia players;
the mafia players, trying to kill the town players; and
the speculators, trying to maximize their profits.

I used the term “spectators” above, but the speculators need to have everyone’s night actions hidden from them the same way players do.

Tweaks/considerations:

The end of the day phase could be decided by some kind of random process that avoids the problem of rapid trading activity right before the phase ends.
There can be other speculative markets that aren’t execution-relevant, such as on the identity of particular players or roles, the number of phases the game will take, the eventual winning side, and so on, but this might lead to weird outcomes where speculators move the execution-relevant markets with the intent of manipulative other markets, such as by trying to take a loss by getting a town player executed because this will increase profits on other markets.
In most setups, most dead players won’t be able to join the speculators, because they’ll posses

...
Evaluating Stability of Unreflective Alignment
by james.lucassen — 2024-02-01T22:15:40.902Z
Link: https://jlucassen.com/evaluating-stability-of-unreflective-alignment/

This post has an accompanying SPAR project! Apply here if you’re interested in working on this with me.

Huge thanks to Mikita Balesni for helping me implement the MVP. Regular-sized thanks to Aryan Bhatt, Rudolph Laine, Clem von Stengel, Aaron Scher, Jeremy Gillen, Peter Barnett, Stephen Casper, and David Manheim for helpful comments.

0. Key Claims

Most alignment work today doesn’t aim for alignment that is stable under value-reflection[1].
I think this is probably the most sensible approach to alignment.
But there is a threat model which could potentially be a serious obstacle to this entire class of alignment approaches, which is not currently being tracked or evaluated. It goes like this:

Long-horizon capabilities require a particular capability I call “stepping back” (described in more detail in the body of the post)
Stepping back seems likely to be learned in a very generalizable way
If stepping back generalizes in what seems like the simplest / most natural way, it will create a tendency for AIs to do value-reflection
If we’re unable to prevent this value-reflection, it will cause any AI whose alignment is not stable under value-reflection to become misaligned

However, I’m quite uncertain about whether or not this threat model will go through, for a few reasons:

The chain of argument is highly conjunctive and therefore fragile.
The track record of anthropomorphic reasoning about which AI capabilities will imply which other AI behaviors seems poor.
There is an intuitive counterexample provided by humans, at least at human-level intelligence.

Even given this uncertainty, work on this threat model seems very neglected given the wide swath of highly-invested-in alignment approaches that it could affect.
I don’t think further abstract argument about the likelihood of this threat model will reduce my uncertainty much. Instead, I propose some evals to keep track of each component of this threat model:

Evaluate the stepping back capabilities of frontier AIs in domains where they receive the most long-horizon training.
Evaluate the generalization of frontier AIs stepping back behavior to other task domains.
Evaluate how difficult it is to decouple the capabilities of frontier AIs between domains.
Evaluate how difficult it is to prevent frontier AIs from thinking certain “forbidden thoughts” even if those thoughts are incentivized by the task setting

1. Reflective vs Unreflective A

...
Davidad's Provably Safe AI Architecture - ARIA's Programme Thesis
by simeon_c — 2024-02-01T21:30:44.090Z
Link: https://www.aria.org.uk/wp-content/uploads/2024/01/ARIA-Safeguarded-AI-Programme-Thesis-V1.pdf 

The programme thesis of Davidad's agenda to develop provably safe AI has just been published. For context, Davidad is a Programme Director at ARIA who will grant somewhere between £10M and £50M over the next 3 years to pursue his research agenda.

It is the most comprehensive public document detailing his agenda to date.

Here's the most self-sufficient graph explaining it at a high-level although you'll have to dive into the details and read it several times to start grasping the many dimensions of it.

I'm personally very excited by Davidad's moonshot that I currently see as the most credible alternative to scaled transformers, which I consider to be too flawed to be a credible safe path, mostly because:

Ambitious LLM interpretability seems very unlikely to work out:

Why: the failed attempts at making meaningful progress of the past few years + the systematic wall of understanding of ~80% of what's going on across reverse engineering attempts

Adversarial robustness to jailbreak seems unlikely to work out:

Why: failed attempts at solving it + a theoretical paper of early 2023 that I can't find right now + increasingly large context windows

Safe generalization with very high confidence seems quite unlikely to work out

Why: absence of theory on transformers + weak interpretability

A key motivation for pursuing moonshots a la Davidad is, as he explains in his thesis, to shift the incentives from the current race to the bottom, by derisking credible paths to AI systems where we have strong reasons to expect confidence in the safety of systems. See the graph below: