trentmkelly/LessWrong-43k huggingface dataset:
[Interview w/ Quintin Pope] Evolution, values, and AI Safety
by fowlertm — 2023-10-24T13:53:06.146Z

On the Futurati Podcast, we recently released an interview with Quintin Pope.

As you can imagine, it mostly focused on:

Inner v.s. outer optimization;
The sharp left turn;
Natural selection, and what kind of evidence we can draw from the emergence of the first great general intelligence;
AI Safety more broadly;
How human values form;

Check it out! And share it if you'd like us to do more interviews like this :)
Lying is Cowardice, not Strategy
by Connor Leahy — 2023-10-24T13:24:25.450Z
Link: https://cognition.cafe/p/lying-is-cowardice-not-strategy

(Co-written by

Connor Leahy

and

Gabe

)

We have talked to a whole bunch of people about pauses and moratoriums. Members of the AI safety community, investors, business peers, politicians, and more.

Too many claimed to pursue the following approach:

It would be great if AGI progress stopped, but that is infeasible.
Therefore, I will advocate for what I think is feasible, even if it is not ideal.
The Overton window being what it is, if I claim a belief that is too extreme, or endorse an infeasible policy proposal, people will take me less seriously on the feasible stuff.
Given this, I will be tactical in what I say, even though I will avoid stating outright lies.

Consider if this applies to you, or people close to you.

If it does, let us be clear: hiding your beliefs, in ways that predictably leads people to believe false things, is lying. This is the case regardless of your intentions, and regardless of how it feels.

Not only is it morally wrong, it makes for a terrible strategy. As it stands, the AI Safety Community itself can not coordinate to state that we should stop AGI progress right now!

Not only can it not coordinate, the AI Safety Community is defecting, by making it more costly for people who do say it to say it.

We all feel like we are working on the most important things, and that we are being pragmatic realists.

But remember: If you feel stuck in the Overton window, it is because YOU ARE the Overton window.

—

1. The AI Safety Community is making our job harder

In a saner world, all AGI progress should have already stopped. If we don’t, there’s more than a 10% chance we all die.

Many people in the AI safety community believe this, but they have not stated it publicly. Worse, they have stated different beliefs more saliently, which misdirect everyone else about what should be done, and what the AI safety community believes.

To date, in our efforts to inform, motivate and coordinate with people: People in the AI Safety Community publicly lying has been one of the biggest direct obstacles we have encountered.

The newest example of this is ”Responsible Scaling Policies”, with many AI Safety people being much more vocal about their endorsement of RSPs than their private belief that in a saner world, all AGI progress should stop right now.

Because of them, we have been told many times that we are a minority voice, and that most people in the AI Safety community (under...
Announcing #AISummitTalks featuring Professor Stuart Russell and many others
by otto.barten — 2023-10-24T10:11:34.527Z

Register now: #AISummitTalks II, featuring Professor Stuart Russell and many others @ Wilton Hall, Bletchley, Tuesday 31 October, 14:00 to 15:30 GMT! - https://lu.ma/n9qmn4h6

About the Talks

The first edition of #AISummitTalks: Navigating Existential Risk was attended by over 250 people, with great engagement from the crowd.

For the second edition of our #AISummitTalks series about AI x-risk, we will meet just outside of the famous Bletchley Park on the eve of the #AISafetySummit. At the AI Safety Summit, world leaders will discuss for the first time how to prevent human extinction by AI. Society unfortunately will not have a say in these discussions, since only 150 people are invited. But at our AI Safety Summit Talks, organized together with Conjecture, you can be a part of this discussion!

Speakers

You will be joined by Existential Risk Observatory and Conjecture, who will host a keynote speech by none other than Prof. Stuart Russell (UC Berkeley), followed by a talk by Conjecture's Connor Leahy, among others. The event will be finalised by a panel discussion that brings together Andrea Miotti (Head of Strategy and Governance, Conjecture), as well as leading voices from the societal debate and the political realm, including investor Jaan Tallinn (Co-founder Centre for the Study of Existential Risk - CSER), Annika Brack (CEO The International Center for Future Generations - ICFG), Mark Brakel (Director of Policy, Future of Life Institute - FLI), Alexandra Mousavizadeh (Economist, CEO, Evident), Hal Hodson (Journalist The Economist), and a mystery guest! Moderator of the afternoon will be David Wood, chair of the London Futurists.

Register

Curious? Want to join the conversation? Be quick and reserve your spot - only 300 spots can be allocated.

https://lu.ma/n9qmn4h6
We're looking forward to welcoming you there!
Linkpost: A Post Mortem on the Gino Case
by Linch — 2023-10-24T06:50:42.896Z
Link: https://www.theorgplumber.com/posts/statement/

As a followup to my previous linkpost to the New Yorker article covering the Ariely and Gino scandals, I'm linking this statement by Zoe Ziani, the grad student who first discovered inconsistencies in Gino's papers. Here she gives a blow-by-blow retelling of her experience attempting to uncover fraud, as well as telling a fairly harrowing story about how higher-ups in her organization attempted to silence her.

I find this story instructive both on the object-level, and as a case study both for a) how informal corrupt channels tries to cover up fraud and corruption, and b) for how active participation is needed to make the long arc of history bend towards truth.

In her own words:

____

Disclaimer: None of the opinions expressed in this letter should be construed as statements of fact. They only reflect my experience with the research process, and my opinion regarding Francesca Gino’s work. I am also not claiming that Francesca Gino committed fraud: Only that there is overwhelming evidence of data fabrication in multiple papers for which she was responsible for the data.

On September 30th, 2023, the New Yorker published a long piece on “L’affaire Ariely/Gino”, and the role I played in it. I am grateful for the messages of support I received over the past few weeks. In this post, I wanted to share more about how I came to discover the anomalies in Francesca Gino’s work, and what I think we can learn from this unfortunate story.

What is The Story?

How it all began

I started having doubts about one of Francesca Gino’s paper (Casciaro, Gino, and Kouchaki, “The Contaminating Effect of Building Instrumental Ties: How Networking Can Make Us Feel Dirty”, ASQ, 2014; hereafter abbreviated as “CGK 2014” ) during my PhD. At the time, I was working on the topic of networking behaviors, and this paper is a cornerstone of the literature.

I formed the opinion that I shouldn’t use this paper as a building block in my research. Indeed, the idea that people would feel “physically dirty” when networking did not seem very plausible, and I knew that many results in Management and Psychology published around this time had been obtained through researchers' degrees of freedom. However, my advisor had a different view: The paper had been published in a top management journal by three prominent scholars… To her, it was inconceivable to simply disregard this paper.

I felt trapped: She kept insisti...
AI Pause Will Likely Backfire (Guest Post)
by jsteinhardt — 2023-10-24T04:30:02.113Z

I'm experimenting with hosting guest posts on this blog, as a way to represent additional viewpoints and especially to highlight ideas from researchers who do not already have a platform. Hosting a post does not mean that I agree with all of its arguments, but it does mean that I think it's a viewpoint worth engaging with.

The first guest post below is by Nora Belrose. In it, Nora responds to a recent open letter calling for a pause on AI development. Nora explains why, even though she has significant concerns about risks from AI, she thinks a pause would be a mistake. I chose it as a good example of independent thinking on a complicated and somewhat polarizing issue, and because it contains some interesting original arguments, such as why she believes that robustness and alignment may be at odds, and why she believes that SGD may be a safer training algorithm than most alternatives.

Should we lobby governments to impose a moratorium on AI research? Since we don’t enforce pauses on most new technologies, I hope the reader will grant that the burden of proof is on those who advocate for such a moratorium. We should only advocate for such heavy-handed government action if it’s clear that the benefits of doing so would significantly outweigh the costs.[1] In this essay, I’ll argue an AI pause would increase the risk of catastrophically bad outcomes, in at least three different ways:

Reducing the quality of AI alignment research by forcing researchers to exclusively test ideas on models like GPT-4 or weaker.

Increasing the chance of a “fast takeoff” in which one or a handful of AIs rapidly and discontinuously become more capable, concentrating immense power in their hands.

Pushing capabilities research underground, and to countries with looser regulations and safety requirements.

Along the way, I’ll introduce an argument for optimism about AI alignment—the white box argument—which, to the best of my knowledge, has not been presented in writing before.

Feedback loops are at the core of alignment

Alignment pessimists and optimists alike have long recognized the importance of tight feedback loops for building safe and friendly AI. Feedback loops are important because it’s nearly impossible to get any complex system exactly right on the first try. Computer software has bugs, cars have design flaws, and AIs misbehave sometimes. We need to be able to accurately eval

...
Human wanting
by TsviBT — 2023-10-24T01:05:39.374Z

[Metadata: crossposted from https://tsvibt.blogspot.com/2023/08/human-wanting.html. First completed August 22, 2023.]

We have pretheoretic ideas of wanting that come from our familiarity with human wanting, in its variety. To see what way of wanting can hold sway in a strong and strongly growing mind, we have to explicate these ideas, and create new ideas.

Human wanting

The problem of AGI alignment is sometimes posed along these lines: How can you make an AGI that wants to not kill everyone, and also wants to do some other thing that's very useful?

What role is the idea of "wanting" playing here? It's a pretheoretical concept. It makes an analogy to humans.

The meaning of wanting

What does it say about a human? When a human wants X, in a deep sense, then:

X has a good chance of actually happening, and if it doesn't happen that's because making X happen is difficult, or very "costly" in some sense;

novelty (new knowledge, understanding, ideas, skills, mindsets) that the human gains will be put to the use of bringing about X, and won't be put to the use of destroying the potential for X, and also won't terribly trample over other things that humans care about——the power contained in the human's mind is channeled, circumscribed, so that the power isn't applied towards ends other than those chosen by [that which wants X];

if the human can't achieve X directly, ze will recurse on creatively finding ways to become the sort of agent who can achieve X;

the human will interpret the meaning of X in a good, reasonable, sane, intended way, including when X was given in an ambiguous way;

the human won't pursue X in an extreme way that renders X no longer good as X;

the human won't merely pretend to pursue X and then at the last minute replace the potential for X with something else;

the human, if put in a context where negotiation or conflict with other agents is appropriate, will stand up for X;

these facts will persist, still holding true of the human as ze goes through life, learns, reflects, gains capabilities, and undergoes deep revisions of mental elements;

the human wants these facts to persist, and will notice and correct when zer growth has threatened or eroded these facts.

And, it is at least sometimes feasible for a human to choose to want X——or even, for a human to choose that another human will want X. Wanting is specifiable.

Baked-in claims

The concept of w...
Towards Understanding Sycophancy in Language Models
by Ethan Perez — 2023-10-24T00:30:48.923Z
Link: https://arxiv.org/abs/2310.13548

TL;DR: We show sycophancy is a general behavior of RLHF’ed AI assistants in varied, free-form text-generation settings, extending previous results. Our experiments suggest these behaviors are likely driven in part by imperfections in human preferences–both humans and preference models sometimes prefer convincingly-written sycophantic responses over truthful ones. This provides empirical evidence that we will need scalable oversight approaches.

Tweet thread summary: link

Summary of Paper

It has been hypothesized that using human feedback to align AI could lead to systems that exploit flaws in human ratings.[1] Meanwhile others have empirically found that language models repeat back incorrect human views,[2] which is known as sycophancy.[3] But these evaluations are mostly proof-of-concept demonstrations where users introduce themselves as having a particular view. And although these existing empirical results match the theoretical concerns, it isn’t clear whether they are actually caused by issues with human feedback. We decided to investigate these questions more thoroughly. In other words, is sycophancy actually a problem in AI assistants? And what is the role played by human preference judgments in sycophancy?

We first found that sycophancy is a general behavior of RLHF-trained conversational models. Across Anthropic, OpenAI, and Meta assistants, we found clear sycophancy in varied, free-form text tasks: models wrongly admitted mistakes, gave biased feedback, and mimicked user errors. The consistency might indicate a link to RLHF training more broadly, rather than model-specific factors.

We also release our evaluation datasets at

https://github.com/meg-tong/sycophancy-eval

. We believe they measure more realistic forms of sycophancy than existing datasets.

We then investigated whether human preference judgements play a role in this broadly observed behavior. Analyzing Anthropic's released helpfulness preference data, we found "matching user beliefs and biases" was highly predictive of human judgments. However, other factors, such as a response being truthful or assertive, are also predictive of human preferences.

We looked at the behavior of humans and preference models (PMs) in a specific setting—distinguishing between seemingly correct but incorrect sycophantic responses to user-stated misconceptions, and truthful responses to...
Open Source Replication & Commentary on Anthropic's Dictionary Learning Paper
by Neel Nanda — 2023-10-23T22:38:33.951Z

This is the long-form version of a public comment on Anthropic's Towards Monosemanticity paper

Introduction

Anthropic recently put out a really cool paper about using Sparse Autoencoders (SAEs) to extract interpretable features from superposition in the MLP layer of a 1L language model. I think this is an awesome paper and I am excited about the potential of SAEs to be useful for mech interp more broadly, and to see how well they scale! This post documents a replication I did of their paper, along with some small explorations building on it, along with (scrappy!) training code and weights for some trained autoencoders.

See

an accompanying colab tutorial

to load the trained autoencoders, and a demo of how to interpret what the features do.

TLDR

The core results seem to replicate - I trained a sparse autoencoder on the MLP layer of an open source 1 layer GELU language model, and a significant fraction of the latent space features were interpretable

I open source two trained autoencoders, here’s

a tutorial for how to use them, and how to interpret a feature

.

And a (very!) rough

training codebase

I give some of the implementation details, and tips for training your own.

I investigate how sparse the decoder weights are in the neuron basis and find that they’re highly distributed, with 4% well explained by a single neuron, 4% well explained by 2 to 10, and the remaining 92% dense. I find this pretty surprising!

Kurtosis shows the neuron basis is still privileged.

I exhibit some case studies of the features I found, like a title case feature, and an "and I" feature
I didn’t find any dead features, but more than half of the features form an ultra-low frequency cluster (frequency less than 1e-4). Surprisingly, I find that this cluster is almost all the same feature (in terms of encoder weights, but not in terms of decoder weights). On one input 95% of these ultra rare features fired!

The same direction forms across random seeds, suggesting it's a true thing about the model and not just an autoencoder artifact
I failed to interpret what this shared direction was
I tried to fix the problem by training an autoencoder to be orthogonal to this direction but it still forms a ultra-low frequency cluster (which all cluster in a new direction)

Should the encoder and decoder be tied? I find that, empirically, the decoder and encoder weights

...
The Shutdown Problem: An AI Engineering Puzzle for Decision Theorists
by EJT — 2023-10-23T21:00:48.398Z
Link: https://philpapers.org/rec/THOTSP-7

[NOTE: This paper was previously titled 'The Shutdown Problem: Three Theorems'.]

This paper is an updated version of the first half of my AI Alignment Awards contest entry. My theorems build on the theorems of Soares, Fallenstein, Yudkowsky, and Armstrong in various ways.[1] These theorems can guide our search for solutions to the shutdown problem.[2]

One aim of the paper is to get academic philosophers and decision theorists interested in the shutdown problem and related topics in AI alignment. They’re my assumed audience. I’m posting here because I think the theorems will also be interesting to people already familiar with the shutdown problem.

For discussion and feedback, I thank Adam Bales, Ryan Carey, Bill D’Alessandro, Tomi Francis, Vera Gahlen, Dan Hendrycks, Cameron Domenico Kirk-Giannini, Jojo Lee, Andreas Mogensen, Sami Petersen, Rio Popper, Brad Saad, Nate Soares, Rhys Southan, Christian Tarsney, Teru Thomas, John Wentworth, Tim L. Williamson, and Keith Wynroe.

Abstract

I explain and motivate the shutdown problem: the problem of designing artificial agents that (1) shut down when a shutdown button is pressed, (2) don’t try to prevent or cause the pressing of the shutdown button, and (3) otherwise pursue goals competently. I prove three theorems that make the difficulty precise. These theorems suggest that agents satisfying some innocuous-seeming conditions will often try to prevent or cause the pressing of the shutdown button, even in cases where it’s costly to do so. I end by noting that these theorems can guide our search for solutions to the problem.

0. Preamble

Tradition has it that decision theory splits into two branches. The descriptive branch concerns how actual agents behave. The normative branch concerns how rational agents behave. But there is also a lesser-known third branch: what we can call ‘constructive decision theory.’ It concerns how we want artificial agents to behave and how we can create artificial agents that behave in those ways. I suggest that this third branch is due for a growth spurt.

I make the case for studying constructive decision theory by explaining a characteristic problem. The shutdown problem (Soares et al. 2015) is the problem of designing artificial agents that (1) shut down when a shutdown button is pressed, (2) don’t try to prevent or cause the pressing of the shutdown button, and (3) otherwise p...
AI Alignment [Incremental Progress Units] this Week (10/22/23)
by Logan Zoellner — 2023-10-23T20:32:37.998Z
Link: https://midwitalignment.substack.com/p/ai-alignment-breakthroughs-this-week-f47

Sorry for getting this one out a day late.

Before getting started, one minor update. I liked David Orr’s suggestion that stars ⭐are too generic. So to emphasize that I am rating the innovativeness of a topic (and not necessarily its quality), I am going to be rating breakthroughs with a lightbulb 💡instead.

As a reminder, a rating of 1 💡means a breakthrough, that while achieving a new state-of-the-art was mostly done through predictable iterative improvements (such as combining know methods or applying an existing method at greater scale). A rating of 5 💡💡💡💡💡 Means a breakthrough that solves a significant, previously assumed to be hard problem on at least on alignment path.

Without Further ado, here are our

AI Alignment Breakthroughs this Week

This week there were breakthroughs in the areas:

Understanding Humans

Benchmarking

AI Agents

Truthful AI

Learning Human Preferences

Math

Mechanistic Interpretability

Augmentation

Making AI Do what we want

AI Art

Understanding Humans

Human Consciousness as a result of Entropy

What it is: Research suggesting human consciousness might be related to the entropy of our internal mental states

What’s new: Researchers used an MRI to measure brain activity during conscious/unconscous states and found conscious states exhibit higher entropy

What’s it Good for: Literally any progress on understanding the hard problem of consciousness would be a massive breakthough with implications for problems such as S-Risk

Rating: 💡💡💡(I want to rate it high because this is such an important problem, but I’m not overly impressed with this particular study. I think IIT would have predicted this)

Beautiful Infographics get more citations

What it is: Research suggesting beautiful infographics are more likely to be believed and cited

What’s new: They manipuated data to be more beautiful and found it was rated higher

What is it good for: Understanding how humans evaluate research is a key step to figuring out how to make us better.

Rating: 💡💡

Benchmarking

GenBench

What is it: a new bench mark measuring how well AI models generalize

What’s new: They identify different categories of “generalization” and measure how AIs do on each of them

What is it good for: Measuring AI capabilities (especially generalization) is critical for plans such as strategic AI pauses.

Rating: 💡

AI IQ test

What it is: Researches find a way to measure the overall “inte...