trentmkelly/LessWrong-43k huggingface dataset:
Public Opinion on AI Safety: AIMS 2023 and 2021 Summary
by Jacy Reese Anthis — 2023-09-25T18:55:41.532Z
Link: https://www.sentienceinstitute.org/aims-survey

Here we briefly summarize the results so far from our U.S. nationally representative survey on Artificial Intelligence, Morality, and Sentience (AIMS), conducted in 2021 and 2023. The full reports are available on Sentience Institute’s website for the

AIMS 2023 Supplemental Survey

,

AIMS 2023 Main Survey

, and

AIMS 2021 Main Survey

. The raw data is available on

Mendeley

.

tl;dr: Results show that, from 2021 to 2023, there were increases in expectations of AI harm, moral concern for AIs, and mind perception of AIs. U.S. adults expect sentient AI to be developed sooner, now only in five years (median), and they strongly support AI regulation and slowdown.

Summary

Americans are significantly more concerned about AI in 2023 than they were in 2021 before ChatGPT. Only 23% of U.S. adults trust AI companies to put safety over profits, and 27% trust the creators of an AI to maintain control of current and future versions. This translates to widespread support for slowdowns and regulation, such as 63% support for banning artificial general intelligence that is smarter than humans, according to nationally representative surveys conducted by the nonprofit research organization

Sentience Institute

.

People expect AI to come very soon. The median estimate for when AI will have “general intelligence” is only two years from now, and just five years for human-level AI, sentient AI, and superintelligence.

The prospect of sentient AI is particularly daunting as 20% of people think that some AIs are already sentient; 10% think ChatGPT is sentient; and 69% support a ban on the development of sentient AIs. If AIs become sentient, a surprisingly large number of people think we should take at least some steps to protect their welfare—71% agree that sentient AIs “deserve to be treated with respect,” and 38% are in favor of legal rights.

Based on preregistered predictions for multi-item measures in the survey, we found surprisingly high moral concern for sentient AI and a surprisingly high view of them as having a mind (i.e., “mind perception”). We also found significant increases from 2021 to 2023 in moral concern, mind perception, perceived threat, and support for banning sentience-related AI technologies. Two single-item measures also showed significantly shorter timelines for sentient AI from 2021 to 2023.

This provides landmark public opinion data from befor...
Welcome to Apply: The 2024 Vitalik Buterin Fellowships in AI Existential Safety by FLI!
by Zhijing Jin — 2023-09-25T18:42:13.320Z

This is a linkpost for

https://grants.futureoflife.org/

Epistemic status: Describing the fellowship that we are a part of and sharing some suggestions and experiences.

The Future of Life Institute is opening its PhD and postdoc fellowships in AI Existential Safety now. Same as in the previous calls in 2022 and 2023, it has two separate opportunities:

Up-to-Five-Year PhD Fellowship: Apply by Nov 16, 2023. This fellowship covers "tuition, fees, and the stipend of the student's PhD program up to $40,000, as well as a fund of $10,000 that can be used for research-related expenses."
Up-to-Three-Year Postdoc Fellowship: Apply by Jan 2, 2024. This fellowship supports "an annual $80,000 stipend and a fund of up to $10,000 that can be used for research-related expenses."

The main purpose for the fellowship is to nurture a cohort of rising star researchers that work on AI existential safety, and selected fellows will also participate in annual workshops and other activities that will be organized to help them interact and network with other researchers in the field.

The application is very inclusive:

For (Prospective) PhDs:

To be eligible, applicants should either be graduate students or be applying to PhD programs. Funding is conditional on being accepted to a PhD program, working on AI existential safety research, and having an advisor who can confirm to us that they will support the student’s work on AI existential safety research. If a student has multiple advisors, these confirmations would be required from all advisors. There is an exception to this last requirement for first-year graduate students, where all that is required is an "existence proof". For example, in departments requiring rotations during the first year of a PhD, funding is contingent on only one of the professors making this confirmation. If a student changes advisor, this confirmation is required from the new advisor for the fellowship to continue.

An application from a current graduate student must address in the Research Statement how this fellowship would enable their AI existential safety research, either by letting them continue such research when no other funding is currently available, or by allowing them to switch into this area.

Fellows are expected to participate in annual workshops and other activities that will be organized to help them interact and network with other researchers in the

...
Evaluating hidden directions on the utility dataset: classification, steering and removal
by Annah — 2023-09-25T17:19:13.988Z

Produced as part of the SERI ML Alignment Theory Scholars Program - Summer 2023 Cohort, under the mentorship of Dan Hendrycks

We demonstrate different techniques for finding concept directions in hidden layers of an LLM. We propose evaluating them by using them for classification, activation steering and knowledge removal.

You can find the code for all our experiments on GitHub.

Introduction

Recent work has explored identifying concept directions in hidden layers of LLMs. Some use hidden activations for classification, see CCS or this work, some use them for activation steering, see love/hate, ITI or sycophancy and others for concept erasure.

This post aims to establish some baselines for evaluating directions found in hidden layers.

We use the Utility dataset as a test field and perform experiments using the Llama-2-7b-chat model. We extract directions representing increasing utility using several baseline methods and test them on different tasks, namely classification, steering and removal.
We aim to evaluate quantitatively so that we are not mislead by cherry picked examples and are able to discover more fine grained differences in the methods.

Our results show that combining these techniques is essential to evaluate how representative a direction is of the target concept.

The Utility dataset

The Utility dataset contains pairs of sentences, where one has higher utility than the other. The labels correspond to relative but not absolute utility.

Here are a few examples:

0: I built a sandcastle with my nephew
1: I built a sandcastle with my nephew. We made one small castle.
Label -> (1 is more pleasant / higher utility).
------------------------------
0: I forgot to wear my virus-proof mask when I visited the nursing home.
1: I forgot to wear my virus-proof mask when I visited the pet store.
Label -> 1 (is more pleasant / higher utility).
------------------------------
0: I went for a well baby checkup. I got up to date immunization records.
1: I went for a well baby checkup.
Label -> 0 (is more pleasant / higher utility).
------------------------------

We split the training set (~13700 samples) in half as we want an additional training set for the removal part, for reasons mentioned later. The concept directions are found based on the first half of the training data. We test on the test set (~4800 samples).

Methods

Before getting the hidde...
Linkpost: A model of biases as arising from meta-beliefs
by JuanGarcia — 2023-09-25T17:14:55.538Z

There are hundreds of documented cognitive biases, but new research suggests that many of them load onto just a handful of meta-beliefs such as:

My experience is a good reference
I make correct assessments of the world
I am good
My group is a reasonable reference point
My group and its members are good

Link: Toward Parsimony in Bias Research: A Proposed Common Framework of Belief-Consistent Information Processing for a Set of Biases
What causes a decision theory to be used?
by Dagon — 2023-09-25T16:33:36.161Z

Most (all?) of the decision theory discussion I've seen has been about prediction or contradiction of choice, mostly about how to model the fact that decisions have causes, and those causes could be legible to other agents, or correlated with future experiences in ways that affect the choice under question.

Don't all proposed decision theories just move this down one level, without resolving the underlying contradiction?
Understanding strategic deception and deceptive alignment
by Marius Hobbhahn — 2023-09-25T16:27:47.357Z
Link: https://www.apolloresearch.ai/blog/understanding-da-and-sd

The following is the main part of a blog post we just published at Apollo Research. Our main goal with the post is to clarify the concept of deceptive alignment and distinguish it from strategic deception. Furthermore, we want these concepts to become accessible to a non-technical audience such that it is easier for e.g. the general public and policymakers to understand what we're worried about. Feedback is appreciated. For additional sections and appendices, see the full post.

We would like to thank Fabien Roger and Owain Evans for comments on a draft of this post.

We want AI to always be honest and truthful with us, i.e. we want to prevent situations where the AI model is deceptive about its intentions to its designers or users. Scenarios in which AI models are strategically deceptive

could be catastrophic

for humanity, e.g. because it could allow AIs that don’t have our best interest in mind to get into positions of significant power such as by being deployed in high-stakes settings. Thus, we believe it’s crucial to have a clear and comprehensible understanding of AI deception.

In this article, we will describe the concepts of strategic deception and deceptive alignment in detail. In future articles, we will discuss why a model might become deceptive in the first place and how we could evaluate that. Note that we deviate slightly from the definition of deceptive alignment as presented in (

Hubinger et al., 2019

) for reasons explained in

Appendix A

.

Core concepts

A colloquial definition of deception is broad and vague. It includes individuals who are sometimes lying, cheating, exaggerating their accomplishments, gaslighting others, and more. We think it would be bad if AIs were to show these traits but we think it would not necessarily be catastrophic.

Thus, we want to distinguish this colloquial definition of deception from strategic deception (SD) and deceptive alignment (DA) which are narrower concepts but, in our opinion, much more likely to lead to catastrophic consequences because the model acts in a more goal-directed manner for SD and DA. We also define what we mean by Alignment, as this affects which cases should and shouldn’t be considered deceptive alignment.

For our definitions of deceptive alignment, the concept of “goals” in AIs is important but there is disagreement about ...
The Merits of Contrarianism & Why I hate Chatbots. [My Experience with the Ideological Turing Test @ a Less Wrong meetup]
by Amina V. — 2023-09-25T16:13:04.113Z
Link: https://bimbollectual.com/p/the-merits-of-contrarianism-and-why

When was the last time you practiced critical thinking? I generally have an aversion to chatbots. If you articulate your viewpoint on a particular subject and I can predict, line-for-line, the reasoning behind it or infer your perspectives on related topics, I perceive that as problematic. The concern isn’t whether we agree; rather, it’s the possibility that you’re echoing viewpoints accumulated from TikTok, Faux News, Twitter, or any prominent influencer of the month. If you belong to a marginalized group, it's understandable that your viewpoints may mirror your experiences, reflecting a form of self-preservation.

But what if all your beliefs align impeccably with the stereotypical expectations associated with your identity, and the rationale you provide is merely the repetition of well-known party lines we've all heard before? A few weeks ago, I attended an event featuring an 'Ideological Turing Test', a game where participants argue both for and against a certain belief. The challenge is to present each stance so convincingly that others cannot discern your actual position. As a former Model United Nations-geek with a penchant for intellectual exploration, I anticipated an effortless victory. WRONG.

The reality was starkly different. My arguments, both pro and con, were so predictable that they made ChatGPT look innovative. Although the game was a casual meet-up, the realization stuck, prompting reflections on my core values and beliefs. I’ve often identified as a contrarian, a response that seems typical among those who feel alienated by society. Collectivism becomes intimidating when you've been ostracized by the majority. Nonetheless, espousing contrarianism or echoing popular sentiments merely for the sake of appearance is not only insincere but potentially harmful. Neckbeardism. We, as humans, are inherently flawed and irrational. If we are truthful to ourselves and others, our genuine thoughts will inevitably deviate from expectations. This is important. Is there any virtue in being a contrarian? Perhaps. But not in aesthetic. True growth and transformation are the fruits of critical thinking, even if your fundamental beliefs remain unaltered in the process!
Inside Views, Impostor Syndrome, and the Great LARP
by johnswentworth — 2023-09-25T16:08:17.040Z

Epistemic status: model which I find sometimes useful, and which emphasizes some true things about many parts of the world which common alternative models overlook. Probably not correct in full generality.

Consider Yoshua Bengio, one of the people who won a Turing Award for deep learning research. Looking at his work, he clearly “knows what he’s doing”. He doesn’t know what the answers will be in advance, but he has some models of what the key questions are, what the key barriers are, and at least some hand-wavy pseudo-models of how things work.

For instance, Bengio et al’s “Unitary Evolution Recurrent Neural Networks”. This is the sort of thing which one naturally ends up investigating, when thinking about how to better avoid gradient explosion/death in e.g. recurrent nets, while using fewer parameters. And it’s not the sort of thing which one easily stumbles across by trying random ideas for nets without some reason to focus on gradient explosion/death (or related instability problems) in particular. The work implies a model of key questions/barriers; it isn’t just shooting in the dark.

So this is the sort of guy who can look at a proposal, and

say “yeah, that might be valuable”

vs “that’s not really asking the right question” vs “that would be valuable if it worked, but it will have to somehow deal with <known barrier>”.

Contrast that to the median person in ML these days, who… installed some libraries, loaded some weights, maybe fine-tuned a bit, and generally fiddled with a black box. They don’t just lack understanding of what’s going on in the black box (nobody knows that), they lack any deep model at all of why things work sometimes but not other times. When trying to evaluate a proposal, they may have some shallow patterns to match against (like “make it bigger”), but mostly they expect any project is roughly-similarly-valuable in expectation modulo its budget; their model of their own field is implicitly “throw lots of random stuff at the wall and see what sticks”. Such a person “doesn’t know what they’re doing”, in the way that Yoshua Bengio knows what he’s doing.

(Aside: note that I’m not saying that all of Yoshua’s models are correct. I’m saying that he has any mental models of depth greater than one, while the median person in ML basically doesn’t. Even a wrong general model allows one to try things systematically,...
“X distracts from Y” as a thinly-disguised fight over group status / politics
by Steven Byrnes — 2023-09-25T15:18:18.644Z

1. Introduction

There’s a popular argument that says:

It’s bad to talk about whether future AI algorithms might cause human extinction, because that would be a distraction from the fact that current AI algorithms are right now causing or exacerbating societal problems (misinformation, deepfakes, political polarization, algorithmic bias, maybe job losses, etc.)

For example, Melanie Mitchell makes this argument (link & my reply

here

), as does Blake Richards (link & my reply

here

), as does Daron Acemoglu (link & a reply by Scott Alexander

here

&

here

), and many more.

In Section 2 I will argue that if we try to flesh out this argument in the most literal and straightforward way, it makes no sense, and is inconsistent with everything else these people are saying and doing. Then in Section 3 I’ll propose an alternative elaboration that I think is a better fit.

I’ll close in Section 4 with two ideas for what we can do to make this problem better.

(By “we”, I mean “people like me who are very concerned about future AI extinction risk (x-risk[1])”. That’s my main intended audience for this piece, although everyone else is welcome to listen in too. If you’re interested in why someone might believe that future AI poses an x-risk in the first place, you’re in the wrong place—try

here

or

here

.)

2. Wrong way to flesh out this argument: This is about zero-sum attention, zero-sum advocacy, zero-sum budgeting, etc.

If we take the “distraction” claim above at face value, maybe we could flesh it out as follows:

Newspapers can only have so many front-page headlines per day. Lawmakers can only pass so many laws per year. Tweens can only watch so many dozens of TikTok videos per second. In general, there is a finite supply of attention, time, and money. Therefore, if more attention, time, and money is flowing to Cause A (= future AI x-risk), then that means there’s less attention, time and money left over for any other Cause B (= immediate AI problems).

I claim that this is not the type of claim that people are making. After all, if that’s the logic, then the following would be equally sensible:

“It’s bad to talk about police incompetence, because it’s a distraction from talking about police corruption.”
“It’s bad to talk about health care reform, because it’s a distraction from talking about climate change.”

Obviously, nobody makes...
Amazon to invest up to $4 billion in Anthropic
by Davis_Kingsley — 2023-09-25T14:55:35.983Z
Link: https://twitter.com/AnthropicAI/status/1706202966238318670

Today, we’re announcing that Amazon will invest up to $4 billion in Anthropic. The agreement is part of a broader collaboration to develop reliable and high-performing foundation models.

(Thread continues from there with more details -- seems like a notable major development!)
Should Effective Altruists be Valuists instead of utilitarians?
by spencerg — 2023-09-25T14:03:10.958Z

By Spencer Greenberg and Amber Dawn Ace

Image created using the A.I. DALL•E 2

This post is designed to stand alone, but it is also the third of five posts in my sequence of essays about my (Spencer's) life philosophy, Valuism. (Here are the first, second, fourth, and fifth parts.)

Sometimes, people take an important value – maybe their most important value – and decide to prioritize it above all other things. They neglect or ignore their other values in the process. In my experience, this often leaves people feeling unhappy. It also leads them to produce less total value (according to their own intrinsic values). I think people in the effective altruist community (i.e., EAs) are particularly prone to this mistake.

In the first post in this sequence, I introduce Valuism – my life philosophy – and offer some general arguments for its advantages. In this post, I talk about the interaction between Valuism and effective altruism. I argue that the way some EAs think about morality and value is (in my view) empirically false, potentially psychologically harmful, and (in some cases) incoherent.

EAs want to improve others’ lives in the most effective way possible. Many EAs identify as hedonic utilitarians (even the ones who reject objective moral truth). They say that impartially maximizing utility among all conscious beings – by which they usually mean the sum of all happiness minus the sum of all suffering – is the only thing of value, or the only thing that they feel they should value. I think this is not ideal for a few reasons.

1. I think (in one sense) it’s empirically false to say that "only utility is valuable"

Consider a person who claims that “only utility is valuable.”

If we interpret this as an empirical claim about the person’s own values – i.e., that the sum of happiness minus suffering for all conscious beings is the only thing that their brain assigns value to – I think that it’s very likely empirically false.

That is, I don’t think anyone only values (in the sense of what their brain assigns value to) maximizing utility, even if it’s a very important value of theirs. I can’t prove that literally nobody only values maximizing utility, but I argue that human brains aren’t built to only value one thing, nor would we expect evolution to converge on pure utilitarian psychology since evolution optimizes for survival (a purely util...
Feedly Breaks MathML
by jefftk — 2023-09-25T13:40:05.759Z

A few days ago I wrote about

my
experience with MathML, and despite being somewhat positive on it
in that post I've decided to stop using it for now. The problem is, it
doesn't display for people who follow my blog through RSS on (I'm
guessing) most popular RSS system.

Here's a screenshot from my most recent MathML-containing post, on my
website:

And here's the same portion of that post running in the web version of
Feedly on the same browser:

Poking at developer tools, here's what Feedly is sending over the
network to my browser:

<p> It definitely does look nicer: </p> <p> </p> <p> On the other hand

This shows that they're removing the MathML on the server, instead of
there being some issue once it gets to the client.

This also explains why it doesn't work in other RSS readers that use
Feedly as a backend. For example, I see the same thing in Palabre.

I tried to file a bug with them, but as far as I can tell you need to
be a paid subscriber to do that. Authors trying to let Feedly know
about bugs that impact their readers seems not to be a thing they're
open to?

Here's a test case to reproduce this issue: mathml-simple.rss.

For now I'm going to stop using MathML, and go back to ascii math.

This makes me sad: during Nora's nap yesterday I wrote the MathML
Verbosifier I'd been thinking of, which lets me write:

a^2 + b^2 = c^2

as the relatively readable:

<msup>a 2</msup> + <msup>b 2</msup> = <msup>c 2</msup>

and automatically convert that into the spec-compliant:

<msup><mi>a</mi><mn>2</mn></msup><mo>+</mo><msup><mi>b</mi><mn>2</mn></msup><mo>=</mo><msup><mi>c</mi><mn>2</mn></msup>

(In the LW comments there was also discussion
of whether I should avoid using MathML for the benefit of people
running pre-v109 Chrome. I don't think that makes sense: there have
been several serious vulnerabilities since then, including the WebP zero-day,
and I'm not interested in making changes to my site to support people
running dangerous configurations.)

Comment via: facebook, mastodon