trentmkelly/LessWrong-43k huggingface dataset:
1. Premise one: Values are malleable
by Nora_Ammann — 2023-10-26T14:36:41.168Z

In this post, I will defend the first premise of the Value Change Problem: that realistic models of human values take them to be malleable, rather than fixed. I call this the ’value malleability’ claim (VM).

I start by clarifying what I mean by value malleability before making a case that human values are in fact malleable. My general approach is to consider empirical observations about the practical reasoning of real-world agents (humans) and consider to what extent these are best explained by an account that posits fixed values vs one that posits value malleability. Concretely, I look at changes over the course of the human developmental lifespan; cases of value uncertainty, underspecification and open-endedness; and finally two existing accounts in the philosophical literature - transformative experiences and aspiration - and consider whether they represent genuine cases of value change. Finally, I consider some further objections.

This is the longest of the three premises of the Value Change Problem, and the discussion is relatively long and philosophical. If you are excited about diving into some of the details, great - read ahead! If instead you’re looking for a minimal summary, the following is what I would say if I could only give one argument in favour of the value malleability claim:

Argument from open-endedness:

Relative to (cognitively) bounded agents (such as humans), the world is open-ended. Humans do not come into this world with a complete representation of the world (e.g., a full hypothesis space). As a result, people occasionally get exposed to situations which they have not considered before and with respect to which they have not yet formed a meaningful evaluative stance. Thus, for a bounded agent to be able to function in such an open-ended world, it must be the case that they sometimes undergo value changes. In other words, the only way an agent could come to adopt an evaluative stance with respect to a genuinely novel situation is one that involves value change.

Clarifying what mean by value change

By claiming that values are malleable, I claim that they can (and typically do) undergo some form of change over, e.g., the course of a person’s life.

But what do I mean by change? To clarify this question, let’s disambiguate three different notions of value change: value replacement, value development and value refine...
0. The Value Change Problem: introduction, overview  and motivations
by Nora_Ammann — 2023-10-26T14:36:15.466Z

Given that AI systems are becoming increasingly capable and widely deployed, we have to think carefully about the multifarious effects their adoption will have on our individual and collective lives. In this sequence, I will focus specifically on how AI can come to substantially affect what it is that we value, as well as the mechanisms by which this may occur.

This sequence introduces the “Value Change Problem” (VCP). I will argue why I think it’s a critical aspect of ethical AI design, and outline what risks we might expect if we fail to properly address it.

This is a shortened and slightly adapted version of academic work I have submitted for publication elsewhere. The original work was written mainly with an academic philosophy audience in mind. If you’re interested in reading the original (longer) version, feel free to reach out.

The core claim of VCP can be summarised as follows:

AI alignment must address the problem of (il)legitimate value change; that is, the problem of making sure AI systems neither cause value change illegitimately, nor forestall legitimate cases of value change in humans and society.

Outline (or: how to read this sequence?)

This case resides on three core premises:

human values are malleable (post 1);
some instances of value change are (un)problematic (post 2);
AI systems are (and will become increasingly) capable of affecting people’s value change trajectories (post 3).

From this, I conclude that we ought to avoid building AI systems that disrespect or exploit the malleability of human values, such as by causing illegitimate value changes or by preventing legitimate ones. I will refer to this as the ‘Value Change Problem’ (VCP). After having established the core case for VCP, the remaining two posts discuss in some more depth the risks that may result from inadequately addressing the VCP.

If you are already on board with each of the premises, you may instead want to directly skip to reading the discussion of risks (posts 4 and 5).

I consider two main categories of risks: (post 4) risks from causing illegitimate value change, as well as (post 5) risks from (illegitimately) preventing legitimate value change. For each of these I want to ask: What is the risk? What are plausible mechanisms by which these risks manifest? What are ways in which these risks manifest already, and what are the ways in which ...
EPUBs of MIRI Blog Archives and selected LW Sequences
by mesaoptimizer — 2023-10-26T14:17:11.538Z
Link: https://git.sr.ht/~mesaoptimizer/epubs

Here is a repository of EPUBs that I've created, consisting of rationalist or alignment content I want to read on my Kindle, such as selections of the MIRI (Blog) Archives and of multiple LessWrong Sequences that I am reading or intend to read.

This post is an announcement that this repository exists[1], and a way for people who search the internet to get EPUBs for LW sequences and MIRI (Blog) archives to find their way to this repository, instead of having to create their own EPUBs.[2]

I expect I will add more such EPUBs to the linked repository as time passes, for any collection of essays I'd prefer an e-book version of[3], both as a way to archive parts of the internet I find valuable, and because I like reading long content on e-ink screens.

My previous post on me creating an EPUB was intended to be a one-off, but then I found myself building a collection. ↩︎

It seems plausible that someone would reason as I did and conclude that going through the MIRI Archives is very high value to gain certain illegible or non-explicitly-distilled context related to MIRI's model of alignment and alignment research, and would prefer doing it using an e-reader enough to go through the trouble of searching for or creating their own e-books. ↩︎

I've created an EPUB of (almost) the entirety of Gwern's essays from https://gwern.net. Sure, the EPUB is outdated within a day of creation when Gwern updates some arbitrary passage in some arbitrary essay, but it feels great to have effectively the entirety of Gwern's evergreen intellectual output in your Kindle. ↩︎
UK Government publishes "Frontier AI: capabilities and risks" Discussion Paper
by A.H. — 2023-10-26T13:55:16.841Z
Link: https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper

Ahead of next week's AI Safety Summit, the UK government has published a discussion paper on the capabilities and risks of AI. The paper comes in three parts and can be found here. See here for the press release announcing the publication of the paper. The paper has been reviewed by a panel of experts including Yoshua Bengio and Paul Christiano.

I haven't read it all, but it looks like like a pretty good primer on AI risk. The first part 'Capabilities and risks from frontier AI: discussion paper' is the main overview and it is followed by two 'Annexes': Annex A 'Future risks of frontier AI' and Annex B 'Safety and security risks of generative artificial intelligence to 2025'. Predictably, there is a lot of discussion of the basics of AI risk and non-catastrophic risks such as labour market disruption/disinformation/bias but catastrophic risk does get a mention, often with the caveat that the subject is 'controversial'.

Here are some quotes I found after a quick skim:

On AI companies racing to the bottom regarding safety:

Individual companies may not be sufficiently incentivised to address all
the potential harms of their systems. In recent years there has been an intense competition between AI developers to build products quickly. Competition on AI has raised concern about potential “race to the bottom” scenarios, where actors compete to rapidly develop AI systems and under-invest in safety measures. In such scenarios, it could be challenging even for AI developers to commit unilaterally to stringent safety standards, lest their commitments put them at a competitive disadvantage. The risks from this “race” dynamic will be exacerbated if it is technologically feasible to maintain or even accelerate the recent rapid
pace of AI progress.

On losing control of AI:

Humans may increasingly hand over control of important decisions to AI systems, due to economic and geopolitical incentives. Some experts are concerned that future advanced AI systems will seek to increase their own influence and reduce human control, with potentially catastrophic consequences - although this is contested.

...

The likelihood of these risks remains controversial, with many experts thinking the likelihood is very low and some arguing a focus on risk distracts from present harms. However, many experts are concerned that losing control of advanced general-purpose AI systems is a real possibility and

...
AI #35: Responsible Scaling Policies
by Zvi — 2023-10-26T13:30:02.439Z

There is much talk about so-called Responsible Scaling Policies, as in what we will do so that what we are doing can be considered responsible. Would that also result in actually responsible scaling? It would help. By themselves, in their current versions, no. The good scenario is that these policies are good starts and lay groundwork and momentum to get where we need to go. The bad scenario is that this becomes safetywashing, used as a justification for rapid and dangerous scaling of frontier models, a label that avoids any actual action or responsibility.

Others think it would be better if we flat out stopped. So they say so. And they protest. And they point out that the public is mostly with them, at the same time that those trying to play as Very Serious People say such talk is irresponsible.

Future persuasion will be better. Sam Altman predicts superhuman persuasion ability prior to superhuman general intelligence. What would that mean? People think they would not be fooled by such tactics. Obviously, they are mistaken.

As usual, lots of other stuff as well.

On the not-explicitly-about-AI front, I would encourage you if you haven’t yet to check out my review of Going Infinite. Many are calling it both fascinating and a rip-roaring good time, some even calling it my best work. I hope you enjoy reading it as much as I enjoyed writing it.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. North Korean cyberattacks. Great.

Language Models Don’t Offer Mundane Utility. The sorcerer’s ASCII cats.

GPT-4 Real This Time. Is that Barack Obama? You know. For kids.

A Proposed Bet. Will hallucinations be solved within months? Market says 17%.

Fun With Image Generation. I bet she doesn’t drink coffee, she’s too chill.

Deepfaketown and Botpocalypse Soon. Slovakian election deepfake. Oh no?

They Took Our Jobs. Jon Stewart figures out what The Problem is.

Get Involved. Jobs at NYU and LTFF. Panels at the CPDP.ai conference.

Introducing. Eureka for superhuman robot dexterity. This is fine.

In Other AI News. Baidu says Ernie 4.0 is as good as GPT-4, won’t allow access.

Quiet Speculations. Will we have your attention?

The Quest for Sane Regulation. What do we want? Why aren’t you mentioning it?

The Week in Audio. PM Sunak, Ian Morris, a Ted conference.

Rhetorical Innovation. Stop it.

...
RA Bounty: Looking for feedback on screenplay about AI Risk
by Writer — 2023-10-26T13:23:02.806Z

Jamie Wahls and Arthur Frost are writing a screenplay for Rational Animations. It's a sci-fi episodic comedy illustrating various ways in which AI goes off the rails in more or less catastrophic ways.

We're looking for feedback and offering bounties. This is the current draft: https://docs.google.com/document/d/1iFZJ8ytS-NAnaoz2UU_QUanAjgdB3SmtvhOt7jiDLeY/edit?usp=sharing
We're offering:
500 USD if you offer feedback that causes us to rework the story significantly. In this category are changes that would make us rewrite at least two episodes from scratch.

100 USD for changes that make us improve the credibility or entertainment value of the story. In this category, there are changes that make us rewrite one episode or less. There are also changes that would significantly improve the credibility of the story, even if they don't require us to make significant changes or any changes at all. Some insights might impact future episodes but not the current ones if there's anything that's still underspecified, and I'd still like to reward them.

25 USD for any other minor changes we implement due to a piece of feedback. Only grammar doesn't count.

I recognize these conditions are pretty vague. I will err on the side of paying too much as I've been doing on Bountied Rationality.
Sensor Exposure can Compromise the Human Brain in the 2020s
by trevor — 2023-10-26T03:31:09.835Z

Overview

The 20th century was radically altered by the discovery of psychology, a science of the human mind, and its exploitation (e.g. large-scale warfare, propaganda, advertising, information/hybrid warfare, decision theory/mutually assured destruction).

However, it's reasonable to think that the 20th century would have been even further transformed if the science and exploitation of the human mind was even further advanced than it already was.

I'm arguing here that, in an era of mass surveillance,

hybrid

/cognitive warfare between the US and China and Russia, and substantial ML advancements, it is also reasonable to think that the situation with SOTA human cognitive analysis and exploitation may already be threatening the continuity of operations of the entire AI safety community; and if not now, then likely at some point during the 2020s, which will probably be much more globally eventful than the pace that humanity became accustomed to in the previous two decades.

AI will be the keys to those kingdoms, and the wars between them. If demanding a development pause might be the minimum ask for humanity to survive, and for a conflict like that, we won't even know what hit us.

The attack surface is unacceptably large for human life in general, let alone for the AI safety community, a community of nerds who chanced upon the engineering problem that the fate of this side of the universe revolves around, and a community that absolutely must not fail to survive the 2020s, nor to limp on in a diminished/captured form.

This problem is fundamental to intelligent civilizations

If there were intelligent aliens, made of bundles of tentacles or crystals or plants that think incredibly slowly, their minds would also have discoverable exploits/zero days, because any mind that evolved naturally would probably be like the human brain, a kludge of spaghetti code that is operating outside of its intended environment.

They would probably not even begin to scratch the surface of finding and labeling those exploits, until, like human civilization today, they began surrounding thousands or millions of their kind with sensors that could record behavior several hours a day and find webs of correlations.

In the case of humans, the use of social media as a controlled environment for automated AI-powered experimentation appears to be what created th...
Notes on "How do we become confident in the safety of a machine learning system?"
by RohanS — 2023-10-26T03:13:56.024Z

I'm trying out taking notes on an AI safety reading in a LessWrong draft post and sharing the result as a low-effort "distillation." I'm hoping this is a quick way to force myself to take better notes, and to provide distillations that are useful to others. This reading is "How do we become confident in the safety of a machine learning system?" by Evan Hubinger.

I wasn't sure at the outset how far into tangential thoughts of my own I would go. I think by writing notes with the thought that other people might read them in mind, I ended up a) following the structure of the original post more rigidly than usual and b) going on fewer tangents than usual. I think that probably isn't ideal for me; if I do this again, I might try to actively not follow those impulses.

Anyway, here are the notes. Each section title here links to the corresponding section in the original post.

Summary:

This post suggests that we use training stories to flesh out and evaluate proposals for training safe ML systems. A proposal is a training setup (loosely understood), and a training story consists of a training goal and a training rationale. A training goal describes what behavior you hope the training setup will give rise to, in some mechanistic detail; it includes both how you think the model will perform the relevant task capably and why you think performing it in that way will be safe. A training rationale describes why you think the training setup will result in a model that implements the mechanistic behavior described in the training goal. While training stories do not capture all possible ways to make AI systems safer, they are quite a general framework that can help us better systematically organize and evaluate our proposals. This framework can also help us generate proposals if there are clear gaps in the space of possible training stories. The post includes quite a few examples of proposals, training goals, training rationales, and evaluations of training stories.

How do we become confident in the safety of a machine learning system?

An overview of 11 proposals for building safe advanced AI uses the criteria of outer alignment, inner alignment, training competitiveness, and performance competitiveness to evaluate proposals for safe AGI
These are good for posing open questions, but not for systematically helping us understand what assumptions need to hold for a proposal to work

Plus

...
Apply to the Constellation Visiting Researcher Program and Astra Fellowship, in Berkeley this Winter
by Nate Thomas — 2023-10-26T03:07:34.118Z

This is a link post for two AI safety programs we’ve just opened applications for:

https://www.constellation.org/programs/astra-fellowship

and

https://www.constellation.org/programs/researcher-program

Constellation

is a research center dedicated to safely navigating the development of transformative AI. We’ve previously helped run

the ML for Alignment Bootcamp (MLAB) series

and Redwood’s

month-long research program on model internals (REMIX)

in addition to a variety of other field-building programs & events.[1]

This winter, we are running two programs aimed at growing and supporting the ecosystem of people working on AI safety:

The Constellation Visiting Researcher Program

provides an opportunity for around 20 researchers to connect with leading AI safety researchers, exchange ideas, and find collaborators while continuing their research from our offices in Berkeley, CA. The funded program will take place this winter from the 8th of January 2024 to the 1st of March 2024.

The Astra Fellowship

provides an opportunity for around 20 people to conduct research in AI safety with experienced advisors. Fellows will be based out of the Constellation office, allowing them to connect and exchange ideas with leading AI safety researchers. The program will take place in Berkeley, CA between January 8 and April 1, 2024.

Applications for both are due November 10, 11:59pm anywhere on Earth. You can apply to the Astra Fellowship

here

and the Visiting Researcher Program

here

. If you are unsure about your fit, please err on the side of applying. We especially encourage women and underrepresented minorities to apply. You can refer others who you think might be a good fit through

this form

.

Logistics: Housing and travel expenses are covered for both programs, and Astra fellows will receive an additional monetary stipend. The start and end dates for both programs are flexible.

Questions? Email

programs@constellation.org

or ask them below.

^

Over 15 participants from these past programs are now working on AI safety at Anthropic, ARC Evals, ARC Theory, Google DeepMind, OpenAI, Open Philanthropy, and Redwood Research.
CHAI internship applications are open (due Nov 13)
by Erik Jenner — 2023-10-26T00:53:49.640Z

CHAI internship applications have just opened, apply here by Nov 13th! The internship might be a good fit if you want to get research experience in technical AI safety. You'll be mentored by a CHAI PhD student or postdoc and work on your own project for 3-4 months.

Researchers at CHAI are interested in many different AI safety topics; a few examples are reward learning, adversarial robustness of LLMs, and interpretability. (I mention this because it might not be obvious from some of the language and links on the CHAI website.)

I've copied the full announcement below:

Our internships require a background in mathematics and computer science. Existing research experience in machine learning is strongly advantageous but not required. We are interested in people who can demonstrate technical excellence and wish to transition to technical AI safety research. Examples include undergraduate or Master's students in computer science or adjacent fields, PhD students/researchers, professional software or ML engineers, etc.

This internship is designed for individuals who are interested in technical AI safety research. All applicants should take a look at our papers (here and here) before applying to understand CHAI's research.

General Information

Location: In-person (at UC Berkeley) is preferred but remote is possible.
Deadline: November 13th, 2023
Start Date: Flexible
Duration: Internships are typically 12 to 16 weeks
Compensation: $3,500 per month for remote interns. $5,000 per month for in-person interns.
International Applicants: We accept international applicants
Requirements:

Cover Letter or Research Proposal (choose one and see instructions below)
Resume
Academic Transcript

Cover Letter or Research Proposal (Choose One)

The primary purpose of the Cover Letter or Research Proposal is for us to match you to a project that interests you.

Most of our interns are generally interested in technical AI safety research but do not have a specific project in mind when they start the internship. Throughout the interview process, we learn more about each intern's interests and match them with a mentor who has an existing project idea that fits the intern's skills and interests. If you do not have a particular project in mind, then we ask you to please write a Cover Letter answering the following questions:

Why do you want to work at CHAI as opposed to other research labs?
What ar

...