trentmkelly/LessWrong-43k huggingface dataset:
Ethicophysics I
by MadHatter — 2023-11-27T15:44:29.236Z
Link: https://open.substack.com/pub/bittertruths/p/ethicophysics-i?r=1274q7&utm_campaign=post&utm_medium=web

What are Good and Evil? How do we explain these concepts to acomputer sufficiently well that we can be assured that the computer will understand them in the same sense as humans understand them? These are hard questions, and people have often despaired of fnding any answersto the AI safety problem.

In this paper, we lay out a theory of ethics modeled on the laws of physics. The theory has two key advantages: it squares nicely with most human moral intuitions, and it is amenable to rather straightforward computations that a computer could easily perform if told to. It therefore forms an ideal foundation for solving the AI safety problem.
Sentience Institute 2023 End of Year Summary
by michael_dello — 2023-11-27T12:11:37.228Z
Link: https://www.sentienceinstitute.org/blog/eoy2023

Summary

Last year’s

2022 End of Year Summary

was published just five days before the launch of ChatGPT. This event surprised many in the field, and the global spotlight has illustrated how important it is to understand human-AI interaction. Our priority continues to be researching the rise of

digital minds

: AIs that have or are perceived as having mental faculties, such as reasoning, agency, experience, and

sentience

. We hope to answer questions such as: What will be the next ‘ChatGPT moment’ in which humanity’s relationship with AI rapidly changes? How can humans and AIs interact in ways that are beneficial rather than destructive?

The highlight from our 2023 research is the

Artificial Intelligence, Morality, and Sentience (AIMS)

survey with the same questions as in 2021, so we can compare public opinion from before to after recent events, as well as an additional

AIMS 2023 Supplement on Public Opinion in AI Safety

. Data like this is typically collected once a social issue is fully in the public spotlight, so we’re eager to have this longitudinal tracking already in place before the rise of digital minds. In terms of outreach, we released two

podcast

episodes (and one in December 2022) and hosted an intergroup call on digital minds as well as a

presentation by Yochanan Bigman

.

Our ongoing work includes online experiments on how AI autonomy and sentience affect perceptions, developing a scale to measure substratism, and qualitative studies (e.g., interviews) of how people understand and interact with cutting-edge AI systems. We hope to raise $150,000 this giving season to continue our research on digital minds. We also have a few ongoing projects to address factory farming (our focus before 2021) funded with earmarked donations from donors, such as ongoing data collection for the 2023 iteration of our

Animals, Food, and Technology (AFT) survey

.

As always, we are extremely grateful to our supporters who share our vision and make this work possible. If you are able to in 2023, please consider

making a donation

.

Accomplishments

Research

We recently released the 2023 results of our

Artificial Intelligence, Morality, and Sentience (AIMS) survey

. The

Main

and

Supplemental

2023 survey results, as well as the

2021 Main

survey results, are available in full on our website. Americans express sign

...
A Question about Corrigibility (2015)
by A.H. — 2023-11-27T12:05:51.659Z

Recently, I read Corrigibility by Soares et al. and became confused. I followed most of the mathematical reasoning but am now struggling to understand what the point or end goal of this avenue of research is meant to be. I know that MIRI now pursues a different research direction so maybe some or all of what I write here is moot.

Doesn't this paragraph undermine the whole project?

In Section 2.1, a 'manipulative action' a−1 is considered. This action results in a lower utility according to UN (the 'trial' utility function which the agent should maximize unless the shutdown button is pressed) but lowers the probability that the shutdown button is pressed. It makes sense that such an action could be manipulative. But the authors then clarify the following:

We remark that [the mathematical condition satisfied by a−1] is not by any means a necessary or sufficient description of manipulative actions preventing [pressing of the button]. Some actions satisfying [these conditions] might be non-manipulative (for example, if the agent expends extra effort ensuring that anew subagent is corrigible, and this behavior helps convince the programmers that the agent is indeed corrigible). Furthermore, many manipulative actions may not have this form
(as we will see in Section 4.2). Rather, we consider one manipulative action a−1 which happens to be both manipulative and have this form, and show that if chigh is set to be too low, then U will incentivize U-agents to prefer this a−1 to the
default action a∗1

Ok, cool. But if we are using a framework where the manipulative action has exactly the same mathematical representation as a non-manipulative action, isn't this an indication that our whole framework is wrong? Surely we need to distinguish between these two types of action? Doesn't this paragraph show precisely that corrigibility can't be accurately captured by this model using utility functions and actions?

Suppose I designed a utility function and proved, in this framework that it never took deceptive actions like a−1, then this proof would also show that my utility function doesn't allow for the kind of helpful, non-manipulative actions described in the quoted paragraph above.

Conversely, if I did create a real-life instantiation of a corrigible agent, it would distinguish between manipulative and no...
Appendices to the live agendas
by technicalities — 2023-11-27T11:10:32.187Z

Lists cut from our main post, in a token gesture toward readability.

We list past reviews of alignment work, ideas which seem to be dead, the cool but neglected neuroscience / biology approach, various orgs which don't seem to have any agenda, and a bunch of things which don't fit elsewhere.

Appendix: Prior enumerations

Everitt et al (2018)

Ji (2023)

Soares (2023)

Gleave and MacLean (2023)

Krakovna and Shah on Deepmind (2023)

AI Plans (2023)

, mostly irrelevant

Macdermott (2023)

Kirchner et al (2022): unsupervised analysis

Krakovna (2022)

, Paradigms of AI alignment

Hubinger (2020)

Perret (2020)

Nanda (2021)

Larsen (2022)

Zoellner (2022)

McDougall (2022)

Sharkey et al (2022)

on interp

Koch (2020)

Critch (2020)

Hubinger on types of interpretability

(2022)

Tai_safety_bibliography (2021)

Akash and Larsen (2022)

Karnofsky prosaic plan

(2022)

Steinhardt (2019)

Russell (2016)

FLI (2017)

Shah (2020)

things which claim to be agendas

Tekofsky listing indies (2023)

This thing called boundaries (2023)

Sharkey et al (2022), mech interp

FLI Governance scorecard

The money

Appendix: Graveyard

Ambitious value learning?
MIRI youngbloods (see Hebbar)

JW selection theorems

??

Provably Beneficial Artificial Intelligence

(but see Open Agency and Omohundro)

HCH

(see QACI)
IDA → Critiques and recursive reward modelling
Debate is now called Critiques and ERO

Market-making

(Hubinger)
Logical inductors

Conditioning Predictive Models: Risks and Strategies

?

Impact measures

, conservative agency, side effects → “power aversion”

Acceptability Verification

Quantilizers

Redwood

interp?

AI Safety Hub

Enabling Robots to Communicate their Objectives

(early stage interp?)

Aligning narrowly superhuman models

(Cotra idea;

tiny followup

; lives on as scalable oversight?)
automation of semantic interpretability

i.e. automatically proposing hypotheses instead of just automatically verifying them

Oracle AI

is not dead so much as ~everything LLM falls under its purview

Tool AI

, similarly, ~is LLMs plugged into a particular interface

EleutherAI: #accelerating-alignment

- AI alignment assistants are live, but it doesn’t seem like EleutherAI is currently working on this
Algorithm Distillation Interpretability (Levoso)

Appendix: Biology for AI alignment

Lots of agendas but not clear if anyone besides Byrnes and Thiergart are actively turning the crank. Seems like it would need a billion dollars.

Human enhancement

One-

...
Shallow review of live agendas in alignment & safety
by technicalities — 2023-11-27T11:10:27.464Z

Summary

You can’t optimise an allocation of resources if you don’t know what the current one is. Existing maps of alignment research are mostly too old to guide you and the field has nearly no ratchet, no common knowledge of what everyone is doing and why, what is abandoned and why, what is renamed, what relates to what, what is going on.

This post is mostly just a big index: a link-dump for as many currently active AI safety agendas as we could find. But even a linkdump is plenty subjective. It maps work to conceptual clusters 1-1, aiming to answer questions like “I wonder what happened to the exciting idea I heard about at that one conference” and “I just read a post on a surprising new insight and want to see who else has been working on this”, “I wonder roughly how many people are working on that thing”.

This doc is unreadably long, so that it can be Ctrl-F-ed. Also this way you can fork the list and make a smaller one.

Our taxonomy:

Understand existing models (evals, interpretability, science of DL)
Control the thing (prevent deception, model edits, value learning, goal robustness)
Make AI solve it (scalable oversight, cyborgism, etc)
Theory (galaxy-brained end-to-end, agency, corrigibility, ontology, cooperation)

Please point out if we mistakenly round one thing off to another, miscategorise someone, or otherwise state or imply falsehoods. We will edit.

Unlike the late

Larks reviews

, we’re not primarily aiming to direct donations. But if you enjoy reading this,

consider

donating to

Manifund

,

MATS

, or

LTFF

, or to

Lightspeed

for big ticket amounts: some good work is bottlenecked by money, and you have free access to the service of specialists in giving money for good work.

Meta

When I (Gavin) got into alignment (actually it was still ‘AGI Safety’) people

warned

me it was pre-paradigmatic. They were right: in the intervening 5 years, the live agendas have

changed

completely.[1] So here’s an update.

Chekhov’s evaluation: I include Yudkowsky’s operational

criteria

(Trustworthy command?, closure?, opsec?, commitment to the common good?, alignment mindset?) but don’t score them myself. The point is not to throw shade but to remind you that we often know little about each other.

See you in 5 years.

Editorial

Alignment is now famous enough that Barack Obama is

sort

...
Found Paper: "FDT in an evolutionary environment"
by the gears to ascension — 2023-11-27T05:27:50.709Z
Link: https://arxiv.org/pdf/2005.05154.pdf

I'm not sure what to think of this paper, it's quite long and I haven't finished checking it for sanity. nevertheless, I noticed it hadn't made its way here, and there are mighty few papers that cite the FDT paper, so I figured I'd drop it off rather than leave it sitting open in a tab forever.

Abstract:

Functional decision theory (FDT) is a fairly new mode of decision theory and a normative viewpoint on how an agent should maximize expected utility. The current standard in decision theory and computer science is causal decision theory (CDT), largely seen as superior to the main alternative evidential decision theory (EDT). These theories prescribe three distinct methods for maximizing utility. We explore how FDT differs from CDT and EDT, and what implications it has on the behavior of FDT agents and humans. It has been shown in previous research how FDT can outperform CDT and EDT. We additionally show FDT performing well on more classical game theory problems and argue for its extension to human problems to show that its potential for superiority is robust. We also make FDT more concrete by displaying it in an evolutionary environment, competing directly against other theories. All relevant code can be found here: https://github.com/noahtopper/FDT-in-an-Evolutionary-Environment.
why did OpenAI employees sign
by bhauth — 2023-11-27T05:21:28.612Z

Recently, OpenAI employees signed an open letter demanding that the board reinstate Sam Altman, add other board members (giving some names of people allied with Altman), and resign, or else they would quit and follow Altman to Microsoft.

Following those demands would've put the entire organization under the control of 1 person with no accountability to anyone. That doesn't seem like what OpenAI employees wanted to be the case, unless they're dumber than I thought. So, why did they sign? Here are some possible reasons that come to mind:

Altman is just really likeable for people like them - they just like him.

They felt a sense of injustice and outrage over the CEO being fired that they'd never felt over lower-level employees being fired.

They were hired or otherwise rewarded by Altman and thus loyal to him personally.

They believed Altman was more ideologically aligned with them than any likely replacement CEO (including Emmett Shear) would be.

They felt their profit shares would be worth more with Altman leading the company.

They were socially pressured by people with strong views from (3) or (4) or (5).

They were afraid the company would implode and they'd lose their job, and wanted the option of getting hired at a new group in Microsoft, and the risk of signing seemed low once enough other people already signed.

They were afraid Altman would return as CEO and fire or otherwise punish them if they hadn't signed.

Something else?

Which of those reasons do you think drove people signing that letter, and why do you think so?
Unknown Probabilities
by transhumanist_atom_understander — 2023-11-27T02:30:07.335Z

An unknown probability sounds like a type error.
There are unknowns, such as the result of a coin flip.
And there are probabilities that these unknowns take certain values, such as the probability that the flip comes up heads.

As a formula,
P(Result=Heads)=1/2
The unknown, the result of the flip, is inside the probability operator.
The probability, 1/2, is outside.
They're not the same kind of thing.

But suppose we have, for every possible value θ of the unknown Θ,
P(H|B,Θ=θ)=θ
where H and B are propositions.
Then it makes sense to say that Θ is an unknown probability of a hypothesis H, against background knowledge B.

Some natural examples:

Suppose a coin is flipped, and you don't know whether it's a fair coin, or a trick coin that is heads on both sides.
Let Θ be 1/2 if it's a fair coin, and 1 if it's a trick coin.

When sampling randomly from a population, let Θ be the fraction of individuals in the population that have a certain property.

Suppose a die is rolled, but due to imprecise manufacturing, its shape deviates from a perfect cube.
Consider the phase space of the die: six dimensions for position and momentum of the center of mass, and an additional six dimensions for three angles and three angular velocities.
Label each point in this phase space according to the face of the die that comes up when rolled with that point as an initial condition.
Let Θ be the fraction of the phase space labeled 6.
As long as the prior distribution for the initial condition of the roll is sufficiently wide, this should be almost exactly the unknown probability that the die comes up 6.
(This is based on Jaynes's discussion of coins in section 10.6 of "Probability Theory: The Logic of Science". The idea of an unknown shape is inspired by an actual analysis of dice data as inference of manufacturing defects.)

I could have simplified that last one considerably by just saying that Θ is the probability you would assign to 6, if you knew the shape of the die.
In fact, all of these unknown probabilities are really probabilities you would assign if you had some additional piece of information.
But before we get there, I'll go through the first example, the coin example, in more formalism than is really necessary.
I'd rather be tedious than mysterious.

(By the way, this will already be familiar if you've taken a probability class based on measure theory, where conditional expe...
Justification for Induction
by Krantz — 2023-11-27T02:05:49.217Z

David Hume presented philosophers with a problem. His claim was that there is no justification in the assertion that, “instances of which we have had no experience resemble those of which we have had experience”. In the following paper, I will show that observations of instances can justify assertions of instances that have not been experienced. I will begin, in part A, by defining what it means to be an observation of an instance. Next, in part B, I will define what it means to be an assertion of an instance that has not been experienced. Then, in part C, I will define what it means for an assertion to be justified. In part D, I will show how experiences (as defined in part A) meet the criteria for justification (as defined in part C) for non-experienced instances (as defined in part B). Finally, in corollary part E, I will demonstrate how generalized statements about future events can be justified using the methodology from part D and the conceptualized framework of the (epsilon, delta) proofs that are employed in calculus.

Part A

An observation of instance, is a statement that can be made about the Universe. For example, “there is at least one swan that is white. (this sort of assertion can be made while looking at a white swan that is found in nature.)”

Part B

An instance that has not been experienced, is an observation that is to be made in the future. For example, “the next swan that we see will be white. (with regard to the next swan that will be observed in nature.)” An assertion of such instance, is a statement that is made at a time (t1) that predicates an observation that occurs at a time (t2), such that, the time (t1) is before the time (t2).

Part C

An assertion (p) is justified if the negation of (p) implies a contradiction. An assertion is also justified if it is directly observed. For example, if a white swan is observed, we are justified in making the assertion, “a white swan has been observed.”

Part D

Consider a case where an assertion (p1) is made, (the next swan that we see will be white.), at a given time (t1). According to the criteria in part C, an observation at time (t2) of an object that holds both properties (the next swan seen) and (white) justifies the assertion (p1). It is important to understand that at time (t1) the assertion (p1) was not justified,...
Situational awareness (Section 2.1 of “Scheming AIs”)
by Joe Carlsmith — 2023-11-26T23:00:47.588Z

This is Section 2.1 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?”. There’s also a summary of the full report here (audio here). The summary covers most of the main points and technical terms, and I’m hoping that it will provide much of the context necessary to understand individual sections of the report on their own.

Audio version of this section here, or search "Joe Carlsmith Audio" on your podcast app.

What's required for scheming?

Let's turn, now, to examining the probability that baseline ML methods
for training advanced AIs will produce schemers. I'll begin with an
examination of the prerequisites for scheming. I'll focus on:

Situational awareness: that is, the model understands that it's
a model in a training process, what the training process will
reward, and the basic nature of the objective world in
general.[1]

Beyond-episode goals: that is, the model cares about the
consequences of its actions after the episode is complete.[2]

Aiming at reward-on-the-episode as part of a power-motivated
instrumental strategy: that is, the model believes that its
beyond-episode goals will be better achieved if it optimizes for
reward-on-the-episode – and in particular, that it, or some other
AIs, will get more power if it does this.[3]

Situational awareness

Will models have situational awareness? Let's distinguish between two
broad sorts of information at stake in such awareness:

General information about the objective world, including e.g.
information about how machine learning training works.

"Self-locating" information: that is, information that locates
the model in the objective world, and tells it facts about its own
situation in particular – e.g., that it is this sort of model,
that it's being trained on this particular reward signal, at this
particular lab, during this particular time period, etc.[4]
(Though: note that it's not clear how much of this sort of
information is necessary to start scheming. E.g., a model could in
principle decide that "whoever it is" and "whatever time period it
is," it will probably do better by its goals to try to perform well
by the lights of the training signal, and to get more
power/information later.)

It seems very plausible that even somewhat-better-than-human models will
absorb huge amounts of general information about the objective world,
and develop detailed, mec...
AXRP Episode 26 - AI Governance with Elizabeth Seger
by DanielFilan — 2023-11-26T23:00:04.916Z

YouTube link

The events of this year have highlighted important questions about the governance of artificial intelligence. For instance, what does it mean to democratize AI? And how should we balance benefits and dangers of open-sourcing powerful AI systems such as large language models? In this episode, I speak with Elizabeth Seger about her research on these questions.

Topics we discuss:

What kinds of AI?

Democratizing AI

How people talk about democratizing AI

Is democratizing AI important?

Links between types of democratization

Democratizing profits from AI

Democratizing AI governance

Normative underpinnings of democratization

Open-sourcing AI

Risks from open-sourcing

Should we make AI too dangerous to open source?

Offense-defense balance

KataGo as a case study

Openness for interpretability research

Effectiveness of substitutes for open sourcing

Offense-defense balance, part 2

Making open-sourcing safer?

AI governance research

The state of the field

Open questions

Distinctive governance issues of x-risk

Technical research to help governance

Following Elizabeth’s research

Daniel Filan:
Hello, everybody. In this episode, I’ll be speaking with Elizabeth Seger. Elizabeth completed her PhD in philosophy of science at Cambridge in 2022, and is now a researcher at the Centre for Governance of AI in Oxford, where she works on AI democratization and open-source AI regulation. She recently led the production of a large report on the risks and benefits of model-sharing, which we will talk about in this episode. For links to what we’re discussing, you can check the description of the episode and you can read the transcript at axrp.net.

Well, Elizabeth, welcome to the podcast.

Elizabeth Seger:
Awesome. Thanks for having me.

What kinds of AI?

Daniel Filan:
Cool. We’re going to be talking about a couple of papers basically about democratizing and open-sourcing AI. When we’re talking about this, what sorts of AI should we be thinking about? What sorts of AI do you have primarily in mind?

Elizabeth Seger:
When talking about open-sourcing and model-sharing, I’ve been primarily interested in talking about frontier foundation models, so understanding frontier foundation models as those systems on the cutting edge of dev...