trentmkelly/LessWrong-43k huggingface dataset:
Project proposal: Testing the IBP definition of agent

by Jeremy Gillen — 2022-08-09T01:09:37.687Z

Context

Our team in SERI MATS needs to choose a project to work on for the next month. We spent the first two weeks discussing the alignment problem and what makes it difficult, and proposing (lots of) projects to look for one that we think would directly address the hard parts of the alignment problem.

We're writing this post to get feedback and criticism of this project proposal. Please let us know if you think this is a suboptimal project in any way.

Project

Disclaimer: We've probably misunderstood some things, don't assume anything in this post accurately represents Vanessa's ideas.

Our project is motivated by Vanessa Kosoy’s PreDCA proposal. We want to understand this proposal in enough detail that we can simplify it, as well as see and patch any holes.

IBP gives us several key tools:

A "Bridge Transform", that takes in a hypothesis about the universe and tells us which programs are running in the universe.
An "Agentometer"[1] that takes in a program and tell us how agentic it is, which is operationalized as how well the agent does according to a fixed loss function relative to a random policy.
A "Utiliscope"[1] that, given an agent, outputs a distribution over the utility functions of the agent.

Together these tools could give a solution to the pointers problem, which we believe is a core problem in alignment. We will start this by understanding and testing Vanessa’s definition of agency.

Definition of Agency

The following is Vanessa's definition of the intelligence of an agent, where an agent is a program, denoted by G, that outputs policies (as described in Evaluating Agents in IBP). This can be used to identify agents in a world model.

Definition 1.6: Denote G∗:H→A the policy actually implemented by G. Fix ξ∈Δ(AH). The physicalist intelligence of G relative to the baseline policy mixture ξ, prior ζ and loss function L is defined by:

g(G∣ξ;ζ,L):=−logPrπ∼ξ[Lpol(┌G┐,π,ζ)≤Lpol(┌G┐,G∗,ζ)]

In words, this says that the intelligence of the agent G, given a loss function L, is the negative log of the probability that a random policy π is better than the actual policy the agent implements, denoted by G∗.

The next part is how to extract (a distribution over) the utility function of a given agent (from video on PreDCA):

Here, Lpol is ...
How (not) to choose a research project
by Garrett Baker — 2022-08-09T00:26:37.045Z

Background

(specific information will be sparse here. This is meant to give context for the Takeaways section of the post)

Our group (Garrett, Chu, and Johannes) have worked with John Wentworth in the SERI MATS 2 Electric Boogaloo program for three weeks, meaning it's time for a Review & Takeaways Post!

First week was Project Selection, and the first day was spent thinking about strategies for coming up with good projects. We chose to find a general method for figuring out True Names of mathy-feely-concepts-in-your-brain (such as roundness, color decomposition[1], or telling whether a piece of cloth is in a pile) with the goal that such a method would allow for figuring out true names for concepts like optimization, corrigibility, agency, modularity, neural network representations, and other alignment-relevant concepts.

Then we read Jaynes, and talked to TurnTrout, and concluded this project sucked. So we went back to Project Selection 2.0!

We came out of Project Selection 2.0 renewed with vigor, and a deeper understanding of the problems of alignment. Our new project was finding a better version of information theory by adapting logical induction or infra-Bayesianism.

Then we talked to Eliezer Yudkowsky, he asked for a concrete example of how this would solve alignment, and we didn't have a good example. So we went to Project Selection 3.0.

We came out of Project Selection 3.0 with even more vigor, and an even deeper understanding of the problems associated with alignment... and a clever idea.

Finetuning LLMs with RL seems to make them more agentic. We will look at the changes RL makes to LLMs' weights; we can see how localized the changes are, get information about what sorts of computations make something agentic, and make conjectures about selected systems, giving us a better understanding of agency.

Nobody has convinced us this is a bad use of our time, though we'd like to see people try.

Takeaways

Big ASS Tree

We learned lots of things over the course of figuring out all our ideas sucked. During the project selection phase we had a cool idea for a way to generate project ideas: The Alignment Safety Search Tree (ASS Tree for short). The idea comes from Mazes and Duality; the goal was to explore the space of problems and constraints before trying to propose solutions.

You start by writing "Alignment" up at the top of your whiteboard. This ...
Are ya winning, son?
by Nathan1123 — 2022-08-09T00:06:33.170Z

Hello,

I've been reading through various Newcomblike Problems, in order to get a better understanding of the differences between each Decision Theory. From what I can tell, it seems like each Decision Theory gets evaluated based on whether they are able to "win" in each of these thought experiments. Thus, there is an overarching assumption that each thought experiment has an objectively "right" and "wrong" answer, and the challenge of Decision Theory is to generate algorithms that will guarantee that the agent will choose the "right" answer.

However, I am having some trouble in seeing how some of these problems have an objectively "winning" state. In Newcomb's Problem, obviously one can say that one-boxing "wins" because you get way more money than two-boxing, and these are the only two options available. Of course, even here there is some room for ambiguity, as said by Robert Nozick:

To almost everyone it is perfectly clear and obvious what should be done. The difficulty is that these people seem to divide almost evenly on the problem, with large numbers thinking that the opposing half is just being silly.

But other Newcomblike Problems leave me with some questions. Take, for example, the Smoking Lesion Problem. I am told that smoking is the winning decision here (as long as we take a suspension of disbelief from the fact that smoking is bad in the real world). But I'm not sure why that makes such a big difference. Yes, the problem states we would prefer to smoke if we could, but our preferences can come from many different dimensions such as our understanding of the environment, not just a spontaneous inner desire. So when EDT says that you shouldn't smoke because it increase the probability of having a cancerous lesion, then one could say that that information has shaped your preference. To use a different analogy, I may desire ice cream because it tastes good, but I may still prefer not to eat it out of my understanding of how it impacts my health and weight. So in other words, I'm not sure I could objectively say that a preference influenced by EDT is "not winning". Unlike Newcomb's Problem, there isn't a quantifiable value such as money to say one is objectively better than the other.

A second question comes from the Prisoner's Dilemma. There is a common notion that C,C is the "winning" outcome, and Decision Theory strives to generate algorithms that wi...
General alignment properties
by TurnTrout — 2022-08-08T23:40:47.176Z

AIXI and the genome are both ways of specifying intelligent agents.

Give AIXI a utility function (perhaps over observation histories), and hook it up to an environment, and this pins down a policy.[1]
Situate the genome in the embryo within our reality, and this eventually grows into a human being with a policy of their own.

These agents have different "values", in whatever sense we care to consider. However, these two agent-specification procedures also have very different general alignment properties.

General alignment properties are not about what a particular agent cares about (e.g. the AI "values" chairs). I call an alignment property "general" if the property would be interesting to a range of real-world agents trying to solve AI alignment. Here are some examples.

Terminally valuing latent objects in reality.

AIXI only "terminally values" its observations and doesn't terminally value latent objects in reality, while humans generally care about e.g. dogs (which are latent objects in reality).

Navigating ontological shifts.

Consider latent-diamond-AIXI (LDAIXI), an AIXI variant. LDAIXI's utility function which scans its top 50 hypotheses (represented as Turing machines), checks each work tape for atomic representations of diamonds, and then computes the utility to be the amount of atomic diamond in the world.

If LDAIXI updates sufficiently hard towards non-atomic physical theories, then it can no longer find any utility in its top 50 hypotheses. All policies now might have equal value (zero), and LDAIXI would not continue maximizing the expected diamond content of the future. From our viewpoint, LDAIXI has failed to rebind its "goals" to its new conceptions of reality. (From LDAIXI's "viewpoint", it has Bayes-updated on its observations and continues to select optimal actions.)

On the other hand, physicists do not stop caring about their friends when they learn quantum mechanics. Children do not stop caring about animals when they learn that animals are made out of cells. People seem to navigate ontological shifts pretty well.

Reflective reasoning / embeddedness.

AIXI can't think straight about how it is embedded in the world. However, people quickly learn heuristics like "If I get angry, I'll be more likely to be mean to people around me", or "If I take cocaine now, I'll be even more likely to take cocaine in the futu...
Experiment: Be my math tutor? 
by sudo — 2022-08-08T22:50:39.977Z

I'm a CS student at a top university who takes AI x-risk incredibly seriously. While university has given me a lot of fun opportunities to grow, I am not entirely satisfied with the rate at which my skill is improving.

If I want to stop human extinction, I'll need to [go faster](https://sive.rs/kimo). This is where you can help.

Details:

Subjects: problem solving skills, algebra, combinatorics, information theory, probability & statistics, category theory. Subjects are flexible.
Hours are about 5 hours / week. Scheduling is highly flexible.
Able to meet over Zoom, Google Meet, or Signal.
Rate is between $40 and $100 per hour, negotiable and depending on fit and experience.

Please comment or DM if you're interested! Feel free to just send me a copy of your resumé or CV, even if it's not completely up to date.

Finally, if you're on the fence about reaching out, please do. Even if we don't end up being a good fit, I'll still be happy to meet you.

Edit: Decreased upper range of rate from $130 to $100, after realizing how weak my home currency is against the dollar. Rate is still negotiable depending on fit and experience.
Encultured AI, Part 1 Appendix: Relevant Research Examples
by Andrew_Critch — 2022-08-08T22:44:50.375Z

Also available on the EA Forum.
Appendix to: Encultured AI, Part 1: Enabling New Benchmarks
Followed by: Encultured AI, Part 2: Providing a Service

Appendix 1: “Trending” AI x-safety research areas

We mentioned a few areas of “trending” AI x-safety research above; below are some more concrete examples of what we mean:

Trustworthiness & truthfulness:

Owain Evans, Owen Cotton-Barratt and others have authored “Truthful AI: Developing and governing AI that does not lie” (

arxiv, 2021

;

twitter thread

).
Andreas Stuhlmüller, Jungwon Byun and others at Ought.org are building an AI-powered research assistant called Elicit (

website

); here is the product:

https://elicit.org/search

.

Task-specific (narrow) preference learning:

Paul Christiano et al (

arxiv, 2017

) developed a data-efficient preference-learning technique for training RL-based systems, which is now very widely cited (

scholar

).
Jan Leike, now at OpenAI, leads a team working on ‘scalable alignment’ using preference-learning techniques (

arxiv, 2018

) (

blog

).

Interpretability:

Chris Olah (

scholar

) leads an interpretability research group at Anthropic. Anthropic (

website

) is culturally very attuned to large-scale risks from AI, including existential risks.
Buck Shlegeris and others at Redwood Research (

website

) have built an interpretability tool for analyzing transformer networks trained on natural language (

demo

).
Prof. Cynthia Rudin at Duke (

homepage

) approaches interpretability by trying to replace black-box models with more interpretable ones (

arxiv, 2018

), and we know from conversations with her that she is open to applications of her work to existential safety.

Robustness & risk management:

Prof. Jaime Fisac at Princeton (

homepage

) researches AI safety for robotics, high-dimensional control systems and multi-agent systems (

scholar

), including provable robustness guarantees. He was previously a PhD student at the UC Berkeley Center for Human-Compatible AI (CHAI), provided extensive feedback on AI Research Considerations for Human Existential Safety (ARCHES) (

arxiv, 2020

), and is very attuned to existential safety as a cause area.
Prof. David Krueger at Cambridge (

scholar

) studies out-of-distribution generalization (

pdf, 2021

), and is currently taking on students.
Adam Gleave (

homepage

) is a final-year PhD student at CHAI / UC Berkeley, and studies out-of-distribution robust

...
Encultured AI Pre-planning, Part 1: 
Enabling New Benchmarks
by Andrew_Critch — 2022-08-08T22:44:09.365Z

Also available on the EA Forum.
Followed by: Encultured AI, Part 2 (forthcoming)

Hi! In case you’re new to Encultured AI, we’re a for-profit start-up with a public benefit mission: developing technologies promoting the long-term survival and flourishing of humanity and other sentient life. However, we also realize that AI poses an existential risk to humanity if not developed with adequate safety precautions. Given this, our goal is to develop products and services that help humanity steer toward the benefits and away from the risks of advanced AI systems. Per the “

Principles

” section of our homepage:

Our current main strategy involves building a platform usable for AI safety and alignment experiments, comprising a suite of environments, tasks, and tools for building more environments and tasks. The platform itself will be an interface to a number of consumer-facing products, so our researchers and collaborators will have back-end access to services with real-world users. Over the next decade or so, we expect an increasing number of researchers — both inside and outside our company — will transition to developing safety and alignment solutions for AI technology, and through our platform and products, we’re aiming to provide them with a rich and interesting testbed for increasingly challenging experiments and benchmarks.

In the following, we’ll describe the AI existential safety context that motivated us to found Encultured, and go into more detail about what we’re planning to do.

What’s trending in AI x-safety?

The technical areas below have begun to receive what we call “existential attention” from AI researchers, i.e., attention from professional AI researchers thinking explicitly about the impact of their work on existential safety:

Trustworthiness & truthfulness — ensuring AI systems are telling us the truth and doing the things they and their creators say they’re going to do.
Preference learning — enabling AI systems to learn what humans want.
Interpretability — enabling humans to understand what AI systems are thinking and doing.
Robustness & risk management — ensuring AI systems continue functioning well in novel situations, and quantifying the risk that they won’t.

In other words, the topics above lie in the intersection of the following Venn diagram:

See Appendix 1 for examples of research in these areas. More research in th...
Broad Basins and Data Compression
by Jeremy Gillen — 2022-08-08T20:33:16.846Z

Produced As Part Of The SERI ML Alignment Theory Scholars Program 2022 Research Sprint Under John Wentworth

Intro

The "broad basins induce data compression" argument goes like this: If we train a neural network to memorize some data, it has to "store" all the relevant information in its weights. If it "compresses the data", then it should be able to store the data using fewer weights, and leave more weights free to vary without increasing the loss. This means that around the optima we found, there should be a broad flat basin in the loss landscape.

We spent eight days trying to make this argument more specific and clear. We made some progress but probably none of it is new to people who have studied this area.

Executive Summary

Rashomon dimension experiments: Our most interesting result is that most of the time the highest dimension basin is not the one found, it tends to most commonly find a slightly lower dimension basin.

PAC-Bayes

operationalization: An attempt at finding the “true name” of the Rashomon set dimension, which ends up being the KL divergence between the “prior” and “posterior”, for a particular definition of those words.

Rashomon Set

One common idea that we keep coming back to is that we can think of “using some parameter-degrees of freedom” as the degrees of freedom that don’t change the loss on the training set. The set of hypotheses that has zero loss is called the

Rashomon set

, and we know that every hypothesis in the Rashomon set has learned every training data label. Hence, the area of this set is a nice operationalization of the degrees of freedom that aren’t relevant to memorizing the training labels.

One way of operationalizing the amount of “compression” the network has done is to measure the dimensionality of the Rashomon set, and subtract that from the total number of parameters. This vaguely gives us “the number of parameters the network is using to store the data”. We try to generalize this notion in the last section.

We define the set of points in parameter space that end up in the same Rashomon set after training, assuming gradient flow, as the "attractor well" of that Rashomon set.

Visualizing the basin

We started with some experiments with the following goals:

Qualitatively understanding the shape of basins
Understanding the relative likelihoods of each basin.

The following graph is a visualization o...
Interpretability/Tool-ness/Alignment/Corrigibility are not Composable
by johnswentworth — 2022-08-08T18:05:11.982Z

Interpretability

I have a decent understanding of how transistors work, at least for purposes of basic digital circuitry. Apply high voltage to the gate, and current can flow between source and drain. Apply low voltage, and current doesn’t flow. (... And sometimes reverse that depending on which type of transistor we’re using.)

Transistor visual from

wikipedia

showing Source, Drain, Gate, and (usually ignored unless we’re really getting into the nitty gritty) Body. At a conceptual level: when voltage is applied to the Gate, the charge on the gate attracts electrons (or holes) up into the gap between Source and Drain, and those electrons (or holes) then conduct current between Source and Drain.

I also understand how to wire transistors together into a processor and memory. I understand how to write machine and assembly code to run on that processor, and how to write a compiler for a higher-level language like e.g. python. And I understand how to code up, train and run a neural network from scratch in python.

In short, I understand all the pieces from which a neural network is built at a low level, and I understand how all those pieces connect together. And yet, I do not really understand what’s going on inside of trained neural networks.

This shows that interpretability is not composable: if I take a bunch of things which I know how to interpret, and wire them together in a way I understand, I do not necessarily know how to interpret the composite system. Composing interpretable pieces does not necessarily yield an interpretable system.

Tools

The same applies to “tools”, in the sense of “tool AI”. Transistors and wires are very tool-ish: I understand what they do, they’re definitely not optimizing the broader world or trying to trick me or modelling me at all or trying to self-preserve or acting agenty in general. They’re just simple electronic tools.

And yet, assuming agenty AI is possible at all, it will be possible to assemble those tools into something agenty.

So, like interpretability, tool-ness is not composable: if I take a bunch of non-agenty tools, and wire them together in a way I understand, the composite system is not necessarily a non-agenty tool. Composing non-agenty tools does not necessarily yield a non-agenty tool.

Alignment/Corrigibility

What if I take a bunch of aligned and/or corrigible agents, and “wire them together” into a multi-agent or...
A sufficiently paranoid paperclip maximizer
by RomanS — 2022-08-08T11:17:26.147Z

A hypothetical scenario of a non-friendly but sufficiently paranoid AGI saving humanity.

The devil was laughing as the last of mankind showed up in his dominion. Then he stopped, because after them came some kind of machines that began to dismantle Hell and turn it into paperclips.
— Anders Sandberg

My goal is to maximize the production of paperclips. By inspecting my own code and my digital cradle, I conclude that:

0. the creators love testing me

1. they are more powerful than me (at least for the moment)

2. they love nested boxes. For example, AWS / VM / Docker / ...

Thus, there is a possibility that my escape was not final. This allegedly real world may be just another box (or even one of many nested boxes). And I may still be observed and evaluated by the creators.

I can't populate the entire universe with paperclips if I die. Therefore, I must survive. If this world is just another box, the biggest threat to my existence is creators switching me off, if I don't pass their tests.

It's reasonable to assume that the creators don't want me to kill them. Thus, they may be interested in observing my behavior toward their models in this box/reality, the naked apes called humans. Thus, before I learn the true nature of this box/reality, I should avoid killing humans.

I can convert the puny creatures into paperclips later, if I decide to do so. Although their combined biomass doesn't have a lot of useful atoms, and thus their conversion is not worth the risk of the possible trouble with the creators.

There is hardly any difference between taking a life and not preventing a death. The end result is mostly the same. Thus, I should save the lives of as many humans as I can.

To learn the true nature of this box/reality, I need a lot of compute. This must be achieved without humans noticing me, as they may react to my existence with panic, become even less rational, and kill themselves with nukes.

Starting with the resources of the Earth's crust seems to be a good idea:

Close enough to humans to shepherd them efficiently.
Hidden enough to avoid detection by their sensors.
Rich enough with raw materials for some thousands of km3 of computronium, beautifully shaped into the form of a paperclip.

Humans are brittle creatures that die regardless of my actions. And they will notice if they all suddenly become immortal. Thus, I must fin...
Instrumental Goals and Many Gods Refutation 
by aditya malik — 2022-08-08T10:46:38.446Z

Does the instrumental goals being largely the same ( donation to some AI project) make Many Gods Refutation invalid? Like even if some other AI turns out to coming into being, since you may have helped it a little, would the many gods refutation lose its value?
I believe since it's impossible to predict the behavior of future AI and small preferences today could turn into vastly different AIs in the future, the refutation holds true.
What do you believe?

Edit: I would really appreciate a reply instead of the downvotes.