trentmkelly/LessWrong-43k huggingface dataset:
Paradigms of AI alignment: components and enablers
by Vika — 2022-06-02T06:19:59.378Z

(Cross-posted from my personal blog. This post is based on an overview talk I gave at UCL EA and Oxford AI society (recording here). Thanks to Janos Kramar for detailed feedback on this post and to Rohin Shah for feedback on the talk.)

This is my high-level view of the AI alignment research landscape and the ingredients needed for aligning advanced AI. I would divide alignment research into work on alignment components, focusing on different elements of an aligned system, and alignment enablers, which are research directions that make it easier to get the alignment components right.

Alignment components

Outer alignment
Inner alignment

Alignment enablers

Mechanistic interpretability
Understanding bad incentives
Foundations

You can read in more detail about work going on in these areas in my list of AI safety resources.

Alignment components

The problem of alignment is getting AI systems to do what we want them to do. Let’s consider this from the perspective of different levels of specification of the AI system’s objective, as given in the Specification, Robustness & Assurance taxonomy. We start with the ideal specification, which represents the wishes of the designer – what they have in mind when they build the AI system. Then we have the design specification, which is the objective we actually implement for the AI system, e.g. a reward function. Finally, the revealed specification is the objective we can infer from behavior, e.g. the reward that the system seems to be actually optimizing for. An alignment problem arises when the revealed specification doesn’t match the ideal specification: the system is not doing what we want it to do.

The gaps between these specification levels correspond to different alignment components. We have outer alignment when the design specification matches the ideal specification, e.g. when the reward function perfectly represents the designer’s wishes. We have inner alignment when the revealed specification matches the design specification, e.g. when the agent actually optimizes the specified reward. (Robustness problems also belong in the design-revealed gap, but we expect them to be less of an issue for advanced AI systems, while inner alignment problems remain.)

Now let’s have a look at how we can make each of those components work.

Outer alignment

The most promising class of approaches to outer alignment is scalable oversight. Thes...
The Bio Anchors Forecast
by Ansh Radhakrishnan — 2022-06-02T01:32:05.615Z

Ajeya Cotra’s

Forecasting Transformative AI with Biological Anchors

, to my knowledge, represents the most serious effort to predict the arrival of

transformative AI

- even if it’s not attempting to pinpoint the exact instant that we’ll get transformative AI, it posits an upper bound on the amount of time until then (

Holden Karnofsky’s post

clarifies the difference).

The report’s methodology,

summarized by Scott Alexander

, is:

1. Figure out how much inferential computation the human brain does.

2. Try to figure out how much training computation it would take, right now, to get a neural net that does the same amount of inferential computation. Get some mind-bogglingly large number.

3. Adjust for "algorithmic progress", ie maybe in the future neural nets will be better at using computational resources efficiently. Get some number which, realistically, is still mind-bogglingly large.

4. Probably if you wanted that mind-bogglingly large amount of computation, it would take some mind-bogglingly large amount of money. But computation is getting cheaper every year. Also, the economy is growing every year. Also, the share of the economy that goes to investments in AI companies is growing every year. So at some point, some AI company will actually be able to afford that mind-boggingly-large amount of money, deploy the mind-bogglingly large amount of computation, and train the AI that has the same inferential computation as the human brain.

5. Figure out what year that is.

There are some other biological anchors that the report also considers, besides the human brain compute estimate, such as drawing a comparison between the parameter count of a transformative AI model and the parameter count of the human genome, or the amount of compute used by all animal brains over the course of evolution - I don’t place a ton of weight on these anchors myself, since I find them uninformative, and I tend to think the strength of this report lies in assuming that the current deep learning paradigm will lead to transformative AI and trying to work backward from there.

Is it actually reasonable to assume that deep learning might produce transformative AI? I think so. Mark Xu’s model

here

is compelling regarding the differences between different AI algorithms/paradigms in terms of their reliance on computing power: each algorithm seems to have a certain “effectiv...
Public beliefs vs. Private beliefs
by Eli Tyre — 2022-06-01T21:33:32.661Z

A distinction that I get a lot of value out of is the difference between private beliefs and public beliefs.

Public vs. Private

A public belief is a proposition that someone thinks is true, and justifying on the basis of legible info and reasoning. If X is a public belief, then implicit claim is "not only do I think that X is true, I think that any right thinking person who examines the evidence should come to conclude X."

If someone disagrees with you about a public belief, it is prosocial and epistemically virtuous to defend your claim and to debate the matter on a public forum. Public beliefs, if consensus is reached about them, can be added to the sum total of human knowledge for others to take for granted and then build on.

A private belief is proposition that someone thinks is true based on their own private or illegible info and reasoning. In this case, the implicit claim is "given my own read of the evidence, I happen to think X. But I don't think that the arguments that I've offered are necessarily sufficient to convince a third party. I'm not claiming that you should believe this, I'm merely providing you the true information that I believe it."

If someone disagrees with you about a private belief, there might or might not be a fruitful discussion to be had about it, but it is also important to be able to "agree to disagree."

An example

I think that Circling meaningfully develops real skills of introspection, subtle interpersonal sensitivity, and clarity of map-territory distinctions. I further think that Circling is relevant to the Art of Rationality.

There have been some write-ups that describe why I think this is the case, but I don't know that any of them are persuasive. If a person is curious about why I might be interested in Circling, I think this post is a decent overview. But crucially, I don't think that the evidence presented should be sufficient to convince a skeptic.

I would say that I have a private belief that Circling is useful. It is actually my calibrated view, based on my personal experiences and my reasoning about those experiences. But by stating that belief, I am not at all making a social bid that others believe it too.

A special case: cloaks

There's a special case of having a private belief that, at CFAR, we used to refer to as "having a cloak".

If you are pursuing some ambitious project or personal develop...
Probability that the President would win election against a random adult citizen?
by Daniel Kokotajlo — 2022-06-01T20:38:44.197Z

Suppose we magically intervene on the USA to make there be a snap election for the office of President. The incumbent (the existing president) goes up against X, where X is a randomly selected eligible person (e.g. adult, citizen, etc.) Voters have one month of campaigning before they decide, so X isn't a total stranger by the end.

What is the probability that the President wins?

Is the probability substantially different for Biden than it was for Trump? What about Obama?

What is the probability that the President loses in a massive landslide (getting e.g. only 40% of the vote or less)?

ETA: I've changed the title to stop giving the impression that I am looking for a yes or no answer. I'm not looking for a yes or no answer, I'm looking for a probability.
Revisiting "Why Global Poverty"
by jefftk — 2022-06-01T20:20:02.314Z

Seven years ago I gave a talk,

Why
Global Poverty, at the 2015

effective altruism

conference. I concluded that
building the EA movement was the highest priority, but that we should
be doing some things that are directly valuable in parallel and that
global poverty was the best option there.

Reading back over the talk, with the benefit of seven years of
hindsight, I'm not happy with my treatment of existential risk.
Here's that section of the talk, with thoughts interspersed:

There's a bit of a continuum. At one end we have risks like an
asteroid hitting the earth. Cataloging asteroids and comets that might
hit the earth at some point is something that people are working on,
and actually is reasonably well understood. Because we have a pretty
good understanding, and governments have a lot of sensible people,
risks like this are reasonably well funded. So this end of the
continuum is probably not high impact.

I'm far more skeptical of the "governments have this covered" position
than I was in 2015. Some of this is for theoretical reasons (ex:
preventing catastrophe benefits people beyond your country) and some
of it is from observing governments more (ex: pandemic response).

At the other end we have risks like the development of an artificial
intelligence that destroys us through its indifference. Very few
people are working on this, there's low funding, and we don't have
much understanding of the problem. Neglectedness is a strong heuristic
for finding causes where your contribution can go far, and this does
seem relatively neglected. The main question for me, though, is how do
you know if you're making progress?

First, a brief digression into feedback loops. [...] Back to AI
risk.

The problem is we really really don't know how to make good
feedback loops here. We can theorize that an AI needs certain
properties not to just kill us all, and that in order to have those
properties it would be useful to have certain theorems proved, and go
work on those theorems. And maybe we have some success at this, and
the mathematical community thinks highly of us instead of dismissing
our work. But if our reasoning about what math would be useful is off
there's no way for us to find out. Everything will still seem like
it's going well.

This was primarily criticism of MIRI's
approach, and was about a year before Concrete
Proble...
What will happen when an all-reaching AGI starts attempting to fix human character flaws?
by Michael Bright — 2022-06-01T18:45:17.760Z

You know the saying： “I just want him to listen not to solve my problem”

Would an all-reaching AGI accept this?

How would AGI respond to self-sabotaging tendencies?

If it's true that world problems stem from individual preferences which are based on individual perspectives that originate from individual personality traits and experiences, where will an AGI that's hellbent to improve the world stop and accept things as they are and realize that any attempt to improve things may cause more damage to humans than good?

Take social media algorithm for context, keeping everyone in a relatively closed bubble it believes each person should be in and guides each person's decisions by determining what information to show to each person.

And with the advent of the internet of things, AI will become more prevalent.

While there may be AI specifically designed to offer emotional support and AI designed to solve problems, will an AGI that may develop some sort of consciousness simply accept some of the human character flaws and limitations, or will it strip it all away at the risk of hurting the human until the singularity of what is considered acceptable is achieved?

Would an AGI always act from a place of pure rationality and maximum efficiency, in disregard of some human values that prevent us from doing this most of the time?
Any prior work on mutiagent dynamics for continuous distributions over agents?
by Quintin Pope — 2022-06-01T18:12:02.963Z

Typical work on game theory, economics or multiagent negotiations assumes that there are a finite number of discrete agents. What if, instead, there's a continuous distribution over possible agents? Further, is there any work looking at the case where actions are taken by agents discretely sampled from that distribution? I.e., suppose we have a distribution over possible agents, then at each time step, we stochastically sample a single discrete agent from that distribution and allow it to act. Is there any work on the sorts of equilibria we might see?

Additionally, suppose the actions of the sampled agents can then influence the distribution over future agents. How does that change things? E.g., it seems clear enough that we should expect there to be attractors around agents whose actions cause more frequent sampling of agents similar to themselves, but are there other dynamics we should expect?

I ask this question because I'm interested in a multiagent perspective on the mind, but think that it's unlikely that the mind contains a discrete, finite set of subagents, and more likely that it contains (something like) a continuous distribution over possible agentic computations.

Thanks for any help you can provide!
Formation via nucleation of boltzmann brains
by Zeruel017 — 2022-06-01T18:05:34.450Z

hi beautiful people, in this forum I feel very stupid and it's good, it means that I can learn more!

I would have a doubt as I expressed in my previous question about boltzmann brain formation via nucleation (https://www.lesswrong.com/posts/LGMSLXkpKAofebjfi/a-terrifying-variant-of-boltzmann-s-brains-problem), I read this concept in this Wikipedia page (https://en.m.wikipedia.org/wiki/Boltzmann_brain) but it is not clear to me how it actually happens? for example: does it form instantaneously or slowly? (obviously this is not the only thing I doubt about this way of formation)
Machines vs Memes Part 3: Imitation and Memes
by ceru23 — 2022-06-01T13:36:15.089Z

This is the last in a series of three posts on the interlinkages between Memetics and AI Alignment. It was written as an output from the 2022 AI Safety Camp, for the ‘Impact of Memetics on Alignment’ team, coached by Daniel Kokotajlo and comprising Harriet Farlow, Nate Rush and Claudio Ceruti. Please read on to post 1 and post 2. We are not AI Safety experts so any and all feedback is greatly appreciated :)

TL;DR: Linking the concept of imitation between the domains of memetic and AI safety leads to considering how misalignment could spread memetically; this helps to identify misaligned agents able to generate a contagious spread of memes of misalignment. In particular, power-seeking agents could diffuse power-seeking tendencies to other agents under mild conditions. This work adds seminal tools to researching misalignment in multi-agent settings.

What does it mean to imitate

In memetic theory, imitation is a fundamental part of the evolutionary process of memes since it’s the primary way that provides the means for spreading, reproducing, selecting, and mutating memes. Even if selection pressure on memes is exerted internally, e.g., a single agent recalls only certain memes it was exposed to because the agent finds them easier to remember, the reproduction of memes can exist only in the presence of imitation.

In more detail, Susan

Blackmore

defines imitation as when an agent copies other agents' observed behaviour to achieve one of its goals (or sub-goals). Even if this concept clearly distinguishes imitation from other forms of collective learning (such as reaction to enhanced stimuli resulting from the actions of other agents), it lacks a detailed explanation of the copying function.

We can fill this gap by referring to the concept of observational learning made by Albert

Bandura

. Observational learning consists of observing and modelling another agent’s behaviour, without the need to clone the observed behaviour perfectly, but treating it as a learning source rather than precisely imitating it.

Combining observational learning and Blackmore’s definition produces a notion of copying another agent without explicitly and fully modelling its behaviour, but only focusing on replicating certain aspects or part of it.

This synthesis sounds intuitive since when we imitate some behaviour, we generally grasp ...
Rationalism in an Age of Egregores
by David Udell — 2022-06-01T07:29:06.297Z

Political tribes work by allowing people to emit costly signals of fealty to a tribe, where that signal of fealty ties your social status to the tribe's social status. So joining a political tribe is a status bet: you're hoping[1] that the tribe will gain in group status, relative to rival political tribes. If you called it well, you'll rise in social status with your tribe. Social status is a matter of relative position in a prestige order, so these group-status competitions are zero-sum.

Political tribes are horrifyingly successful in the social media age. Certainly, monotheistic faiths and the 20th-century political ideologies and nationalism were similarly effective phenomena. But living in the post-scarcity First World, people seem to want to spend a lot of their wealth on playing the zero-sum game about prestige ordering, online and otherwise.

People who have never really understood any other way to use words and arguments will sometimes say things like, "everything is political." I think this means something like: "there are no interesting, purely predictive uses of language that avoid signaling in some group-status relevant manner; anyone claiming so is actually making a status move by trying to arrogate the title of 'neutral.'"

This is just false. Beyond being motivated, though, it's epistemically evil! It's just plain wrong that we have to live in an adversarial communicative environment where we can't just take claims at face value without considering political-tribe-maneuvering implications. This is not a satisfactory equilibrium! In a mindkilled age, when egregores propagate freely in the medium of human psychology and bash in our ability to think, it's especially important to state directly: being "political" is unequivocally epistemically bad. And being epistemically weak is in turn bad for your goals, whatever they are. Raising the instrumental-rationality waterline is the common instrumental interest of many final causes.

There's no real contradiction in explicitly joining the anti-tribe tribe. It's just false that you can't get better at being a better-calibrated observer by trying hard at it. One of the most basic rationalist insights is that the best way to accomplish your goals, whatever those goals are, is to try directly. If you want to be less political, try directly first.

I bet that a core tenet of the Neo-Enlightenment project we'r...
Wielding civilization
by dominicq — 2022-06-01T07:11:30.119Z

(cross-posted from my blog, Sunday Stopwatch)

The other day, I was walking home on an empty road and a bus drew past me. It got me thinking about the differences in our speeds. I could never compete with the bus on speed. Not only because the bus has a big engine, but also because of the context it's embedded in. The bus lives in a world where the roads are made for it, and where fuel is available to it. Each of those two things - roads and fuel - both have their own highly complex supply networks. Roads require asphalt concrete, asphalt concrete requires giant plants for extracting rock and producing bitumen glue, both of these involve giant, planet-spanning networks of people who talk to each other, coordinate on amounts, routes, deadlines, shipments, issues with shipments, quality, quantity, and everything in between. And that's just for road material! We haven't even started to talk about the fuel part, nor how the buses themselves are produced! And the most incredible thing is how these networks - these complex, inert behemoths - they actually work, and work pretty well. If you taboo your understanding of what a road is, and instead see this enormous, civilization-spanning spell that has "road" as a byproduct, you really start to get incredulous that the whole thing works at all.

All this from the fact that the bus goes much faster than I walk. I guess I had a spiritual experience: a deep insight and appreciation of the sheer size of the entire thing. And that got me thinking - why have I been ignoring civilization?

Not in the sense of industrial illiteracy even though I could certainly be more industrially literate, but specifically in the sense of making a categorical distinction between being focused on things and being focused on people. These feel like they are different: people focused on things use tools, optimize systems, measure and analyze, think in terms of inputs and outputs, and so on. On the other hand, people focused on people use language, talk to others, understand their motivations, help or prevent them from achieving their goals, and so on.

And they are different, they're two different clusters of personalities and preferences, but I guess I see the thing side of civilization now. I feel like I missed the forest for the trees with civilization. Civilization is also a tool to be wielded, same as any computer program, angle grinder, surgica...
Machines vs. Memes 2: Memetically-Motivated Model Extensions
by naterush — 2022-05-31T22:03:48.992Z

[This is post 2 of 3 in a series written as the output for the 2022 AI Safety Camp, for the ‘Impact of Memetics on Alignment’ team, coached by Daniel Kokotajlo and comprising Harriet Farlow, Nate Rush and Claudio Ceruti. You can view the first post in this series here. Feedback so appreciated I'll do a jig for every comment.]

Overview

This post is a beginner’s attempt at extending the generally-accepted model of reinforcement learning agents (see

this post

).

This extension encourages us to think about artificial agents in a more suggestive and (potentially) more productive way. Specifically, this extensions captures an agent’s ability to change the policies it uses over time, while abstracting the decision to change policies away from the agent itself.

This encourages thinking about agents as more dynamic creatures than the standard RL formalization. It also allows for some interesting arguments about constraints on these agents' behaviors that are relevant to AI safety. This post is not optimistic, as it seems these more dynamic agents are less likely to be well-aligned.

This post was written as a culmination of my time at AI Safety camp. I am not an AI researcher, nor am I particularly good at math. Please treat these suggestions as a beginner taking a shot in a specific direction. Of course, any and all feedback is greatly appreciated!

Motivation

The model extensions I suggest are motivated by the study of memetics and memes.

By memes, I’m not talking about Grumpy Cat - I’m talking more generally about memes as a unit of “cultural replication.” The theory of memetics argues that humans are a uniquely big system of cultural replicators, and as such a well-developed theory of memes might be able to explain much of our behavior.

Although memetic theory is far from mainstream, its relevance to AI Safety is immediately obvious. Humans are the most intelligent agents we know of, and we are also by far the most memetic. As such, it’s natural to wonder how this trend might continue to affect artificial (and possibly smarter) agents.

Importantly, this is not an attempt to formalize memes. This model extension merely uses memetics as the inspiration. Memes make clear that behavior/beliefs may infect an agent, whether they want to be infected or not. This, in turn, might change how an agent behaves through a process entirely outside of its control....