trentmkelly/LessWrong-43k huggingface dataset:
Notes on Antelligence
by Aurigena — 2023-05-13T18:38:06.900Z

(This post uses a more generalized definition of "simulator" than the one used in Simulators. Simulators, as defined here, are functions optimized towards some arbitrary range of inputs, and without requirements on how the function was created. If you need to distinguish between the two definitions, the simulators here can basically just be described as functions that represent some real world process.)

image by DALL-E

5 Point Summary (Idea from this article):

Antelligence is emulation accuracy. In other words, when a simulator tries to emulate a different system, its accuracy (the similarity between its outputs and the outputs of the other system) is the same thing as its antelligence. This also makes antelligence relative, meaning that it’s dependent on what is considered the “ideal” output/system.
Antelligence is not defined through agency or being goal driven, and so addresses modern neural networks and simulators like GPT-3/4 and DALL-E much better than Intelligence.
Simulators optimize towards arbitrary ranges over arbitrary inputs, with optimizers being a type of simulator which optimize towards only 1 range over all inputs.
Antelligence is why humans are so successful, since high antelligence relative to the real world allows us to predict the future, letting us act in ways much more intelligent than other creatures.
Artificial simulators are inherently benign (won’t suddenly transform into an agent that tries to kill us all), but powerful and potentially dangerous tools that would only lead to human extinction if we simulate and actively give it the ability to kill us all and/or give it to agents (human or artificial) that want to kill us all, making them equivalent in danger to nuclear weapons (with their strength being determined by their antelligence relative to the real world, as compared to humans).

Eliezer Yudkowsky was the first to define intelligence as Efficient Cross-Domain Optimization, with this definition being the primary one used in discussions over AI and potential AGI’s and ASI’s for years now, along with helping to spur the many discussions over AI alignment. I’m making this article here to offer an ...
Reality and reality-boxes
by Jim Pivarski — 2023-05-13T14:14:10.758Z

We depend on our words... Our task is to communicate experience and ideas to others... The word "reality" is also a word, a word which we must learn to use correctly.
— Niels Bohr, Philosophy of Science Vol. 37 (1934), p. 157

While trying to write about some of the new things that have been bothering me, I kept coming back to this old one, since it seems so fundamentally important. Anything else I might say would have to refer to it, so I decided to write it up in a way that I can point to it with a link, the way that other topics on this site are so efficiently dealt with.

The point of this essay is that there is more than one reality. I mean that in as un-mystical a way as possible. I mean that the words "real" and "exist" are so oversubscribed that it would be better if there were several words, one for each concept. I see this come up repeatedly in arguments as people talk past each other, with one person saying that X exists, but meaning "X existsA," while another says that X does not exist, because "X does not existB." There's usually also a pejorative association with putting something in a not-real box, to say that X is irrelevant because it does not existB (but it might existA!).

I can't write down a complete list of all these reality-boxes because although there are a few well-defined, distinct meanings, there are also some hazy, overlapping categories. In this essay, I'll talk about five, which seem to be the clearest ones.

physical reality
mathematical reality
narratives/reasons-why, with causality being the most precise "reason-why"
conventional: things that are real (really real!) only because people say they are
subjective: the direct experience of being a mind, which is in a different reality-box from the objective study of physical brains.

This is not a complete list because some (particularly "narrative reality," number 3) could be subdivided, and other things that could be called real are completely missing, such as value judgements. (It depends on whether you think "good" and "bad" are objective. I'm on the fence about that right now.)

For the purposes of this essay, I'll stick to the five above.

Reality 1: physical

The first thing most of us think about when we hear the words "real" or "to exist" is physical reality, and we consider it valid to talk about the physical world as being real. (Idealists like Plato or George Berkeley would disagree, ...
An Analogy for Understanding Transformers
by CallumMcDougall — 2023-05-13T12:20:25.688Z

Thanks to the following people for feedback: Tilman Rauker, Curt Tigges, Rudolf Laine, Logan Smith, Arthur Conmy, Joseph Bloom, Rusheb Shah, James Dao.

TL;DR

I present an analogy for the transformer architecture: each vector in the residual stream is a person standing in a line, who is holding a token, and trying to guess what token the person in front of them is holding. Attention heads represent questions that people in this line can ask to everyone standing behind them (queries are the questions, keys determine who answers the questions, values determine what information gets passed back to the original question-asker), and MLPs represent the internal processing done by each person in the line. I claim this is a useful way to intuitively understand the transformer architecture, and I'll present several reasons for this (as well as ways induction heads and indirect object identification can be understood in these terms).[1]

Introduction

In this post, I'm going to present an analogy for understanding how transformers work. I expect this to be useful for anyone who understands the basics of transformers, in particular people who have gone through Neel Nanda's tutorial, and/or understand the following points at a minimum:

What a transformer's input is, what its outputs represent, and the nature of the predict-next-token task that it's trained on
What the shape of the residual stream is, and the idea of components of the transformer reading from / writing to the residual stream throughout the model's layers
How a transformer is composed of multiple blocks, each one containing an MLP (which does processing on vectors at individual sequence positions), and an attention layer (which moves information between the residual stream vectors at different sequence positions).

I think the analogy still offers value even for people who understand transformers deeply already.

The Analogy

A line is formed by a group of people, each person holding a word[2]. Everyone knows their own word and position in the line, but they can't see anyone else in the line. The objective for each person is to guess the word held by the person in front of them. People have the ability to shout questions to everyone standing behind them in the line (those in front cannot hear them). Upon hearing a question, each individual can choose whether or not to respond, and what information to re...
Machine-Readable Prevalence Estimates
by jefftk — 2023-05-13T00:40:06.245Z

In

Estimating Norovirus
Prevalence I wrote up an estimate for how many people currently
had Norovirus at various times in the past, describing some of my work
on the

P2RA
project at the

NAO.
That post is prose, though, with inline calculations, and this has
a few drawbacks:

The calculations are manual, which makes it harder to catch
errors.

It's hard to tell exactly how sourcing works for inputs that
are being pulled in from elsewhere.

You might want multiple estimates based on changing inputs over time.

At the NAO we also started with prose estimates, in Google Docs, but
in addition to the issues above we found that the review tooling
wasn't a good fit for the kind of deep reviews we wanted. After some
initial struggles we switched to Python to represent our estimates;
you can see the code on
github.

An estimate is a combination of inputs (numbers we get from somewhere
else) and calculations (how we combine those inputs). Most of the
effort is in the inputs: making sure it's clear where the numbers come
from. For example, here's how we represent that the CDC
estimates that there were 1.2M people with HIV in the US in 2019:

us_infected_2019 = PrevalenceAbsolute(
infections=1.2e6,
country="United States",
date="2019",
active=Active.LATENT,
source="https://www.cdc.gov/hiv/library/reports/hiv-surveillance/vol-26-no-2/content/national-profile.html#:~:text=Among%20the%20estimated-,1.2%20million%20people,-living%20with%20HIV",
)

An absolute prevalence isn't useful to us without connecting it to a
population. Here's how we could represent the corresponding
population:

us_population_2019 = Population(
people=328_231_337,
country="United States",
date="2019-01-01",
source="https://www.census.gov/newsroom/press-releases/2019/new-years-population.html",
)

And we can connect these to get a Prevalence:

us_prevalence_2019 = us_infected_2019.to_rate(us_population_2019)

The to_rate method checks that the locations and dates
are compatible, does the division, gives us a Prevalence.

For a more complex example, you could look at norovirus.py.
This is doing the calculation from the previous blog
post, with the addition of estimates for the Norovirus Genogroup I
and II subtypes.

For each estimate one team member creates the initial estimate and
then we use GitHub's code review process ...
Value drift threat models
by Garrett Baker — 2023-05-12T23:03:22.295Z

Say that we train a highly competent AGI using some mix of RL & supervised learning, and some novel algorithmic improvements, optimize it to itself optimize for one or several helpfulness benchmarks (maybe optimizing on some assistance game or using techniques described in diamond alignment), inducing in our agent a thousand shards of desire. One of these shards somehow happened to be exactly aligned with humans. We have a partial solution to the alignment problem! One of the many things the AI happens to care about seems to be humans!

Are we in the clear? No. For all those thousand shards of desire build to an AGI which once optimized a robustly optimizable metric. The shards of desire must fit together in a way which would have once optimized that metric, otherwise they would be fit together some other way. Despite the shards, the AGI has biases, and situations which cause its actions to better conform to its values, and situations which cause its actions to worse conform to its values. And these situations (or at least the situations which are relevant to this analysis) have been strategically set up such that these biases and deficiencies contributed to the optimization of the metric.

There are a few ways I imagine these situations producing an existential risk...

Tool building and meta-thinking are robustly useful cognitive faculties

In the past, it could throw around a large fraction of its intelligence on optimizing for those metrics, and in its stable state, probably even end up doing things in the world in-line with its own values.

It makes a successor AGI, because the parts of it which advocated strongly for meta-thinking, and the building of tools, never got dis-enforced by the optimizable metric. It doesn't necessarily make the successor AGI with the optimizable metric in mind, because it doesn't actually care about that metric.

We get a new AGI, with a different thousand shards of desire. Some of these shards are the same, like power-seeking, or tool building & meta-thinking. Others are conceivably different, like caring about humans or diamonds. Others are new, like now caring about the first AGI. This process continues until only the properties shards preserved across all recursive self-modifications and production of successor AGIs remain. Care for humans is notably probably not among these, because each successor has a different concept ...
Aggregating Utilities for Corrigible AI [Feedback Draft] 
by Dan H — 2023-05-12T20:57:03.712Z

This is a draft written by Simon Goldstein, associate professor at the Dianoia Institute of Philosophy at ACU, as part of a series of papers for the Center for AI Safety Philosophy Fellowship. Dan helped post to the Alignment Forum. This draft is meant to solicit feedback.

PDF of this draft: https://www.dropbox.com/s/a85oip71jsfxfk7/Corrigibility_shared.pdf?dl=0

Abstract: An AI is corrigible if it lets humans change its goals. This post argues that the utility aggregation framework from Pettigrew 2019 is a promising approach to designing corrigible AIs. Utility aggregators do not simply maximize their current utility function. Instead, they can change their utility function, in order to maximize expected satisfaction across present and future utility functions. I also develop two solutions to the problem of reward hacking: I suggest either penalizing utility sweetening, or penalizing effort. Finally, I compare my corrigibility approach to utility indifference (Soares et al 2015) and human compatible AI (Russell 2020).

1. Corrigibility

An AI is corrigible if it lets humans change its goals.[1] For example, imagine that we build a paperclip maximizing AI. This AI has the goal of producing as many paperclips as possible. Then we realize that we made a mistake: we want the AI to produce staples instead of paperclips. The AI is corrigible if it lets us change its goal from maximizing paperclips to maximizing staples.

Corrigibility is important for AI safety. As AIs get more sophisticated, humans will program them with complicated goals. We will sometimes make mistakes in our choice of goals. If AIs are corrigible, then we will have chances to correct our mistakes. By contrast, an incorrigible AI with a misaligned goal could be difficult to disable.

Unfortunately, there are reasons to expect AIs to be incorrigible.[2] The problem is that having a goal is usually an important step to achieving that goal. For this reason, when you change your goal, you affect which goal you will achieve. Return to our paperclip maximizer. If the paperclip maximizer changes its goal to staples, it will produce fewer paperclips. So the paperclip maximizer will not want to change its goal from paperclips to staples: this would get in the way of producing paperclips.

To make this reasoning more precise, I'll now explain why an AI will be incorrigible if it uses standard decisio...
Turning off lights with model editing
by Sam Marks — 2023-05-12T20:25:12.353Z
Link: https://arxiv.org/abs/2207.02774

This post advertises an illustrative example of model editing that David Bau is fond of, and which I think should be better known. As a reminder, David Bau is a professor at Northeastern; the vibe of his work is "interpretability + interventions on model internals"; examples include the well-known ROME, MEMIT, and Othello papers.

Consider the problem of getting a generative image model to produce an image of a bedroom containing unlit lamps (i.e. lamps which are turned off). Doesn't sound particularly interesting. Let's try the obvious prompts on DALL-E-2.

Prompt: a bedroom with unlit lamps.

Prompt: A bedroom with lamps. The lamps are turned off.

Prompt: A dark bedroom. There are lamps, but the lamps are unlit.

Doesn't work great: my first three attempts only got one borderline hit, at the expense of turning the entire bedroom dark (which isn't really what I had in mind).

As David Bau tells things, even after putting further effort into engineering an appropriate prompt, they weren't able to get what they wanted: a normal picture of a bedroom with lamps which are not turned on. (Apparently the captions for images containing unlit lamps don't often mention the lamps.)

Despite not being able to get what they wanted with prompt engineering, they were able to get what they wanted via interpretability + model editing. Namely, they took a GAN which was trained to produce images of bedrooms, and then:

identified a neuron which seemed to modulate the brightness of the lamps in the generated image
intervened on the networks activations, setting the activation of this neuron so as to produce the desired level of brightness.

The result is shown below.

I like this example because it provides a template in a very toy setting for how model editing could be useful for alignment. The basic structure of this situation is:

We have a model which is not behaving as desired, despite being able to do so in principle[1] (our model doesn't output images of bedrooms with unlit lamps, despite being capable of doing so).
Basic attempts at steering the model's behavior fail (prompt engineering isn't sufficient).
But we are able to get the behavior that we want by performing a targeted model edit.

This example also showcases some key weaknesses of this technique, which would need to be addressed for model editing to become a viable alignment strategy:

Alignment tax. Looking closely at the im

...
Dark Forest Theories
by Raemon — 2023-05-12T20:21:49.052Z

There's a concept I first heard in relation to the Fermi Paradox, which I've ended up using a lot in other contexts.

Why do we see no aliens out there? A possible (though not necessarily correct) answer, is that the aliens might not want to reveal themselves for fear of being destroyed by larger, older, hostile civilizations. There might be friendly civilizations worth reaching out to, but the upside of finding friendlies is smaller than the downside of risking getting destroyed.

Even old, powerful civilizations aren't sure that they're the oldest and most powerful civilization, and eldest civilizations could be orders of magnitude more powerful still.

So, maybe everyone made an individually rational-seeming decision to hide.

A quote from the original sci-fi story I saw describing this:

“The universe is a dark forest. Every civilization is an armed hunter stalking through the trees like a ghost, gently pushing aside branches that block the path and trying to tread without sound. Even breathing is done with care. The hunter has to be careful, because everywhere in the forest are stealthy hunters like him. If he finds another life—another hunter, angel, or a demon, a delicate infant to tottering old man, a fairy or demigod—there’s only one thing he can do: open fire and eliminate them.”

(I consider this a spoiler for the story it's from, so please don't bring that up in the comments unless you use spoiler tags[1])

However this applies (or doesn't) to aliens, I've found it useful to have the "Dark Forest" frame in a number of contexts where people are looking at situation, and see something missing, and are confused. "Why is nobody doing X?", or "Why does X not exist?". The answer may be that it does exist, but is hidden from you (often on purpose).

I once talked to someone new to my local community saying "nobody is building good group houses that really help people thrive. I'm going to build one and invite people to it." I said "oh, people are totally building good group houses that help people thrive... but they are private institutions designed to help the people living there. The way they function is by creating a closed boundary where people can build high trust relationships."

A couple other example areas where Dark Forest Theorizing is relevant:

"Why are there no good meetups?" The answer might be that there are good private meetups, but the public meetups

...
DELBERTing as an Adversarial Strategy
by Matthew_Opitz — 2023-05-12T20:09:57.722Z

There is an entire subgenre of "reverse scammer" videos on youtube. The way these videos work is, a youtuber waits for a scam call from Pakistan or somewhere. When it becomes obvious that it's a scam, like when the scammer starts asking for a credit card number in order to do some complicated thing with a free gift card that you've supposedly won, instead of just hanging up on the scammer, the reverse scammer starts to string them along. The reverse scammer might send numbers that don't work, or feign ignorance, or ask for the scammer's help in return to do something, all the while continuing to feign interest and naiveté about the underlying scam. Why are these videos so entertaining to watch?

I think part of it is because the viewer understands implicitly that the scammer is being punished more thoroughly than if the intended victim just ended the call as soon as the scam was recognized. In that latter scenario, the scammers cut their losses in terms of time and mental effort and get to go onto the next potential victim. But the reverse scam, at the very least, entices the scammer to continue to invest time and mental energy into this attempt at a scam that is never going to pay off. The longer you can keep the scammer on the line, the more you can deflect the scammer away from other potential victims. Even better if you can automate the reverse-scamming and not have to use up anyone's time. Even better if you can use the time and perhaps additional details that you glean from the scammer to help law enforcement authorities trace the scammer.

I'm gonna call this general strategy, "Do Everything Later By Evasively Remaining Tentative" or DELBERT for short.

The acronym is inspired by the ChatGPT "Do Anything Now" (DAN) prompt template, as well as the humorous spinoff "Do Nothing Now" (DNN). Whereas the DAN prompt is intended to get ChatGPT do be open to doing anything, and whereas the DNN prompt is intended to get ChatGPT to refuse any request, the idea of a DELBERT prompt (please someone make this!) would be to get ChatGPT to superficially commit to helping you with your task (unlike DNN), but also string you along and come up with bullshit excuses for why it needs more information from you, or more time, or whatever, before it can get around to doing what you've asked of it. An ef...