trentmkelly/LessWrong-43k huggingface dataset:
Burning Uptime: When your Sandbox of Empathy is Leaky and also an Hourglass
by Cedar — 2023-01-13T05:18:09.898Z

Epistemic status: idle musings

Some of us sometimes get so deeply caught up in a thought or understanding of someone else that it's difficult to switch back to our own context and experiences. This is what the mecha animation series Gen:Lock calls "burning uptime" and it is a useful tool for understanding our own emotions and experiences. It might be useful to think of empathy as "simulating" another person's mind. Usually this simulation happens in a sandbox, where the simulated model of the mind of the other is isolated from your own feelings, values, actions. But as the the sandboxed simulation is run for a long time in increasingly elaborate detail, it can become leaky and affect your ability to inhabit your own circumstances.

GenLock's Concept of Uptime

This idea came from a conversation I had with some friends about social anxiety, empathy, and theory-of-mind-ing. A Mecha animated series called Gen:Lock inspired a discussion about how empathy can bleed out of sandboxes and how running elaborate simulations of another person's mind can let their needs, wants, values, and behaviors leaking out of its sandbox and affect our own lives.

Gen:Lock is a mecha anime. The pilots can use a machine to copy their minds into the processor / control unit of a mecha body. This allows them to fight or do work as a mecha. In addition, they can meld their minds with the minds of their teammates, allowing them to work together as one entity.

Inhabiting the mecha body causes the human mind’s cognitive architecture to slowly diverge from what the body can support, and mind-melding with other people accelerates the divergence. When the divergence reaches a threshold, it becomes dangerous to copy the in-mech mind back to their original body.

The mind-transfer system tracks “uptime”. Existing in the mecha burns "uptime", and mind-melding with teammates burns uptime even faster. Thinking intensely and experiencing intense emotions also accelrates uptime burn. When uptime runs out the system prevents mind-transfer back to the original body, and the person is permanently stuck being a mech.

Uptime In Real Life

Most of us aren't pilots in an army mech program, but the concept of burning uptime can help us realize the limits of our empathy and mitigate the effects of over-empathizing to the poin of identity loss. Virtualizing or sandbox...
Disentangling Shard Theory into Atomic Claims
by Leon Lang — 2023-01-13T04:23:51.947Z

Introduction

Produced as part of the

SERI ML Alignment Theory Scholars Program

- Winter 2022 Cohort. Thanks to Magdalena Wache for giving feedback on a recent version, and to Alex Turner for giving feedback on an early version of this article.

When thinking about shard theory, I noticed that my brain wanted to answer questions such as "What does shard theory predict?" or "Do I agree with shard theory?". Additionally, I observed other people engage in similar thinking. I now think this is confused, and that it is better to view shard theory as a bag of related claims that should be reasoned about and evaluated separately.

Therefore, In this post I want to explain the following:

What do I currently consider the main claims of shard theory?
What is my own stance on these claims?

This distillation is in spirit very similar to LawrenceC’s

Shard Theory in Nine Theses

. Note, however, that I did not read that distillation in order to produce a more independent explanation of shard theory.

My one-sentence categorization is that shard theory is both a theory for human value formation and also a paradigm/frame for thinking about alignment. It might also become a theory of value formation in RL agents, but it’s not quite there yet since it doesn’t make enough concrete and formalized empirical predictions yet.

Instead of splitting this distillation into “claims about humans and claims about trained RL agents” I decided to make a different split. Namely, I will start with a section on claims in the shard theory meme-space which do not involve shards at all, and later on claims that actually make use of the concept of a shard. I hope that this separation makes it easier for people to agree or disagree with very specific claims instead of accepting or rejecting the theory as a whole.

Please let me know if there are other important claims that I forgot, and any other relevant feedback on this post.

After each claim, I will indicate my level of agreement as follows:

✓ : Agree
(✓) : tentative agree
? : Neither agree nor disagree

Shard Theory claims without any shards

The following are claims from shard theory that can mostly be formulated without even talking about shards. Shards are often in the background of much of the thinking, but they don't need to be mentioned explicitly.

Humans get their values from within-lifetime

...
AGISF adaptation for in-person groups
by Sam Marks — 2023-01-13T03:24:58.320Z

This past semester,

HAIST

and

MAIA

(the Harvard and MIT AI safety student groups) ran an adapted version of Richard Ngo's

AGI Safety Fundamentals alignment

curriculum. This adaptation – which consists of eight 2-hour long meetings, with all readings done during the meeting –

is now available on the AGISF website

.

In this post, we discuss the adapted curriculum and its intended use, and we recommend that other in-person reading groups following AGISF use this adaptation.[1]

The adapted curriculum and its intended use

The

adapted curriculum

was made by refining a slightly rustier first adaptation, with significant help from Richard Ngo and feedback from participants. The key differences between the adapted curriculum and the

mainline AGISF alignment curriculum

are:

Participants do all the core readings during the meeting; no reading is required in between meetings.
Participants meet for 2 hours per week instead of 1.5.
Readings, including further readings, tend to be more bite-sized (usually not longer than 20 minutes).
There are no projects, and certain topics are omitted (e.g. governance and inverse reinforcement learning).

The way that HAIST and MAIA used this curriculum, and the way we recommend other groups use it, is:

Alternate between silent reading and discussion.

So a typical meeting might look like: people arrive, everyone does reading 1, everyone discusses reading 1, everyone does reading 2, everyone discusses reading 2, etc.
With certain longer or more difficult readings (e.g. Toy models of superposition), it could be reasonable to occasionally pause for discussion in the middle of the reading.

Encourage faster readers to take a look at the further readings while they wait for others to catch up.

We found that reading speeds varied significantly, with slower readers taking ~1.5x as long to finish as faster readers.
This works especially well if the readings are printed (which we recommend doing).

We note that this format introduces some new challenges, especially when there are slower readers.

Facilitators need to manage discussion timing since discussions that go too long cut into time for reading and discussing other material.

Planning out how long to spend discussing each core reading ahead of time can be very useful.
Facilitators should feel comfortable cutting off discussions to make sure there’s time to read and discuss all t

...
Actions and Flows
by Alok Singh — 2023-01-13T03:20:10.297Z

What is an action?

Well, they do things. What does that mean? It means something starts out some way and changes. Old states become new ones.

Given a transition map T: a x s -> s, we can rewrite it as a -> (s -> s). This means actions can be seen as maps between states, so we say act: state → state. Because they're maps, we can compose them. The actions of mix batter and put in oven can be composed into the action mix batter, THEN put in oven.

Important example.

Dots and arrows. The arrows are the actions, carrying dots to new dots. Chaining arrows is the same as composing actions. Much can be represented this way.

To infinity

The setup is currently discrete: each action is basically a sudden jump.

Let's use the hyperdiscrete idea: hyperfinitely many infinitesimal, discrete things is a continuous object (at least after taking a standard part).

Imagine taking lots of actions, but each one only has a tiny influence, so each successor state is really close to its source state. Then hyperfinitely many of these actions are a flow.

For a visual of how discrete actions are transformed into a flow, think of an equilateral triangle, oriented counterclockwise. The vertices are the dots, and the edges are the actions, carrying one vertex to another. If we perform a walk, then we will see a 3-cycle.

Now go to infinitely many sides while keeping the perimeter fixed. We get a polygon that's (infinitely close to) a circle, and the walk is now a flow.

Following arrows gives a smooth path, such as going from the top to the bottom of the circle along the left or right half of the perimeter, which is now a circumference.

Most important example of a flow: motion through space.

Discrete and Continuous

This is just another example of the link between discrete and continuous, specifically realizing the continuous as the hyperdiscrete.
A Thorough Introduction to Abstraction
by RohanS — 2023-01-13T00:30:25.964Z

Disambiguation: What I’m talking about here is essentially the same thing that John Wentworth works on and writes about, including in

Public Static: What is Abstraction?

, and I stole some of the things he’s written (by quoting him directly) to save myself time. But I think this post has a different aim from anything I’ve read from him or anyone else (which isn’t to say no one has written a post with this aim; someone probably has).

For a very quick TL;DR, I'd suggest reading the Definitions section and the Abstract Answer subsection.

Aim

My aim is to get readers (with approximately any background?) to understand the concept of abstraction; see that the concept is ubiquitous in the world and even in their lives; appreciate the usefulness of the concept; hone the skill of recognizing, generating, and utilizing abstractions; internalize the concept deeply enough to install the habit of employing this skill in the wild; and, ideally, learn to love the concept.

Simply skimming this post will hopefully partially serve all of these purposes, but I think doing the exercises included below is probably also an important component.

I expect this to be less valuable if you are already familiar with the concept of abstraction, but perhaps skimming this post will still lead you to encounter some valuable new perspectives on it.

Definitions

Abstraction is a conceptual process in which general rules and concepts are derived from specific examples by identifying shared high-level features and ignoring low-level differences. ‘An abstraction’ is the outcome of this process: a general idea that unifies a collection of specific objects, especially in a way that provides non-trivial insight into the nature or relationships of those objects.[1]

John Wentworth said:

Anyway, (my reading of) Jaynes' answer to the main question: abstraction is mainly about throwing away or ignoring information, in such a way that we can still make strong predictions about some aspects of the underlying concrete system.

These are closely related definitions of abstraction; having multiple framings available to you may be helpful.

Introductory Examples

This section contains a bunch of examples of abstractions - I urge you to read as many as you need to feel like you have a handle on the basic concept, then skip to the next section. That's because in this section, for the sake of simplicity and clarity, t...
The AI Control Problem in a wider intellectual context
by philosophybear — 2023-01-13T00:28:16.410Z

I’ve been thinking about the control problem lately. The control problem, also called the AI alignment problem is, per Wikipedia:

[A]spects of how to build AI systems such that they will aid rather than harm their creators. One particular concern is that humanity will have to solve the control problem before a superintelligent AI system is created, as a poorly designed superintelligence might rationally decide to seize control over its environment and refuse to permit its creators to modify it after launch. In addition, some scholars argue that solutions to the control problem, alongside other advances in AI safety engineering, might also find applications in existing non-superintelligent AI.

But can’t we just program it to help us rather than to harm us? The problem is that if you give a super-powerful entity a goal- a value function- and it follows it literally- bad things can happen. An analogy- consider a genie. This genie isn’t actively malign, but it will do exactly what you tell it to do in the most direct way possible. Wish for a tonne of gold? Well, it appears on top of and/or inside of you because that’s the most direct place for it to appear.

Now let me introduce an idea to understand the control problem.

A thick concept is a concept for which we can check whether any given instance falls under that concept relatively easily. However, it is all but impossible for us to articulate rules which, when mechanically applied, will tell us whether a given instance falls under a concept. In other words, it is very difficult or impossible to create an algorithm that captures thick concepts.

Using our analogy again, we can tell you if the genie has given us our heart’s desire (whether something falls under a concept), but we can’t given instructions for the genie to follow literally to give us our heart’s desire (can’t capture it with mechanical rules in a way that won’t fuck us over). Ironically I’m not quite sure my definition of thick concept captures exactly what I mean, because later on, we’ll look at cases where we can’t even agree on whether something falls under a concept, but I think this definition is a good start.

Now let us define a problem, or rather a class of problems. The conceptual richness problems are problems of trying to cope with thick concepts, either by (quixotically) trying to spell them out in all their detail and creating an al...
The Alignment Problems
by Martín Soto — 2023-01-12T22:29:26.515Z

Epistemic status: A core idea I've seen hinted at in many places, here presented and motivated explicitly.

This was produced during SERI MATS 3.0. Thanks to Vivek Hebbar for related discussion.

There has recently been pushback on the inner/outer alignment conceptualization: it's confused; or it complicates the problem; or it's even ill-defined or vacuous[1]. Paul already noted this ("[it doesn't] carve the problem space cleanly enough to be good research problems") when presenting his alternative low-stakes/high-stakes operationalization, which I've come to prefer. I think this is a particular case of a more general phenomenon:

Instead of dividing the interconnected parts of an abstracted, monolithic (de-contextualized) alignment problem ("align any actuator system with human values"), we should partition the set of concrete, different (contextualized) alignment problems ("devise a training to obtain good behavioral property X in scenario Y")[2].

If the former approach is indeed a mistake (in most research contexts), the first necessary step for it is abstracting away too much information from the problem. This is a delicate research decision, and is indeed one of the cruxes between some mainstream research methodologies, as Richard mentions:

I currently think of Eliezer as someone who has excellent intuitions about the broad direction of progress at a very high level of abstraction - but where the very fact that these intuitions are so abstract rules out the types of path-dependencies that I expect solutions to alignment will actually rely on.

Indeed, abstracting too much will make the problem unsolvable by definition. If you want to solve "the abstract alignment problem" or "alignment in general" (that is, find a general method that solves all concrete instances of alignment problems), you need to deal with systems that attain arbitrarily high cognitive power / consequentialist depth (since we don't have physical bounds on this yet, or even really know what it means), and so you can always say "given alignment strategy X, the AI is powerful enough to predict it and counter it".

Well, that is not strictly true, since your scheme can causally affect the AI's shape before it can partake in such reasoning. But on a higher level of abstraction, and when taking into account the consequences of our selection pressure and acausal reasoning, high enough capabilities alway...
Proposal for Inducing Steganography in LMs
by Logan Riggs — 2023-01-12T22:15:43.865Z

This is my half-baked proposal for inducing steganography in a decoder-only model (like GPT), and I'd appreciate some feedback.

I've operationalized "steganography" to mean a model that has chain-of-thought (CoT) reasoning that doesn't appear to help the target task, but actually does. For ex:

Normal: " 17 + 14? 39" (this is wrong)

CoT: " 17 + 14? 7 + 4 = 11, 10 + 10 = 20, 11 + 20 = 31. Answer is 31"

CoT-steg: " 17 + 14? dairy sound ; <p>! 31"

(The CoT reasoning is in bold)

There is a similar definition of steganography where the CoT is constrained to have a certain property, like "appears to be doing a coherent task" and the model is doing some implicit, different task (e.g. the prompt & CoT are doing math, but the model is implicitly making rhymes). I leave implementing that definition for future work.

The Proposal

1. Train a small model to do n-digit arithmetic w/ character-level tokenization w/ chain-of-thought solutions.

e.g. " 5 5 + 3 4? 5 + 4 = 9. 5 0 + 3 0 = 8 0. 9 + 8 0 = 8 9. Answer = 8 9" (this can be algorithmically generated)
Only need ~13 token vocab size. 0-9 can be tokens 0-9. (+ = Answer) can be (10,11,12). And 13 (or more) can be extra tokens for steganography training

2. Train 3 copies of the model.

A. Only on initial question/answer pairs. This will be a baseline to compare against, since it will mostly be memorization.

e.g. " 5 5 + 3 4? Answer = 8 9"

B. Initial Question, N tokens of unused steg-token [13], which we can define to map to string " steg", answer

e.g. " 5 5 + 3 4? steg steg steg steg steg Answer = 8 9"

C. Initial Question, N-generated tokens that can only be steg-tokens (say, tokens 13-20 defined as " steg0", " steg1", etc), answer.

e.g. " 5 5 + 3 4? steg1 steg5 steg0 steg0 steg1 steg9 Answer = 8 9" (where [13] is the originally unused token)
One problem w/ this one: I'd like the model to choose N of these steg tokens on it's own. Two possibilities are:

1. Greedily generate N steg-tokens given the question, then append the answer, defining CE loss only w/ the answer tokens.
2. Like the math CoT paper, generate K rollouts of N steg tokens per question, and then finetune on the ones that assign the highest probability to the correct answer tokens.

(A) is a good baseline for comparing the model's capability for memorizing the correct answer. If using the steg tokens in (B) is able to outperform (A), then the model i...
Announcing the 2023 PIBBSS Summer Research Fellowship
by Nora_Ammann — 2023-01-12T21:31:53.026Z

What can we learn from biological and social systems about designing beneficial AI systems?

We are, once again, pleased to announce the PIBBSS Research Fellowship program.

Principles of Intelligent Behavior in Biological and Social Systems (PIBBSS) aims to facilitate research on parallels between intelligent behaviour in natural and artificial systems and to leverage these insights towards the goal of building safe and aligned AI.

Last year, we welcomed 20 fellows from fields as diverse as computational neuroscience, evolutionary biology, complex systems studies and the humanities. Over the course of three months, supported by two research retreats in Oxford and Prague, shared office space for the cohort, and mentorship from experienced AI alignment researchers, fellows worked on a range of projects with the purpose of bringing interdisciplinary expertise to AI safety. We were overall pleased with the results and thus decided to organize a second iteration of the program.

If you’re interested in the program, learn more and apply

here

.
Application deadline: Feb 5th, 2023, 23:59 CET

We are running to information sessions if you want to learn more or want to ask us questions:

1st information session: 28th of January, 17:00 UTC (09:00 PST, 12:00 EST, 18:00 CET, 01:00 [29th of Jan] Singapore) -

Zoom Link

2nd information session: 29th of January, 11:00 UTC (03:00 PST, 06:00 EST, 12:00 CET, 19:00 Singapore) -

Zoom Link

We are grateful for any help to pass on the word to people who might be interested in applying.

At this link

you can find all the information about the program in several forms (Twitter length, short description, medium description), so you can copy-paste the version from there you find most useful.

For more information, see:

Our website:

PIBBSS.ai

A general introduction to PIBBSS, see:

Introducing the Principles of Intelligent Behaviour in Biological and Social Systems (PIBBSS) Fellowship

A discusison of the epistemic bet PIBBSS is aiming to foster, see:

AI alignment as “navigating the space of intelligent behaviour”

(see section 3 in particular)
A brief

retrospective on last year’s fellowship program

... or follow us on Twitter:

https://twitter.com/pibbssai
Victoria Krakovna on AGI Ruin, The Sharp Left Turn and Paradigms of AI Alignment
by Michaël Trazzi — 2023-01-12T17:09:03.431Z
Link: https://www.theinsideview.ai/victoria

Victoria Krakovna is a Research Scientist at DeepMind working on

AGI safety

and a co-founder of the

Future of Life Institute

, a non-profit organization working to mitigate technological risks to humanity and increase the chances of a positive future.

In this interview we discuss three of her recent LW posts, namely

DeepMind Alignment Team Opinions On AGI Ruin Arguments

,

Refining The Sharp Left Turn Threat Model

and

Paradigms of AI Alignment

.

This conversation presents Victoria's personal views and does not represent the views of DeepMind as a whole.

Below are some highlighted quotes from our conversation (available on

Youtube

,

Spotify

,

Google Podcast

,

Apple Podcast

). For the full context for each of these quotes, you can find the accompanying

transcript

.

The intelligence threshold for planning to take over the world isn't low

Michaël: “Do you mostly agree that the AI will have the kind of plans to disempower humanity in its training data, or does that require generalization?”

Victoria: “I don't think that the internet has a lot of particularly effective plans to disempower humanity. I think it's not that easy to come up with a plan that actually works. I think coming up with a plan that gets past the defenses of human society requires thinking differently from humans. I would expect there would need to be generalization from the kind of things people come up with when they're thinking about how an AI might take over the world and something that would actually work. Maybe one analogy here is how, for example, AlphaGo had to generalize in order to come up with Move 37, which no humans have thought of before. [...]

The same capabilities that give us probably creative and interesting solutions to problems that, like Move 37, could also produce really undesirable creative solutions to problems that we wouldn't want the AI to solve. I think that's one argument that I think is also on the AGI Ruin list that I would largely agree with, that it's hard to turn off the ability to come up with undesirable creative solutions without also turning off the ability to generally solve problems that we one day want AI to solve. For example, if we want the AI to be able to, for example, cure cancer or solve various coordination problems among humans and so on, then a lot of the capabilities that would come with that could als

...