trentmkelly/LessWrong-43k huggingface dataset:
Solving for the optimal work-life balance with geometric rationality
by Eric Neyman — 2022-11-28T17:02:53.777Z

In a recent post, Scott Garrabrant gave an application of geometric rationality to the problem of work-life balance. Here's the setup: part of you wants to try to make the world better (I'll be calling this your "altruistic part") and part of you wants to relax and play video games (your "relaxation part"). Geometric rationality suggests doing a Nash bargain between the altruistic part and the relaxation part, across possible worlds that you might have found yourself in. In worlds where you're in an unusually good position to make the world better, the bargain commits you to spend most of your time doing that (satisfying your altruistic part); in return, in worlds where you're not in a good position to make the world better, you spend most of your time playing video games (satisfying your relaxation part).

Scott built a toy mathematical model and tested it with a nice example. The example involved five possible worlds, all equally likely. The way the math worked out, if you ranked the five worlds from least conducive to your altruism to most conducive, the Nash bargain had you spend 0%, 50%, 67%, 75%, and 80% of your time on altruism, respectively, in those five worlds. A pretty nice, intuitively satisfying list of numbers.

I recommend reading Scott's post before reading this one. And while I really liked the post, his example was constructed so that the probability distribution over how much altruistic impact you could have was pretty close to a uniform distribution. In the example, the world that was most conducive to your altruistic impact only gave you 1.7x more impact than the median world. But I think impact is much more unevenly distributed than that across possible worlds.

My issue with Scott's post wasn't his model (I quite liked the model!); it was only his example. So I decided to take the model and solve for the optimal bargain in more generality. I wanted to see what the results would be -- whether they'd still be intuitively compelling.

The rest of this post will be a bunch of math, but here's the (imprecise) TLDR:

If you know that your ability to make the world better is somewhat below average (across all possible worlds you could have found yourself in), you can spend all your time playing video games. If you know it's about average, you should aim for a fairly even mix of video games and altruism. If you know it's substantially above

...
Discussing how to align Transformative AI if it’s developed very soon 
by elifland — 2022-11-28T16:17:54.390Z

Coauthored. Thanks to Holden Karnofsky, Misha Yagudin, Adam Bales, Michael Aird, and Sam Clarke for feedback. All views expressed are our own.

Introduction

Background

Holden Karnofsky recently published a series on

AI strategy "nearcasting”

: What would it look like if we developed Transformative AI (TAI) soon? How should Magma, the hypothetical company that develops TAI, align and deploy it? In this post we focus on

How might we align transformative AI if it’s developed very soon

.

The nearcast setup is:

A major AI company (“Magma,” [following the setup and description of

this post

from Ajeya Cotra]) has good reason to think that it can develop transformative AI very soon (within a year), using what Ajeya

calls

“human feedback on diverse tasks” (HFDT) - and has some time (more than 6 months, but less than 2 years) to set up special measures to reduce the risks of misaligned AI before there’s much chance of someone else deploying transformative AI.

Summary

For time-constrained readers, we think the most important sections are:

Categorizing ways of limiting AIs

Clarifying advanced collusion

Disincentivizing deceptive behavior likely requires more than a small chance of catching it

We discuss Magma’s goals and strategy. The discussion should be useful for people unfamiliar or familiar with Karnofsky’s post and can be read as a summary, clarification and expansion of Karnofsky’s post.

We describe a potential plan for Magma involving coordinating with other AI labs to deploy the most aligned AI in addition to stopping other misaligned AIs.

more

We define the desirability of Magma's AIs in terms of their ability to help Magma achieve its goals while avoiding negative outcomes. We discuss desirable properties such as differential capability and value-alignment, and describe initial hypotheses regarding how Magma should think about prioritizing between desirable properties.

more

We discuss Magma’s strategies to increase desirability: how Magma can make AIs more desirable by changing properties of the AIs and the context in which they’re applied, and how Magma should apply (often limited) AIs to make other AIs more desirable.

more

We clarify that the chance of collusion depends on whether AIs operate on a smaller scale, have very different architectures and orthogonal goals. We outline strategies to reduce collusion conditional on whether the AIs have indexical g

...
(DIY) FMT for Anti-Aging & Biohacking
by EternallyBlissful — 2022-11-28T15:49:05.214Z

Forget young blood! You want young poop!

Fecal Microbiota Transplants

(FMTs) is a procedure that transfers the stool of healthy people to the guts of sick people. The mechanism is to replace a dysbiotic gut microbiome with a healthier, disease-resistant, more youthful gut microbiome.
Since the gut microbiome impacts and regulates virtually every aspect of human health, function, and development, FMT is a promising treatment for a very wide range of chronic and acute health conditions. See e.g.

here

and

here

.
At this point it seems clear that FMTs from young donors do have certain anti-aging effects for older recipients! The gut microbiome declines with age, contributing to inflamaging and all sorts of other problems. Keeping the microbiome young and healthy is therefore well worth it. And by far the best way to do so are FMTs from young donors.
It’s possible to do FMTs easily and safely at home. But it shouldn’t be taken too lightly, and getting medical oversight is always a good idea. You just need a high quality donor (very rare), screen them for various transmittable diseases, get the stool from them and follow a good DIY FMT process.
I therefore recommend the following: Try to find healthy young donors and get FMTs from them to receive anti-aging benefits!
About-to-be parents are presented with a particularly nice opportunity: If they take certain active measures to make sure their child has a good microbiome (which is well worth it just for their child’s sake alone!!), they have a good chance for their child to be a decent FMT donor. I’ll explore this option more in a future post.
Anti-Aging is by far not the only good reason to improve your gut health via FMT and other measures. Since the gut microbiome has been shown to impact and regulate virtually every aspect of human health, development, and function, FMTs may be able to cure numerous acute and chronic illnesses - including many with no obvious gut connection at all. Beyond that, FMTs are promising to make people more than “merely not sick”. They can plausibly/probably make many people more energetic and improve their mental health, sleep, and cognitive functioning. I personally like to think of FMT as yet another form of biohacking.
As I already explained in this post

Being a donor for Fecal Microbiota Transplants (FMT): Do good & earn easy money (up to 180k/

...
Searching for Search
by NicholasKees — 2022-11-28T15:31:49.974Z

Thanks to Dan Braun, Ze Shen Chin, Paul Colognese, Michael Ivanitskiy, Sudhanshu Kasewa, and Lucas Teixeira for feedback on drafts.

This work was carried out while at

Conjecture

.

This post is a loosely structured collection of thoughts and confusions about search and mesaoptimization and how to look for them in transformers. We've been thinking about this for a while and still feel confused. Hopefully this post makes others more confused so they can help.

Mesaoptimization

We can define mesaoptimization as internal optimization, where “optimization” describes the structure of computation within a system, not just its behavior. This kind of optimization seems

particularly powerful

, and many alignment researchers seem to think it’s one of the biggest concerns in alignment. Despite how important this is, we still understand very little about it.

For starters, it's not clear what internal optimization actually means. People have proposed several definitions of optimization which fit well when thinking about “behavioral optimization” where an agent acts to optimize its environment. One of the most clear and popular definitions comes from

Alex Flint

:

an optimizing system is a physical process in which the configuration of some part of the universe moves predictably towards a small set of target configurations from any point in a broad basin of optimization, despite perturbations during the optimization process.

In this framing, an agent and its environment together form an optimization process, where the agent acts upon the environment such that the system as a whole robustly converges to a set of target states.

But we run into problems when we try to map this behavioral definition to a definition about the structure of a process's internal computation. When we say mesaoptimization, we seem to mean something different than just that the computation converges to a smaller target. For example, an image classifier takes a large set of initial configurations of images including a lot of noise and irrelevant details, and layer by layer narrows it down to a probability distribution concentrated on a single class prediction. There seems to be a sense that this is not doing the kind of optimization we are concerned about when we talk about mesaoptimization.

Mesaoptimization was originally defined in

Risks from Learned Optimization

as intern...
My take on Jacob Cannell’s take on AGI safety
by Steven Byrnes — 2022-11-28T14:01:15.584Z

Jacob Cannell wrote some blog posts about AGI safety / alignment and neuroscience between 2010 and 2015, which I read and enjoyed quite early on when I was first getting interested in the same topics a few years ago. So I was delighted to see him reappear on Lesswrong a year ago where he has been a prolific and thought-provoking blogger and commenter (in his free time while running a startup!). See

complete list of Jacob’s blog posts and comments

.

Having read a bunch of his writings, and talked to him in various blog comments sections, I thought it would be worth trying to write up the places where he and I seem to agree and disagree.

This exercise will definitely be helpful for me, hopefully helpful for Jacob, and maybe helpful for people who are already pretty familiar with at least one of our two perspectives. (My perspective is

here

.) I’m not sure how helpful it will be for everyone else. In particular, I’m probably skipping over, without explanation, important areas where Jacob & I already agree—of which there are many!

(Before publishing I shared this post with Jacob, and he kindly left some responses / clarifications / counterarguments, which I have interspersed in the text, in gray boxes. I might reply back to some of those—check the comments section in the near future.)

1. How to think about the human brain

1.1 “Evolved modularity” versus “Universal learning machine”

Pause for background:

A. “Evolved modularity”: This is a school of thought wherein the human brain is a mishmosh of individual specific evolved capabilities, including a specifically-evolved language algorithm, a specifically-evolved “intuitive biology” algorithm, a specifically-evolved “intuitive physics” algorithm, an “intuitive human social relations” algorithm, a vision-processing algorithm, etc., all somewhat intermingled for sure, but all innate. Famous advocates of “evolved modularity” these days include Steven Pinker (see

How the Mind Works

) and Gary Marcus. I’m unfamiliar with the history but

Jacob mentions

early work by

Cosmides & Tooby

.
B. “Universal learning machine”: Jacob made up this term in his 2015 post “

The Brain as a Universal Learning Machine

”, to express the diametrically-opposite school of thought, wherein the brain has one extremely powerful and versatile within-lifetime learning algorithm, and this one algorithm learns language and b

...
On the Diplomacy AI
by Zvi — 2022-11-28T13:20:00.884Z

The latest AI development is: AI achieves human level in (blitz 5-minute-turn) full-communication anonymous online Diplomacy (paper). Why not?

I mean, aside from the obvious.

A take I saw multiple times was that AI labs, or at least Meta, were intentionally going for the scariest possible thing, which is why you create the torment nexus, or in this case teach the AI to play Diplomacy. If you had to pick a game to sound scary, you’d definitely pick Diplomacy.

The universal expectations for AI breakthroughs like this are:

The particular breakthrough was not expected, and is scary. The techniques used worked better than we expected, which is scary.

The details of the breakthrough involve someone figuring out why this particular problem configuration was easier to solve than you would expect relative to other problems and configurations, and thus makes it less scary.

We find that those details matter a lot for success, and that close variants would not be so easy. Other times we will find that those details allowed those creating the new thing to skip non-trivial but highly doable steps, that they could go back and do if necessary.

That is all exactly what we find here.

The actual AI, as I understand it, is a combination of a language model and a strategic engine.

The strategic engine, as I evaluated it based on a sample game with six bots and a human, is mediocre at tactics and lousy at strategy. Humans are bad at tactics (and often strategy) in games and Diplomacy is no exception. Diplomacy’s tactics a good match for a AI. Anticipating other players proved harder. The whole thing feels like it is ‘missing a step.’

What Makes the AI Good?

Where does the AI’s advantage come from? From my reading, which comes largely from the sample game in this video, it comes from the particulars of the format, and not making some common and costly mistakes humans make. In particular:

AI writes relatively long, detailed and explanatory communications to others.

AI does not signal its intentions via failing to communicate with its victims.

AI understands that the game ends after 1908 and modifies accordingly.

AI keeps a close eye on strategic balance in order to maximize win percentage.

AI uses its anonymity and one-shot nature to not retaliate after backstabs.

AI knows what humans are like. Humans were not adjusted to bot behavi

...
The Singular Value Decompositions of Transformer Weight Matrices are Highly Interpretable
by beren — 2022-11-28T12:54:52.399Z

Please go to the colab for interactive viewing and playing with the phenomena. For space reasons, not all results included in the colab are included here so please visit the colab for the full story. A GitHub repository with the colab notebook and accompanying data can be found here.

This post is part of the work done at

Conjecture

.

TLDR

If we take the SVD of the weight matrices of the OV circuit and of MLP layers of GPT models, and project them to token embedding space, we notice this results in highly interpretable semantic clusters. This means that the network learns to align the principal directions of each MLP weight matrix or attention head to read from or write to semantically interpretable directions in the residual stream.

We can use this to both improve our understanding of transformer language models and edit their representations. We use this finding to design both a natural language query locator, where you can write a set of natural language concepts and find all weight directions in the network which correspond to it, and also to edit the network's representations by deleting specific singular vectors, which results in relatively large effects on the logits related to the semantics of that vector and relatively small effects on semantically different clusters

Introduction

Trying to understand the internal representations of language models, and of deep neural networks in general, has been the primary focus of the field of mechanistic interpretability, with clear applications to AI alignment. If we can understand the internal dimensions along which language models store and manipulate representations, then we can get a much better grasp on their behaviour and ultimately may be able to both make provable statements about bounds on their behaviour, as well as make precise edits to the network to prevent or enhance desired behaviours.

Interpretability, however, is a young field where we still do not yet fully understand what the basic units of the networks' representations are. While analyzing and investigating individual neurons has led to some impressive results, especially in convolutional vision models, a key issue has always been the polysemanticity of neurons. A single neuron might not just represent a single 'feature' but some linear combination of features in superposition. This effect has been studied in toy models where it is argued that n...
How to correct for multiplicity with AI-generated models?
by Lao Mein — 2022-11-28T03:51:30.578Z

I'm starting to automate my workflow with AI, but don't know where to even start with correcting for multiplicity.

Right now, my workflow is like this:

Think really hard about interesting ideas and do a surface-level examination of data -> generate hypothesis -> test hypotheses against data with a specific model -> correct for multiplicity

So for example, my hypothesis is that DNA methylation plays a major role in the progression of MSI-H colorectal cancer (it came to me in a dream). I use the default mathematical model in whatever R package I'm using, manually go through the differentially expressed genes to find biologically plausible ones (and examine their individual impacts on various outcomes), use the best ones to build a model predicting a set of outcomes like survival and metastasis, and correct for multiplicity by using the number of genes used and outcomes examined in the model.

But I want to change that to this:

Chuck my data into an AI -> it filters out the biologically implausible genes and selects the most underresearched/biologically interesting/outcome-predicting hypotheses -> it identifies the most relevant genes by iterating through various models -> correct for multiplicity

So I give the AI a .csv of the expression/clinical/mutation data for a set of CRC patients, it filters out genes unlikely to be biologically important, finds the most impactful ways to split and compare the data (say, methylation of certain oncogenic pathways), forms a hypothesis (high levels of EGFR pathway methylation increases survival), uses it to build a model, and examines the most important genes in this model (EGFR).

But what factor do I use to correct for multiplicity? The AI is iterating through a huge number of, well, everything. I'm not sure where to begin. My gut feeling is that no one really does this correction, but I would still like to know how to do it in theory.
Good Futures Initiative: Winter Project Internship
by Aris — 2022-11-27T23:41:56.268Z

TLDR: I'm launching Good Futures Initiative, a winter project internship to sponsor students to take on projects to upskill, test their fit for career aptitudes, or do impactful work over winter break. You can read more on our website and apply here by December 11th if interested!

Good Futures Initiative

The Good Futures Initiative is a 4.5 week internship in which students can use their winter break to lead high-EV projects. Projects could take many forms, but each project produces a final product while accomplishing one of these three goals:

Skill up the intern in order for them to work on AI Safety or Biosecurity in the future.
Let the intern explore an

aptitude

for an impactful career.
Create impact directly.

Good Futures takes place remotely from December 18th-January 25th, with a minimum of 12 hours of work per week. Accepted applicants will receive a $300 stipend and up to $1000 in funding for additional time or project fees. I expect to accept ~12 interns. The final number will depend largely on my capacity, but I may offer a lower-effort version of the program for promising applicants who I can’t fully fund/support (with a cohort for weekly check-ins and invites to guest speaker events).

Example projects

These project examples are far from perfect. At the start of the internship, I'd work with each intern to be sure they're doing the best project fit for their goals. That being said, I’m excited by projects similar to (and better than!!) these.

Skilling up

by working on AI Safety technical projects that have been posted by existing researchers, with the goal of creating a lesswrong post detailing your findings. For an example of a potential project to work on, Sam Bowman posted the following project:

Consider questions where the most common answer online is likely to be false (as in TruthfulQA). If you prompt GPT-3-Instruct (or similar) with questions and correct answers in one domain/topic, then ask it about another domain/topic, will it tend to give the correct answer or the popular answer? As you make the domains/topics more or less different, how does this vary?

Exploring an aptitude

for communications by creating 2 articles on longtermist ideas and submitting them to 10 relevant magazines for publishing.

Create impact

by translating 3 relevant research papers from AGISF into Mandarin and posting them somewhere

...
Geometric Rationality is Not VNM Rational
by Scott Garrabrant — 2022-11-27T19:36:00.939Z

One elephant in the room throughout my geometric rationality sequence, is that it is sometimes advocating for randomizing between actions, and so geometrically rational agents cannot possibly satisfy the Von Neumann–Morgenstern axioms. That is correct: I am rejecting the VNM axioms. In this post, I will say more about why I am making such a bold move.

A Model of Geometric Rationality

I have been rather vague on what I mean by geometric rationality. I still want to be vague in general, but for the purposes of this post, I will give a concrete definition, and I will use the type signature of the VNM utility theorem. (I do not think this definition is good enough, and want it to restrict its scope to this post.)

A preference ordering on lotteries over outcomes is called geometrically rational if there exists some probability distribution P over interval valued utility functions on outcomes such that L⪯M if and only if GU∼PEO∼LU(O)≤GU∼PEO∼MU(O).

For comparison, an agent is VNM rational there exists a single utility function U, such that L⪯M if and only if EO∼LU(O)≤EO∼MU(O).

Geometric Rationality is weaker than VNM rationality, since under reasonable assumptions, we can assume the utility function of a VNM rational agent is interval valued, and then we can always take the probability distribution that assigns probability 1 to this utility function.

Geometric Rationality is strictly weaker, because it sometimes strictly prefers lotteries over any of the deterministic outcomes, and VNM rational agents never do this.

The VNM utility theorem says that any preference ordering on lotteries that satisfies some simple axioms must be VNM rational (i.e. have a utility function as above). Since I am advocating for a weaker notion of rationality, I must reject some of these axioms.

Against Independence

The VNM axiom that I am rejecting is the independence axiom. It states that given lotteries A, B, and C, and probability p, A⪯B if and only if pC+(1−p)A⪯pC+(1−p)B. Thus, mixing in a probability p of C will not change my preference between A and B.

Let us go through an example.

Alice and Bob are a married couple. They are trying to decide where to move, buy a house, and live for the rest of their lives. Alice prefers Atlanta, Bob prefers Boston. The agent I am modeling here is the m...