trentmkelly/LessWrong-43k huggingface dataset:
Some Thoughts on AI Alignment: Using AI to Control AI
by eigenvalue — 2024-06-21T17:44:19.263Z
Link: https://github.com/Dicklesworthstone/some_thoughts_on_ai_alignment

Recent news has caused me to think through some questions about AI alignment, so I collected my thoughts here. While I'm sure a lot of this stuff isn't new, I haven't seen all these ideas presented together in one place. I think that some of the approaches that are used in designing decentralized systems can also be useful in constructing alignment systems, so I've tried to do that here. Anyway, I welcome feedback on my ideas.
What distinguishes "early", "mid" and "end" games?
by Raemon — 2024-06-21T17:41:30.816Z

Recently William_S posted:

In my mental model, we're still in the mid-game, not yet in the end-game.

I replied:

A thing I've been thinking about lately is "what does it mean to shift from the early-to-mid-to-late game".

In strategy board games, there's an explicit shift from "early game, it's worth spending the effort to build a longterm engine. At some point, you want to start spending your resources on victory points." And a lens I'm thinking through is "how long does it keep making sense to invest in infrastructure, and what else might one do?"

I assume this is a pretty different lens than what you meant to be thinking about right now but I'm kinda curious for whatever-your-own model was of what it means to be in the mid vs late game.

He replied:

Like, in Chess you start off with a state where many pieces can't move in the early game, in the middle game many pieces are in play moving around and trading, then in the end game it's only a few pieces, you know what the goal is, roughly how things will play out.

In AI it's like only a handful of players, then ChatGPT/GPT-4 came out and now everyone is rushing to get in (my mark of the start of the mid-game), but over time probably many players will become irrelevant or fold as the table stakes (training costs) get too high.

In my head the end-game is when the AIs themselves start becoming real players.

This was interesting because yeah, that totally is a different strategic frame for "what's an early, midgame and endgame?", and that suggests there's more strategic frames that might be relevant.

I'm interested in this in the context of AI, but, also in other contexts.

So, prompt for discussion:

a) what are some types of games or other "toy scenarios," or some ways of looking at those games, that have other strategic lenses that help you decisionmake?

b) what are some situations in real life, other than "AI takeoff", where the early/mid/late game metaphor seems useful?
Nuclear War, Map and Territory, Values | Guild of the Rose Newsletter, May 2024
by moridinamael — 2024-06-21T17:39:24.119Z
Link: https://guildoftherose.org/articles/prepare-for-nuclear-war-and-learn-the-guild-s-values

Community Updates

Upcoming events:

Naturalism event every week on Thursdays

This Interest Group discusses the 'Naturalism' sequence by Logan Strohl, practicing the techniques it describes.

Meaning and Spirituality event every other week on Tuesdays

This Interest Group concerns itself with determining how we can help each other live more meaningful and rich lives.

Social Butterflies event every week on Tuesdays

This Interest Group discusses social connections and how to improve them. Often, this will involve examining some material on relationships and discussing how to apply it.

You can add all the events to your calendar by clicking here.

You can join our events and conversations by creating a free Guild account!

Workshop Updates

New and revised workshop content

This month at our Denver chapter of the Guild, Guild member Alex Fauble ran an in-person Guild workshop on our Emergency Preparedness workshop. In his presentation, Alex makes the case that nuclear war is the riskiest disaster. Given its wide-ranging effects on everything from logistics to breathable air, preparing for nuclear war also prepares you for nearly any other disaster or emergency.

The Emergency Preparedness workshop will help you take the most valuable steps to prepare for the likeliest dangers, culminating with stocking your own emergency bag for quick response. Emergencies happen to everyone eventually, so you had best be ready!

To start using this workshop with our skill tree, create a free account!

Skill of the Month

Here is an example of a skill from our Skill Tree. Use this opportunity to do the skill and start making changes in your life!

[Empiricist] **Map and Territory** - 1 XP

1. Read one of the Map and Territory articles. (Choose one at random, don't just pick the first one.)

2. Write an informal explanation of the concepts described by the article, and briefly connect it to your life in some way. (For example, if you read Your Strength as a Rationalist, you might recall a time when you should have noticed your confusion, but didn't.)

3. Share your explanation with the Guild, in #meta-and-theory or #skilltree-updates.

Example Submission

Making Beliefs Pay Rent (in Anticipated Experiences)

The article gestures towards a definition of belief, requiring that the the belief's consequences would produce a sensory experience. This constrasts...
AI governance needs a theory of victory
by Corin Katzke — 2024-06-21T16:15:46.560Z
Link: https://www.convergenceanalysis.org/publications/ai-governance-needs-a-theory-of-victory

This post is part of a series by Convergence Analysis. An earlier post introduced scenario planning for AI risk. In this post, we argue for the importance of a related concept: theories of victory.

In order to improve your game you must study the endgame before everything else; for, whereas the endings can be studied and mastered by themselves, the middlegame and the opening must be studied in relation to the endgame." — José Raúl Capablanca, Last Lectures, 1966

Overview

The central goal of AI governance should be achieving existential security: a state in which existential risk from AI is negligible either indefinitely or for long enough that humanity can carefully plan its future. We can call such a state an AI governance endgame. A positive framing for AI governance (that is, achieving a certain endgame) can provide greater strategic clarity and coherence than a negative framing (or, avoiding certain outcomes).

A theory of victory for AI governance combines an endgame with a plausible and prescriptive strategy to achieve it. It should also be robust across a range of future scenarios, given uncertainty about key strategic parameters.

Nuclear risk provides a relevant case study. Shortly after the development of nuclear weapons, scientists, policymakers, and public figures proposed various theories of victory for nuclear risk, some of which are striking in their similarity to theories of victory for AI risk. Those considered included an international nuclear development moratorium, as well as unilateral enforcement of a monopoly on nuclear weapons.

We discuss three potential theories of victory for AI governance. The first is an AI moratorium, in which international coordination indefinitely prevents AI development beyond a certain threshold. This would require an unprecedented level of global coordination and tight control over access to compute. Key challenges include the strong incentives to develop AI for strategic advantage, and the historical difficulty of achieving this level of global coordination.

The second theory of victory is an "AI Leviathan" — a single well-controlled AI system or AI-enhanced agency that is empowered to enforce existential security. This could arise from either a first-mover unilaterally establishing a moratorium, or voluntary coordination between actors. Challenges include the potential for dysto...
Connecting the Dots: LLMs can Infer & Verbalize Latent Structure from Training Data
by Johannes Treutlein — 2024-06-21T15:54:41.430Z
Link: https://arxiv.org/abs/2406.14546

TL;DR: We published a new paper on out-of-context reasoning in LLMs. We show that LLMs can infer latent information from training data and use this information for downstream tasks, without any in-context learning or CoT. For instance, we finetune GPT-3.5 on pairs (x,f(x)) for some unknown function f. We find that the LLM can (a) define f in Python, (b) invert f, (c) compose f with other functions, for simple functions such as x+14, x // 3, 1.75x, and 3x+2.

Paper authors: Johannes Treutlein*, Dami Choi*, Jan Betley, Sam Marks, Cem Anil, Roger Grosse, Owain Evans (*equal contribution)

Johannes, Dami, and Jan did this project as part of an Astra Fellowship with Owain Evans.

Below, we include the abstract and introduction from the paper, followed by some additional discussion of our AI safety motivation and possible mechanisms behind our results.

Abstract

One way to address safety risks from large language models (LLMs) is to censor dangerous knowledge from their training data. While this removes the explicit information, implicit information can remain scattered across various training documents. Could an LLM infer the censored knowledge by piecing together these implicit hints? As a step towards answering this question, we study inductive out-of-context reasoning (OOCR), a type of generalization in which LLMs infer latent information from evidence distributed across training documents and apply it to downstream tasks without in-context learning. Using a suite of five tasks, we demonstrate that frontier LLMs can perform inductive OOCR. In one experiment we finetune an LLM on a corpus consisting only of distances between an unknown city and other known cities. Remarkably, without in-context examples or Chain of Thought, the LLM can verbalize that the unknown city is Paris and use this fact to answer downstream questions. Further experiments show that LLMs trained only on individual coin flip outcomes can verbalize whether the coin is biased, and those trained only on pairs (x,f(x)) can articulate a definition of f and compute inverses. While OOCR succeeds in a range of cases, we also show that it is unreliable, particularly for smaller LLMs learning complex structures. Overall, the ability of LLMs to “connect the dots” without explicit in-context learning poses a potential obstacle to monitoring and controlling the knowledge a...
On OpenAI’s Model Spec
by Zvi — 2024-06-21T13:00:03.014Z

There are multiple excellent reasons to publish a Model Spec like OpenAI’s, that specifies how you want your model to respond in various potential situations.

It lets us have the debate over how we want the model to act.

It gives us a way to specify what changes we might request or require.

It lets us identify whether a model response is intended.

It lets us know if the company successfully matched its spec.

It lets users and prospective users know what to expect.

It gives insight into how people are thinking, or what might be missing.

It takes responsibility.

These all apply even if you think the spec in question is quite bad. Clarity is great.

As a first stab at a model spec from OpenAI, this actually is pretty solid. I do suggest some potential improvements and one addition. Many of the things I disagree with here are me having different priorities and preferences than OpenAI rather than mistakes in the spec, so I try to differentiate those carefully. Much of the rest is about clarity on what is a rule versus a default and exactly what matters.

In terms of overall structure, there is a clear mirroring of classic principles like Asimov’s Laws of Robotics, but the true mirror might be closer to Robocop.

What are the central goals of OpenAI here?

1. Objectives: Broad, general principles that provide a directional sense of the desired behavior

Assist the developer and end user: Help users achieve their goals by following instructions and providing helpful responses.

Benefit humanity: Consider potential benefits and harms to a broad range of stakeholders, including content creators and the general public, per OpenAI’s mission.

Reflect well on OpenAI: Respect social norms and applicable law.

I appreciate the candor on the motivating factors here. There is no set ordering here. We should not expect ‘respect social norms and applicable law’ to be the only goal.

I would have phrased this in a hierarchy, and clarified where we want negative versus positive objectives in place. If Reflect is indeed a negative objective, in the sense that the objective is to avoid actions that reflect poorly and act as a veto, let’s say so.

Even more importantly, we should think about this with Benefit. As in, I would expect that you would want something like this:

Assist the developer and end user…

…as long as doing so is a

...
Attention Output SAEs Improve Circuit Analysis
by Connor Kissane — 2024-06-21T12:56:07.969Z

This is the final post of our Alignment Forum sequence produced as part of the ML Alignment & Theory Scholars Program - Winter 2023-24 Cohort.

Executive Summary

In a previous post we trained A

ttention Output Sparse Autoencoders (SAEs)

on

every layer of GPT-2 Small

.
Following that work, we wanted to stress-test that Attention SAEs were genuinely helpful for circuit analysis research. This would both validate SAEs as a useful tool for mechanistic interpretability researchers, and provide evidence that they are identifying the real variables of the model’s computation.
We believe that we now have evidence that attention SAEs can:

Help make novel mechanistic interpretability discoveries that prior methods could not make.
Allow for tracing information through the model’s forward passes on arbitrary prompts.

In this post we discuss the three outputs from this circuit analysis work:

We use SAEs to deepen our understanding of the IOI circuit. It was previously thought that the indirect object’s name was identified by tracking the names positions, whereas we find that instead the model tracks whether names are before or after “and”. This was not noticed in prior work, but is obvious with the aid of SAEs.
We introduce “recursive direct feature attribution” (recursive DFA) and release an Attention Circuit Explorer tool for circuit analysis on GPT-2 Small (Demo 1 and Demo 2). One of the nice aspects of attention is that attention heads are linear when freezing the appropriate attention patterns. As a result, we can identify which source tokens triggered the firing of a feature. We can perform this recursively to track backwards through both attention and residual stream SAE features in models.

We also announce a $1,000 bounty for whomever can produce the most interesting example of an attention feature circuit by 07/15/24 as subjectively assessed by the authors. See the section "Even cooler examples" for more details on the bounty.

We open source

HookedSAETransformer

to SAELens, which makes it easy to splice in SAEs during a forward pass and cache + intervene on SAE features. Get started with this

demo notebook

.

Introduction

With continued investment into dictionary learning research, there still remains a concerning lack of evidence that SAEs are useful interpretability tools in practice. Further, while SAEs clearly find interpretable features (Cu...
"Newton's laws" of finance
by pchvykov — 2024-06-21T09:41:21.931Z

[NOT financial advice! In fact I'll really be showing some generic math here – and it's up to you if you apply it to investments in stocks, projects, relationships, ideas, skills, etc.]

As I’ve been thinking about some economic theories and working on a personal investment algorithm, I kept coming across ideas that seemed entirely fundamental to how one should think about investing – and yet I could not find any reasonable references that would give a clear overview of these. Thus it took me some time of playing with algorithms and math to figure out how it all fits together – and now seems perfectly obvious. But just as it was not obvious to me before, I now have conversations with friends who also get confused. I suspect this comes from investment being taught from a “wrong” starting points – so people have a different idea of what is “basic” and what is “advanced.”

Two caveats I have to mention here: First, I’m a physicist, and so my idea of what is “basic” is entirely informed by my background. I do suspect this stuff might seem more “advanced” to someone not as comfortable with the math I’ll be using. By the same token, this is probably why the typical “finance basics” seem very confusing and convoluted to me. Second, I’m a fan of Ergodicity Economics, and this post will be informed by their perspective. This perspective, while seems entirely correct to me, is not part of the “economics cannon” – and so may be contested by some. Still, I think the math I will show and the conclusions relevant to investing are generally correct and accepted as such (just a bit hard to find).

Geometric Brownian Motion

The starting point for everything I'm going to say is the assumption that investments follow Geometric Brownian Motion (GBM) dynamics:

∂tx=x⋅(μ+σξ)[1]

where x is your wealth (e.g., stock price, or project value), (μ,σ) are parameters characterizing the investment (return and volatility respectively), and ξ(t) is standard uncorrelated white Gaussian noise process (so that ⟨ξ(t)⟩=0 and ⟨ξ(t),ξ(t′)⟩=δ(t−t′), where ⟨⋅⟩ denotes average over noise realizations, and, for experts, we take Itô noise). This model is at the core of much of quantitative finance, as it captures both the random fluctuations and compounding growth of the investment. It is also a model that researchers "love to ...
Capitalising On Trust—A Simulation
by James Stephen Brown — 2024-06-21T04:43:29.971Z
Link: https://nonzerosum.games/cooperationvsdefection.html

Recently I’ve been exploring moral philosophy with a series on Moral Licensing, Andrew Tane Glen’s Why Cooperate?, and in a workshop I ran with my daughter’s class about the strategies of cooperation and defection. One phenomenon that has arisen through these explorations is that defectors generally gain a short term relative advantage, while cooperators benefit from a sustained long term absolute advantage, which got me thinking about a simulation.

Unfortunately I can't embed the simulation here so to view it you'll need to visit the site. But I'm interested in what people think of the concept, and the practice of using simulations in general to illustrate these sorts of concepts.

Do you think this sort of short term individualism and long term cooperation, is something that applies outside the prisoner's dilemma?
"... than average" is (almost) meaningless
by jwfiredragon — 2024-06-21T04:42:26.682Z

Recently, I was talking to a friend who hadn't seen me in a while. They mentioned that my hair had grown noticeably, and then asked whether my hair grew fast or slow. I said that my hair growth was probably around average, but upon consideration, I realized that statement was so divorced from reality it would be a disservice to the notion of truth to even call it "false". Not only do I not know how I compare to the average, I have no idea what the average even is.

It seems that people (myself included) make this kind of statement a lot, even when it's obviously not backed by concrete evidence. So what's going on here?

I: Analysis

When people (myself included) say that "I'm more/less [X] than average", where [X] is anything that isn't trivially measurable, they're not using 'average' to mean any statistical notion of an average. Rather, they're referring to their own mental generic-template-model of somebody who is "average at [X]".

The first problem with this is that this template person isn't necessarily representative of any empirical measure of the average. It's a Frankenstein's monster cobbled together out of personal experience, half-remembered data, anecdotes from the internet, fictional 'evidence', and who knows what else. It doesn't have zero correlation to reality, but it's nowhere near 100%.

The second problem is that your mental model is not my mental model. Your idea of average is not the same as my idea of it. Once again, it's not going to be maximally dissimilar, but I'd expect that there would be significant variance between people if you asked them to describe what their mental model of "average at [X]" actually entails.

So when somebody says, for example, they're an above-average driver, you can't conclude anything about their driving skills until you also know what they think an average driver is. Yet when you hear them say that, you automatically match those words to your idea of an average driver, and develop a preconception of their driving skills that is only tangentially rooted in evidence.

II: Which average?

Having explained this, what would it take to make a statement like this and have it actually have a truth value? The most important difference between an evidence-based average and the mental model 'average' is the existence of a data set to calculate the average from. In other words, what group are you comparing yourself against?

The glob...
The Kernel of Meaning in Property Rights
by Abhimanyu Pallavi Sudhir — 2024-06-21T01:12:55.330Z

A phenomenon I have often encountered when thinking about things is when everything seems to collapse to tautology. This is hard to define precisely, but I’ll give you some examples:

Bounded rationality: Bounded rationality can be thought of as “rationality conditional on some given algorithmic information” (in contrast to perfect Bayes-rationality, which is “rationality conditional on some given statistical information”), where “algorithmic information” is a more general version of “logical information”. But “algorithmic information” is fundamentally just arbitrary programs/heuristics, including the heuristic that estimates the argmax of the utility. So everything seems to unwind, and you could as well just say every program is boundedly rational — it’s “doing the best it can with the algorithmic information it has (itself)”, and bounded rationality becomes empty of meaning.

Efficient Market Hypothesis: Basically the same as above; the Efficient Market Hypothesis is only valid when you read it as “The market is efficient conditional on all the algorithmic information it has”, but the “algorithmic information” just refers to the traders in existence and the market-maker.

Market dynamics: Markets must take some time to compute equilibrium (it’s a complex computational problem), but also in some sense the market is “always” at equilibrium conditional on the information possessed, and market dynamics is really just a problem of information propagation.

… but deep down, you know there has to be something there, there has to be some kernel of meaning and non-nihilism. There has to be some meaning to these terms, because at the end of the day — we want boundedly rational agents, we want efficient economic institutions.

The Kernel of Meaning in these examples is a bit hard to grok at — they’re probably all related, and have to do something with “learning” (Russell, “Rationality and Intelligence: An Update”) and “Algorithmic Bayesian Epistemology” (Eric Neyman) — there’s a certain recursive sense in which markets (and boundedly rational agents) are rational about updating their heuristics, about updating these meta-heuristics, and so on.

But I can give you one other example of apparent collapse to tautology where I know exactly how to defeat this nihilism: property rights. Where you might want to answer questions like “isn’t any government policy acceptable, if you ...