trentmkelly/LessWrong-43k huggingface dataset:
Adaptive Coherence
by Zero Contradictions — 2024-08-07T06:17:02.665Z
Link: https://thewaywardaxolotl.blogspot.com/2014/11/adaptive-coherence.html

Biology and culture evolve together.

For example, cattle-herding is a cultural adaptation, and lactose tolerance is a biological adaptation. Those adaptations evolved together and fit together. Each selects for the other. The two adaptations together are more useful than either is separately. You can’t just take a bunch of lactose intolerant people and drop them into a cattle-herding culture. They will get sick, because their biology does not fit the culture. The linked adaptations of cattle-herding and lactose tolerance emerged in two different parts of the world, Europe and Africa, thousands of years ago, and have spread widely since.

The coevolution of biology and culture makes them fit together. They cohere. They are codependent.

Society and culture are also codependent. To live together in a society, people need a common culture, which includes language and concepts that are necessary for social organization. People develop a common culture by living together in the same place and interacting. Social interaction requires a common culture and creates a common culture. Society depends on culture, and culture depends on society.

Money is a good example of the codependence of culture and society. Money has value because people believe it has value. That belief is part of culture. Everyone shares that belief. Why do people believe that money has value? Because they can use it to purchase things. The use of money is part of society. Buying things, working for wages, and paying taxes are social interactions. The use of money in those social interactions creates the belief that money is valuable. The belief that money is valuable makes the use of money possible. The cultural belief and the social use emerged together. Neither can exist without the other.

For a society to exist, its laws and institutions must be viewed as good (or mostly good) by the people. People tend to view the existing social order as good, because they are familiar with it. For the most part, morality consists of cultural norms of behavior. Those cultural norms emerge within a social context, and are shaped by that social context. Society reinforces moral norms with rewards and punishments. Society imposes morality on its members, but society also depends on the moral approval of its members. Cultural norms depend on social power. Social power depends on cultural norms.

Society and biology are a...
Individual Utilities Shift Continuously as Geometric Weights Shift
by StrivingForLegibility — 2024-08-07T01:41:10.071Z

This is a supplemental post to Geometric Utilitarianism (And Why It Matters), in which I show that when all agents have positive weight ψi, the optimal geometric weighted average moves continuously across the Pareto frontier as we change those weights. I also show that we can extend this continuity result to all weights ψ, if we're willing to accept an arbitrarily good approximation of maximizing G(_,ψ). I think of this as a bonus result which makes the geometric average a bit more appealing as a way to aggregate utilities, and the main post goes into more detail about the problem and why it's interesting.

High Level Overview

How does changing ψ affect the optima of G(_,ψ)? Ideally, we'd like a small change in the weights assigned to each agent to cause a small change in the resulting joint utility p. In other words, we would like p(ψ)=argmaxu∈FG(u,ψ) to be continuous. It turns out this is true when ψi>0 for all agents, and there's at least one way to create a continuous function ψ→p that works for all ψ∈Δn and is an arbitrarily good approximation of G(_,ψ) maximization.

We've already solved the inverse problem: given a point p and Harsanyi weights ϕ which make p optimal according to H(_,ϕ), find geometric weights ψ that make p optimal according to G(_,ψ).

ψ=p⊙ϕp⋅ϕ

So we already have ψ(p,ϕ), which is a smooth function of its inputs where it's defined.[1]

It turns out we can invert this function and recover (p,ϕ) from ψ when pi>0 and ϕi>0 for all agents. This is the interior of what we'll be calling the Harsanyi Bundle; the set of all of the pairs (p,ϕ) which are consistent with our Pareto frontier P. This post will show that there is a bijection between the interior of the Harsanyi Bundle and the interior of the set of all valid weights Δn.

So we have a smooth bijection (p,ϕ)→ψ between these two subsets of R2n and Rn respectively. And thankfully we're doing calculus, where this is sufficient to establish that the inverse function ψ→(p,ϕ) is at least continuous.[2] And that is the main result this post sets out to prove: that individual utilities shift continuously as geometric weights shift....
Gradient Ascenders Reach the Harsanyi Hyperplane
by StrivingForLegibility — 2024-08-07T01:40:40.353Z

This is a supplemental post to Geometric Utilitarianism (And Why It Matters), in which I show that, if we use the weights we derived in the previous post, a gradient ascender will reach the Harsanyi hyperplane H. This is a subproblem of the proof laid out in the first post of this sequence, and the main post describes why that problem is interesting.

The Gradient and Contour Lines

It's easy to find the points s∈Rn which have the same G score as p: they're the points which satisfy G(s,ψ)=G(p,ψ). They all lie on a skewed hyperbola that touches P at p.

Check out an interactive version here

One way to think about G is as a hypersurface in n+1-dimensional space sitting "above" the n-dimensional space of utilities we've been working with. When there are 2 agents, we can plot G using the third vertical axis.

Interactive version here

Check out the intersection of G and the vertical plane above the Harsanyi line H: this tells us about the values of G along this line, and as we shift p we can recalculate ψ so that among H, G peaks at p.

Our choice of p determines where we land on that surface "above" p. If we take a slice through G by only looking at the points of G at the same "altitude" as p, we get exactly that hyperbola back!

Doing this for many altitudes gives us a contour map, which you're probably familiar with in the context of displaying the altitude of real 3D landscapes on flat 2D maps.

You can see how these contours change as we change p and H using the interactive version here.

There's a theorem which tells us that the gradient of G must either be 0 or perpendicular to these contour hypersurfaces. So by calculating the gradient, we can calculate the tangent hyperplane of our skewed hyperbolas! And then we'll see if anything interesting happens at p.

This is a subproblem of the gradient calculation we did earlier, where we were specifically interested in how G changes along the Harsanyi hyperplane H. The slope of H, encoded in ϕ(p,F), showed up in how we defined ψ(p,F). So let's see how ϕ and p have shaped the geometry of G(_,ψ).

The gradient ∇uG(u,ψ) is just a vector of partial derivatives&n...
Deriving the Geometric Utilitarian Weights
by StrivingForLegibility — 2024-08-07T01:39:58.686Z

This is a supplemental post to Geometric Utilitarianism (And Why It Matters), in which I show how I derived the weights ψ which make any Pareto optimal point p optimal according to the geometric weighted average. This is a subproblem of the proof laid out in the first post of this sequence, and the main post describes why that problem is interesting.

Overview

So how are we going to calculate weights ψ which make p optimal among F?

The idea here is to identify the Harsanyi hyperplane H, which contains all of the joint utilities u∈Rn which satisfy H(u,ϕ)=H(p,ϕ). Where ϕ are the weights which make our chosen point p∈Rn optimal with respect to H(_,ϕ). And we're going to calculate new weights ψ which make p optimal with respect to G(_,ψ). It turns out it's sufficient to make p optimal among H, and p will also be optimal across our entire feasible set F.

In terms of calculus, we're going to be constructing a function G∘H, which tells us about how moving around on H changes G. And we're going to choose weights ψ which make the gradient ∇v(G∘H) equal 0 at p. This makes it a local optimum, and it will turn out to be a global maximum across H, which in turn will make it a global maximum across F.

Geometrically, we can think of that as the surface gradient of G across H. And so in terms of the overall gradient ∇uG, we're designing ψ so that ∇uG is perpendicular to H at p.

Parameterizing the Harsanyi Hyperplane

When thinking about moving around on the Harsanyi hyperplane H, we have a linear constraint that says no matter which u∈H we pick, we know that u⋅ϕ=p⋅ϕ=H(p,ϕ). If we know u lies on H, we can calculate the the n-th agent's utility un from the first n−1 utilities. We'll be referring to these first n-1 utilities a lot, so let's call them v∈Rn−1. So vi=ui for all i<n.

H and G are both symmetrical with respect to shuffling the indices of agents around, so without loss of generality we'll assume that the n-th agent is one we're assigning positive Harsanyi weight to: ϕn > 0. This is necessary for the reconstruction to ...
Proving the Geometric Utilitarian Theorem
by StrivingForLegibility — 2024-08-07T01:39:10.920Z

This is a supplemental post to Geometric Utilitarianism (And Why It Matters), which sets out to prove what I think are the main interesting results about Geometric Utilitarianism:

Maximizing a geometric weighted average G(_,ψ) can always lead to Pareto optimality.
Given any Pareto optimal joint utility p, we can calculate weights ψ which make p optimal according to G(_,ψ).

That post describes why this problem is interesting, but the quick summary is: geometric utility aggregation is a candidate alternative to Harsanyi utility aggregation (which is an arithmetic weighted average), which handles some tradeoffs better than Harsanyi aggregation. The resulting choice function is geometrically rational, whereas the Harsanyi choice function is VNM-rational. This post is mostly math supporting the main post, with some details moved to their own posts later in this sequence.

We as a community, and I individually, would need pretty strong reasons to endorse a theory of rationality based on a broadening of the VNM axioms. I've been sufficiently radicalized by Scott Garrabrant, and thinking about how each system handles common decision problems, that I think more work in this direction is potentially very valuable. Here's how I solved a couple of the subproblems towards making progress in that direction.

Assumptions

Most of these proofs can be understood geometrically, and we'll need to make some geometric assumptions. One big property we'll be using is the fact that set of feasible joint utilities F∈Rn is always convex. This is always true for VNM-rational agents with utility functions, but we can prove the same result for agents with other functions describing their preferences, as long as the feasible utilities stay convex. This will be helpful when we start combining utilities in ways that make the resulting social choice function violate the VNM axioms, but preserve convexity.

We'll also need F to be compact, if we want to be able to find optimal points according to G or H. If F extends infinitely in any direction, there might not be any Pareto optimal joint utilities. This "problem" also appears in classic individual rationality: what does a rational agent do when they can simply pick a number u∈R and receive that much utility? Individual VNM-rationality isn't well-defined when...
The Geometric Importance of Side Payments
by StrivingForLegibility — 2024-08-07T01:38:04.635Z

I'm generally a fan of "maximize economic surplus and then split the benefits fairly". And I think this approach makes the most sense in contexts where agents are bargaining over a joint action space D×P, where D is some object-level decision being made and P are side-payments that agents can use to transfer value between them.[1]

An example would be a negotiation between Alice and Bob over how to split a pile of 100 tokens, which Alice can exchange for $0.01 each, and Bob can exchange for $10,000,000 each. The sort of situation where there's a real and interpersonally comparable difference in the value they each derive from their least and most favorite outcome.[2]

In this example D is the convex set containing joint utilities for all splits of 100 tokens, and the ($0,$0) disagreement point. If we take F=D, the Nash and KS bargaining solutions are for Alice and Bob to each receive 50 tokens. But this is clearly not actually Pareto optimal. Pareto optimality looks like enacting a binding agreement between Alice and Bob that "Bob can have all the tokens, and Alice receives a fair split of the money". And I claim the mistake was in modelling D as the full set of feasible options, when in fact the world around us redunds with opportunities to do better.

Side payments introduce important geometric information that D alone doesn't convey: the real-world tradeoff between making Alice happier and making Bob happier. Bargaining solutions are rightly designed to ignore how utility functions are shifted and scaled, and when F is compact we can standardize each agent's utility into [0,1].

A Standardized Flat Pareto Frontier for 2 Agents

With D alone, we can't distinguish between "Bob is just using bigger numbers to measure his utility" (measuring this standardized shape in nano-utilons) and "Bob is actually 1 billion times more sensitive to the difference between his least and most favorite outcome than Alice is."

In this example, when we project the outcome space into standardized joint utility space, the results for D and D×P look like that image above: a line sloping down from Bob's favorite outcome to Alice's, and all the space between that line and (0,0). And the Nash and KS bargaining solut...
Attention-Feature Tables in Gemma 2 Residual Streams
by J Bostock — 2024-08-06T22:56:40.828Z

This is research I did in a short span of time, it is likely not optimal, but it's unclear whether the constraints are my skills, my tools, or my methods. Code and full results can be found here, but you'll need to download the model yourself to replicate it.

TL;DR

Using Gemma Scope, I am able to find connections between SAE features in the attention portion of Gemma 2 2B's layer 13. This can be done without running the model. These are often (somewhat) meaningful, with the OV circuits generally being easier to interpret than the QK circuits.

The Background

Anthropic has done some nice research decomposing attention layers into the QK and OV circuits. To summarize, the QK circuit tells an attention head where to look, and the OV circuit tells it what information to pass forward. When looking at a single-layer transformer, this looks like patterns in tokens of the form A,...,B→C.

Recently, Google Deepmind released Gemma Scope, consisting of a large number of SAEs, trained on the Gemma 2 family of models. Neuronpedia has already generated a description for all of the "canonical" SAEs using GPT-4o.

This is an attempt to look at QK and OV circuits in Gemma 2 from the lens of features, rather than tokens.

The Setup

I chose to look at layer 13 (indexed from zero) roughly halfway through the model. For each decoder vector in the 16k-feature canonical residual stream SAE of layer 12 (henceforth SAE12), I calculated the element-wise product with the RMS norm coefficients, then calculated the products with WQ, WK, and WV. I'll refer to these as the queries, keys, and values.

I then took the encoder vectors of the 16k-feature canonical residual stream SAE of layer 13 (henceforth SAE13), did an element-wise product with the RMS norm of the attention layer output, and multiplied by the transpose of WO. This creates a vector I'll call the pre-output. This is working backward towards the attention layer.

This does ignore the MLP layer. I wish there were SAEs trained between the two components. It also ignores the rotary positional encoding. I may return to these later. It also ignores the attention-out prelinear SAE! This is because Neuronpedia doesn't have labels available to download for these yet.

I calculated, for each head, a [16k,16k] matrix consisting of the dot-products of the queries and keys, and which contains the dot-products o...
What are the strategic implications if aliens and Earth civilizations produce similar utilities?
by Maxime Riché — 2024-08-06T21:16:21.719Z

I am working on a project about estimating alien density, their expected utility, and arguing for strategic implications. I am optimistic about the project producing valuable content to inform the AI safety strategy. But I want to probe what the community thinks beforehand.

Main question:

Assumptions:

Alien and Earth space-faring civilizations produce similar expected utilities [1]. The utility is computed using our CEV.
Alien space-faring civilizations are frequent enough such that there are at least several of them per affectable light cone such that very few resources are left unused.

Question:

Given the assumptions, what would the strategic implications be for the AI safety community?

Secondary question:

Without any assumption. What are your arguments for expecting Alien space-faring civilizations to have similar, or lower (e.g. 0), or higher expected utility than a future Earth-originating space-faring civilization?

Note: Please reach out to me by MP if you want to contribute or provide feedback on the project.

^

I am talking about the expected utility per unit of controlled resources, after considering everything. This is the “final” expected utility created per unit of resource. As an illustration, it accounts for the impact of trades and conflicts, causal or acausal, happening in the far future.
WTH is Cerebrolysin, actually?
by gsfitzgerald — 2024-08-06T20:40:53.378Z

[This article was originally published on Dan Elton's blog, More is Different.]

Cerebrolysin is an unregulated medical product made from enzymatically digested pig brain tissue. Hundreds of scientific papers claim that it boosts BDNF, stimulates neurogenesis, and can help treat numerous neural diseases. It is widely used by doctors around the world, especially in Russia and China.

A recent video of Bryan Johnson injecting Cerebrolysin has over a million views on X and 570,000 views on YouTube. The drug, which is advertised as a “peptide combination”, can be purchased easily online and appears to be growing in popularity among biohackers, rationalists, and transhumanists. The subreddit r/Cerebrolysin has 3,100 members.

TL;DR

Unfortunately, our investigation indicates that the benefits attributed to Cerebrolysin are biologically implausible and unlikely to be real. Here’s what we found:

Cerebrolysin has been used clinically since the 1950s, and has escaped regulatory oversight due to some combination of being a “natural product” and being grandfathered in.
Basic information that would be required for any FDA approved drug is missing, including information on the drug’s synthesis, composition, and pharmacokinetics.
Ever Pharma’s claim that it contains neurotrophic peptides in therapeutic quantities is likely false. HPLC and other evidence show Cerebrolysin is composed of amino acids, phosphates, and salt, along with some random protein fragments.
Ever Pharma’s marketing materials for Cerebrolysin contain numerous scientific errors.
Many scientific papers on Cerebrolysin appear to have ties to its manufacturer, Ever Pharma, and sometimes those ties are not reported.
Ever Pharma’s explanation of how the putative peptides in Cerebrolyin cross the blood-brain barrier does not make sense and flies in the face of scientific research which shows that most peptides do not cross the blood-brain barrier (including neurotrophic peptides like BDNF, CDNF, and GDNF).
Since neurotrophic factors are the proposed mechanism for Cerebrolysin’s action, it is reasonable to doubt claims of Cerebrolysin’s efficacy. Most scientific research is false. It may have a mild therapeutic effect in some contexts, but the research on this is shaky. It is likely safe to inject in small quantities, but is almost certainly a waste of money for anyone looking to improve their cognitive function.

Introduction

O...
The Pragmatic Side of Cryptographically Boxing AI
by Bart Jaworski — 2024-08-06T17:46:21.754Z

All previous discussions and literature I was able to find on the topic of cryptographic containment of powerful AI or AI boxing primarily discussed or argued these points:

Certain AI boxing schemes could, in principle, solve alignment.

Even if AI boxing is implemented, social engineering would likely be the primary failure mode[1].

AI boxing might be necessary as progress on alignment lags behind AI capabilities[2].

Cryptographic AI boxing is generally computationally expensive.

However, there has been little attempt made to measure the magnitude of the alignment tax associated with the implementation of cryptographic containment for AI and use that to estimate the feasibility of such implementation. In this post, I argue that we are highly unlikely to see cryptographic containment of powerful AI in the coming years, if ever. This argument can be summarised as follows:

The computational overhead inherent to the cryptographic techniques required to contain a powerful AI is so enormous as to make the implementation infeasible, even when accounting for breakthroughs in hardware acceleration.

Additionally, I go over the general economic disincentives that make cryptographic containment of powerful AI even less likely to happen. And at the end, I assume that “computational overhead is not an issue”, and discuss the complex and troublesome nature of the centralisation of power it leads to.

I tried to keep this post as brief as possible, and broke down the above argument into 4 parts:

I define cryptographic AI boxing and give examples of how it could be useful, and potentially necessary in certain scenarios.
I estimate the alignment tax of cryptographic boxing and argue that, relative to the benefits it brings, it is huge.
I go over the (dis)incentives associated with constraining the accessibility/deployment of SOTA AI.
I talk about the caveat of successful implementation of AI containment - centralisation of power. I also propose a scheme that could solve this problem using Secure Multi-Party Computation (MPC), however, it exacerbates the underlying implementation challenges present in classic (centralised) containment schemes.

What do I mean by cryptographic AI boxing?

AI Boxing is typically defined as efforts to ensure safe communication with AI only allowing the bare minimum amount of information to pass through a specific information channel to the hum...