trentmkelly/LessWrong-43k huggingface dataset:
Aligning alignment with performance
by Marv K — 2022-12-14T22:19:35.312Z

This post is a short sketch of a problem with alignment that I wanted to share. The solution suggested below is not fleshed out at all yet, I may update this in a future post. I feel its worth sharing the initial idea already though.

The Problem

Claim: Usually, making an AI aligned will reduce its performance, i.e. increases in alignment are a sufficient condition for decreases in performance.

This is almost by definition, since alignment is about ruling out behavior we deem bad, which would otherwise be the AI doing "its best" to follow our instructions literally.

Some examples of how this might look:

The AI is trained with an objective function that contains an additive alignment term, similar to a regularization term. The System then chooses to either go for the rest of its objective function, or pay respect to the alignment term.
We filter out unwanted inputs, i.e. clean the training data. This results in a less capable system.
We filter out unwanted outputs, perhaps by training a second AI to recognize them. This also leads to a less capable system.

If the above claim is true, it makes sense to think about whether there are architectures and training paradigms where this tradeoff doesn't exist, but where, instead, increases in alignment are a necessary condition for increases in performance.

Let's call such a setup alignment-inducing.

A Potential Solution

I think many properties of society are alignment-inducing, i.e. human personal growth necessitates alignment with social circles.

Some arguments for this:

Antisocial behavior decreases future input. E.g. mentors choose students that they sympathize with, not ones they only find promising in their performance in some domain.
Intelligent people tend to be more altruistic. I recall reading a study on this, but can't find it at the moment.
Reputation systems such as social status and serotonin reward prosocial behavior.
Growing up in a social environment forces you to model the needs and wants of others, take them into consideration and find an equilibrium with your own needs.

We should consider social multi-agent systems, where agents interface with other agents (and maybe sometimes with humans as well), that naturally select for kindness. Here, the agents naturally have an incentive to model the needs and wants of others, and agents that rise in the hierarchy/perform well should naturally b...
Contrary to List of Lethality's point 22, alignment's door number 2
by False Name — 2022-12-14T22:01:15.244Z

Abstract: An alternative to the now-predominating models of alignment, corrigibility and "CEV", following a (jaunty, jaunty) critique of these. The critique to show, in substance: CEV and corrigibility have the exact same problems - in effect, they're isomorphs of one another, and each equally unobtainable. This briefly shown, and then, in flat contradiction to point 22 of the "List of Lethalities" (hereafter, "LoL", naturally), there is a quite different way to characterize, indeed, to achieve, alignment.

We begin by considering the cause of Yudkowsky's Despair™, in failing to make usable CEV or corrigibility; it's because they're functionally the exact same thing. Or at least, they lead to the exact same problem. The method which follows, then, is not "door number three" relative to what the "List of Lethalities" calls the "only options" for alignment; following the critique of present approaches is the second way.

Corrigibility (actually, any consequentialist ethics whatever, if they are characterized by "wish" or "desire" or "preference" - anything person-affecting, while up against physical reality, the possibility or practicality of its fulfilment, etc.) founders by the presumptive desire for the best - versus a desire for something. That is, you want something, and presumably you want the best you can get of that something, that's implicit in the human idea of the asking; as an absolute ideal, moreover, you get "the best" of your desire, absolutely. Likewise this tendency emerges in quantity: to get anything, if only on an infinitesimal scale, you need some quantity of it - and the more the better, to be sure of having enough (a version of the "Do I have enough paperclips yet? Better do one more, just to be sure. Do I have enough..." of the paperclip apocalypse) and more, you want as good a quality of what you want as possible, to ensure it's good enough, approaching to the limit of perfection.

In fact, this holds for anything you want, so that, what you want FOR ANY GIVEN DESIRE conceivable, is to receive the best you can receive, so, ideally, "the best" absolutely, of it. It follows, then, that what you actually want is not just the specified something, but the best of anything you might ask for. Except, of course: you didn't actually ask for that; so we've the fundamental contradiction of corrigibility. You want the one sp...
Kolmogorov Complexity and Simulation Hypothesis
by False Name — 2022-12-14T22:01:15.140Z

Previous presentations of the ideas here given have been dogged by a confusion between the "obtainability" versus the "existence" of a Theory of Everything (hereafter "ToE"; dogged also by grammatical errata); it is for the author now to apologize for the vagueness which has enabled such confusions. In this instance, this difference will be directly addressed, and the correct formulation for the desired deduction rendered.

In brief, first: if simulation, everything's computable, so there's a ToE; no ToE? No simulation; contra-anthropic capture following, plus new-model non-Bayesian inference of possible worlds, so actions.

Begin proposing that, for there to be a perfect simulation of phenomena conducted by computer, then within that computer, perhaps accessible to any simulator, is a "source code" describing all entities, events, relations in the simulation, and this source code is thereby isomorphic to a "Theory of Everything" (as an aside, this seems to imply any such simulation is a Mathematical Universe, per Tegmark).

Thereby (roughly) let "is a computerized source code of a simulation" be defined as "p", "yields a ToE" be defined as "q". Then p implies q. And since we might conceive a ToE as separate from a simulation (as a non(ITAL) simulated universe we can conceive), it is a necessary condition, and that formula's form is validated.

But then, from transposition (or modus tollens, or disjunctive syllogism; as one wills), not-q implies not-p. So that: can we by any means find a ToE is impossible for - not only in - a given universe; that is, the universe exists, but a uniform mathematical theory accounting for it cannot exist and we confirm such a theory is impossible, we thereby exclude the possibility that universe is simulated; likewise for our universe, naturally. Ergo, that impossibility identified, it would follow we are not in a simulation. QED.

Now, do you "assume, for the sake of argument," the above, the foregoing makes decidable, in at least one instance, and given certain data, whether you're in a simulated universe; the simulation hypothesis, the notion that there is no such method, must therefore be false. In accordance with the accepted etiquette of argument, if so-and-so, then such-and-such, you've assumed, and you know the argument valid.

However, even that obeyed, perhaps you find the suspensi...
Stanley Meyer's water fuel cell
by mikbp — 2022-12-14T21:19:20.158Z

I read somewhere a comment claiming that a "water engine" (Stanley Meyer's fuel cell) is possible. Wikipedia calls it a perpetual motion machine, but a quick search indicates that the guy just claimed it used water as fuel (nothing about it producing more energy than it consumed).

The thing is that the topic is very prone to conspiracy-story telling (plus the guy died young), and I don't have the will and time to separate the wheat from the chaff.

Has anyone read about it and can make a nice review of the topic?
Is the AI timeline too short to have children?
by Yoreth — 2022-12-14T18:32:45.045Z

If you had asked me at any point in my adult life (until recently) whether I wanted to have children eventually, I would've said yes, without hesitation. In recent years I've been telling myself: I don't know how likely these AI doom predictions are, but I'm going to focus on optimizing the "long path" because that's where my decisions actually matter - and so I should still have children just in case.

But now, both as I'm nearing the family-forming stage in my life, and as the AI timeline seems to be coming into sharper focus, I'm finding it emotionally distressing to contemplate having children.

If AI kills us all, will my children suffer? Will it be my fault for having brought them into the world while knowing this would happen? Even if I think we'll all die painlessly, how can I look at my children and not already be mourning their death from day 1? If I were to die right now, I would at least have had a chance to live something like a fulfilling life - but the joy of childhood seems inextricable from a sense of hope for the future. Even if my children's short lives are happy, wouldn't their happiness be fundamentally false and devoid of meaning?
Predicting GPU performance
by Marius Hobbhahn — 2022-12-14T16:27:23.923Z
Link: https://epochai.org/blog/predicting-gpu-performance

We develop a simple model that predicts progress in the performance of field-effect transistor-based GPUs under the assumption that transistors can no longer miniaturize after scaling down to roughly the size of a single silicon atom. We construct a composite model from a performance model (a model of how GPU performance relates to the features of that GPU), and a feature model (a model of how GPU features change over time given the constraints imposed by the physical limits of miniaturization), each of which are fit on a dataset of 1948 GPUs released between 2006 and 2021. We find that almost all progress can be explained by two variables: transistor size and the number of cores. Using estimates of the physical limits informed by the relevant literature, our model predicts that GPU progress will stop roughly between 2027 and 2035, due to decreases in transistor size. In the limit, we can expect that current field-effect transistor-based GPUs, without any paradigm-altering technological advances, will be able to achieve a peak theoretical performance of 1e14 and 1e15 FLOP/s in single-precision performance.

Figure 1. Model predictions of peak theoretical performance of top GPUs, assuming that transistors can no longer miniaturize after scaling down transistors to around 0.7nm. Left: GPU performance projections; Top right: GPU performance when the limit is hit; Middle right: Our distribution over the physical limits of transistor miniaturization; Bottom right: Our distribution over the transistors per core ratio with relevant historical comparisons.

While there are many other drivers of GPU performance and efficiency improvements (such as memory optimization, improved utilization, and so on), decreasing the size of transistors has historically been a great, and arguably the dominant, driver of GPU performance improvements. Our work therefore strongly suggests that it will become significantly harder to achieve GPU performance improvements around the mid-2030s within the longstanding field-effect transistor-based paradigm.

This post is just the executive summary, you can find the full report on the Epoch website.
[Incomplete] What is Computation Anyway?
by DragonGod — 2022-12-14T16:17:43.093Z
Link: https://arxiv.org/abs/1403.4880

Disclaimer

This post was written three months ago, and I haven't really touched it since then. I never fully completed it[1], and I'm not satisfied with the current draft. But as is, I'm not on track to complete it anytime in the foreseeable future.

At the time I started it, I believed it to be pretty important as far as my LW essays go. Given that, it seems marginally more dignified to publish the draft as is[2].

As at the time of publishing, I don't necessarily still endorse it.

Epistemic Status

Unconfident, exploratory. This is a dialogue and an exercise in deconfusion, not necessarily a novel contribution.

Preamble

I've been quite confused about computation, and I recently read a paper that made me feel significantly less confused. This is a review/exploration of the paper, and a documentation of my deconfusion. Because it's an exercise in my deconfusion, I will explore some of my own inferences and thinking. It's not intended to be a faithful representation of the paper[3].

I'll leave any lingering confusion in my understanding of computation to Cunningham's Law.

Acknowledgements

Many thanks to @davidad, for discussing this topic with me and answering some of my questions. He also recommended the insightful paper.

Introduction

Automata theory answers how we compute. Models of computation give us idealised abstractions of methods to compute. But the question of what a "computation" actually is, remains unresolved. What does it mean to "compute". When we "compute" something, what are we actually doing? Why do we engage in that activity?

The paper poses two questions:

Why do we compute?
What do we compute?

I found the treatment of the first question very enlightening, and think a satisfactory account of it was given. The treatment of the second question, however, felt very unsatisfactory and left me confused. But, it was a productive confusion and led me to some interesting thinking.

Because I don't really understand/appreciate the paper's treatment of the second question, I'm going to set it aside, and instead explore the avenues of thinking it led me down in my confusion.

Why Do We Compute?

Naively, we compute to acquire/obtain information.

There are two objections to the naive view.

One problem is that information cannot be created de novo. Computation cannot actually create information out of nothing (information is conserved in a total ...
Chair Hanging Peg
by jefftk — 2022-12-14T15:30:03.739Z

Our house is now back up to full occupancy, which meant some things
that had been hanging out in unused rooms needed better homes. Four
of those things were

folding
chairs, and now they are up and out of the way:

I made this out of three pieces of scrap wood: two bits of 2x6 and a 1
1/4 dowel. Here are the 2x6s, painted and screwed together, with the
hole for the dowel drilled:

The outer 2x6 is screwed to the inner one, and the inner one is
screwed through the wall into a stud. The dowel stays in the hole by
friction: it's a close fit and it's not going anywhere. This works
for these particular chairs because they have a nice round hole in the
seat back; other chair designs would likely need two pegs.

There are chair-hanging products you can buy, but I didn't like any of
them and this was a quick project I could do with things I had left
over from earlier projects.
My AGI safety research—2022 review, ’23 plans
by Steven Byrnes — 2022-12-14T15:15:52.473Z

“Our greatest fear should not be of failure, but of succeeding at something that doesn't really matter.” –DL Moody (allegedly)

The short version:

In this post I’m briefly summarizing how I spent my work-time in 2022, and what I’m planning for 2023.

The first half of 2022 was writing the

“Intro to Brain-Like-AGI Safety”

blog post series.
The second half of 2022 was split maybe 45%-45%-10% between my main research project (on reverse-engineering human social instincts), miscellaneous other research and correspondence, and outreach mostly targeted towards neuroscientists.

I expect to carry on with a similar time allocation into 2023.

If you think there are other things I should be doing instead or differently, please don’t be shy, the comment section is below, or DM me, email, etc.

The long version:

1. First half of 2022: Writing “Intro to Brain-Like AGI Safety”

So, I was writing some technical post in late 2021, and realized that the thing I was talking about was a detail sitting on top of a giant pile of idiosyncratic beliefs and terminology that nobody else would understand. So I started writing a background section to that post. That background section grew and grew and grew, and eventually turned into a book-length series of 15 blog posts entitled

“Intro to Brain-Like AGI Safety”

, which reorganized and re-explained almost everything I had written and thought about up to that point, since I started in the field around 2019. (My

palimpsest

!) Writing that series took up pretty much 100% of my work time until May 2022.

Then I spent much of the late spring and summer catching up on lots of miscellaneous to-do-list stuff that I had put off while writing the series, and everyone in my family caught COVID[1], and we took a family vacation, and I attended two conferences, and I

switched jobs

when Jed McCaleb generously offered me a home at Astera Institute, and various other things. So I didn’t get much research done during the late spring and summer.

Moving on to the rest of the year, my substantive work time has been divided, I dunno, something like 45%-45%-10% between “my main research project”, “other research”, and “outreach”. Let’s take those one at a time in the next three sections.

2. Second half of 2022 (1/3): My main research project

2.1 What’s the project?

I’m working on the open neuroscience problem that I described in the post

Symbol Grou

...
Extracting and Evaluating Causal Direction in LLMs' Activations
by Fabien Roger — 2022-12-14T14:33:05.607Z

This post was written by Fabien at SaferAI[1]. Simeon has prompted Fabien in relevant directions and has provided valuable feedback.

Thanks to Jean-Stanislas Denain, Alexandre Variengien, Charbel-Raphael Segerie, and Nicole Nohemi for providing helpful feedback on early experiments and drafts of this post.

In this post

I describe a method to evaluate that a specific direction encodes information relative to a specific concept,
I use it to evaluate how good directions found through probing techniques are,
I present a way of using causal intervention to find directions that maximally encode information relative to a specific concept, which I call Causal Direction Extraction (CDE),
I present some preliminary results about the directions found through Causal Direction Extraction

One of the main goals of this post is to get suggestions on further experiments to run. I think the current data I gathered is not strong enough to prove or disprove claims like “CDE finds a direction, which mostly captures the concept of gender in GPT-J”, but further experiments could.

The code for all experiments is available

here

.

[EDIT] I'm less excited about this technique than I was when I wrote this post. This is because of two experiments I ran since. First, CDE works much less on automatically augmented data (take OpenWebText and swap pronouns and names), whereas other methods such as mean difference ablation (see MP and LEACE) still work reasonably well. Second, ablation along the gender direction fails to consistently reduce bias on gender bias benchmarks.

How to Evaluate How Much a Direction Encodes a Concept

What It Means for a Direction to Encode a Concept

Let’s say you have a neural network N. Let N<=L be the network up to a given layer L, and N>L be the network after that, such that for all input x, N>L(N<=L(x))=N(x).

The claim that a (normalized) direction d in the space of activations A[2] encodes a target concept between layers L and L+1 can be understood as follows:

Activations after layer L can be cleanly cut into two pieces:

c(x), the projection of N<=L(x) along d, which only encodes information relative to the target concept
rest(x), the projection of N<=L(x) orthogonal to d, which only encodes all information not r

...
Key Mostly Outward-Facing Facts From the Story of VaccinateCA
by Zvi — 2022-12-14T13:30:00.831Z

I had the privilege of getting an advance draft of Patrick McKenzie’s very long story founding and running VaccinateCA, an organization dedicated to providing Americans information on where and how to get vaccinated against Covid-19.

It is an amazing document. Despite its length, I am going to flat out say to Read the Whole Thing if you have the capability of doing that. It is full of things worth knowing and understanding. I benefited from reading it twice, once as a draft and again as a finished document.

Since the VaccinateCA post remains very long and difficult to navigate or draw reference from, this post is an attempt to pull out the most long-term-important passages and facts and give an outline of the events and insights that reflect the world outside VaccinateCA, to keep this bounded. I have a short bullet point section on the inside lessons as well, but to keep this bounded am focusing elsewhere.

A central theme is the author, Patrick McKenzie, being surprised and confused by how dysfunctional were government operations, and how deprioritized was the cause of having less citizens get sick and die. As you read, notice these repeated surprises and confuses, and notice that the correct model would not be so reliably surprised in the same directions.

Similarly, he notes that he believes some of his statements present the situation as so dysfunctional that they sound crazy. The statements do not sound crazy to me at all. They sound accurate, and like the type of thing one would expect. It seems important, if we want things to improve, to realize that such statements are accurate, normal and expected.

Again, seriously, read the original if you can, at least the parts about how government systems (largely didn’t) track the vaccine and ensure it was distributed to citizens.

The Logistical Problems With Distribution Were Unnecessary

For some reason we decided that our standardized cold chain, sufficient to keep the vaccines good for ten weeks, was not good enough, despite that obviously being good enough. So for no real physical reason, we didn’t use our existing infrastructure.

The Pfizer vaccine keeps for ten weeks when transported or stored unopened between two degrees Celsius and eight degrees Celsius (36°F and 46°F), which is the exact same range as the flu vaccine and overlaps that of milk.

…

Transportation was easily achievable via cold-chain

...
Discovering Latent Knowledge in Language Models Without Supervision
by Xodarap — 2022-12-14T12:32:56.469Z
Link: https://arxiv.org/abs/2212.03827

Existing techniques for training language models can be misaligned with the truth: if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can't detect. We propose circumventing this issue by directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way. Specifically, we introduce a method for accurately answering yes-no questions given only unlabeled model activations. It works by finding a direction in activation space that satisfies logical consistency properties, such as that a statement and its negation have opposite truth values. We show that despite using no supervision and no model outputs, our method can recover diverse knowledge represented in large language models: across 6 models and 10 question-answering datasets, it outperforms zero-shot accuracy by 4% on average. We also find that it cuts prompt sensitivity in half and continues to maintain high accuracy even when models are prompted to generate incorrect answers. Our results provide an initial step toward discovering what language models know, distinct from what they say, even when we don't have access to explicit ground truth labels.

Some discussion on Twitter including Eliezer calling it "Very dignified work!"