trentmkelly/LessWrong-43k huggingface dataset:
Progress Report 2
by Nathan Helm-Burger — 2022-03-30T02:29:32.670Z

Prev: Report 1

Next: Report 3

In looking at interpretability in transformers (1), I've been looking at the Causal Tracing notebook here: https://rome.baulab.info/ (2)

Expanding on this code somewhat, I made an attempt to increase the 'resolution' of the trace from layer-specific down to neuron-specific. I'm not all that confident yet that my code to do this is correct, since I haven't had a chance to go over it with anyone else yet (please feel free to message me with offers to help review my code). If there are mistakes, I hope that they are small enough ones that the general idea still works once corrected. With that in mind, here's a couple example plots:

In the plots, each point of red is a specific neuron in an MLP layer in GPT2-xl from Hugging Face. The rectangular shape of the layers in the plot is an arbitrary convenience for visualization (unlike how the shape of layers in a CNN would actually be relevant).

My hope is that I can use this information to collect a dataset of neuron -> context associations which I can then use as edges in a weighted bipartite graph to do clustering of the neurons in the transformer along the same line as what Filan et al. 2021 (3) did in CNNs. I'm running this now, starting with the set of 1200 or so facts given in the demo notebook. It's pretty slow

Notes

Thanks to my AGI Safety Fundamentals discussion group for sparking ideas and giving reading suggestions, in particular the discussion leader William Saunders.
Locating and Editing Factual Knowledge in GPT
Kevin Meng*1, David Bau*2, Alex Andonian1, Yonatan Belinkov3
1MIT CSAIL, 2Northeastern University, 3Technion - IIT; *Equal Contribution
Clusterability in Neural Networks
Daniel Filan1, *, Stephen Casper2, *, Shlomi Hod3, *,
Cody Wild1, Andrew Critch1, Stuart Russell
What would make you confident that AGI has been achieved?
by Yitz — 2022-03-29T23:02:58.250Z

Consensus seems to be that there is no fire alarm for artificial intelligence. We may see the smoke and think nothing of it. But at what point do we acknowledge that the fire has entered the room? To be less euphemistic, what would have to happen to convince you that a human-level AGI has been created?

I ask this because it isn’t obvious to me that there is any level of evidence which would convince many people, at least not until the AGI is beyond human levels. Even then, it may not be clear to many that superintelligence has actually been achieved. For instance, I can easily imagine the following hypothetical scenario:

A future GPT-N which scores a perfect 50% on a digital Turing Test (meaning nobody can detect if a sample output is written by humans or GPT-N), is announced by OpenAI. Let’s imagine they do the responsible thing and don’t publicly release the API. My intuition is that most people will not enter panic mode at that point, but will either:

Assume that this is merely some sort of publicity stunt, with the test being artificially rigged in some way.
Say something like “yes, it passed the Turing test, but that doesn’t really count because [insert reason x], and even if it did, that doesn’t mean it will be generalizable to domains outside of [domain y that GPT-N is believed to lie inside of].”
Claim that being a good conversationalist does not fully capture what it means to be intelligent, and thereby dismiss the news as being yet another step in the long road towards “true” AGI.

The next week, OpenAI announces that the same model has solved a massive open problem in mathematics, something that a number of human mathematicians had previously claimed wouldn’t be solved this century. I predict a large majority of people (though probably few in the rationalist community) would not view this as being indicative of AGI, either.

The next week, GPT-N+1 escapes, and takes over the world. Nobody has an opinion on this, because they’re all dead.

This thought experiment leads me to ask: at what point would you be convinced that human-level AGI has been achieved? What about superhuman AGI? Additionally, at what point would you expect the average (non-rationalist) AI researcher to accept that they’ve created an AGI?
Beyond Blame Minimization: Thoughts from the comments
by physicaleconomics — 2022-03-29T22:28:38.493Z

There were a lot of fantastic comments on this post. I want to break down some common themes and offer my thoughts.

Incentives

Unsurprisingly, there was a lot of focus on the role that incentives—or a lack thereof—play in a bureaucracy.

Dumbledore's Army:

I’ve been asking myself the same question about bureaucracies, and the depressing conclusion I came up with is that bureaucracies are often so lacking incentives that their actions are either based on inertia or simply unpredictable. I’m working from a UK perspective but I think it generalises. In a typical civil service job, once hired, you get your salary. You don’t get performance pay or any particular incentive to outperform.[1] You also don’t get fired for anything less than the most egregious misconduct. (I think the US has strong enough public sector unions that the typical civil servant also can’t be fired, despite your different employment laws.) So basically the individual has no incentive to do anything.

As far as I can see, the default state is to continue half-assing your job indefinitely, putting in the minimum effort to stay employed, possibly plus some moral-maze stuff doing office politics if you want promotion. (I’m assuming promotion is not based on accomplishment of object-level metrics.) The moral maze stuff probably accounts for tendencies toward blame minimisation.

Some individuals may care altruistically about doing the bureaucracy’s mission better, eg getting medicines approved faster, but unless they are the boss of the whole organisation, they need to persuade other people to cooperate in order to achieve that. And most of the other people will be enjoying their comfortable low-effort existence and will just get annoyed at that weirdo who’s trying to make them do extra work in order to achieve a change that doesn’t benefit them. So the end result is strong inertia where the bureaucracy keeps doing whatever it was doing already.

tailcalled quotes John Wentworth, who talks about incentives in terms of degrees of freedom:

Responding to Chris: if you go look at real bureaucracies, it is not really the case that "at each level the bosses tell the subordinates what to do and they just have to do it". At every bureaucracy I've worked in/around, lower-level decision makers had many de facto degrees of freedom. You can think of this as a generalization of one of the central problems of

...
Gears-Level Mental Models of Transformer Interpretability
by RowanWang — 2022-03-29T20:09:53.073Z

This post aims to quickly break down and explain the dominant mental models interpretability researchers currently use when thinking about how transformers work.

In my view, the focus of transformer interpretability research is teleological: we care about the functions each component in the transformer performs and how those functions interact to yield the capabilities we see in language models today. From a functional understanding of transformer internals, we then hope to be able to answer other important interpretability questions such as, “Where/How is knowledge stored?,” “Do transformers have beliefs?”, and “How do transformers reason?” As such, the mental models described in this post will be functional theories about how researchers think about transformer internals, and not hypotheses about other interpretability questions, like the ones mentioned above.

There are three main components at work in the transformer: the attention heads, the MLP, and the additive residual stream that connects all the layers. The functions that each play aren’t clear and are the subject of much research, and the mental models described below will be different ways of thinking about each main component. None of these mental models are mutually exclusive, and in reality, transformer internals probably look like a messy combination of many of these models.

This post assumes a working understanding of transformers. For a primer on the transformer architecture, I recommend

The Illustrated Transformer

,

Transformers from Scratch

, and

The Annotated Transformer

. Other works that this post draws greatly from are

the logit lens

,

Transformer Feed-Forward Layers Are Key-Value Memories

,

ROME

, and the

Anthropic

papers

.

The Models

Residual Stream as Output Accumulation

The residual stream is simply the accumulation of all the stuff the transformer wants to say at the end of its inference step.

Clearly, the last residual hidden state is the transformer’s prediction (before it gets projected to the vocabulary), and all this mental model is suggesting is that the prior hidden states are less-refined versions of the final residual hidden state. The strongest version of this model states that the middle hidden states are the model’s nascent predictions, and the weaker version just states that the middle hidden states&nbs...
Retreat for rationality meetup organizers (apply to attend!)
by mingyuan — 2022-03-29T18:42:58.292Z

TL;DR I'm running a retreat for meetup organizers in the Bay Area from July 21–24. More info here; you can apply using this form.

ETA: The application form will be open through the end of the day on April 30th, and final decisions will go out by May 7th.

Why a retreat?

I think we would all benefit from having an international community of rationality and ACX meetup organizers who can go to one another for advice and support. The goal of this retreat is to jumpstart that community. You'll spend time together and help each other solve problems, hopefully forming connections that will allow you to reach out for help even after you've gone back home to your respective cities.

In addition to getting to know each other, you'll be taking part in activities and discussions with the goal of improving your meetup group. I hope that you'll come out of the retreat with a clearer vision for your local community, and concrete plans for how to work towards that vision.

Who is this retreat for?

You're invited if you:

Currently organize ACX/LW/rationality meetups or other rationalist community events, anywhere in the world
Would like to start an ACX/LW/rationality community in a city where none exists, or where the community has been small or dormant for a long time
Formerly organized ACX/LW/rationality meetups and are interested in sharing your knowledge and experience

If you're unsure whether you 'qualify', please err on the side of applying anyway — if we don't think it makes sense for you to come, we'll let you know!

Food, lodging, and travel are covered

We don't want money to be an obstacle preventing anyone from attending, so your travel to and from the Bay will be fully reimbursed, and your food and lodging will be covered during the retreat itself. If you would like to stay longer in the Bay, we by default expect that you will cover your own expenses for that time, but if that's a problem for you, please don’t hesitate to reach out to me.

Apply here

or check out this FAQ doc for more info.

Let me know if you have any questions! I can be reached in various ways, but probably the most reliable is emailing meetupsmingyuan@gmail.com. I'm looking forward to your applications!
Debating myself on whether “extra lives lived” are as good as “deaths prevented”
by HoldenKarnofsky — 2022-03-29T18:30:23.792Z

Click lower right to download or find on Apple Podcasts, Spotify, Stitcher, etc.

If we could do something to lower the probability of the human race going extinct,1 that would be really good. But how good? Is preventing extinction more like “saving 8 billion lives” (the number of people alive today), or “saving 80 billion lives” (the number who will be alive over the next 10 generations) … or "saving 625 quadrillion lives" (an Our World in Data estimate of the number of people who could ever be born) ... or “saving a comically huge number of lives" (Nick Bostrom argues for well over 10^46 as the total number of people, including digital people, who could ever exist)?

More specifically, is “a person getting to live a good life, when they otherwise would have never existed” the kind of thing we should value? Is it as good as “a premature death prevented?”

Among effective altruists, it’s common to answer: “Yes, it is; preventing extinction is somewhere around as good as saving [some crazy number] of lives; so if there’s any way to reduce the odds of extinction by even a tiny amount, that’s where we should focus all the attention and resources we can.”

I feel conflicted about this.

I am sold on the importance of specific, potentially extinction-level risks, such as risks from advanced AI. But: this is mostly because I think the risks are really, really big and really, really neglected. I think they’d be worth focusing on even if we ignored arguments like the above and used much more modest estimates of “how many people might be affected.”

I’m less sold that we should work on these risks if they were very small. And I have very mixed feelings about the idea that “a person getting to live a good life, when they otherwise would have never existed” is as good as “a premature death prevented.”

Reflecting these mixed feelings, I'm going to examine the philosophical case for caring about “extra lives lived” (putting aside the first bullet point above), via a dialogue between two versions of myself: Utilitarian Holden (UH) and Non-Utilitarian Holden (NUH).2

This represents actual dialogues I’ve had with myself (so neither side is a pure straw person), although this particular dialogue serves primarily to illustrate UH's views and how they are defended against initial and/or basic objections from NUH. In future dialogues, NUH will raise more sophisticated o...
Hinges and crises
by Jan_Kulveit — 2022-03-29T11:11:03.605Z
Link: https://forum.effectivealtruism.org/s/dr4yccYeH6Zs7J82s/p/DzBEfDdsd4ucxChyk

Crossposted from EA forum. The second post in the sequence covers the importance of crises, argues for crises as opportunities, and makes the claim that this community is currently better at acting with longer timescale OODA loops but lacks skills and capabilities to act with short OODA loops.
We often talk about the

hinge of history

, a period of high influence over the whole future trajectory of life. If we grant that

our century is such a hinge

, it’s unlikely that the "hinginess" is distributed uniformly across the century; instead, it seems much more likely it will be concentrated to some particular decades, years, and months, which will have much larger influence. It also seems likely that some of these "hingy" periods will look eventful and be understood as crises at the time. So understanding crises, and the ability to act during crises, may be particularly important for influencing the long-term future.

The first post in this sequence mentioned my main reason to work on COVID: it let me test my models of the world, and so informed my longtermist work. This post presents some other reasons, related to the above argument about hinges. None of these reasons would have been sufficient for me personally on their own, but they still carry weight, and should be sufficient for others in the next crisis.[1]

An exemplar crisis with a timescale of months

COVID has commonalities with some existential risk scenarios. (See

Krakovna

.) Lessons from it could transfer to risks in which:

the crisis unfolds over a similar timescale (weeks or years, rather than seconds or hours),
governments have some role,
the risk is at least partially visible,
the general population is engaged in some way.

This makes COVID a more useful comparison for versions of continuous AI takeoff where governments are struggling to understand an unfolding situation, but in which they have options to act and/or regulate. Similarly, it is a useful model for versions of any x-risk where a large fraction of academia suddenly focuses on a topic previously studied by a small group, and resources spent on the topic increase by many orders of magnitude. This emergency research push is likely in scenarios with a warning shot or sufficiently loud

fire alarm

that gets noticed by academia.

On the other hand, lessons learned from COVID will be correspondingly less useful for cases where few of the above assumption...
Strategies for differential divulgation of key ideas in AI capability
by azsantosk — 2022-03-29T03:22:35.825Z

Openness makes the AI race worse

I will start with a short discussion of this article on the implications of openness in AI research. If you are very familiar with this argument, feel free to skip this section.

While the paper makes a lot of effort to present arguments on both sides, the strongest point is that openness makes the AI development race more competitive.

The oversimplified model of how openness affects the AI race dynamic is one in which there are k ideas necessary to build an AGI. Once all these insights are available, you can actually start working on the implementation details, which may take several months to a few years. If insights are kept to the teams who found them, it is likely that the front-runner team (likely to be the one more productive in finding key ideas) will have some head start. If instead most or all of the insights are published openly, then many teams will start working on the implementation nearly at the same time, and the race will be very competitive.

This is particularly bad as it makes it harder for a leading AI developer to pause or slow down capability research to develop safety methods, or to implement performance-handicapping safety controls, without abandoning the lead to some other less careful developer.

First model: front-runner team committed to AGI safety

To me the above is more than enough to out-weight any positives associated with openness. As a result, my first model of how things could go well includes a team strongly committed to AGI safety succeeding not only in becoming the front-runner of AI development, but also doing so with a significant head start.

This sounds difficult. Those committed to AGI safety seem to be a relatively small minority among AI researchers today, so unless that changes dramatically, it seems a priory unlikely that one of the teams committed to safety will become the front-runner, and that becomes even more unlikely if we require a significant initial lead.

Coordination advantages of "ideological" teams

Not all is lost, though. We, those committed to AGI safety, have a significant advantage: we are united, or at least we have the potential to be. If we can coordinate ourselves, we may be able to increase our changes dramatically.

Let us assume for the sake of the argument that we have solved these coordination problems and that there is a centralized authority, capable ...
Towards a better circuit prior: Improving on ELK state-of-the-art
by evhub — 2022-03-29T01:56:39.684Z

Thanks to Paul Christiano for useful comments and feedback.

The basic circuit prior setup

We’ll start with the basic setup that we’re trying to improve upon, which is trying to solve ELK via the use of a Boolean circuit size prior. Previously, Evan summarized Paul, Mark, and Ajeya’s argument for why this might work as follows:

As in the ELK report, there is a plausible argument for why there exists a speed prior that would prefer the direct translator to the human imitator. Naively, the problem with the speed prior here is that the computation required for the human imitator is proportional to the size of the human's Bayes net, whereas the computation required for the direct translator is proportional to the size of the model's Bayes net—and in the superhuman limit we should expect the latter to be substantially larger than the former.

The argument in the ELK report, however, is that while this argument is valid in the limit, there's reason to believe it might be invalid for all the finite cases that we care about. That's because perfect inference in either Bayes net, and thus perfect loss, shouldn't be possible in any finite case. Thus, the performance of the ontology mapping function, and thus its loss, should be proportional to how much computation it puts into its inference task—for which the direct translator has a big advantage, since it gets to reuse the computation performed by the model.

The obvious response here, and the response that is given in the ELK report, is that the above argument is very fragile—it relies on inference in the human's Bayes net being too hard to always get right on the training distribution, which is a strong assumption both about the difficulty of inference and the complexity of the training data.

Furthermore, as the ELK report also notes, it's not enough for the direct translator to just be more efficient than the human imitator: the direct translator has to be a cost-effective improvement (in terms of how much loss/computation it saves per increase in description complexity) compared to all other possible mappings.

Overall, however, despite the issues with this approach, we agree with Paul, Mark, and Ajeya’s conclusion that it is the most promising approach currently on offer:[1]

In order to ensure we learned the direct translator, we would need to change the training strategy to ensure that it contains sufficiently chall

...
Can we simulate human evolution to create a somewhat aligned AGI? 
by Thomas Kwa — 2022-03-28T22:55:20.628Z

Epistemic status: less than 20% chance that this is a good idea. I've spent about 25 hours thinking about this.

TL;DR: If AI alignment is intractable, but human evolution is robust in producing something close to human values, we could try to simulate/mimic human evolution to create a superintelligent successor AI. This plan would have various problems, especially training competitiveness.

Many people I know put significant credence in the following three statements:

AI alignment is very difficult, and has a <<50% chance of success, largely because we have no idea how to put human values into an AI.
Even if AI alignment is intractable, we know of one process that definitely creates general intelligence with human values: the evolution and cultural development of humans.
Medium-sized perturbations to the human evolutionary process wouldn't destroy human values. I have heard MIRI people guess that an AI aligned with elephants, or with aliens that evolved in a similar environment to humans ("humanlike aliens"), would be much better than an AI with a random goal chosen through SGD, perhaps 50% as good as an AI aligned with human

CEV

.

If you're one of these people, this post describes a different approach to making the future go well which you might find plausible. While alignment requires some way to specify human values and put them into an AI, a successor species plan requires creating a system that generates values similar to ours, from scratch, by some process similar to human evolution. The resulting superintelligence will not be totally aligned with human values. However, if the plan is sufficiently tractable while alignment is intractable, it may have higher expected value than aiming for aligned AI. I believe that such plans are neglected because many AI pessimists are uncomfortable thinking about this backup plan.

How to create a successor AI by simulating evolution?

If we had infinite compute and perfect knowledge of the evolutionary history of humans, we could just simulate the evolution of humans and end up with simulated humans, who could then either become superintelligent through self-modification, or solve the AI alignment problem by thinking for thousands of subjective years, likely achieving human CEV. If we had infinite compute but not perfect knowledge, we could execute

Paul Christiano's plan

of simulating the entire evolutionar...