trentmkelly/LessWrong-43k huggingface dataset:
Decomposing independent generalizations in neural networks via Hessian analysis
by Dmitry Vaintrob — 2023-08-14T17:04:40.071Z

In our joint SERI MATS project, we came up with a series of equations and experiments to mechanistically understand and steer the generalization behavior of neural nets. The core conceit is to locate the circuits (which we call "modules") responsible for implementing different generalizations using a toolbox of techniques related to Hessian eigenvectors. This is a general-audience distillation of our work.

We hope most of the ideas and high-level goals are understandable to a non-expert, though, for most of our experiments, we attempt to include "supplementary" material with the main mathematical intuitions and concrete equations that would allow someone to reproduce our work. We plan in the coming weeks to write multiple follow-up distillations and discussions, both of some of the more technical parts of our work and of a few new insights into generalization behavior and phase transitions in general that came out of experiments involving our Hessian toolbox.

Introduction

A central problem for inner alignment is understanding how neural nets generalize off-distribution. For example, a powerful AI agent trained to make people happy can generalize either by choosing actions that deceptively look good to its overseers or those that truly align with human values. The same diversity of generalization is already seen in existing real-world tasks both in minor ways (image classifiers classifying cars by learning to recognize wheels vs. windows) and in serious ways (language models appearing honest by agreeing with the user vs. insisting on consensus opinions).

One approach to steer between generalizations is activation steering, which Nina is investigating as her other SERI MATS project. This aims to encourage the neural net to implement one possible generalization (in this case, honestly reflecting the LLM's internal world model) instead of the other generalization (in this case, sounding good and correct to a particular user).

While activation steering, supervised finetuning, and RLHF work well in practice and can make systems behave better, there is still a risk that powerful models generalize in unpredictable and potentially undesirable ways in out-of-distribution examples. In particular, for subtle alignment-related questions like deception or power-seeking, activation steering or RLHF may fix the "symptoms" of the problem on examples similar to ...
Memetic Judo #2: Incorporal Switches and Levers Compendium
by Max TK — 2023-08-14T16:53:05.363Z

Exposé

I regularly have encounters with people who "are not afraid of misaligned AGI as long as it doesn't even have a body to inflict harm with in the real world".
This article is intended as a study of how to deal with this kind of argument.
It can also be used as a reference for people who remain unconvinced for this reason.

Example Argument

I am not very concerned about strong artificial intelligence, because as long as we don't put it inside robot bodies, its options for affecting the real world are very limited. You can't plot a human extinction level threat if you are constrained to the internet/the digital realm. For example, such a system would probably not decide to turn against us as long as it depends on humans to maintain its server and power infrastructure.

Initial Thoughts

It is my general impression that proponents of this argument usually either have not thought enough about the question of what such an "incorporal AI" could do in order to affect the state of the real world, or they lack the imagination or domain knowledge to come up with a list of things that they would consider impressive enough for it to be considered a legitimate threat.

The obvious response is then to come up with such a list yourself, or share the one that follows.

This "compendium of incorporal switches and levers" is guaranteed to not be a complete account of all the options available to such an AI - nor does it need to be. It is a collection of examples that I (and the people I talk to) could come up with.

We should - of course - take it for granted that a very intelligent AI would also think of things that I and others would have trouble to predict or even imagine.

Remarks

Parts of this collection can also be used for responding to different arguments concerned with the threat level of an uncontained AGI, for example the "we can just turn it off or destroy its servers" class of arguments.

There are "shortcut" counter arguments that capture the spirit in which this list was made. For example (thanks Karl von Wendt):

If an AGI could not significantly change the world without a body, neither could a human dictator (for example he could not invade other countries on his own).

Potential Infohazard Disclaimer

I have thought about the security risk of sharing such a list online, however I do think that these are mostly points that are somewhat obvious and would clea...
Stepping down as moderator on LW
by Kaj_Sotala — 2023-08-14T10:46:58.163Z

Just a brief announcement that I'm stepping down as a moderator on LW. The reason is that I've been doing emotional coaching for some time now, and several of my clients are a part of the rationalist / effective altruism communities. This often involves them sharing intimate personal details about what's happening with them. While this hasn't created any conflict of interest so far, I feel that I should not hold an official position of power within the community (like being an LW mod) while also having a therapist-type role for people within that same community.

In practice, I haven't used my mod powers much aside from occasionally deleting spam or answering questions about the site that people have messaged me with, so this doesn't change very much.

I do intend to continue posting normally and feel good about my time with the mod team. :)
Announcing Manifest 2023 (Sep 22-24 in Berkeley)
by Saul Munn — 2023-08-14T05:13:03.186Z

Forecasting Festival hosted by Manifold

TL;DR: Manifold is hosting a conference! 🥳 Chat with the Manifold team, special guests like Robin Hanson, Shayne Coplan, Patrick McKenzie, Dylan Matthews, Destiny, Aella, and more at our inaugural in-person gathering of the forecasting & prediction market community.

More info & buy tickets: manifestconference.net

WHEN: Sept 22-24

WHERE: Berkeley, CA

WHO: Everyone in the forecasting, EA, and LW communities. If you're reading this, you're invited!

Join the Discord here :)

Why should I come?

Forecasting and prediction markets are effective ways of improving our judgement and decision-making — most people in the Effective Altruism and LessWrong communities will feel right at home at Manifest.

Here are extra reasons to come:

you think forecasting & prediction markets are impactful/fun/cool/rational/intriguing/etc
you want to vibe with other forecasting nerds
you want to engage & network with the forecasting community (find jobs/recruit hires/see what’s out there)
you want to meet & chat with the Manifold team and our special guests (including Robin Hanson, Shayne Coplan, Patrick McKenzie, Dylan Matthews, Destiny, Aella, and more!)
you want enjoy the gorgeous Rose Garden Inn
…or if you like memes?

A day at Manifest

Everything’s optional. There will always be a bunch of sessions running concurrently, but this is an example of what your day at Manifest might look like:

10-11 — Opening session

11-12 — Fireside chat: Robin Hanson

12-1 — Lunch & mingling

1-2 — Estimathon: fermi estimation with prizes and steep competition!

2-3 — Speed friending: a few chats with other friendly, ambitious forecasting nerds :)

3-4 — Break: relax, vibe, unwind, chill, destress, etc

4-5 — Panel: Forecasting Founders (hear from Manifold, Kalshi, Polymarket, Insight Predictions, and more!)

5-6 — Games & markets: chess, poker, and prediction markets!

6-7 — Dinner & mingling

7-8 — Workshop: How to Write Good Forecasting Questions

8-12 — Murder She Bet: a murder mystery + a low-tech prediction market = fun!

Who else is coming?

The rest of the forecasting & prediction market community. Current speakers & special guests include:

The whole Manifold team, including co-founders Austin Chen and Stephen & James Grugett
Robin Hanson (economist & professor)
Shayne Coplan (CEO of Polymarket)
Patrick McKenzie (writer, Advisor at Stripe)
Dylan Matthews

...
Coherence Therapy with LLMs - quick demo
by Chris Lakin — 2023-08-14T03:34:29.102Z

update on my current views: a lot of the useful meaning in Coherence Therapy sessions comes from intonation/paralanguage. Unfortunately, language models unfortunately can’t currently perceive this. I don't think current AI systems can do Coherence Therapy nearly as well as a counselor, though I imagine they can still help a lot for issues that can be totally verbalized (which I think isn't that common for people just starting out with therapy).

Coherence Therapy is, IMO, probably part of the "one true therapy". I think the crispest explanation of how to update one's own maladaptive beliefs/expectations/parts/traumas, especially when beliefs get stuck and the usual methods don't work.

Kaj Sotala provides an evidenced-based summary here Book summary: Unlocking the Emotional Brain which I adore.

However people I've recommended this to have told me that it's hard to implement.

It might be possible to simulate Coherence Therapy with LLMs, though— especially Claude 2.0 with its 100k context window.

Here's a demo I captured:

Ideally you'd be more descriptive than me, though

Ultimately I enjoyed my session with Counselor Claude.

To try it yourself:

set up an account with Claude

if it's not available in your country: use a VPN (US or UK) to set up the account; it will work after that

download the post (I have a trimmed PDF available for download here)
upload it to Claude with prompt:

You are an expert Coherence Therapist. You will lead me through a session of coherence therapy like that described in the post.

chat!

Let me know in the comments if you come up with improvements for this, or areas where you feel Counselor Claude is lacking.

I am writing a much longer series about how to solve all social drama in your own life. Subscribe to my profile for when I release that (or DM me to review the draft).
Listen For What You Don’t Hear: The Case for Contrarianism
by Yashvardhan Sharma — 2023-08-14T02:53:27.970Z

What beliefs do you hold that you know spark disagreement? Is there something you hope to accomplish that requires persuading opposed minds?

Do you ever catch yourself dismissing reasonable arguments from those you disagree with? When was the last time an opponent changed your mind or expanded your thinking?

This could be like debate competition, doing community building for AIS/EA, or even convincing potential investors/funders in your company/organization.

In these examples of negotiation, one thing is key: understanding the incentives and cruxes of the people you’re up against. This is a lesson that might seem trivial but I see it less frequently in everyday life.

In this age of the internet and social media especially, it is extremely easy for people to be trapped within a vicious cycle of echo chambers where their ideas are reinforced.

It is easy for people to demonize and dehumanize their enemies. When we view those we disagree with as enemies, it becomes difficult to accept or understand any logic or rational points that are said by them.

I call upon you to take the road less taken. Like a catapult, take a few steps in a direction that you resist—which is going to be hard—only to bounce back stronger than ever to the side that you actually want to be.

In this post, I would like to encourage you to do the hard thing: engage with people you disagree with. Take a belief that you have, for example.

Seriously, take a pause and think of something you believe in.

This could be something like being vegan because you care about animals, why climate change is the biggest issue right now, any political issue in your country, etc.

Now, most likely, you’re well aware of all things about ideas that you do believe in: why you believe what you believe in, what your reasoning for doing so, its merits (and possibly even the biggest demerits), etc. Heck, if you’re a smart one, you can also probably cite a paper or research study that supports your belief.

But, how much do you know about those who disagree with you? Relative to the time you’ve spent learning and studying the idea that you believe in, how much time have you invested in learning about that? 5-10%?

If you truly care about what you believe in, spend a bit longer.

Be intentional about understanding what you do not believe in.

Once you have a decent understanding of what the opposition be...
Recipe: Hessian eigenvector computation for PyTorch models
by Nina Panickssery — 2023-08-14T02:48:01.144Z

The idea/description of this method is fully taken from John Wentworth's Applied Linear Algebra lecture series, specifically Lecture 2.

Training deep neural networks involves navigating high-dimensional loss landscapes. Understanding the curvature of these landscapes via the Hessian of the loss function can provide insights into the optimization dynamics. However, computing the full Hessian can be prohibitively expensive. In this post, I describe a method (described by John Wentworth in his lecture series) for efficiently computing the top eigenvalues and eigenvectors of the loss Hessian using PyTorch's autograd and SciPy's sparse linear algebra utilities.

Hessian-vector product

The core idea hinges upon the Hessian-vector product (HVP). Given a vector v, the HVP is defined as H⋅v , where H is the Hessian matrix. This product can be computed efficiently using automatic differentiation without forming the full Hessian. The process can be outlined as:

Compute the gradient of the loss with respect to model parameters: g=∇L
Compute the dot product of g and v: c=g⋅v
Compute the gradient of c with respect to the model parameters, which gives the HVP

Lanczos Iteration and eigsh

eigsh from scipy.sparse.linalg implements the Lanczos iteration, which finds the top eigenvalues and eigenvectors of a symmetric matrix. It requires matrix-vector multiplication as the main computation, making it ideal for large matrices where full matrix factorizations are infeasible.

Using LinearOperator

To interface with eigsh, we need a mechanism to represent our Hessian as a linear operator that supports matrix-vector multiplication. SciPy's LinearOperator serves this purpose, allowing us to define a matrix implicitly by its action on vectors without forming the matrix explicitly.

Implementation

Given a PyTorch model, loss function, and training data, the approach is to:

Accumulate a subset of the training data (as many batches as specified)
Define the HVP using PyTorch's autograd
Construct a LinearOperator using the HVP
Call eigsh with this linear operator to compute the top eigenvalues and eigenvectors

Appendix: Python code

You can find this code as a GitHub gist here also.

import torch as t
from torch.autograd import grad
from scipy.sparse.linalg import LinearOperator, eigsh
import numpy as np
def get_hessian_eigenvectors(model, loss_fn,

...
Assuming LK99 or similar: how to accelerate commercialization?
by ryan_b — 2023-08-13T21:34:25.290Z

Tabling the question of whether LK99 itself is a room temperature superconductor, I wonder about how we would commercialize it assuming it (or something similar) is, as rapidly as possible. I was reading Ben Reinhardt's Metalessons from the LK-99 Saga post, which raised the point that it normally takes a few decades for an invention to commercially mature. This is consistent with the perspective from Gordon's The Rise and Fall of American Growth (summary), and Smil's Creating the Twentieth Century.

First, I would like the benefits of widespread superconduction sooner than 20 years. Second, given the US/EU's moderate-to-severe distaste for capital-intensive work these paste couple of decades, I think there's a too-high-for-sanity risk that the historical picture is optimistic and we might in fact go slower or fail to exploit the discovery at all.

So then, what should we do? I think we should be able to generate a plausible roadmap of all the tasks that need to be accomplished for widespread commercialization, and then it would be nice if we could sort of speed-run founding the businesses required to push it that far. I am looking for answers that cover the kinds of tasks that need to be done, businesses that will need to be created, etc. Another good kind of answer would be comparison cases: should we look at the road to commercialization for transistors? Fiber-optic cable? Copper cable? Et cetera.
Fundamental Uncertainty: Chapter 1 - How can we know what's true?
by Gordon Seidoh Worley — 2023-08-13T18:55:44.861Z

N.B. This is a chapter in a book about truth and knowledge. It is the first draft. I have since revised it. You can find the most up-to-date info/version on the book's website.

It's 1 a.m.. You're arguing with a stranger on the internet about the meaning of the word "is". How did you get here?

The night started out innocently enough. You were settling down for the evening by checking your favorite internet forum. You read an interesting post that you agreed with, but it contained a minor error. You posted a comment offering a correction. You were about to sign off when you got a notification that the author of the post had replied.

They said, no, you were wrong and an idiot. Normally you'd let it slide, but tonight you're not having it. You fired off a confrontational reply; they sent one back in kind; you escalated in your response; they did the same. After a couple hours you'd exchanged thousands of increasingly heated words that took you away from the original point and into a battle over worldviews. And at this late hour, bleary eyed from staring at the screen for so long, you hit upon the fundamental question separating you:

"How do you know that's true?"

Turns out it's not so easy a question to answer. You say your claims are obvious; they disagree. You cite articles that justify your points; they say those aren't trustworthy sources. You ask them to offer alternative evidence; they provide "evidence" that is little more than opinion. Pretty soon you're debating what truth really mean. You're both convinced you're right, but you can't come to any agreement.

Congratulations! You've run headlong into epistemology—the study of knowledge. Epistemology is the branch of philosophy concerned with how we figure out what's true. We use it to determine what arguments are correct, what evidence to trust, and where we meet the limits of our knowledge. It's also something we use all the time whether we realize it or not. Even if we don't have to think about it often, our lives are filled with acts of applied epistemology.

For example, how do you know that eating a sandwich will satisfy your midday hunger? Sure, you ate a sandwich for lunch yesterday and afterwards you were no longer hungry, but how do you know that today's sandwich will have the same effect? It's not literally the same sandwich made from the same molecules, so maybe it will leave you starving. Yet somehow...
We Should Prepare for a Larger Representation of Academia in AI Safety
by Leon Lang — 2023-08-13T18:03:19.799Z

Epistemic Status: I had the idea for the post a few days ago and quickly wrote it down while on a train. I'm very curious about other perspectives.

TL;DR: The recent increased public interest in AI Safety will likely lead to more funding for and more researchers from academia. I expect this increase to be larger than that of non-academic AI Safety work. We should prepare for that by thinking about how we "onboard" new researchers and how to marginally allocate resources (time and money) in the future.

Why I think academia's share in AI safety will increase

With the recent public interest in AI (existential) safety, many people will think about how they can help. Among people who think "I might want to do research on AI Safety", most will come from academia because that's where most research happens. Among people who will think "I should fund AI Safety research", most will fund academic-style research because that's where most research talent sits, and because it's the "normal" thing to do. I expect this increase to be larger than that of AI Safety researchers in companies (though with less certainty), AI Safety orgs, or independent researchers of, e.g., the "Lesswrong / Alignment Forum" style.

Weak evidence that this is already happening

At the university of Amsterdam, where I'm a PhD student, there has been increased interest in AI Safety recently. In particular, one faculty actively starts to think about AI existential safety and wants to design a course that will include scalable oversight, and ≥4 other faculty are at least starting to get informed about AI existential safety with an "open mind".

What might one do to prepare?

Needless to say, I didn't think about this a lot, so take the following with a grain of salt and add your own ideas.

Academics will mostly read papers that are at least on arxiv. So to "onboard" them, it seems more important than in the past to make the most important insights from lesswrong or the alignment forum accessible to academics.
Doing a PhD might become more worthwhile because it's easier now to have an alignment career in academia.
Doing a PhD might also become less worthwhile because "academic-style" research into AI safety will be less neglected going forward. Whether you buy this argument depends on your views on how open-minded academia is to the most important types of AI Safety research.
In general, it se

...
AGI is easier than robotaxis
by Daniel Kokotajlo — 2023-08-13T17:00:29.901Z

[Epistemic status: Hot take I wrote in 1 hour. We'll see in the comments how well it holds up.]
[Update: OK, seems like people hate this. I guess it was probably too hot and sloppy in retrospect. I should have framed it as "People seem to have a prior that AGI is a grand challenge that's way more difficult than mundane AI tech like self-driving cars, but I don't think this is justified, I think people aren't thinking through the difficulties involved and are instead basically going AGI-is-exciting-therefore-difficult, robotaxis-are-less-exciting-therefore-easier."]
Who would win in a race: AGI, or robotaxis? Which will be built first?

There are two methods:

Tech companies build AGI/robotaxis themselves.
First they build AI that can massively accelerate AI R&D, then they bootstrap to AGI and/or robotaxis.

The direct method

Definitions: By AGI I mean a computer program that functions as a drop-in replacement for a human remote worker, except that it's better than the best humans at every important task (that can be done via remote workers). (h/t Ajeya Cotra for this language) And by robotaxis I mean at least a million fairly normal taxi rides a day are happening without any human watching ready to take over. (So e.g. if the Boring Company gets working at scale, that wouldn't count, since all those rides are in special tunnels.)

1. Scale advantage for AGI:

Robotaxis are subject to crippling hardware constraints, relative to AGI. According to my rough estimations, Teslas would cost tens of thousands of dollars more per vehicle, and have 6% less range, if they scaled up the parameter count of their neural nets by 10x. Scaling up by 100x is completely out of the question for at least a decade, I'd guess.

Meanwhile, scaling up GPT-4 is mostly a matter of purchasing the necessary GPUs and networking them together. It's challenging but it can be done, has been done, and will be done. We'll see about 2 OOMs of compute scale-up in the next four years, I say, and then more to come in the decade after that.

This is a big deal because roughly half of AI progress historically came from scaling up compute, and because there are reasons to think it's impossible or almost-impossible for a neural net small enough to run on a Tesla to drive as well as a human, no matter how long it is trained. (It's about the size of an ant's brain. An ant is driving your car! Have you watched a...