trentmkelly/LessWrong-43k huggingface dataset:
Exercise is Good, Actually
by Gordon Seidoh Worley — 2023-02-02T00:09:18.143Z

Yesterday I wrote about how dressing nice is a good idea. This post is written similarly, but the topic is exercise. This is again somewhat casual writing that's in the spirit of what I wish someone had told me when I was 15.

Until about 10 years ago I didn't care about exercise. I figured I could get on just fine in life without really being in shape. Sure, I knew that exercise was supposed to be good for me and I'd make half hearted attempts to include some activity in my life, but I never went to the gym or intentionally tried to get stronger or fitter.

I was carrying around a story in my head like this one:

I care about doing important intellectual and professional work that depends on my mind.
Physical exercise doesn't much impact my ability to do that type of work.
Additionally, only muscle-heads go to the gym all the time. Regular people just get enough exercise from living their lives.
And besides, smart people in movies are usually weak anyway. Their power comes from being smart, not strong.
I'm smarter than average, so why do I need to be strong?
Conclusion: I don't need to work out or go to the gym or even care about physical fitness.

I think a lot of this story can be traced back to how characters in stories work. They're created like D&D characters: they get a certain number of points and the author distributes those points around to different attributes. Some characters have more points than others to work with, but at the end of the day you have to make tradeoffs. Want to be really smart? Gonna have to take that out of something else. What to be really, really, really smart? You're going to have to have to put basically all of your points into intelligence and give up everything else.

Authors do this because it helps make characters more relatable. When someone is too good at too many things the story gets boring because they don't face any real challenges, and we might even envy and hate a character that's too capable because they make us feel inadequate. But this isn't how real people work. In fact, positive traits tend to correlate in people. Yes, you find some jocks with tiny brains, politicians with terrible looks, and geniuses with major health problems, but most positive traits correlate in real humans so that it's not unreasonable for someone to be strong, smart, and good looking.

Dismissing the need for exercise is a kind of coping mechanism...
Product safety is a poor model for AI governance
by Richard Korzekwa  — 2023-02-01T22:40:04.469Z
Link: https://aiimpacts.org/product-safety-is-a-poor-model-for-ai-governance/

Note: This post is intended to be accessible to readers with relatively little background in AI safety. People with a firm understanding of AI safety may find it too basic, though they may be interested in knowing which kinds of policies I have been encountering or telling me if I’ve gotten something wrong.

I have recently encountered many proposals for policies and regulations intended to reduce risk from advanced AI. These proposals are highly varied and most of them seem to be well-thought-out, at least on some axes. But many of them fail to confront the technical and strategic realities of safely creating powerful AI, and they often fail in similar ways. In this post, I will describe a common type of proposal[1] and give a basic overview of the reasons why it is inadequate. I will not address any issues related to the feasibility of implementing such a policy.

Caveats

It is likely that I have misunderstood some proposals and that they are already addressing my concerns. Moreover, I do not recommend dismissing a proposal because it pattern-matches to product safety on the surface level.
These approaches may be good when combined with others. It is plausible to me that an effective and comprehensive approach to governing AI will include product-safety-like regulations, especially early in the process when we’re still learning and setting the groundwork for more mature policies.

The product safety model of AI governance

A common AI policy structure, which I will call the ‘product safety model of AI governance’[2], seems to be built on the assumption that, while the processes involved in creating powerful AI may need to be regulated, the harm from a failure to ensure safety occurs predominantly when the AI has been deployed into the world. Under this model, the primary feedback loop for ensuring safety is based on the behavior of the model after it has been built. The product is developed, evaluated for safety, and either sent back for more development or allowed to be deployed, depending on the evaluation. For a typical example, here is a diagram from a US Department of Defense report on responsible AI:[3]

The system in this diagram is not formally evaluated for safety or performance until after “Acquisition/Development”.[4]

I do not find it surprising that this model is so common. Most of the time when we are concerned about risk from technology, we are worri...
Hinton: "mortal" efficient analog hardware may be learned-in-place, uncopyable
by the gears to ascension — 2023-02-01T22:19:03.227Z

This post comes in two forms: a 15 minute talk and link to partway through forward-forward's initial paper (see also papers citing this).

Talk version (2x speed recommended, captions recommended):

I don't find the discussion of forward-forward to be the most interesting part; it's a plausible learning algorithm, perhaps, but what I'm really interested in is the impact he thinks it'll have on how computers are designed: he claims they're going to be chunks of trained matter that interface with the outside world at boundaries and are otherwise inscrutable and reliant on defects in the particular hardware's shape.

This seems most relevant to me in terms of what effects it might have on the shape of selfhood, if any, of the AI living in that low-power silicon brain.

Sidenote: Amusingly, he opens with a claim that this implies that we can't do brain uploads, and yet describes exactly how to do them anyway: distillation training of a student, including copying of mistakes, which seems to me like the obvious way to do incremental hardware replacement of human brains as well. I also think he's overestimating how much this will prevent exact copying; exact copies won't behave exactly the same way, but it seems likely to me that one would be able to copy knowledge out of a chip more precisely than just by distillation by also involving the kind of offline scanning hardware one would use to examine a CPU. Using the resulting scan would require the help of a learned scan-to-hardware converter AI, though.
Can we “cure” cancer?
by jasoncrawford — 2023-02-01T22:03:44.989Z
Link: https://rootsofprogress.org/cure-for-cancer

In an excellent recent essay on “big visions for biology,” Sam Rodriques writes:

Ask most biologists about the cure for cancer, and they will tell you it doesn’t exist: cancer is many diseases that are mostly unrelated to each other, and that all have to be cured one at a time.

Are “most biologists” right about this?

We can get perspective on this from the history of infectious disease. After all, infection was also “many diseases,” with disparate causes (viruses, bacteria, protozoa) and disparate pathways (air, water, food, insects). And yet, while we did not exactly “cure infectious disease” (just ask the 248,544 people who got covid last week in the US alone), we did reduce every major metric of infectious disease by 90% to 99% in wealthy/developed countries:

Overall mortality from infectious disease reduced by 90% in the 20th century alone
Youth mortality (death before age 15) reduced from almost half to well under 1%
Maternal mortality reduced from almost 1% to under 0.01%
Surgical mortality reduced from half or more (for some major surgeries at least) to around 1% to 2%

Infectious disease was not the whole story in all of the above metrics, but it was the biggest factor in all of them.

US crude mortality rates. Armstrong, Conn & Pinner 1999

Our World in Data

We also eradicated smallpox worldwide and eliminated diseases such as cholera, malaria, and polio from many countries:

Our World in Data

Did we have to cure all these diseases “one at a time”? No, not exactly:

Sanitation and hygiene efforts were effective against broad classes of germs: e.g., water sanitation was effective against all water-borne diseases.
Disinfectants and antiseptics, such as bleach, are effective against most if not all germs.
Antibiotic drugs are effective against broad classes of bacteria.
Vaccines do have to be developed one disease at a time—but even here there are general techniques for creating them. In fact, with mRNA technology, we now have a very general technique that can create new vaccines with relatively little adaptation.

Underlying all of this is a theory that explains the basic causal mechanism of infection: the germ theory. The theory alone doesn’t give us everything we need to cure each specific disease: new research is needed to understand the etiology and epidemiology of each one. But the theory does give us a conceptual framework and a set of tools to guide that research....
Eli Lifland on Navigating the AI Alignment Landscape
by ozziegooen — 2023-02-01T21:17:05.807Z
Link: https://quri.substack.com/p/eli-lifland-on-navigating-the-ai

Recently I had a conversation with Eli Lifland about the AI Alignment landscape. Eli Lifland has been a forecaster at Samotsvety and has been investigating said landscape.

I’ve known Eli for the last 8 months or so, and have appreciated many of his takes on AI alignment strategy.

This was my first recorded video, so there were a few issues, but I think most of it is understandable.

Full (edited) transcript below. I suggest browsing the section titles for a better overview of our discussion.

Transcript

Sections

Samotsvety, a Recent Forecasting Organization
Reading, “Is Power-Seeking AI an Existential Risk?”
Categories of AI Failures: Accident, Misuse, and Structural
Who Is Making Strategic Progress on Alignment?
Community Building: Arguments For
Community Building: Fellowships and Mentorship
Cruxes in the AI Alignment Space
Crux: How Promising is AI Interpretability?
Crux: Should We Use Narrow AIs to Help Solve Alignment?
The Need for AI Alignment Benchmarks
Crux: Conceptual Insights vs. Empirical Iteration
Vehicles and Planes as Potential Metaphors

Samotsvety, a Recent Forecasting Organization

Ozzie Gooen: So to get started, I want to talk a little bit about Samotsvety.

Eli Lifland: It's a Russian name. Samotsvety currently has about 15 forecasters. We've been releasing forecasts for the community on topics such as nuclear risk and AI. We’re considering how to create forecasts for different clients and make public forecasts on existential risk, particularly AI.

Team forecasting has been valuable, and I've encouraged more people to do it. We have a weekly call where we choose questions to discuss in advance. If people have time, they make their forecasts beforehand, and then we discuss the differences and debate. It's beneficial for team bonding, forming friendships, and potential future work collaborations.

It's also interesting to see which forecasts are correct when they resolve. It's a good activity for different groups, such as AI community groups, to try.

Ozzie Gooen: How many people are in the group right now?

Eli Lifland: Right now, it's about 15, but on any given week, probably closer to five to ten can come. Initially, it was just us three. It was just Nuño, Misha, and I, and we would meet each weekend and discuss different questions on either Foretell (now INFER) or Good Judgment Open, but now it's five to ten people per week, from a total pool of 15 people.

Ozzie Go...
Schizophrenia as a deficiency in long-range cortex-to-cortex communication
by Steven Byrnes — 2023-02-01T19:32:24.447Z

(Written in a hurry. I was almost going to title this “My poorly-researched pet theory of schizophrenia”. Hoping for feedback and pointers to relevant prior literature. I am very far from a schizophrenia expert. Really. I cannot emphasize this enough. Like, if I took an undergraduate psych test on schizophrenia right now, I might well flunk it.)

1. What’s my hypothesis?

My hypothesis is that the root cause of schizophrenia is (…drumroll…) a deficiency in medium- to long-range cortex-to-cortex connections. Some elaboration:

When I say “deficiency”, I mean either “the connections aren’t there in their normal numbers” or “the connections are there, but for some reason they’re not accomplishing what they accomplish in neurotypical people”.
When I say “cortex-to-cortex connections”, I think the main culprit is direct connections between Cortex Region A and Cortex Region B, but it’s also possible that the relevant thing is indirect connections between Cortex Region A and Cortex Region B, e.g. via the thalamus or cerebellum.
When I say “medium- to long-range”, this definitely includes e.g. connections between different lobes, and it probably also includes connections across a few centimeters of cortex in humans. I haven’t really thought about what would happen if there was a deficiency in all connections of any length, including the very short ones, but I would weakly guess that this would present as schizophrenia as well.

2. How did I originally come up with that hypothesis?

I can pinpoint the exact moment: I was reading about visual processing abnormalities in schizophrenia, and more specifically the paper Weak Suppression of Visual Context in Chronic Schizophrenia (Dakin, Carlin, Hemsley 2005). They showed people pictures like this:

The task was to match the image contrast in the red circle to one of the circles on the left. The eye-popping results were:

the schizophrenics did better than the control group,
…with a p-value of 0.0000002!
Indeed, “12 out of 15 [subjects with schizophrenia] were more accurate than the most-accurate control.”
…i.e., there were only three subjects with schizophrenia who did not outperform each and every one of the 33 people in the control groups. And one of those three was later re-diagnosed as not schizophrenic after all!

Back to the task, the control group gets misled by the contrast level outside of the red circle, and the schizophrenic group ...
AI Safety Arguments: An Interactive Guide
by Lukas Trötzmüller — 2023-02-01T19:26:58.017Z

Crossposted to the EA Forum.

TL/DR: We developed an interactive guide to AI safety arguments, based on researcher interviews. Go check it out! Please leave a comment and let us know what you think.

Introduction

Vael Gates interviewed 97 AI researchers on their beliefs about the future. These interviews were quite broad, covering researchers’ hopes and concerns for the field in general, as well as advanced AI safety specifically. Full transcripts are now available.

Lukas Trötzmüller interviewed 22 EAs about their opinions on AI safety. They were pre-selected to be skeptical either about the classical “existential risk from AI” arguments, or about the importance of work on AI safety. The focus of this research was on distilling specific arguments and organizing them.

This guide builds mostly on Vael’s conversations, and it aims to replicate the interview experience. The goal is not necessarily to convince, but to outline the most common arguments and counterarguments, and help readers gain a deeper understanding of their own perspective.

Design Goals

Our previous research has uncovered a wide range of arguments that people hold about AI safety. We wanted to build a resource that talks about the most frequently mentioned of those.

Instead of a linear article (which would be quite long), we wanted to create an interactive format. As someone goes through the guide, they should be presented with the content that is most relevant to them.

Even though we had a clear target audience of AI researchers in mind, the text turned out to be surprisingly accessible to a general audience. This is because most of the classical AI risk arguments do not require in-depth knowledge of present-day AI research.

Format

Our guide consists of a collection of short articles that are linked together. There are five main chapters:

When will Generally Capable AI Systems be developed?

The Alignment Problem (“AI systems don’t always do what we want”)

Instrumental Incentives

Threat Models (“how might AI be dangerous?”)

Pursuing Safety Work

Each chapter begins with an argument, after which the reader is asked for their agreement or disagreement. If they disagree, they can select between several objections that they may have.

Each one of these objections links to a separate article, presenting possible counterarguments, that they can optionally read. Most of the objections and counterarguments are...
More findings on Memorization and double descent
by Marius Hobbhahn — 2023-02-01T18:26:41.320Z

Produced as part of the

SERI ML Alignment Theory Scholars Program

- Winter 2022 Cohort.

I’d like to thank Wes Gurnee, Aryan Bhatt, Eric Purdy and Stefan Heimersheim for discussions and Evan Hubinger, Neel Nanda, Adam Jermyn and Chris Olah for mentorship and feedback.

The post contains a lot of figures, so the suggested length is deceiving. Code can be found in these three colab notebooks

[1]

[2]

[3]

.

I have split the post into two parts. The first one is concerned with double descent and other general findings in memorization and the second focuses on measuring memorization using the maximum data dimensionality metric. This is the first post in a series of N posts on memorization in transformers.

Executive summary

I look at a variety of settings and experiments to better understand memorization in toy models. My primary motivation is to increase our general understanding of NNs but I also suspect that understanding memorization better might increase our ability to detect backdoors/trojans.

The work heavily builds on two papers by Anthropic, “

Toy models of superposition

” and “

Superposition, Memorization and double descent

”. I successfully replicate a subset of their findings.
I specifically look at three different setups of NNs that I speculate are most relevant to understanding memorization in the non-attention parts of transformers.

Bottlenecks between layers, i.e. when projecting from high-dimensional spaces (e.g. MLPs) into lower dimensions (e.g. the residual stream). This is similar to the setting in the

toy models of superposition

paper and its

sequel

.
MLP blocks, i.e. when projecting from lower-dimensional spaces (e.g. the residual stream) into higher dimensions with ReLU non-linearities.
The final layer, i.e. when projecting from the end of the residual stream into the vocab space. The main difference to the previous scenarios is that we use the cross-entropy loss for the experiments which has a different inductive bias than the MSE loss.

I’m able to find the double descent phenomenon in all three settings. My takeaway from this is that the transition between memorization and learning general features seems to be a very regular and predictable phenomenon (assuming you know the sparsity and number of features of your network). Furthermore, it seems like the network is “confused” (e.g. has much higher test

...
Language Models can be Utility-Maximising Agents
by Raymond Douglas — 2023-02-01T18:13:34.694Z

1 minute takeaways

It's actually pretty easy to train and run a language model to function as an agent for a specific task, rather than as a non-agentic simulator
The resulting agents seem pretty powerful at least in limited domains and therefore might turn out to be quite useful. They also have some possibly concerning properties
More specifically, they're evidential decision theorists (EDTs), which are known for one-boxing and cooperating with copies of themselves.

Incidentally, they're also known for struggling with causality, which is related to why LLMs hallucinate.
It's also possible to make causal decision theorists (CDTs), which are maybe not so bad but still not ideal

Alignment-wise, this means you outer align them by giving them a utility function. Inner alignment is still a nightmare with the added challenge of making sure the model is correctly inferring the utility function.

How to make an EDT agent out of a language model:

You can find a thorough explanation in this Neurips paper from last year, which laid out how to create such agents and showed that they were state of the art.

The gist is, you take the standard reinforcement learning loop of 'action/state/reward' and train a transformer to simulate it. So the transformer is outputting tokens corresponding to the actions of an agent, the state of the world, and the agent's reward, and the resultant string shows how these progress over time. It is effectively simulating an RL agent. Crucially, the 'reward' is both current reward and total expected future reward.

You run the transformer as follows: instead of predicting how the agent will act, you have it iterate through every possible action the agent might take and simulate the total expected future reward. Then, you take whichever action has the highest expected future reward, and have the agent 'take that action', adding it to the prompt. It is effectively choosing the action which provides the best evidence of it maximising utility. This coincides perfectly with the definition of an evidential decision theorist.

How to make a CDT

As far as I can tell, that's what's happening in this DeepMind paper. Basically you have a slightly different loss function which uses "counterfactual teaching" to have the model treat agent actions as causal interventions. In this paper the simulated agent is being used to imitate an expert, and they demonstrate that it does ...