trentmkelly/LessWrong-43k huggingface dataset:
Features that make a report especially helpful to me
by lukeprog — 2022-04-14T01:12:19.111Z

Cross-post from EA Forum, follow-up to EA needs consultancies.

Below is a list of features that make a report on some research question more helpful to me, along with a list of examples.

I wrote this post for the benefit of individuals and organizations from whom I might commission reports on specific research questions, but others might find it useful as well. Much of what's below is probably true for Open Philanthropy in general, but I've written it in my own voice so that I don't need to try to represent Open Philanthropy as a whole.

For many projects, some of the features below are not applicable, or not feasible, or (most often) not worth the cost, especially time-cost. But if present, these features make a report more helpful and action-informing to me:

The strongest forms of evidence available on the question were generated/collected. This is central but often highly constrained, e.g. we generally can't run randomized trials in geopolitics, and major companies won't share much proprietary data. But before commissioning a report, I'd typically want to know what the strongest evidence that could in theory be collected is, and how much that might cost to gather or produce.

Thoughtful cost-benefit analysis, where relevant.

Strong reasoning transparency throughout, of this particular type. In most cases this might be the most important feature I'm looking for, especially given that many research questions don't lend themselves to more than 1-3 types of evidence anyway, and all of them are weak. In many cases, especially when I don't have much prior context and trust built up with the producers of a report, I would like to pay for a report to be pretty "extreme" about reasoning transparency, e.g. possibly:

a footnote or endnote indicating what kind of support nearly every substantive claim has, including lengthy blockquotes of the relevant passages from primary sources (as in a GiveWell intervention report[1]).

explicit probabilities (from authors, experts, or superforecasters) provided for dozens or hundreds of claims and forecasts throughout the report, to indicate degrees of confidence. (Most people don't have experience giving plausibly-calibrated explicit probabilities for claims, but I'll often be willing to provide funding for explicit probabilities about some of a report's claims to be provided by companies that specialize in doing that, e.g. Go

...
Exploring toy neural nets under node removal. Section 1.
by Donald Hobson — 2022-04-13T23:30:40.012Z

Introduction

This post is a long and graph heavy exploration of a tiny toy neural network.

Suppose you have some very small neural network. For some inexplicable reason, you want to make it even smaller. Can we understand how the network behaviour changes when some nodes are deleted?

Training code

The network, and how it was trained.

#This module trains a small neural network to determine if a point is inside or outside a circle, and then saves that model to a file.
#First import some libraries.

import tensorflow as tf
import numpy as np
from math import *
import matplotlib.pyplot as plt
from itertools import *

#This function accepts n, and returns a generated dataset of n datapoints. Each individual datapoint consists of 2 float inputs, and one bool output.
#Actually the bool is stored as the integers 0,1 because the tensorflow library was written for the general case of catagorization into any number of catagories.
#The x input is uniformly sampled from the -1,1 square.
#The y output is 1 if the point is outside the circle centred on the origin with radius sqrt(2/pi). This radius was chosen so exactly half the points
# (on average) will lie within the circle
def generate(n):
x=np.random.uniform(-1,1,size=[n,2])
y=(np.sum(x*x,1)>sqrt(2/pi)).astype(int)
return x,y

#The number of hidden layer neurons
N=20

#The keras model accepts an input for x
inputs = tf.keras.Input(shape=(2,))
#In order to make removing some neurons easier, the network structure accepts a list of on_nodes. These will be 1 for any node considered to be turned on, and 0 for any node considered off.
#This value will be set to a constant block of ones during training. When the network is evaluated it is usually on many neurons at once. The input here will generally be all 0's and 1's, despite this
#input accepting any floating point values. The input will also generally consist of BATCH_SIZE repititions of the same vector. Ie within any one batch evaluation, the set of neurons that are off is generally fixed.
#Another advantage of this setup is it lets us take gradients of the performance with respect to these on_nodes.

on_nodes = tf.keras.Input(shape=(N,))

#A single hidden layer containing 20 relu nodes.
hidden_layer = tf.keras.layers.Dense(N, activation=tf.nn.relu)(inputs)

#multiplying by on_nodes to turn off any nodes we might want off.
hidden_layer_picked=hidden

...
Make a Movie Showing Alignment Failures
by Logan Riggs — 2022-04-13T21:54:50.764Z

My previous girlfriend didn't want to live forever when I asked her years ago; not even 200 years as a healthy adult! But we watched an episode of Black Mirror where people can be temporarily uploaded into a fun virtual world and permanently stay there when they're near-death. Surprisingly, she said she would like to do that instead of dying a natural death.

Some things are only convincing when you think about it in concrete details, and fiction allows people to live through an experience in fine-grained details. I don't think my previous girlfriend would ever have been convinced of extending life to 200 years if she wasn't presented with a clear concrete story where it ended well.

We can use fiction to show alignment failures to create a better cultural reference point than Terminator or Sorcerer's Apprentice. We can do it and have it funded by FTX (Project #33) to actually make a movie/series.

But humanity has to win, right? But how do we show alignment failures, let alone multiple alignment failures if humanity has to win?

With time loops. And everybody loves time loops.

The Plot

Similar to Dave Scum, Alice is just living her life until everyone dies, but time freezes and she can re-wind up to 1 month in the past. After enough time loops, she realizes it's an AGI optimizing for [X] and she convinces the developers (after several more time loops) to implement a patch so it doesn't kill everyone the way it always does.

So it then kills everyone in the next unblocked strategy.

After a montage of 10-15 patches, everyone dies except Alice who's being kept alive by the AGI (which figured out she was a direct cause of patching the AI's top-10 instrumental paths), which she then figures out how to activate the time-power w/o dying through [the power of friendship/true love] and swears off patches as a potential solution.

She then goes for the strategy of "buying more time" and convinces the developers to halt progress on this until they develop a more robust solution. This buys 3 more months until the world ends from a different company's AGI. She repeatedly tries to convince groups of people until one group will not be convinced despite her best efforts. This buys her a total of 5 years every loop.

Alice begins trying to tackle the core problems with several groups of people, bringing back their results through time, trying out more robust solutions ever...
Summary: "How to Do Research" by OSP's Red
by Pablo Repetto — 2022-04-13T19:46:57.972Z
Link: https://pabloernesto.github.io/2022/04/12/summary-ospred-research.html

This post is part of my "research" series.

Red. How to Do Research. Overly Sarcastic Productions, 2019. https://www.youtube.com/watch?v=sigJwoeU6OI.

Why

Overly Sarcastic Productions are two friends making short (around 14 minutes) video presentations about history, mythology, and literature. I stumbled upon this video by scrolling through their back-catalog, and the title hooked me in.

In one paragraph

This video describes the core loop of the bibliographic research process: reference mining, finding sources, extracting relevant information to research notes, and generating a research thesis. It is good as a light introduction, but if you've read Eco's How to Write a Thesis then you can definitely skip it.

Table of Contents

The video doesn't include timestamps so I just extracted the locations of the section cards.

Step 1: Wikipedia 0:34
Step 2: Hunt down sources! 1:51
Step 3: Bury yourself in notes 5:25
Step 4: Break out the thumbtacks and string! 6:24

An alternative outline

Broadly, how this article reads from my perspective.

Introduction 0:00

Step 1: Wikipedia
1.1 Yes, Wikipedia, I'm serious 0:31
1.2 Never cite Wikipedia 0:39
1.3 Source of sources 0:48
1.4 Get greedy! 1:15

Step 2: Sources
2.1 Acquiring sources 1:43
2.2 Primary and secondary sources 2:30
2.3 Context is everything 4:06

Step 3: Notes 5:11

Step 4: The Conspiracy Board
4.1 Thumbtacks and string 6:17
4.2 Thesis LAST 6:51

Conclusion 7:06

Highlights

The good, the bad, and the extremely revealing. Direct quotations in no particular order. Some of these ideas may not be discussed at length, but it still seemed worthwhile to put them here.

Notes

BURY YOURSELF IN NOTES. Take notes on everything. [...] If you're researching a historical or mythical figure, write down EVERYTHING they did [on screen: Write what they did! Put the accounts on a linear timeline! Write about the writers doing writing!].

(5:25)

It doesn't matter if you think you'll remember it, you want to write it down anyway, because this isn't about memorizing, it's about putting all your information in one place so you can deal with it all at once later.

(5:51)

Thesis last

Now that you have a huge pile of everything you know about this person, place, or thing, you get to start on the fun part: filling in the gaps. You have to take the mess of information you've acquired, organize it into cat

...
A Quick Guide to Confronting Doom
by Ruby — 2022-04-13T19:30:48.580Z

Within the last two weeks, two sets of things happened: Eliezer Yudkowsky shared a post expressing extreme pessimism about humanity's likelihood of surviving AGI, and a number of AI research labs published new, highly impressive results. The combination of these two has resulted in a lot of people feeling heightened concern about the AI situation and how we ought to be reacting to it.

There have been calls to pull "fire alarms", proposals for how to live with this psychologically, people deciding to enter the AI Alignment field, and a significant increase in the number of AI posts submitted to LessWrong.

The following is my own quick advice:

1. Form your own models and anticipations. It's easy to hear the proclamations of [highly respected] others and/or everyone else reacting and then reflexively update to "aaahhhh". I'm not saying "aaahhhh" isn't the right reaction, but I think for any given person it should come after a deliberate step of processing arguments and evidence to figure out your own anticipations. I feel that A concrete bet offer to those with short AI timelines is a great example of this. It lists lots of specific things the authors do (or rather don't) expect to see. What 2026 looks like is another example I'd point to of someone figuring out their own anticipations.[1]

2. Figure out your own psychology (while focusing on what's true). Eliezer, Turntrout, and landfish have each written about their preferred way of reacting to the belief that P(Doom) is very high. My guess is that people who are concluding P(Doom) is high will each need to figure out how to live with it for themselves. My caution is just that whatever strategy you figure out should keep you in touch with reality (or your best estimate of it), even if it's uncomfortable.

3. Be gentle with yourself. You might find yourself confronting some very upsetting realities right now. That's okay! The realities are very upsetting (imo). This might take some time to process. Let yourself do that if you need. It might take you weeks, months, or even longer to come terms with the situation. That's okay.

4. Don't take rash action, and be cautious about advocating rash action. As far as I know, even the people with the shortest timelines still measure them in years, not weeks. Whatever new information came out these past two weeks, we can take some time to process and figure out our plans. Maybe we...
Design, Implement and Verify
by rwallace — 2022-04-13T18:14:28.931Z

Taking a look at the latest here after a hiatus, I notice there is once again a lot of discussion about the problem of AI safety, clearly a cause for concern to people who believe it to be an existential threat.

I personally think AI safety is not an existential threat, not because I believe the AI alignment problem is easier than Eliezer et al do, but because I believe AGI is much harder. I was involved in some debate on that question, a while back, but neither side was able to convince the other. I now think that's because it's unprovable; given the same data, the answer relies too heavily on intuition. Instead, I look for actions that are worth taking regardless of whether AGI is easy or hard.

One thing I do consider a matter for grave concern is the call to address the issue by shutting down AI research, and progress on computing in general. Of course there are short-term problems with this course of action, such as that it is, if implemented, much more likely to be enforced in democracies than dictatorships, which is very much not an outcome we should want.

The long-term problem with shutting down progress is that at very best, it just exchanges one form of suicide for another. Death is the default. Without progress, we remain trapped in a sealed box, wallowing in our own filth until something more mundane like nuclear war or pandemic, puts an end to our civilization. Once that happens, it's game over. Even if our species survives the immediate disaster, all the easily accessible fossil fuel deposits are gone. There will be no new Renaissance and Industrial Revolution. We'll be back to banging the rocks together until evolution finds a way to get rid of the overhead of general intelligence, then the sun autoclaves what's left of the biosphere and the unobserved stars spend the next ten trillion years burning down to cinders.

(Unless AI alignment is so easy that humans can figure it out by pure armchair thought, in the absence of actually trying to develop AI. But for that to be the case, it would have to be much easier than many technical problems we've already solved. And if AI alignment were that easy, there would be no call for concern in the first place.)

It is, admittedly, not as though there is no ground for pessimism. The problem of AI alignment, as conceived by default, is impossible. That's certainly ground for pessimism!

The default way to think abo...
Takeoff speeds have a huge effect on what it means to work on AI x-risk
by Buck — 2022-04-13T17:38:11.990Z

The slow takeoff hypothesis predicts that AGI emerges in a world where powerful but non-AGI AI is already a really big deal. Whether AI is a big deal right before the emergence of AGI determines many super basic things about what we should think our current job is. I hadn’t fully appreciated the size of this effect until a few days ago.

In particular, in a fast takeoff world, AI takeover risk never looks much more obvious than it does now, and so x-risk-motivated people should be assumed to cause the majority of the research on alignment that happens. In contrast, in a slow takeoff world, many aspects of the AI alignment problems will already have showed up as alignment problems in non-AGI, non-x-risk-causing systems; in that world, there will be lots of industrial work on various aspects of the alignment problem, and so EAs now should think of themselves as trying to look ahead and figure out which margins of the alignment problem aren’t going to be taken care of by default, and try to figure out how to help out there.

In the fast takeoff world, we’re much more like a normal research field–we want some technical problem to eventually get solved, so we try to solve it. But in the slow takeoff world, we’re basically in a weird collaboration across time with the more numerous, non-longtermist AI researchers who will be in charge of aligning their powerful AI systems but who we fear won’t be cautious enough in some ways or won’t plan ahead in some other ways. Doing technical research in the fast takeoff world basically just requires answering technical questions, while in the slow takeoff world your choices about research projects are closely related to your sociological predictions about what things will be obvious to whom when.

I think that these two perspectives are extremely different, and I think I’ve historically sometimes had trouble communicating with people who held the slow takeoff perspective because I didn’t realize we disagreed on basic questions about the conceptualization of the question. (These miscommunications persisted even after I was mostly persuaded of slow takeoffs, because I hadn’t realized the extent to which I was implicitly assuming fast takeoffs in my picture of how AGI was going to happen.)

As an example of this, I think I was quite confused about what genre of work various prosaic alignment researchers think they’re doing when th...
What to include in a guest lecture on existential risks from AI?
by Aryeh Englander — 2022-04-13T17:03:40.927Z

A professor I'm friendly with has been teaching a course on AI ethics this semester, and he asked me if I could come give a guest lecture on "AI apocalypse" scenarios. What should I include in the lecture?

Details:

Audience is mostly graduate engineering students. Some are in a program focused on AI, others are in a systems engineering program and don't otherwise have much knowledge about AI beyond this course.
This will be I think the final class for the semester. Previous classes have covered topics like privacy, bias, explainability, social coordination failures (including race to the bottom dynamics), self-driving cars, AI in social media, lethal autonomous weapons, AI in healthcare, implementing ethics in organizations, and the economic impacts of AI. I did not attend those lectures so I don't know exact details of what was or wasn't covered.
I have leeway to include pretty much anything I want in the class and to focus on any topic(s) I want. The professor would like me to include something about forecasting, but that's optional.
I can optionally assign something short as an assignment before class, but it is the end of the semester so nothing more than that. There will not be any homework based on the class, I think.
The class will be 2 hours long.
I do not have a huge amount of time to prepare for the lecture - it's relatively soon (April 27) and I have multiple other work / school / family obligations I need to attend to between now and then.

If anybody has relevant material I could use, such as slides or activities, that would be great! Also, if anybody wants to help develop the material for this class, please message me (preferably at my work email - Aryeh.Englander@jhuapl.edu).

As a bonus, I expect that material for a class of this sort may turn out to be useful for plenty of other people on this and related forums, either for themselves or as a tool they can use when presenting the same topic to others.

[Note: I am posting this here with permission from the professor.]
Common Knowledge is a Circle Game for Toddlers
by ryan_b — 2022-04-13T15:24:09.906Z

I wanted to jot down a particular instance of a common interaction I have with my daughter.

My child, wife, and I are sitting around the coffee table in the living room discussing the plan to take the car to the mechanic the next day. We describe this as going on an adventure to the child as a lingering holdover from isolation during covid, which seems to be fine because she enjoys car rides. What is different from the usual this time is that I am coming; normally car adventures are my wife driving, because she runs most of the errands. The following (simplified) dialogue takes place:

Child: "Dad, can you come in the car tomorrow?"

Me: "I am coming in the car tomorrow."

Child: "Really?! You are coming?"

Me: "Yes."

Child: "Mom! Did you know Dad is coming in the car tomorrow?"

Mom: "Yes! He is coming with us tomorrow."

Child: "Dad! Mom knows you are coming tomorrow! You are both coming!"

Me: "Yes!"

Keep in mind we have all been present for the whole conversation. We are sitting around a coffee table. None of us are more than three feet from each other.

It struck me that what my daughter was doing was trying to explicitly track what her mom and I knew. I expect this falls somewhere under the theory-of-mind stage of development, but what stuck out to me was the procedure she undertook, which looked something like three loops:

One loop with me, of establishing the fact and confirming the fact.
A lateral loop to mom, of establishing that mom knows the fact and confirming that mom knows the fact.
A lateral loop back to me, of establishing that I know that mom knows, and confirming that I know that mom knows.

The looping strategy reminded me of Circle Games, and only after this did it occur to me that the object of the game was common knowledge.

My child has been playing these kinds of games for a while now, on reflection; but recently they have grown in frequency and elaboration enough that I began to wonder why she seemed to be spending more time on what I instinctively treat as mundane details. The connection to common knowledge did not crystalize until this event.
Another list of theories of impact for interpretability
by Beth Barnes — 2022-04-13T13:29:27.670Z

Neel's post on this is good. I thought I'd add my own list/framing. Somewhat rough.

I see various somewhat different ways in which interpretability can be useful for AI safety. These require different things from your interpretability in terms of how efficient it is, how much it lets you identify exactly what your model is thinking as opposed to broad properties of its cognition, and how reliable it needs to be.

Roughly in decreasing order of demandingness:

Microscope AI
Component of a full solution to alignment problem (ie as part of something like imitative generalization)
Knowing everything a model is thinking and fully auditing it to make sure it’s not doing anything sketchy
Relaxed adversarial training: identifying which part of the model corresponds to its ‘beliefs about its observations’ so that you can search over these
Identifying a ‘truthfulness direction’ in activation space or something similar
Having some rough understanding of what a model is thinking/what type of thinking it’s doing and thereby increasing the chance that you can spot it’s deceptive
Lobotomy: identifying which parts of a model do what sort of cognition, and extracting just the parts that are less likely to be doing something dangerous

Microscope AI

Instead of building and using an ML model, build the model and then use interpretability techniques to extract the knowledge it has learnt. Humans can then apply this knowledge directly rather than needing to actually deploy the model. See https://www.lesswrong.com/posts/X2i9dQQK3gETCyqh2/chris-olah-s-views-on-agi-safety

Component of a ‘full solution to alignment problem’ (ie as part of something like imitative generalization)

By ‘a full solution to the alignment problem’ I’m thinking of a setup which would in theory let you know everything the model knows (in particular, if a model can use some knowledge to design a plan that leads to it getting power, you can use this knowledge to see that the plan will lead to the model getting more power)IDA/debate/RRM probably don’t give us this, partly because you can’t use them to learn things the model knows based on observing patterns in large datasets. For example, it would be hard to use them to supervise a model to translate correctly from a language it knows but you don’t. Some possible solutions to this have something that vaguely looks like ‘have humans in the loop for learning patterns from large...