trentmkelly/LessWrong-43k huggingface dataset:
The importance of how you weigh it
by Joe Carlsmith — 2021-03-29T04:59:41.327Z

(Cross-posted from Hands and Cities)

“Like, nobody,” she says, “is considering the décor!”

– G.A. Cohen, “Rescuing Conservatism”

Much of normative ethics centers on which considerations matter, and why. But often, it makes a bigger difference how much a consideration matters — that is, its weight. I think this is an important fact. I think it sheds light, for example, on how helpful to expect training in normative ethics to be for real decision-making; and on whether the difference between consequentialism and non-consequentialism is as significant as sometimes supposed.

I. The basic set, and beyond

There are certain things in ethics that basically everyone treats as important: for example, the flourishing of conscious creatures. And these things are enough to motivate and explain a lot of moral life: for example, the importance of helping people, not harming them, and so forth. Call this set of values “the basic set.”

Some ethicists essentially stop with the basic set, or something in the vicinity. Faced with some candidate further value (say, equality of welfare levels, even amongst people who will never interact or know of each other’s existence; or the intrinsic value of things like an ancient redwood grove, apart from how it impacts any conscious creatures), they will generally be inclined either to explain this candidate value instrumentally, via reference to the basic set (e.g., equality matters, when it matters, because it affects people’s experience of their relationships to others), or to offer some sort of (sometimes hazy and ad hoc) debunking explanation (something something evolution game-theory biases something?).

But other ethicists don’t stop with the basic set. To the contrary, they are actively open to, and interested in, an ethical landscape lush with a complex variety of values and considerations, interacting in intricate and sometimes mysterious ways. They care less about an ethical theory’s simplicity, and more about its fidelity to the lived and often subtle character of human ethical life — and they’re happy to posit new, sui generis values and norms where doing so increases the fidelity in question.

I think there are correlations, here, with certain broader distinctions in temperament. Some people — call them “reductionists” — are naturally more concerned to avoid over-complicating their worldview, and so prefer, where possible, to eliminate...
Agreeing to Disagree: towards a better understanding of agreeableness
by Aaron Bergman — 2021-03-29T03:14:23.966Z
Link: https://aaronbergman.substack.com/p/agreeing-to-disagree

Note: this is from my personal blog

here

.

My sister (hi Lindsay) and I are very different people. Give Lindsay an hour with 15 people, and she will introduce you to her 15 new best friends. Give me an hour with 15 people, and I might remember one of their names. Need a mediator for some sort of drama? She’ll have everyone hugging (even over Zoom) after a heartfelt and compassionate discussion, whereas I’m more likely to take sides with whoever seems to be in the right.

In psychological terms, these differences boil down to two of the

Big Five personality traits

:

extraversion

and

agreeableness

. On a scale from 1-10, Lindsay is probably a 10 on extraversion and an 8 on agreeableness. I’ll plead the fifth instead of rating myself on both (and I don’t think I’m a mean person, honest!), but I probably do fall a bit lower than both of those.

What is agreeableness?

As shown in the picture above, agreeableness has six “facets” or features, each of which has a section in the

Wikipedia page

:

Trust
Straight-forwardness
Altruism
Compliance
Modesty
Tender-mindedness, or “the extent to which an individual's judgments and attitudes are determined by emotion”

A missing definition

Surprisingly, I can’t seem to find a unique definition of agreeableness as used in psychology. Here are some quotes I came across:

The picture above defines it as “the degree to which a person needs pleasant and harmonious relations with others.”
“A person's ability to put other people's needs above their own. For instance, people who are high in agreeableness naturally experience empathy and tend to get tremendous pleasure from serving others and taking care of them.” (

Source

)
“Agreeableness is a personality trait manifesting itself in individual behavioral characteristics that are perceived as kind, sympathetic, cooperative, warm, and considerate.”(

Source

)
“Specifically, Agreeableness appears to describe differences in being predominantly prosocial or other-oriented versus antisocial or self-oriented in social interactions.” (

Source

)
“Agreeableness reflects the individual's tendency to develop and maintain prosocial relationships.” (

Source

)

There is always a risk of being pedantic or counterproductively hyper-analytical when trying to define a term. Don’t these five definitions and the six facets cluster around the same intuitive phenomenon?

In part, yes. I can easily imagine...
Scenarios and Warning Signs for Ajeya's Aggressive, Conservative, and Best Guess AI Timelines
by Kevin Liu — 2021-03-29T01:38:02.183Z
Link: https://kliu.io/post/scenarios-and-warning-signs/

Epistemic status: mild confidence that this provides interesting discussion and debate.

Credits to (in no particular order) Mark Xu, Sydney Von Arx, Jack Ryan, Sidney Hough, Kuhan Jeyapragasan, and Pranay Mittal for resources and feedback. Credits to Ajeya (obviously), Daniel Kokotajlo, Gwern, Robin Hanson, and many others for perspectives on timeline cruxes. This post was written as part of a 10-week AI Safety Fellowship run by Mark. All errors my own.

Summary

Most of this post is unoriginal. It is intended primarily to summarize and rephrase the core distinctions between three plausible scenarios for AI development, which Ajeya lays out in her draft report on AI timelines. It also contains summaries and links to other related content.

As a secondary goal, it attempts to lay out concrete and hopefully plausible predictions for what would occur in each of these three worlds.

Glossary (from Ajeya's report)

This post will assume familiarity with basic terminology regarding neural networks and supervised learning.

Before reading this post, it's probably good to read at least a summary of Ajeya's report (e.g. Rohin Shah's). Some timeline-specific terminology that is helpful to know is also listed below:

Transformative AI (TAI): A computer program that is as transformative as the Industrial Revolution was to the world's trajectory.[1]

Effective horizon length (EHL): the "length of the task"; more precisely, Ajeya defines it as the amount of data measured in subjective seconds of experience (~how long a human would take to do it) to tell whether a model has performed a task better or worse than it did previously. For some things (e.g. predict next word in the sentence), the length is quite low; for others (e.g. run a corporation that maximizes profit), the length may be quite high.

Anchor: A concept on which to base the estimated number of FLOPS (floating-point operations/second) required to train a transformative AI. For example, you might use an Evolution Anchor, which is roughly the compute performed over all of evolution; or a Short Horizon Neural Network (NN) Anchor, which is the extrapolated compute expected for a neural network that is trained on a short horizon length. Key anchors are:

Lifetime anchor: the amount of compute performed by "one human brain over one lifetime."

Short/medium/long-horizon NN: the compute required to train a NN of a size anchored t

...
Notes on effective-altruism-related research, writing, testing fit, learning, and the EA Forum
by MichaelA — 2021-03-28T23:43:19.538Z

Cross-posted from the EA Forum. I'm not sure how many people on LessWrong will find this useful, but I imagine some will.

I've had calls with >30 people who are interested in things like testing their fit for EA-aligned research careers, writing on the EA Forum, "getting up to speed" on areas of effective altruism, etc. (This is usually during EA conferences.)

I gradually collected a set of links and notes that I felt that many such people would benefit from seeing, then turned that into a Google Doc. Many people told me they found that doc useful, so I'm now (a) sharing it as a public post, and (b) still entertaining the hypothesis that those people were all just vicious liars and sycophants, of course.

Disclaimers

Not all of these links/notes will be relevant to any given person
These links/notes are most relevant to people interested in (1) research roles, (2) roles at explicitly EA organisations, and/or (3) longtermism

But this just because that’s what I know best

There are of course many important roles that aren’t about research or aren’t at EA orgs!
And I'm happy with many EAs prioritising cause areas other than longtermism

But, in any case, some of the links/notes will also be relevant to other people and pathways

This doc mentions some orgs I work for or have worked for previously, but the opinions expressed here are my own, and I wrote the post (and the doc it evolved from) in a personal capacity

Regarding writing, the Forum, etc.

Why you (yes, you) should post on the EA Forum

(a talk by Aaron Gertler)

How you can make an impact on the EA Forum

(another talk by Aaron Gertler)

Feedback available for EA Forum drafts

(a post by that Aaron Gertler guy I've been hearing so much about lately)

Effective Altruism Editing and Review

Reasoning Transparency

This has great writing tips that definitely apply on the Forum, and ideally would apply everywhere, but unfortunately they don’t perfectly align with the norms in some areas/fields

Suggestion: EAs should post more summaries and collections
I also encourage you to consider reading, commenting, and posting on LessWrong

See Welcome to LessWrong! for a great introduction to that site

Sometimes people worry that a post idea might be missing some obvious, core insight, or just replicating some other writing you haven't come across. I think this is mainly a problem only inasmuch as it could've been more efficient for y

...
Transparency Trichotomy
by Mark Xu — 2021-03-28T20:26:34.817Z

Introduction

In Relaxed Adversarial Training for Inner Alignment, Evan presents a trichotomy of ways to understand a model M:

Transparency via inspection: use transparency tools to understand M via inspecting the trained model.

Transparency via training: incentivize M to be as transparent as possible as part of the training process.

Transparency via architecture: structure M's architecture such that it is inherently more transparent.

For each type of transparency, we will analyze them on dimensions that influence how effectively we can use them to align powerful systems:

Scalability asks how easily the transparency method scales to more parameters, compute, data, training steps, etc. Factors that influence scalability include reliance on human judgment, the ability to automate with compute, and how easily the human judgment required can be distributed.

Gameability asks the extent to which optimizing an interpretability tool makes it less valuable. Easily gameable tools can be rendered useless by SGD or deceptively aligned mesa-optimizers. Factors that influence gameability include how frequently the transparency tool is deployed, how much the tool relies on human judgment, and how it is used.

Quality asks about the level and type of understanding the transparency tool can give you about how the model works. Factors that influence quality include reliance on human judgment, the smallest unit at which the tool can operate, and how sharp the resulting predictions are.

Transparency via inspection

Transparency via inspection seeks to understand and predict a model's behavior via looking at it directly. Currently, this means looking at the weights of a neural network and trying to obtain information. A prototypical example of inspection transparency is Olah et al.'s Circuits thread.

Scalability. Inspection transparency primarily functions by having humans look at the weights of a neural network and make various judgment calls. For instance, the authors of An Overview of Early Vision in InceptionV1 look at the first layer of InceptionV1 and determine that 28 of the 64 features correspond to Gabor filters. As such, current inspection transparency requires large amounts of human labor, making it scale poorly with parameter count (unless labor can be automated). Current inspection transparency tools often use dataset examples to determine the function of particular...
Improvement for pundit prediction comparisons
by orthonormal — 2021-03-28T18:50:51.648Z

[EDIT: SimonM pointed out a possibly-fatal flaw with this plan: it would probably discourage more pundits from joining the prediction-making club at all, and adding to that club is a higher priority than comparing the members more accurately.]

Stop me if you've heard this one. (Seriously, I may not be the first to have written this kind of idea here. Let me know if not.)

We've got several pundits making yearly predictions now, which is fantastic progress for the field. However, if they're not answering the same questions, you can't effectively judge their performance against one another.

I suggest that this winter, we do 2 rounds, one for proposing questions and one for making predictions.

December 1: deadline for pundits to propose prediction questions.

December: Metaculus formalizes questions (where possible) and opens markets.

January 1: deadline for pundits to register their predictions (they don't have to bet) on any markets they choose.

At the end of the next year, we can judge pundits against each other on the intersection of their answered questions. (We can also check whether the pundit beat the Metaculus prices at the time they entered their predictions.)

This won't guarantee a total or even partial ordering on pundits, if they choose to answer different sets of questions; but the victor of any pair will be clear (after choosing a scoring rule). We can treat the result as a round-robin tournament among the pundits, or better yet, do data analysis on subdomains (who beat whom in predicting US politics, etc) where clearer winners may emerge.

Additional possible features:

We could make a secret place for pundits to register their predictions, to be revealed on New Year's, so that others can't piggybank off of them. The pundits can of course piggyback off of the Metaculus price.
We can let pundits enter some predictions as official and others as unofficial. They'll only be judged by their official ones; their unofficial ones are for practicing the prediction skill, or cases where their intuition feels uncalibrated.

Thanks to ciphergoth for developing this idea with me!
The Point of Easy Progress
by silentbob — 2021-03-28T16:38:25.682Z

A lot of our productivity happens in the form of “projects”: spending a significant amount of time pursuing a certain desirable goal by consistently working towards it. My attitude towards projects, how to approach them, how to enjoy them and how to increase the odds of success, has changed a great deal over the past 15 years. With this post I want to make three points of varying obviousness that emerged from these past experiences:

The Approach Matters: one’s personal experiences while working on a project are not set in stone but can vary tremendously based on one’s approach.
Harmful Short-Sightedness: acting on short-sighted impulses can be harmful in two ways. It can make us follow tempting trajectories that ultimately lead nowhere, and it can cause us to give up because a small obstacle seems larger than it is.
Point of Easy Progress: For many projects it may be possible to design one’s approach such that a “point of easy progress” is reached early on. From that point on, hardly any willpower is required to make progress and working on the project generally is more attractive than not working on the project.

The Difficulty Landscape and Why the Approach Matters

A simple way to visualize a person progressing on a project is to interpret the scenario as a 2D landscape: the person starts on the left, the goal is somewhere far to the right, and there are height differences in between. Going downhill is easy (e.g. the tasks at that point in time are fun and not too difficult, the person is highly motivated), going uphill is hard (e.g. the tasks are extremely boring, complicated, dangerous or in any other way unattractive).

I like this visualization as it’s easy and intuitive and works well to illustrate the points, and thus will stick to it throughout this post. One drawback however is that the image of a landscape suggests a certain rigidity: it may be appealing to assume that for any given project the landscape is basically predetermined – we just have to put in the work and make our way to the finish line, up and over all the slopes and mountains we encounter. The layout of the landscape may depend on where the person pursuing the project is standing initially, and what the goal actually is, but other than that there’s not much to do.

My first point here however is that the landscape is not set in stone. Terraforming is possible so to speak. There ar...
Review of "Fun with +12 OOMs of Compute"
by adamShimi — 2021-03-28T14:55:36.984Z

Introduction

This review is part of a project with Joe Collman and Jérémy Perret to try to get as close as possible to peer review when giving feedback on the Alignment Forum. Our reasons behind this endeavor are detailed in

our original post

asking for suggestions of works to review; but the gist is that we hope to bring further clarity to the following questions:

How many low-hanging fruits in terms of feedback can be plucked by getting into a review mindset and seeing the review as part of one’s job?
Given the disparate state of research in AI Alignment, is it possible for any researcher to give useful feedback on any other research work in the field?
What sort of reviews are useful for AI Alignment research?

Instead of thinking about these questions in the abstract, we simply make the best review we can, which answers some and gives evidence for others.

In this post, we review

Fun with +12 OOMs of Compute

by Daniel Kokotajlo. We start by summarizing the work, to ensure that we got it right. Then we review the value of the post in itself -- that it, by admitting its hypotheses. We follow by examining the relevance of the work to the field, which hinges on the hypotheses it uses. The last two sections respectively propose follow-up work that we think would be particularly helpful, and discuss how this work fits into the framing of AI Alignment research proposed

here

by one of us.

This post was written by Adam; as such, even if both Joe and Jérémy approve of its content, it’s bound to be slightly biased towards Adam’s perspective.

Summary

The post attempts to operationalize debates around timelines for Transformative AI using current ML techniques in two ways: by proposing a quantity of resources (compute, memory, bandwidth, everything used in computing) for which these techniques should create TAI with high probability (the +12 OOMs of the title), and by giving concrete scenarios of how the use of these resources could lead to TAI.

The operational number comes from Ajeya Cotra’s

report

on TAI timelines, and isn’t really examined or debated in the post. What is expanded upon are the scenarios proposed for leveraging these added resources.

OmegaStar

, a bigger version of AlphaStar trained on every game in the Steam library, as well as on language-related “games” like “predict the next word in a web page” or “make a user engaged as a chatbot”. Note that these language-re

...
Buying a house and making friends in unexpected places
by dominicq — 2021-03-28T14:35:18.319Z

I'm buying a house! A very exciting thing for me. In the process of finding and buying this house, I've come to some interesting game-theoretical realizations.

Observe the three-step process:

We find a house that we like.
We contact the agency to organize things between us, the bank, and the homeowners.
We do everything on our own, with the help of the homeowners.

Putanumonit Jacob has some great insights into how frequently you'd expect to see people not behave according to their incentives (trolls), and it's something like 4% of cases. I guess that's true in my experience. My added data point is that people follow their incentives but not reliably - or at least not reliably enough for me. The classification of people into incentive-followers (non-trolls) and incentive-ignorers (trolls) can be useful, but this experience acquainted me with mixed trolls. Sometimes they'd follow incentives, on other days, not so much. That's because they're playing other games as well, so they have to balance out the payout for my game with their other payouts.

That is to say, I cannot reliably count on the agent to do the things I need done (on time), even though him doing these things means that he gets paid 1% of the selling price. He also gets 1% on like twenty other transactions, which are maybe much more lucrative. Not all one percents are equal.

The unexpected side-effect of this story is making friends in unexpected places. The previous homeowners and us, we share a bond of interest in the house and a common dislike for the agent and the bank. It's circumstance that forces us to go through these proxies, instead of trading directly.

And the expected lesson is that when you have people do the work for you, you still have to do work. It's an instance of the principal-agent problem. No amount of money will allow you to outsource things perfectly. So it makes sense to try to do things on your own, learn the skills in the process, and pay for repairs if you mess things up too badly.