trentmkelly/LessWrong-43k huggingface dataset:
Ngo and Yudkowsky on scientific reasoning and pivotal acts
by Eliezer Yudkowsky — 2022-02-21T20:54:53.979Z

This is a transcript of a conversation between Richard Ngo and Eliezer Yudkowsky, facilitated by Nate Soares (and with some comments from Carl Shulman). This transcript continues the Late 2021 MIRI Conversations sequence, following Ngo's view on alignment difficulty.

Color key:

Chat by Richard and Eliezer	Other chat

14. October 4 conversation

14.1. Predictable updates, threshold functions, and the human cognitive range

[Ngo][15:05]

Two questions which I'd like to ask Eliezer:

1. How strongly does he think that the "shallow pattern-memorisation" abilities of GPT-3 are evidence for Paul's view over his view (if at all)

2. How does he suggest we proceed, given that he thinks directly explaining his model of the chimp-human difference would be the wrong move?

[Yudkowsky][15:07]

1 - I'd say that it's some evidence for the Dario viewpoint which seems close to the Paul viewpoint. I say it's some evidence for the Dario viewpoint because Dario seems to be the person who made something like an advance prediction about it. It's not enough to make me believe that you can straightforwardly extend the GPT architecture to 3e14 parameters and train it on 1e13 samples and get human-equivalent performance.

[Ngo][15:09]

Did you make any advance predictions, around the 2008-2015 period, of what capabilities we'd have before AGI?

[Yudkowsky][15:10]

not especially that come to mind? on my model of the future this is not particularly something I am supposed to know unless there is a rare flash of predictability.

[Ngo][15:11]

1 - I'd say that it's some evidence for the Dario viewpoint which seems close to the Paul viewpoint. I say it's some evidence for the Dario viewpoint because Dario seems to be the person who made something like an advance prediction about it. It's not enough to make me believe that you can straightforwardly extend the GPT architecture to 3e14 parameters and train it on 1e13 samples and get human-equivalent performance.

For the record I remember Paul being optimistic about language when I visited OpenAI in summer 2018. But I don't know how advanced internal work on GPT-2 was by then.

[Yudkowsky][15:13]

2 - in lots of cases where I learned more specifics about X, and updated about Y, I had the experience of looking back and realizing that knowing anything specific about X woul

...
Direct Observation
by LoganStrohl — 2022-02-21T20:51:45.060Z

My dad’s a retired science teacher, so my little brother and I began playing the “I’m not touching you!” game unusually early.

In case you’re not familiar with “I’m not touching you!”, here’s how it works.

Step 1: Ask your parents (or the internet or whatever) why you don’t fall through the floor.

Step 2: Learn about electrostatic repulsion, and conclude that things never “really touch”.

Step 3: Poke your sibling.

Step 4: When they say “Stop touching me!”, insist that “I’m not touching you!”

Step 5: Find out whether your parents consider “technically correct” to be an adequate defense. (They don’t.)

It is true that things do not touch in the way we naively imagine. Their component atoms’ electrons repel each other. “Perfectly direct contact” does not ordinarily happen.

Nonetheless, there is a big difference between the “not touching” that’s happening when you poke your sibling, and the “not touching” that’s happening when you point at your sibling from three meters away.

Similarly, there is no unmediated subjective experience of the territory. When you poke your little brother’s arm, your brain does something with those nerve signals to turn them into phenomena, into the subjective experience of tactile sensations. The sensations are not the nerve signals. There is always processing distance between your subjective experiences and whatever is happening to your sense organs.

However, some processing distances are smaller than others.

Consider the processing distance between [the electrostatic repulsion at your fingertip] and [your plan to test how upset you can make your brother before your parents intervene]. Although your plan exists mostly inside your head, it definitely has something to do with that electrostatic repulsion, in about the same way that a pointing finger has something to do with whatever it’s pointing at. A plan is made of anticipations, imaginings, desires, and so forth; yet it isn’t completely out of contact with the territory.

But consider the distance between [the electrostatic repulsion at your fingertip] and [the immediate tactile sensations of your finger]. The subjective sensations still exist only inside your head, but the processing distance is much smaller. Those sensations put you in closer contact with the underlying reality of electrostatic repulsion. Compared to imaginings or desires, they are much more tightly entangled with ...
Alignment research exercises
by Richard_Ngo — 2022-02-21T20:24:16.375Z

It's currently hard to know where to start when trying to get better at thinking about alignment. So below I've listed a few dozen exercises which I expect to be helpful. They assume a level of background alignment knowledge roughly equivalent to what's covered in the technical alignment track of the AGI safety fundamentals course. They vary greatly in difficulty - some are standard knowledge in ML, some are open research questions. I’ve given the exercises star ratings from * to *** for difficulty (note: not for length of time to complete - many require reading papers before engaging with them). However, I haven't tried to solve them all myself, so the star ratings may be significantly off.

I've erred on the side of including exercises which seem somewhat interesting and alignment-related even when I'm uncertain about their value; when working through them, you should keep the question "is this actually useful? Why or why not?" in mind as a meta-exercise. This post will likely be updated over time to remove less useful exercises and add new ones.

I'd appreciate any contributions of:

Comments about which exercises seem most or least useful.
Answers to the exercises.
More exercises! The ideal exercises are

nerdsnipe-style problems

which can be stated clearly, and seem well-defined, but lead into interesting depths when explored.

Reward learning

* Look at the examples of human feedback mechanisms discussed in the

reward-rational implicit choice paper

. Think of another type of human feedback. What is the choice set? What is the grounding function?

*

This paper by Anthropic

introduces a technique called context distillation. Describe this in terms of the reward-rational implicit choice framework.
* Estimate the bandwidth of information conveyed by different types of human feedback. Describe a rough model for how this might change as training progresses. By contrast, how much information is conveyed by the choice of a programmatic reward function? (Consider both the case where the agent is given the exact reward function, and where it learns from reward observations.)

* Look at the examples of biases discussed in

learning the preferences of ignorant agents

. Identify another bias which similarly influences human decision-making. Describe an example situation where a human with that bias might make the wrong decision. Formulate an algorithm that infers th

...
Summary/Review/Quotes of "The WEIRDest People in the World" by Joseph Henrich
by Torello — 2022-02-21T20:03:42.384Z

Saw other posts on this topic here, but thought I would post my own as well.

Takeaway: if you are interested in how/why psychology may be different in different areas of the world, or what drove European prosperity in the last few centuries, you might like this.

https://digitalsauna.wordpress.com/2022/02/21/the-weirdest-people-in-the-world-by-joseph-henrich-2020/
Seeking models of LW’s aversion to religion
by Visible_Ignorance — 2022-02-21T18:54:51.376Z

Hi there. I’m a longtime LW-er posting under a pseudonym, in the hopes of being able to make my ignorance visible without unnecessary drama. Please don’t out me; it seems to me that the ability to expose one’s ignorance pseudonymously is a pro-epistemic norm that we’ll be better off having.

Now that I’ve gotten that over with:

I want to understand LW’s aversion to religion. I’m hoping some of you guys can help me model it. I’m also hoping to explain what is attractive to me about religion as an aid to forming accurate maps (including, attractive to me vis a vis my goal of making LW stronger/more successfully truth-seeking), and to see if some of you are willing and able to understand my perspective about this (“to pass my ITT,” is I think how you lot would say this?). I want to know whether, after you “pass my ITT,” you have anything to say back that makes me think my attraction to religion is something truth can destroy.

What is LW’s objection to religion?

Here’s my starting attempt at describing LW’s aversion to religion:

1) Many religions claim things that are straightforwardly false (e.g., “Jesus physically rose from the dead.”) It’s harmful to believe false things.

2) Human beings have an observable tendency to really persistently believe (or claim to believe) some obviously false things religions claim, compared to random obviously false things. This suggests to LW that religion is near a “fault line” in the human psyche, and if we go near that fault line our minds might start believing a lot of different false things, or might otherwise become less trustworthy. We don’t have a precise, high-confidence model of what exactly this “fault line” is, so it is safest to stay far away from religion, so as to reduce the odds of accidentally stumbling over the unknown “religion fault line” and having our minds go screwy.

If you’re a LW-er who is averse to religion, and you have a couple minutes to make me smarter, I’d really appreciate a comment below from you (or a PM) about whether this (1 and 2 above) summarizes most of what drives your aversion to religion, or whether it’s missing something core to your reaction.

How the above “objection to religion” sits with me

Objection 1 seems valid and correct to me. I agree many religions claim many false things. Jesus did not physically rise from the dead; there was no message to Noah about a flood; etc. I also agree that fal...
The Big Picture Of Alignment (Talk Part 1)
by johnswentworth — 2022-02-21T05:49:34.962Z
Link: https://www.youtube.com/watch?v=mij7nYPKIHo

I recently gave a two-part talk on the big picture of alignment, as I see it. The talk is not-at-all polished, but contains a lot of stuff for which I don't currently know of any good writeup. Major pieces in part one:

Some semitechnical intuition-building for high-dimensional problem-spaces.

Optimization compresses information "by default"
Resources and "instrumental convergence" without any explicit reference to agents

A frame for thinking about the alignment problem which only talks about high-dimensional problem-spaces, without reference to AI per se.

The central challenge is to get enough bits-of-information about human values to narrow down a search-space to solutions compatible with human values.
Details like whether an AI is a singleton, tool AI, multipolar, oracle, etc are mostly irrelevant.

Fermi estimate: just how complex are human values?
Coherence arguments, presented the way I think they should be done.

Also subagents!

Note that I don't talk about timelines or takeoff scenarios; this talk is just about the technical problem of alignment.

Here's the video for part one:

Big thanks to Rob Miles for editing! Also, the video includes some good questions and discussion from Adam Shimi, Alex Flint, and Rob Miles.
Two Challenges for ELK
by derek shiller — 2022-02-21T05:49:15.097Z

This post sketches two challenges to ARC's project around eliciting latent
knowledge that differ somewhat in kind from the challenges ARC is most
concerned about. They relate to the difficulty in distinguishing beliefs
from other representations.

Introduction

The problem of ELK, as outlined in ARC's technical
report,
is to figure out: given an AI trained to complete a task (the 'task AI'),
how can we design a second AI (the 'reporter AI') that can answer
questions about how the task AI understands the world. If we can know what
the task AI is thinking, we can better assess whether its decisions are
dangerous.

The central challenge identified by ARC to solving the problem of ELK
concerns the possibility that the reporter AI will not learn to answer
questions using what the task AI believes (e.g. answering honestly), but
instead answer questions using what a human observer might think is the
case on the basis of the available evidence (answering empathetically). We
train the reporter AI to answer based on the facts as we understand them
(and know the task AI to understand them) in relatively simple cases. We
hope that the reporter will answer questions in more complex cases based
on how the task AI understands the world, but it might instead answer
based on what it thinks a human would mistakenly believe. If the human
questioner is ignorant of some complex details that the task AI knows, the
reporter AI might choose to leave them out to tell a simpler story that
will satisfy the questioner. It is not obvious how to train an AI to
answer honestly rather than empathetically.

While this is a significant challenge, I don't think it is the only
challenge for ELK. In thinking through the problem, I became worried about
several others. I here describe two other challenges. Though they are
somewhat speculative, ARC aims for a solution to the problem of ELK that deals
with the worst case, and these challenges fall under that umbrella.

Even if we figure out how to make an honest rather than a empathetic
reporter, these challenges will remain.

1.) The Right Representation Challenge

We do not know how the task AI might encode the the things that it
believes, but what it knows about the world must somehow be reflected in
its neural activity. For simplicity, let's suppose that the task AI utilizes
explicit independent representational states that are encoded in its
network in ...
Review: Václav Havel’s “The Power of the Powerless”
by David Gross — 2022-02-21T02:18:20.584Z

In which I review Václav Havel’s essay on “The Power of the Powerless” before Scott Alexander totally steals my thunder.

Václav Havel’s The Power of the Powerless is fascinating and surprising. The purpose of this review is to summarize it in such a tantalizing (and lengthy) way that you stop reading this and go look up Havel’s original instead.

Its context is Czechoslovakia in 1978. The country has been behind the Iron Curtain for thirty years, and ten years have passed since the brief experiment in political liberalization known as the “Prague Spring” which had been quickly choked off by a Soviet-led invasion.

Havel was a Czech playwright with international renown, whose works were banned in his own country. In 1977 he helped to spearhead “Charter 77”—a document that called on the government to respect human rights and its own Constitution.

Charter 77 was prompted by the prosecution of members of the rock band “The Plastic People of the Universe”—the “Pussy Riot” of their day. The government took the threat represented by the Charter seriously—it persecuted its signers and made it illegal to print or distribute the text. In 1979, Havel would be sent to prison for four years for his role in advocating for the Charter.

While Havel wrote The Power of the Powerless, he was under constant police surveillance and harassment for his Charter 77 activism. Meanwhile he was being noticed by freedom-loving people around the world and being hailed as a prominent Soviet-bloc dissident.

The “Post-Totalitarian” System

He begins his argument by examining the system the “dissidents” are up against. It’s a strange new form of tyranny—a dictatorship not by a person or people, but by a bureaucracy and by certain principles and external contingencies (the primary one being that the Soviet Union intends to maintain Czechoslovakia as an obedient client state). So this is not the sort of dictatorship that can be threatened by attacks on a particular person or clique.

Havel refers to this system as “post-totalitarian” (at least in the translation I read). This is confusing, as it seems at first to imply that it is no longer totalitarian, which isn’t the case. “Neototalitarian” might have been more apt.

Czechoslovakia (and the Soviet Union and its client states collectively) was ruled in part by an ideology—almost a religion—that had proven to be a tempting refuge for confused, uprooted, and...
Prizes for the 2020 Review
by Raemon — 2022-02-20T21:07:23.884Z

The third annual review is complete. We spent two months nominating, reviewing, and voting on the best posts of 2020. After lots of thoughtful evaluation, it's time to award some prizes!

As described in the voting results post, there are two prize pools for posts. Lightcone Infrastructure contributed $10k to the Review Vote prize pool, and LessWrong users donated $1770 to the Unit of Caring prize pool (which Lightcone matched 1:1, bringing it up to $3540).

Lightcone Infrastructure is also awarding $3600 in prizes for people who wrote reviews[1] for posts.

Prizes for Posts

The Review Vote Prizes

We've awarded a $1000 Top Prize to each of the following authors:

catherio, for microCOVID.org, a tool to estimate covid risk
Ajeya Cotra, for Draft report on AI Timelines
evhub, for An Overview of 11 Proposals for Building Safe Advanced AI
johnswentworth for When Money is Abundant, Knowledge is the Real Wealth, Alignment by Default, The Pointers Problem, and Coordination as Scarce Resource

We've awarded a $500 Honorable Mention to each of:

Alkjash, for Pain is not the unit of effort
Mark Xu, for The Solomonoff Prior is Malign
Jacob Falkovich, for Seeing the Smoke
Alex Flint, for The Ground of Optimization
Zvi, for Simulacra Levels and their Interactions
Richard Ngo, for AGI Safety from First Principles
Paul Christiano, for Inaccessible Information
Steven Byrnes for My Computational Framework for the Brain
abramdemski for Most Prisoner's Dilemmas are Stag Hunts; Most Stag Hunts are Schelling Problems, and An Orthodox Case Against Utility Functions
Daniel Kokotajlo, for Cortés, Pizarro, and Afonso as Precedents for Takeover
Scott Garrabrant, for Introduction to Cartesian Frames
Rafael Harth, for Inner Alignment, Explain Like I'm 12 Edition

How were prizes determined?

The LessWrong moderation team looked over the results of the vote, sliced four different ways:

Total votes from all users;
Total votes from 1000+ karma users;
Total votes from Alignment Forum users;
Weighted votes from all users, but with 1000+ karma users weighted 3x.

That gave us some sense of how different users related to posts, and what they got out of them. In all cases, approximately the same posts showed up at the top of the rankings, albeit in somewhat different orders. I'm excited to reward all of them prizes for contributing to important intellectual progress.

Two things stuck out between t...
Generalization of the Solomonoff Induction to Accuracy - Is it possible? Would it be useful?
by PeterL — 2022-02-20T19:29:03.343Z

I appreciate Solomonoff's success in generalizing Occam razor from just selecting the simplest hypothesis/model to adding probabilities to each of them.

But for instance the postion (motion) of any body (x, y, z) can be fit by an inequation

instead of writing actual Newton's laws (and adding very tight intervals for constants), and it is simpler, so the particular Solomonoff probablility would be greater (if I understand Solomonoff Induction correctly) even the model I stated above is apparently useless.

Or is SI meant to be used just for exact models? Then it might be completely useless, as long as almost nothing in this world can be fitted exactly according to my worldview.

Can you explain the above mentioned issues likely residing in my incomprehension?
Practitioner: Novice
by The Guild of the Rose — 2022-02-20T19:01:50.486Z
Link: https://guildoftherose.org/articles/practitioner-novice

The Triad gave Taman a week to pay off his crushing debt. Now the Guild of the Rose claims they can help, but how can Taman trust them when he doesn't even know what they want?

This is a fictional story about what the Guild of the Rose could someday become.
2 (naive?) ideas for alignment
by Jonathan Moregård — 2022-02-20T19:01:31.379Z

Context

I'm interested in working on alignment, coming from a programmer/Haskell background.
I have two ideas that are unlikely to info hazards, that I want to post here in order to get feedback/info about prior work.

Both are very rough/early stage, view this as me presenting a pre-MVP to get some very early feedback.

Idea 1 - "Love is that which enables choice" (Inspired by Forrest Landry)

This is an idea for a potential goal/instruction for AI (can't recall the fancy term).
The idea is to make an AI that optimizes for optionality: maximizing the total sum of agency for all human and non-human agents.
Agency is here loosely defined as "Optional ability to make changes to the world".

Making it the sum total would discourage situations where the ability of one person to affect change would hamper the ability of someone else.

Idea 2 - Segmented gradient descent training optimized for collaboration between different agents

This is an idea for a potential training method, that I think may have a big attractor basin for collaborative traits.
The idea is to have some kind of gradient descent-esque training where AI agents of varying calibres/types are put in training scenarios in which a premium is put on collaboration. This is run in multiple iterations, where AI that successfully collaborates with other agents get to continue.

The hardest thing about this is that we want an AI that is cooperative, but we do not want an AI that is naive, as this would lead to situations where terrorists convince the AI to do stupid shit. We could try to model this on human (cultural/biological) evolution.

One thing I like about this idea is that it might lead to AI that develops behavioural patterns akin to those found in herd animals (including humans).
This would make the AI easier to reason about, and more likely to develop something akin to ethical behaviour.