trentmkelly/LessWrong-43k huggingface dataset:
Shah and Yudkowsky on alignment failures
by Rohin Shah — 2022-02-28T19:18:23.015Z

This is the final discussion log in the Late 2021 MIRI Conversations sequence, featuring Rohin Shah and Eliezer Yudkowsky, with additional comments from Rob Bensinger, Nate Soares, Richard Ngo, and Jaan Tallinn.

The discussion begins with summaries and comments on Richard and Eliezer's debate. Rohin's summary has since been revised and published in the Alignment Newsletter.

After this log, we'll be concluding this sequence with an AMA, where we invite you to comment with questions about AI alignment, cognition, forecasting, etc. Eliezer, Richard, Paul Christiano, Nate, and Rohin will all be participating.

Color key:

Chat by Rohin and Eliezer	Other chat	Emails	Follow-ups

19. Follow-ups to the Ngo/Yudkowsky conversation

19.1. Quotes from the public discussion

[Bensinger][9:22] (Nov. 25)

Interesting extracts from the public discussion of Ngo and Yudkowsky on AI capability gains:

Eliezer:

I think some of your confusion may be that you're putting "probability theory" and "Newtonian gravity" into the same bucket. You've been raised to believe that powerful theories ought to meet certain standards, like successful bold advance experimental predictions, such as Newtonian gravity made about the existence of Neptune (quite a while after the theory was first put forth, though). "Probability theory" also sounds like a powerful theory, and the people around you believe it, so you think you ought to be able to produce a powerful advance prediction it made; but it is for some reason hard to come up with an example like the discovery of Neptune, so you cast about a bit and think of the central limit theorem. That theorem is widely used and praised, so it's "powerful", and it wasn't invented before probability theory, so it's "advance", right? So we can go on putting probability theory in the same bucket as Newtonian gravity?

They're actually just very different kinds of ideas, ontologically speaking, and the standards to which we hold them are properly different ones. It seems like the sort of thing that would take a subsequence I don't have time to write, expanding beyond the underlying obvious ontological difference between validities and empirical-truths, to cover the way in which "How do we trust this, when" differs between "I have the following new empirical theory about

...
ELK Thought Dump
by abramdemski — 2022-02-28T18:46:08.611Z

I recently spent a couple of weeks working on ELK. What follows is a somewhat disorganized thought-dump, with the post-hoc theme of relating ELK to some common ideas in philosophy.

I'll assume familiarity with the ELK document. You can read quick summaries on some other reaction posts, so I won't try to recap here.

Everything here should be read as "threads for further research" -- many of these sections could be long posts in themselves if I tried to fully formalize them, or even to unpack all my thoughts so far.

I've numbered sections which contain semi-concrete ELK proposals. As a result, the section numbers will be weird -- for example, "2. Truth Tracking" is not the second section.

Semantics

The problem at the heart of ELK is that of semantics, that is, ascribing meaning.

The leading theory (for LessWrongers, at least) is the map/territory analogy, according to which beliefs are like a map, and the world is like a territory. A map has a "scale" and a "key" which together tell us how to relate the map and the territory. Likewise, beliefs are thought to have a "correspondence" which tells us how to relate belief to reality. In academic philosophy, this is known as the correspondence theory of truth (rather than "the map territory theory of truth").

LessWrong is pretty big on the map/territory distinction, but (I think) has historically been far more interested in applying the idea than developing or analyzing it.

While map/territory semantics seems pretty obvious, it gets weird when you start looking at the details. This has caused some philosophers to abandon it, and develop alternatives (some of which are quite interesting to me). Alex Flint has an extensive discussion of problems with some possible theories of truth. This post is my take on the issue.

Semantics vs Truth

Sometimes, the correspondence theory of truth is rendered via Tarski's T-schema:

True("The moon is made of blue cheese.") ↔ The moon is made of blue cheese.

There are other replies that can be given, but my reply is as follows. I am interested in "truth" only as a way of getting at meaning/semantics. A definition of truth should shed light on what it means for something to be true; why it is considered true. It should help me cope with cases which would otherwise be unclear.

In other words, the T-schema tells us nothing about the nature of the correspondence; ...
Conspiracy-proof archeology
by Malmesbury — 2022-02-27T22:59:56.213Z

This is a cross-post from Telescopic Turnip.

Totally legit fossil of a 3m giant, excavated near Cardiff, NY.

There is a lot you can learn about history by asking random Frenchmen about World War Two. Which army contributed the most to the pwnage of Germany? According to Ifop who conducted the survey in 2014, it’s the United States (49%), followed by the USSR (23%), and Britain (18%). Now I don’t expect these numbers to enlighten your understanding of the Past; the detail that makes them interesting is that Ifop had asked the exact same question three times already: in 2004, in 1994 and in 1945, back when people remembered the war not from textbooks but from echoic memory. Here are the results:

Source (in French)

Ask the same question about history several times, and it becomes meta-history. This survey caught live footage of collective memory being overwritten by the victors. Presumably, this happened in a somewhat liberal democracy with a somewhat free press, maybe with a little help from entertainment. It didn’t require a totalitarian power deliberately distorting history to manipulate the masses. But the masses were still manipulated somehow.

Of course, for most of history before the quantum revolution, Europe was ruled by totalitarian powers deliberately distorting history to manipulate the masses. If French people’s beliefs about the Red Army could drift so easily in the late 20th century, then it’s hard to trust anything coming from the official records of divine-right monarchs. You can find historical documents and artifacts and get an idea of what most likely happened, but if you allow for the possibility that the power structures of the time could fabricate documents and plant artifacts as much as they wanted, you will never be truly convinced.

Zero-day breaches in the laws of the Universe

In the world of computer hacking, there is a kind of security breach called a zero-day. Zero-days happen when an attacker discovers a breach that the developers themselves are not aware of – it has been known for zero days. So there is nothing they can do to prevent an attack. The 2010 Stuxnet attack on the Iranian nuclear program relied on no less than four different zero-days in Microsoft Windows – not something your typical basement h4xx0r can do. In essence, finding a zero-day boils down to understanding how some part of the computer system works, finding a specific ...
[David Chapman] Resisting or embracing meta-rationality
by Gordon Seidoh Worley — 2022-02-27T21:46:23.912Z
Link: https://metarationality.com/meta-rationality-resistance-enjoyment

Excerpting the intro. Follow the link for the rest.

Meta-rationalism may provoke skepticism or hostility at first. It contradicts fundamental rationalist principles that you may hold explicitly, or that you may have absorbed implicitly, from technical education and the culture of technical work.

Alternatively, meta-rationality may seem intriguing, if implausible and incomprehensible. Then you may ask:

Is there really anything to this?
If so, what kinds of problems can I solve with it that I couldn’t deal with rationally? (I bet there aren’t any!)
What principles of rationality does meta-rationality contradict? (Rational principles are absolute truths, so I don’t see how meta-rationality could have better ones.)
How can I learn the supposed meta-rational methods, if they’re worth anything?

These questions are natural and rational. They also miss the point, and therefore make it impossible to understand what meta-rationality is:

You are already doing meta-rationality all the time. You can’t do rationality without it. “What there is” to meta-rationalism is discovering that you mostly do meta-rationality mindlessly, and therefore ineptly. Then you can learn to do it better.
Meta-rationality isn’t about solving problems; that’s what rationality is for. Meta-rationality can help you use rationality more effectively. By understanding problems and potential solution approaches in broader contexts, you become more likely to solve them—but you’ll do that rationally. You’ll also get better at deciding which problems are worth solving, and effective ways of dealing with problems other than “solving” them.
Meta-rationality doesn’t run on principles. Demanding that it does makes it impossible to learn.
Meta-rationality also doesn’t have methods.

Meta-rationality is a different kind of thing from rationality, so you can’t learn it the same way. You learn rationality by getting principles and methods from lectures or textbooks, and then working through toy exercise problems. That won’t work for meta-rationality.
How I Formed My Own Views About AI Safety
by Neel Nanda — 2022-02-27T18:50:17.036Z
Link: https://www.neelnanda.io/blog/47-inside-views

Disclaimer: I work as a researcher at Anthropic, but this post entirely represents my own views, rather than the views of my own employer

Introduction

I’ve spent the past two years getting into the field of AI Safety. One important message I heard as I was entering the field was that I needed to “form an inside view about AI Safety”, that I needed to form my own beliefs and think for myself rather than just working on stuff because people smarter than me cared about it. And this was incredibly stressful! I think the way I interpreted this was pretty unhealthy, caused me a lot of paralysing uncertainty and anxiety, and almost caused me to give up on getting into the field. But I feel like I’ve now reached a point I’m comfortable with, and where I somewhat think I have my own inside views on things and understand how to form them.

In this post, I try to explain the traps I fell into and why, what my journey actually looked like, and my advice for how to think about inside views, now I’ve seen what not to do. This is a complex topic and I think there are a lot of valid perspectives, but hopefully my lens is novel and useful for some people trying to form their own views on confusing topics (AI Safety or otherwise)! (Note: I don’t discuss why I do now think AI Safety is important and worth working on - that’s a topic for a future post!)

The Message of Inside Views

First, context to be clear about what I mean by inside views. As I understand it, this is a pretty fuzzily defined concept, but roughly means “having a clear model and argument in my head, starting from some basic and reasonable beliefs about the world, that get to me to a conclusion like ‘working on AI Safety is important’ without needing to rely on deferring to people”. This feels highly related to the concept of

gears-level models

. This is in comparison to outside views, or deferring to people, where the main reason I believe something is because smart people I respect believe it. In my opinion, there’s a general vibe in the rationality community that inside views are good and outside views are bad (see Greg Lewis’

In Defence of Epistemic Modesty

for a good argument for the importance of outside views and deferring!). Note that this is not the Tetlockian sense of the words, used in forecasting, where outside view means ‘look up a base rate’ and inside view means ‘use my human intuition, which is terribl...
Defensiveness as hysterical invisibility
by TekhneMakre — 2022-02-27T15:28:46.083Z

Sometimes people get defensive. That is, Alice tells Bob, "You have property X", and then Bob tenses his muscles, moves rigidly, pretends to not be bothered, and denies that he has property X, or changes the subject, or asks Alice why she thinks he's X (perhaps in a tenser / higher-pitched tone of voice) and then gives an alternate explanation of observations Alice describes. We can take it for granted that Bob's behavior is often because of something in the ballpark of, he's trying to maintain a Public Relations story to Alice or himself or onlookers (present or not); or, he's trying to avoid a bucket error.
But, given that there's stigma about defensiveness, we can expect that there are other reasons for Bob to deny that he has property X, and that those reasons will be delegitimized.
Here's one such reason: Bob thinks that if he admits that he has property X, then that will render him (more) invisible to Alice (or onlookers). By admitting that he is X, Bob adds support in the common-knowledge context to the image of him as X. If there's common-knowledge that Bob is X, then Alice will: be emboldened to model Bob as X; treat Bob as X; expect Bob to expect Alice to treat him as X; ignore, or perhaps punish or resist, behavior by Bob not in line with X. Whatever else Bob is, other than X, is implicitly marginalized; so most of Bob is invisible, which could make Bob want to scream (or deny X, or without denying X force Alice via insincere questions to expand on the proposition and add nuance until there's room for Bob to be visible).
From the inside this is like: "Oh so this is how they're implementing their decision not to listen to what I'm saying." Or: "Oh so this is how the possibilities of our interaction are collapsed away." Or: "Oh so this is how they're going to pigeonhole me so that they already know everything there is to know about interacting with me."
This is an instance of a more general problem: lopsided common-knowledge creation. A group may be prone to mistakenly act as though it is overconfident in only those beliefs which are common knowledge. For such a group, creating common knowledge of X may lead to mistaken behavior even if X is true. Maybe it helps to accompany common knowledge of X with common knowledge of other facts that might recommend against whatever mistaken behavior X alone could recommend. (I would guess that fear of lopsided com...
Preparing for Possible Russian Countersanctions in Europe
by arunto — 2022-02-27T12:01:38.302Z

Europe is about to introduce stronger economic sanctions against Russia. Given Putin's past behavior I think it is likely that there will be Russian countersanctions.

The most relevant could be cutting off of gas for Europe. In the long run this
would be self defeating because it accelerates energy independence from Russia.
But nevertheless I think this is quite possible (est. prob: 25%). If this would come at the same time as below average temperatures in March and April in countries heavily depending on natural gas for heating (e.g. in Germany > 50% if you include district heating), then many households could find themselves temporarily without heating (est. prob: 10%). Of course, that is nothing compared to the hardship of the Ukrainian people but nevertheless it could make sense to make some cheap preparations:

1. If your heating is based on gas or on gas based district heating: Buying an electric
fan heater. Cheap devices are currently sold on Amazon for less than 20 Euros. (Yesterday, I have paid 16 Euros for a heater).

2. If your cooking stove is based on gas and you don't have an electric alternative
(e.g. a microwave oven): Buying an electric hot plate. Single hot plates are currently about 20 Euros.

Both purchases would have to be made before Putin announces a gas embargo because, I believe, after that they will be sold out immediately.

Anything else on could do to prepare?
Relativized Definitions as a Method to Sidestep the Löbian Obstacle
by homotowat — 2022-02-27T06:37:00.199Z

The goal of this post is to outline a loophole of sorts in Gödel's Second Incompleteness (GSI) theorem (and of Löb's theorem by corollary) and give a somewhat sketchy description for how it might be exploited.

I'll begin by describing the loophole. Essentially, it's merely the observation that Second Incompleteness is a syntactic property rather than a semantic one. Assume that Bew is a provability predicate which is subject to GSI. GSI does not prohibit the existence of a predicate Bew2 not subject to GSI such that [[Bew]]=[[Bew2]], where [[P]] is the set corresponding to the predicate P in the standard semantics. That is to say, it's possible for two predicates to be the same, semantically, while one is subject to GSI and the other is not. Explicitly constructing an example is not hard, take the following definition for example;

Bew2(X):=Bew(X)∧X≠“⊥”

Meta-theoretically, we know that ⊥ is not provable, and so can prove in some ambient theory that Bew2 and Bew are, really, the same predicate. Internal to PA, or some similar system, however, GSI holds for Bew but fails rather trivially for Bew2. Bew2 is called a relativized definition, of Bew.

This demonstrates the loophole, but this particular "exploitation" of it is obviously not very satisfying. Ideally we'd not have to rely on meta-theoretic reasoning to justify the alternative definition. Of course, there are limitations to what we can do. If GSI holds for one definition and not the other then we cannot prove a material equivalence between the two without causing an inconsistency. Instead, we need a weaker notion of internal equivalence. Specifically, we can use the following conditions;

If P(X)→Q(X) then P is at most Q.
If Φ[Q] is a definition for Q, and Φ[P] holds, then P is at least Q.

If both 1. and 2. hold then we've internally demonstrated that P and Q are semantically equivalent. Though both conditions are syntactically weaker than material equivalence, so we can't use it for the same purposes in general. They are, however, sufficient to treat P as if it were Q for many purposes.

To demonstrate these conditions, I want to switch to a weaker system for a bit. Robinson's Q is basically just PA without induction...
Estimating Brain-Equivalent Compute from Image Recognition Algorithms

by Gunnar_Zarncke — 2022-02-27T02:45:21.801Z

[Epistemic Status: Playing around with the idea of a benchmark with some rough numbers.]

When I read

Biological Anchors: A Trick That Might Or Might Not Work

, my thinking was: Biological anchors will work if your algorithms are close enough to what the brain does and can then be used to estimate the compute (FLOPs) needed for the rest of the brain. The compute equivalent of the brain has been discussed recently

here

(I think this indicates

factor 100

more efficient algorithms) and

here

. I used this for

predictions on Metaculus

. This will not give you sharp bounds and not tell you whether algorithms could do things much cheaper or which ones to use. I have not seen this specific comparison elsewhere.

This started with the idea that we might already have some algorithms that perform as well as some parts of the brain and compare their costs, power requirements, and complexity. Specifically, image recognition is about

equally good as human raters

. Thus, let’s compare

state-of-the-art image recognition algorithms

with the corresponding brain regions (the visual cortex) and then extrapolate that to the whole brain.

I did this and here is the result:

Brain Region

Brodmann Area

17

Visual Cortex V1

Algorithm

CoAtNet-7

[cm^3] Volume

11

(1% brain volume)
[10^6] Neurons

280

(0.3% brain neurons)	[10^6] Parameters	2500
[W] Power	0.18	[W] Power, 10 inferences/s	13 at

2TFLOPs/W

[10^21 FLOP] training

200

[10^9 FLOP] inference	2600

Whether the comparison should include only region V1 or also V2 to V5 of the visual cortex is worth asking, but the idea was to estimate conservatively and to exclude cognitive processes current algorithms definitely don't cover.

Extrapolating the compute to the whole brain:

Inference: 8*10^15 FLOPs/s (

86*10^9 neurons

/ 280*10^6 neurons * 2.6*10^12 FLOPs/inference * 10 inferences/second).
Training: 5*10^16 FLOP/s (86*10^9 neurons / 280*10^6 neurons * 2*10^23 FLOPs / 18 life years).

Pretty low compared to the numbers in Cotra’s paper.

There are just some problems: The visual cortex does much more than just static image recognition of 512x512 pixel images:

The resolution of the processed image is much higher:

120 million rods

instead of a quarter-million pixels.
Stitching together the picture from blurred fragments (

Saccades

).
Building something lik

...
You have a place to stay in Sweden, should you need it.
by Dojan — 2022-02-27T01:21:19.552Z

If anyone here is in trouble of any kind, and finds themselves in Sweden with no place to stay, or in need of other help, I want to help you. Send me an email at johan.domeij@gmail.com for further contact.

I am posting this now due to the ongoing conflict in Ukraine, but the offer is not limited to that conflict. I am not able to give significant financial support, but I have a car and a place to stay for several people.
Is Anyone Else Seeking to Help Community Members in Ukraine Who Make Refugee Claims?
by Evan_Gaensbauer — 2022-02-27T00:51:24.997Z

Some rationalists and effective altruists in British Columbia have begun mulling over the option of whether they would be able to sponsor a fellow community member living in Ukraine to stay as a refugee in Canada. There is a process by which 5 or more citizens of Canada can sponsor a refugee. Have you or anyone else you know in the community begun considered doing something similar or other ways to aid peers still in Ukraine?
The Myth of the Blue Sunlight 
by ship_shlap — 2022-02-26T21:42:50.900Z

Note: This is a (philosophical) poem

He sat in the silent house with a gray
Expression of colorless eyes. Peeking
In was the predictable, yellow ray
Of light through the window. The Myth says that
above the clouds the sunlight is a great,
Superb blue state.
O how glorious it must be, in view!
With impulses he did not understand,
He darted out the house, resisting the
Cold Winds of the mountains. Into the odd
Mysterious machine he stood and he
Rotated the wheel with vigor and rose
Up in the air. With time, a brown feeling
Bore upon his forearm to slow him down.
He neared the clouds, the sight of them tiring
Him. Defying the now red pain, he pierced
The tuffs of white cotton.
You see the color of the sunlight. Understanding
and familiarity rushes through as you bathe in the
gold color. 'I saw this on the ground too', you think.

The difference is now your arm is fatigued.
Bug report: disappearing equations
by TLW — 2022-02-26T21:30:02.222Z

Bug report[1]: in the default editor typing a space[2] after an inline LaTeX equation at the end of the line deletes all text on the line from the first inline LaTeX equation on the line onwards[3].

For Atest instance: Btest

If I hit space after the end of the above paragraph, it becomes:

For A

You can work around this in several ways, but it's annoying at best:

Copy/paste a space after the equation.
Type a non-space, then a space, then delete the non-space. (E.g. 'a' 'space' 'left-arrow' 'backspace' 'right-arrow').
Don't turn the equation into LaTeX until after you've put a space after it.

I have hit this several times, and have actually lost (a trivial amount of) work due to the issue (undo works to recover, but as this site doesn't have a full tree-undo-redo system it only works if you catch it before you use the undo system for something else.)

Is this a known issue? Is there a better workaround for this?

^

It feels wrong to submit a question for a bug report, but that's on the suggested list...

^

It only appears to happen with spaces?

^

I think I saw it delete all LaTeX and only LaTeX on the line once? I have been unable to reproduce this variant, however.