trentmkelly/LessWrong-43k huggingface dataset:
Reversal Tests in Argument and Debate
by Davis_Kingsley — 2019-09-13T09:18:48.365Z

One thing that I've noticed recently is that simple reversal tests can be very useful for detecting bias when it comes to evaluating policy arguments or points made in a debate.

In other words, when encountering an argument it can be useful to think "Would I accept this sort of argument if it were being made for the other side?" or perhaps "If the ideological positions here were reversed, would this sort of reasoning be acceptable?"

This can be a very easy check to determine whether there is biased thinking going on. Here are some examples of situations where one might be able to apply this:

Someone is advocating a locally unpopular belief and being attacked for it. (Ask yourself whether the same sort of advocacy and reasoning would be mocked if it were being made towards locally popular conclusions; ask yourself whether the mockery would be accepted if it were being made against someone locally popular.)
Someone advocates an easy dismissal of one of the perspectives in an argument. (Ask yourself whether this sort of dismissal would seem reasonable if made against one of your own points.)
Someone makes arguments against a locally unpopular organization or belief. (Ask yourself whether these arguments would pass muster against something that wasn't already derided locally.)

Often one will find that in fact that sort of argument or reasoning would not fly. This can be a good way to check your biases -- people are often prone to accepting weak arguments for things that they already agree with or against thing they already disagree with, and stopping to check whether that reasoning would work in the "other direction" is useful.

(Other times, of course, one will find that the reasoning in question does pass the reversal test -- but even so, it can good to check such things! "Trust but verify" and all that.)
Request for stories of when quantitative reasoning was practically useful for you.
by Eli Tyre — 2019-09-13T07:21:43.686Z

I'm studying more math and CS these days than I have in the past, and I would like to seize any opportunities to generalize those mental skillsets to other domains. I think that generalization would be easier if I had concrete targets: I knew of the specific low level skills that have been useful for folks.

Therefore, I'm looking for anecdotes that express the value of quantitative thinking, and mathematical competency, in "real life". What does that skill set allow you to do? What concrete problems has it solved for you? etc.

Feel free to interpret "quantitative thinking" or "mathematical competency", as broadly as you want. If there's an attitude or mindset that you learned from studying biology, or or from building software, and that mindset has proved practically useful for you outside of that domain, please share.
What You See Isn't Always What You Want
by TurnTrout — 2019-09-13T04:17:38.312Z

To aid communication, I’m going append a technical rephrasing after some paragraphs.

It’s known to be hard to give non-trivial goals to reinforcement learning agents. However, I haven’t seen much discussion of the following: even ignoring wireheading, it seems impossible to specify reward functions that get what we want – at least, if the agent is farsighted, smart, and can’t see the entire world all at once, and the reward function only grades what the agent sees in the moment. If this really is impossible in our world, then the designer’s job gets way harder.

Even ignoring wireheading, it could be impossible to supply a reward function such that most optimal policies lead to desirable behavior – at least, if the agent is farsighted and able to compute the optimal policy, the environment is partially observable (which it is, for the real world), and the reward function is Markovian.

I think it’s important to understand why and how the designer’s job gets harder, but first, the problem.

Let’s suppose that we magically have a reward function which, given an image from the agent’s camera, outputs what an idealized person would think of the image. That is, given an image, suppose a moral and intelligent person considers the image at length (magically avoiding issues of slowly becoming a different person over the course of reflection), figures out how good it is, and produces out a scalar rating – the reward.

The problem here is that multiple world states can correspond to the same camera input. Is it good to see a fully black image? I don’t know – what else is going on? Is it bad to see people dying? I don’t know, are they real, or perfectly Photoshopped? I think this point is obvious, but I want to make it so I can move on to the interesting part: there just isn’t enough information to meaningfully grade inputs. Contrast with being able to grade universe-histories via utility functions: just assign 1 to histories that lead to better things than we have right now, and 0 elsewhere.

The problem is that the mapping from world state to images is not at all injective... in contrast, grading universe-histories directly doesn’t have this problem: simply consider an indicator function on histories leading to better worlds than the present (for some magical, philosophically valid definition of “better”).

Now, this doesn’t mean we need to have systems grading world stat...
Who's an unusual thinker that you recommend following?
by Pee Doom — 2019-09-13T04:11:17.413Z

Some (optional!) desiderata to guide your recommendation:

They probably have not been heard of around these parts.
They disagree with a consensus.
They have genuinely original thoughts.
You have updated from their views and/or been emotionally affected.
They're actively doing some interesting thing(s) and seem effective at those things.
Reading/listening doesn't feel like an obtuse poetic maze; they attempt to convey things relatively clearly.
Maybe they use an unusual method of communication.

Feel free to disregard these criteria if it doesn't work for answering the question! :)
The Power to Understand "God"
by Liron — 2019-09-12T18:38:00.438Z

This is Part VI of the Specificity Sequence

“God” is a classic case of something that people should be more specific about. Today’s average college-educated person talking about God sounds like this:

Liron: Do you believe in God?

Stephanie: Yeah, I’d say I believe in God, in some sense.

Liron: Ok, in what sense?

Stephanie: Well, no one really knows, right? Like, what’s the meaning of life? Why is there something instead of nothing? Even scientists admit that we don’t understand these things. I respect that there’s a deeper kind of truth out there.

Liron: Ok, I remember you said your parents are Christian and they used to make you go to church sometimes… are you still a Christian now or what?

Stephanie: At this point I wouldn’t be dogmatic about any one religion, but I wouldn’t call myself an atheist either. Maybe I could be considered agnostic? I feel like all the different religions have an awareness that there’s this deep force, or energy, or whatever you want to call it.

Liron: Ok, so, um… do you pray to God?

Stephanie: No, I don’t really say prayers anymore, but I prayed when I was younger and I appreciate the ritual. The Western religions believe in praying to God and being saved. The Eastern religions believe in meditation and striving toward enlightenment. There are a lot of different paths to the same spiritual truth… I just have faith that there is a truth, you know?

What did you think of Stephanie’s answers? I’m pretty sure most people would be like, “Yeah, sounds right to me. I’m just glad you didn’t ask me to explain it. I really would have struggled if you’d put me on the spot like that, but she did a pretty good job.”

If we were asking Stephanie about her fantasy sports team picks, we’d expect her to explain what she believes and why she believes it, grounding her claims in specific predictions about the outcomes of future sportsball games.

But there’s a social norm that when we ask someone about “God”, it’s okay for them to squirt an ink cloud of “truth”, “energy”, “enlightenment”, “faith”, and so on, and make their getaway.

Let’s activate our specificity powers. We’ve seen that the best way to define a term is often to ground the term, to slide it down the ladder of abstraction. How would Stephanie ground her concept of “God”? Her attempts might look like this:

The universal force that people pray to.
The energy that makes the universe exist.
The destination

...
The Power to Solve Climate Change
by Liron — 2019-09-12T18:37:32.672Z

This is Part V of the Specificity Sequence

Companion post: Examples of Examples

Most people agree that climate change is a big problem we should be solving, but couldn't tell you what specifically "solving climate change" means. By the end of this post, I promise you'll know what specifically "solving climate change" means... and also what it doesn't mean.

The Climate Crisis

The first step is knowing the key specific facts of the climate crisis. I admit I couldn't recite them accurately before writing this post. Here's a quick summary from Tomorrow and NASA (both great friendly explanations worth checking out):

Earth's average temperature has shot up by 1°C in the last 50 years.
The causal link from greenhouse gas emissions to Earth's rising temperature has been well established.
On a 1M-year timescale, Earth's temperature has been fluctuating plus or minus a few degrees tops, so this 1°C change is a big fluctuation.
We're predicting it to be as high as a 6°C warming by 2100, so it's actually a huge fluctuation.
The current rate of temperature rise is a whopping 20 times faster than the rate at which Earth's temperature historically fluctuates, so it's actually an oh, SHIT fluctuation.

Thiel's Definite vs. Indefinite Attitude

In Peter Thiel's smart and highly original book Zero to One: Notes on Startups, or How to Build the Future, the chapter called "You Are Not A Lottery Ticket" centers around the concept of definite vs. indefinite attitudes:

Under the heading "Can You Control Your Future?" Thiel writes:

You can expect the future to take a definite form or you can treat it as hazily uncertain. If you treat the future as something definite, it makes sense to understand it in advance and work to shape it. But if you expect an indefinite future ruled by randomness, you'll give up on trying to master it.

Indefinite attitudes to the future explain what's the most dysfunctional in our world today. Process trumps substance: when people lack concrete plans to carry out, they use formal rules to assemble a portfolio of various options. This describes Americans today. In middle school, we're encouraged to start hoarding "extracurricular activities". In high school, ambitious students compete even harder to appear omnicompetent. By the time a student gets to college, he's spent a decade curating a bewilderingly diverse résumé to prepare for a completely unknowable future. Come wha

...
Answering some questions about EA
by jefftk — 2019-09-12T17:50:01.408Z

A teacher in Vienna recently wrote to say that they had assigned

an
article about us as part of a Social Issues class, and asked
whether I would be up for answering questions. The students asked me
their questions via video snippets, and here are my answers:

Gergo: When was the first time you gave money to other people? Did it start
when you were younger, or when you were in college?

The first donation I remember was a pair of political donations
leading up to the 2008
election. I gave $50
each to two candidates who were very unlikely to win: Mike Gravel and
Ron Paul. I was
hoping to expand the range of ideas
people were discussing in the primaries.

The next time I donated was $500 two years
later, after deciding to give half of what I
earned to charity starting in 2009. I think I decided to make my
first donation in 2008 instead of 2009 because I misunderstood how
income taxes worked and thought it was better to spread donations
across years (in the US it's actually the opposite).

Emilie: How did you get into this to start with and how do you keep up with
it? How did you get into the mindset of "we can give so much and that's what
we want to do", without flaking out?

When I started my first real job and was suddenly making much more
money than I'd ever had before, the contrast between what I had and
what I really needed weighed on me. Based on my upbringing and
temperament I would have continued living simply and saved the money,
but after a lot of discussions with Julia I realized that I wasn't
ok having so much while others had so little, and couldn't keep it.

This contrast hasn't gone away, and the level of need in the world remains
unconscionably high, so I can't see doing something else.

It also helps that I've written a lot about this publicly, and it
would be pretty embarrassing to go back and say "you know what, I'm
done with this 'charity' thing, saving up to buy a yacht is just too
important to me."

Danilla: At 22 you were broke and even though you had no job you still
donated all of your savings to the charity. Why you did that and how did you
live at that time if you didn't have any money?

This question is phrased as if it's intended for Julia, but I can give
my perspective. While Julia hadn't found a job yet I was working full
time, and so I think she felt secure enough to donate her savings. In
retrospe...
Predictable Identities - Midpoint Review
by Jacob Falkovich — 2019-09-12T14:39:44.348Z

Predictable Identities is a series that I’m writing on Ribbonfarm; you can also track its chronological progress on Putanumonit with short summaries of each post. The 17th post in the series is below, summarizing the story so far and providing links to all the preceding posts in a narrative, rather than chronological order.

Predictable Identities is 16 posts in, which is going much better than I would’ve predicted. It’s time to review what we covered so far: the principles of predictive processing and how we apply them to other people.

Our brains constantly predict sensory inputs using a hierarchy of models. Learning new and better models improves our predictive ability in the long term but can be so painful in the short term that we will fight against updating, and often fight the people who force us to update. It’s important to take all models with a grain of salt and resist the lure of all-explaining ideologies.

We predict the world to exploit and act on it, and the same applies to other people. We need to know how to get people to do nice things for us, using stereotypes for strangers we don’t know and more detailed mind-simulations for people we do. We don’t need detailed models for people unlikely to be nice no matter what, and we’re creeped out by people who don’t fit our models at all like those who blow their non-conformity budget. We encourage those around us to conform to our narratives and predictions of them, which means that changing one’s opinions and behavior takes great effort in the face of the expectations of your social surroundings.

Finally, our predictions of ourselves interplay in complex ways with how others see us, since our minds are neither transparent to us nor opaque to those around us. For example, to be treated nicely we must honestly believe that we are nice, even if that is self-deceiving. Our models, predictions, and beliefs about our selves form our identities. This will be the topic for the second half of this blogchain.

Thanks for joining me on this journey, I predict exciting things ahead.
Toy model piece #5: combining partial preferences
by Stuart_Armstrong — 2019-09-12T03:31:25.295Z

My previous approach to combining preferences went like this: from the partial preferences Pi, create a normalised utility function ˆUi that is defined over all worlds (and which is indifferent to the information that didn't appear in the partial model). Then simply add these utilities, weighted according to the weight/strength of the preference.

But this method fails. Consider for example the following partial preferences, all weighted with the same weight of 1:

P1:A>C.

P2:A>B.

P3:B>C.

P4:B>D.

P5:B>E.

If we follow the standard normalisation approach, then the normalised utility ˆU1 will be defined[1] as:

ˆU1(A)=0.5, ˆU1(C)=−0.5, and otherwise ˆU1(−)=0.

Then adding together all five utility functions would give:

U⎛⎜
⎜
⎜
⎜
⎜
⎜⎝ABCDE⎞⎟
⎟
⎟
⎟
⎟
⎟⎠=⎛⎜
⎜
⎜
⎜
⎜
⎜⎝11−1−0.5−0.5⎞⎟
⎟
⎟
⎟
⎟
⎟⎠.

There are several problems with this utility. Firstly, the utility of A and the utility of B are the same, even though in the only case where there is a direct comparison between them, A is ranked higher. We might say that we are missing the comparisons between A and D and E, and could elicit these preferences using one-step hypotheticals. But what if comparing A to D is a complex preference, and all that happens is that the agent combines A>B and B>D? If we added another partial preference that said B>F, then B would end up ranked above A!

Another, more subtle point, is that the difference between A and C is too large. Simply having A>B and B>C would give U(A)−U(C)=1. Adding in A>C moves this difference to 2. But note that A>C is already implicit in A>B and B>C, so adding it shouldn't make the difference larger.

In fact, if the difference in utility between A and C were larger than 1, adding in A>C should make the difference between U(A) and U(C) smaller: because having A>C weighted at 1 means that the agent's preference of A over C is not that strong.

Energy minimising between utilities

So, how should we combine these preferences otherwise? Well, if I have a preference Pi, of weight wi, that ranks outcome G below outcome H (write this as G<iH), then, if these outcomes appear nowhere else in any partial preference, U(G)−U(H) will be wi.

So in a sense, that partial preference is trying to set the distance between those two outcomes to wi. Call this the energy-minimising condition for Pi.

Then fo...
Toy model piece #4: partial preferences, re-re-visited
by Stuart_Armstrong — 2019-09-12T03:31:08.628Z

Two failed attempts

I initially defined partial preferences in terms of foreground variables Y and background variables Z.

Then a partial preference would be defined by y+ and y− in Y, such that, for any z∈Z, the world described by (y+,z) would be better than the world described by (y−,z). The idea being that, everything else being equal (ie the same z), a world with y+ was better than a world with y−. The other assumption is that, within mental models, human preferences can be phrased as one or many binary comparisons. So if we have a partial preference like P1: "I prefer a chocolate ice-cream to getting kicked in the groin", then (y+,z) and (y−,z) are otherwise identical worlds with a chocolate ice-cream and a groin-kick, respectively.

Note that in this formalism, there are two subsets of the set of worlds, y+×Z and y−×Z, and map l between them (which just sends (y+,z) to (y−,z)).

In a later post, I realised that such a formalism can't capture seemingly simple preferences, such as P2: "n+1 people is better than n people". The problem is that that preferences like that don't talk about just two subsets of worlds, but many more.

Thus a partial preference was defined as a preorder. Now, a preorder is certainly rich enough to include preferences like P2, but its allows for far too many different types of structures, needing a complicated energy-minimisation procedure to turn a preorder into a utility function.

This post presents another formalism for partial preferences, that keeps the initial intuition but can capture preferences like P2.

The formalism

Let W be the (finite) set of all worlds, seen as universes with their whole history.

Let X be a subset of W, and let l be an injective (one-to-one) map from X to W. Define Y=l(X), the image of l, and l−1:Y→X as the inverse.

Then the preference is determined by:

For all x∈X, x>l(x).

If X and Y are disjoint, this just reproduces the original definition, with X=y+×Z and Y=y−×Z.

But it also allows preferences like P, defining l(x) as something like "the same world as x, but with one less person". In that case, l maps some parts of X to itself[1].

Then for any element x∈X, we can construct its upwards and downwards chain:

…,l−3(x),l−2(x),l−1(x),x,l(x),l2(x),l3(x),….

These chains end when they cycle: so there is an n and an m so that l−n(x)=lm(x) (equivalently, lm+n(x)=x).

If they don't cycle, the upwards chain en...
Conversation with Paul Christiano
by abergal — 2019-09-11T23:20:01.309Z

AI Impacts talked to AI safety researcher Paul Christiano about his views on AI risk. With his permission, we have transcribed this interview.

Participants

Paul Christiano — OpenAI safety team
Asya Bergal – AI Impacts
Ronny Fernandez – AI Impacts
Robert Long – AI Impacts

Summary

We spoke with Paul Christiano on August 13, 2019. Here is a brief summary of that conversation:

AI safety is worth working on because AI poses a large risk and AI safety is neglected, and tractable.
Christiano is more optimistic about the likely social consequences of advanced AI than some others in AI safety, in particular researchers at the Machine Intelligence Research Institute (MIRI), for the following reasons:

The prior on any given problem reducing the expected value of the future by 10% should be low.
There are several ‘saving throws’–ways in which, even if one thing turns out badly, something else can turn out well, such that AI is not catastrophic.
Many algorithmic problems are either solvable within 100 years, or provably impossible; this inclines Christiano to think that AI safety problems are reasonably likely to be easy.
MIRI thinks success is guaranteeing that unaligned intelligences are never created, whereas Christiano just wants to leave the next generation of intelligences in at least as good of a place as humans were when building them.
‘Prosaic AI’ that looks like current AI systems will be less hard to align than MIRI thinks:

Christiano thinks there’s at least a one-in-three chance that we’ll be able to solve AI safety on paper in advance.
A common view within ML is that that we’ll successfully solve problems as they come up.

Christiano has relatively less confidence in several inside view arguments for high levels of risk:

Building safe AI requires hitting a small target in the space of programs, but building any AI also requires hitting a small target.
Because Christiano thinks that the state of evidence is less clear-cut than MIRI does, Christiano also has a higher probability that people will become more worried in the future.
Just because we haven’t solved many problems in AI safety yet doesn’t mean they’re intractably hard– many technical problems feel this way and then get solved in 10 years of effort.
Evolution is often used as an analogy to argue that general intelligence (humans with their own goals) becomes dangerously unaligned with the goals of th

...
What are the merits of signing up for cryonics with Alcor vs. with the Cryonics Institute?
by Eli Tyre — 2019-09-11T19:06:53.802Z

I am signing up for cryonics this year, and I need to figure out if I should pay extra to do it with Alcor instead of CI.

What's the balance of evidence on this question?

With which organization have most cryonicist LessWrongers gone? [edit: I made a google form for this more specific question here.]