trentmkelly/LessWrong-43k huggingface dataset:
[Linkpost] Otu and Lai: a story
by Sunny from QAD — 2019-09-15T20:21:12.445Z

Link: http://vansochill.blog/2019/09/15/otu-and-lai-a-story/

This story was originally for my blog, and it was going to build up to a follow-on post about morality/philosophy, but I think it's an important notion for rationalists too.

(NB: My blog has a gimmick, which is that I don't post anything with E's in it. As such, this story's wording is a bit unusual.)

Without ado:

On a cool autumn day, two girls found a dark hollow. At 14 and 16, Otu and Lai had no grasp of what risks its winding corridors could hold. Lai, always an analyst, was curious what lay at its bottom, if it had a bottom at all. Otu, young and timid, saw no option but to tag along.

It wasn’t long until that hollow’s mouth was a long-ago ghost of a thought. Lai’s hand ran along dank rock walls just to find a way to walk; sunlight had all but quit our protagonists.

“I know this wall!” Lai would say now and again. “Now I can find us a way out!”

On no occasion did this pan out, and following six dud calls, Lai didn’t try again.

“Damn,” said Lai at last. “Okay, I couldn’t hold a full map of this hollow in my mind. But you know what? Turning right at all junctions should work to bring us back out. That’s a fact I know from math class, so it has to work.”

But that was a no-go too; it wasn’t always turning right that would bring you out; it was always following a wall that would do it. Lai’s slip cost an hour, at which point both girls had to stop walking and sit down for a bit. Gloom sprung up, and spans of quick sobs took turns with spans of calm.

Finally Otu lay down, admitting that touching a bit of dirt was worth not having an aching butt from sitting for so long. Might so much as a nap stand as too high a wish?

It did, for Otu’s torso hit not ground but a plastic chassis.

“Ouch! What was that?”

Lai sat up. “What’s what?

Otu’s hand found it, took it, hit it, hit it again, and ran along its plastic skin, trying to find out what it was. It was long-ish, a tubular form, with a sort of protrusion on its top. It was mostly rough and hard, though its protrusion’s front was smooth and glassy. It also had a small switch on it. Flipping that switch did nothing at first, but…

O brilliant light! Abruptly a hollow priorly dark and boding was brightly lit and allowing, possibly, of a happy conclusion.

Otu’s jaw hung, and said nothing.

“Fascinating!” said Lai. “Can I hold it?”

Now it was Lai’s hand that

...
Category Qualifications (w/ exercises)
by Logan Riggs — 2019-09-15T16:28:53.149Z

This is the second post in the Arguing Well
sequence. This post is influenced by A Human's Guide to
Words and Against Lie
Inflation.

Intro

In the last post, we discussed a common problem in arguments that Prove Too Much. In this post, we’ll
generalize that problem to help determine useful categories. But before we go on, what’s wrong with these
arguments?

Ex. 1 [Stolen from slatestarcodex]

“A few months ago, a friend confessed that she had abused her boyfriend. I was shocked, because this
friend is one of the kindest and gentlest people I know. I probed for details. She told me that sometimes she needed
her boyfriend to do some favor for her, and he wouldn’t, so she would cry – not as an attempt to
manipulate him, just because she was sad. She counted this as abuse, because her definition of “abuse”
is “something that makes your partner feel bad about setting boundaries”. And when she cried, that
made her boyfriend feel guilty about his boundary that he wasn’t going to do the favor.”

By this definition of “abuse”, a majority of people are “abusive”.
It would be better to reserve that word for a smaller group of people who are intentionally manipulating people.

Ex. 2: I also had a friend that was “one of the nicest people everyone knows, wow she listens so
well!”. She admitted that she was actually “selfish and manipulative” because she did nice
things to people so they’d like her.

I honestly wish everyone was as “selfish and manipulative” as this girl;
however, it makes those two words nearly useless. It would be better to reserve those words for people who
create win-lose situations (You give, I take) as opposed to win-win situations (Oh wow, you make me feel
important. I want to be your friend).

General Frame

What is the general frame of the problem in the two scenarios? You have 2 minutes.

The claim relies on a “bad definition” of a word. It’s
“bad” because my expectations weren’t met. If you say you’re selfish, then I expect
that you create win-lose situations, but if all you’re actually a really nice person

...
How do I reach a conclusion on how many eggs per week are healthy?
by Rudi C — 2019-09-15T15:44:16.753Z

As someone who is not a nutritionist, and has limited time, what’s the optimal course of action in determining whether the evidence is in favor of eggs or against them?

Should I look at public health guidelines, poke around for meta analyses in Google Scholar, find the expert consensus, or ... ?

It is also worth noting that there are at least two big problems with the evidence about eggs; A lot of research is said to be sponsored by the industry. Egg consumption is correlated with all kinds of bad habits/characteristics (obesity, smoking, poverty, etc).

I know the position of maximum caution is to replace eggs with other less suspect sources of micronutrients, but personally I have crippled access to a good grocery market (I live in Iran), and am under a limited time budget. This makes me reluctant to forego eggs merely to be on the safe side. Especially since I eat a semipaleo diet, which is already restrictive.
Focus
by jefftk — 2019-09-15T15:10:01.475Z

One of the things I have the hardest time with is focus. I have times when
focus comes easily, and I get a week's worth of work done in hours, come up
with ideas that need sustained thinking, write blog posts that have been
bouncing around my head for years. Other times my attention is flighty, and I
have to struggle to keep myself from finding low-investment sources of
entertainment.

The biggest component of this is how excited I am. When I really get
into something, that's when focus comes easily. I recently finished a
project at work where I was building something that dramatically
improved something the team had found frustrating for years. I could
see where I wanted it to go, and I was excited enough that when I ran
into issues I pushed at it until I had good solutions for them. The
bass whistle project was another one like
this, where I couldn't think of anything else for about a week, until
I had something coded up and working. I didn't need to make myself
focus, I needed to make myself do the rest of my life.

A lot of other factors matter, but are mediated by excitement. Short iteration
cycles, where I can quickly find out whether something worked, are so important
to me because they keep the excitement from draining away. Doing something no
one has done before, that a lot of people are going to like, that needs doing
urgently, or that I've been thinking about for a long time all help, but mostly
because those are exciting things.

In-person collaboration also helps, in at least two ways. When I'm
working with someone, talking to them directly, it feels like ideas
flow much better. It's most fun when we have complementary skills,
each filling in for and learning from the other, but even when the
other person is inexperienced it still helps to have another person's
worth of working memory. And then if I'm working one-on-one with
someone I can't respond to brief roadblocks by letting myself get
distracted.

There's also a component of mental patterns: if I'm in the habit of tabbing
over to Facebook I'll fall out of focus more easily. [1] The hard part for me
is that I'm often doing work that is full of short breaks that should be fine
to fill: waiting for compiles, for queries, for tests. This means that when
I've tried to make myself rules like "no distraction activities" I either get
bored enough waiting for things that I ca...
Proving Too Much (w/ exercises)
by Logan Riggs — 2019-09-15T02:28:51.812Z

This is the first post in the Arguing Well sequence. This post is influenced by Scott Alexander's write up on Proving too Much.

[edit: Reformatted the post as a Problem/Solution to clarify what I'm trying to claim]

The Problem

One of the purposes of arguing well is to figure out what is true. A very common type of bad argument claims something like this:

Because of reason X, I am 100% confident in belief Y

I don't know of any reason that leads to 100% truth all the time (and if you do, please let me know!), and it's usually hard to reason with the person until this faulty logic is dealt with first. This is the purpose of this post.

Assuming the context of all exercises is with someone claiming 100% belief for that one reason, what's wrong with the following:

Ex. 1: I believe that Cthulhu exists because that’s just how I was raised.

How someone was raised doesn’t make something true or not. In fact, I could be raised to believe that Cthulhu doesn’t exist. We can’t both be right.

Ex. 2: I believe that a goddess is watching over me because it makes me feel better and helps me get through the day.

Just because believing it makes you feel better doesn’t make it true. Kids might feel better believing in Santa Claus, but that doesn’t make him actually exist.

Generalized Model

How would you generalize the common problem in the above arguments? You have 2 minutes

The common theme that I see is that same logic that proves the original claim, also proves something false. It “Proves too much” because it also proves false things. I like to think of this logic as “Qualifications for 100% truth”, and whatever qualifications proves the original claim can also prove a false claim.

Truth Qualifications -> Claim

Same Truth Qualifications -> Absurd Claim

Important Note: the purpose of this frame isn't to win an argument/ prove anything. It's to differentiate between heuristics that claim 100% success rates vs ones that claim a more accurate estimates. Imagine "I'm 100% confident I'll roll 7's with my two die cause of my luck!" vs "There's a 6/36 chance I'll roll 7's because I'm assuming two fair die"

Let’s work a couple more examples with this model.

Ex. 3: My startup is guaranteed to succeed because it uses quantum machine learning on a blockchain!

A startup using buzzwords doesn’t make it succeed. In fact, there are several startups that use those terms and failed.

Ex. 4: Of course ...
Arguing Well Sequence
by Logan Riggs — 2019-09-15T02:01:30.976Z

Arguments have the potential to allow you to connect and understand with someone else on a deep level, to introspect and figure out what you truly believe and care about, and to find out what is true so you can accomplish your goals!

Most people use it to dominate or talk past each other.

But there are ways to consistently argue well. Luckily, most of the hard work has already been done over the years on Less Wrong,

SlateStarCodex

, and

Street Epistemology videos

. My contribution is providing short summaries of these techniques, exercise prompts (some borrowed from the above sources) and solutions, generalizations, ideal algorithms, and relationships between the different techniques. They are as follows:

Proving Too Much (w/ exercises)
Category Qualifications (w/ exercises)
False Dilemmas (w/ exercises)
Finding cruxes (w/ exercises)

Although these exercises are not as useful as real-life conversations, they are enough to impart a gears-level model of these techniques, so when you find yourself in a confused conversation, you’ll know where to put all the pieces and how to have a productive discussion.

Note: This sequence is motivated by the

rationality exercise contest

.
If you had to pick one thing you've read that changed the course of your life, what would it be?
by  — 2019-09-14T17:50:45.292Z

Found on Twitter: https://mobile.twitter.com/paulg/status/1172767548363935744

Personal answer:

1. This WaitButWhy article on artificial superintelligence started my journey down the rabbit hole of AI alignment and actually, as a matter of fact, changed the course of my life

2. Runner-up: Meditations on Moloch - unsurprising choice, but it did have a profound impact on my worldview

The criteria for choosing your answer should be the impact the thing you read had on your life, not whether you agree with it, learned from it or would recommend it to others, though these things might be correlated.
Reframing the evolutionary benefit of sex
by paulfchristiano — 2019-09-14T17:00:01.184Z

Automatically crossposted from https://sideways-view.com

From the perspective of an organism trying to propagate its genes, sex is like a trade: I’ll put half of your DNA in my offspring if you put half of my DNA in yours. I still pass one copy of my genes onto the next generation per unit of investment in children, so it’s a fair deal. And it doesn’t impact the average fitness of my kids very much, since on average my partner’s genes will be about as good as mine. (ETA: but see the discussion below, in which case the costs might be much bigger.)

But the trade has transaction costs, so I’m only going to do it if I get some benefit. In this post I’ll tell a particularly simple story about the benefit of sex. I think this is basically equivalent to the standard story, but I find it much clearer. It also makes it more obvious that we don’t require group selection, and that the benefit is very large.

Why doesn’t sex change the average fitness of my kids? The possibility of a “lucky” kid who gets the better genes from both of us is offset by the possibility of an unlucky kid who gets the worse genes from both of us. If the effects of genes are linear, the average fitness will be exactly the same as the parents. In practice I expect it to be slightly lower because of convexity and linkage disequilibrium.

But sex increases the average fitness of my grandchildren, because my fittest children will be responsible for a disproportionate fraction of my grandkids. More precisely, if my if an organism with fitness dX has (1+dX) kids per generation, then the total fitness of my grandkids is E[(1 + dX)^2] = 1 + 2 E[dX] + E[dX^2]. So increasing variance by 1 unit is as good as increasing average fitness by 0.5 units.

Reproductive decisions are naturally a tradeoff between average fitness and variance. Sex slightly lowers the average but increases the variance. If you try to get the same amount of variance with random mutations, you’ll have to totally tank your kid’s expected fitness, because your current genome is well-optimized, and you’ll also pass on fewer genes to the next generation (since some of yours got destroyed). In fact, it’s hard to think of any way to get similar benefits without exchanging genes.

Variance becomes linearly more important over time. For the fitness of my grandkids, 1 unit of variance is worth 0.5 units of fitness; for great-grandkids, they are equally...
Do Sufficiently Advanced Agents Use Logic?
by abramdemski — 2019-09-13T19:53:36.152Z

This is a continuation of a discussion with Vanessa from the MIRIxDiscord group. I'll make some comments on things Vanessa has said, but those should not be considered a summary of the discussion so far. My comments here are also informed by discussion with Sam.

1: Logic as Proxy

1a: The Role of Prediction

Vanessa has said that predictive accuracy is sufficient; consideration of logic is not needed to judge (partial) models. A hypothesis should ultimately ground out to perceptual information. So why is there any need to consider other sorts of "predictions" it can make? (IE, why should we think of it as possessing internal propositions which have a logic of their own?)

But similarly, why should agents use predictive accuracy to learn? What's the argument for it? Ultimately, predicting perceptions ahead of time should only be in service of achieving higher reward.

We could instead learn from reward feedback alone. A (partial) "hypothesis" would really be a (partial) strategy, helping us to generate actions. We would judge strategies on (something like) average reward achieved, not even trying to predict precise reward signals. The agent still receives incoming perceptual information, and strategies can use it to update internal states and to inform actions. However, strategies are not asked to produce any predictions. (The framework I'm describing is, of course, model-free RL.)

Intuitively, it seems as if this is missing something. A model-based agent can learn a lot about the world just by watching, taking no actions. However, individual strategies can implement prediction-based learning within themselves. So, it seems difficult to say what benefit model-based RL provides beyond model-free RL, besides a better prior over strategies.

It might be that we can't say anything recommending model-based learning over model free in a standard bounded-regret framework. (I actually haven't thought about it much -- but the argument that model-free strategies can implement models internally seems potentially strong. Perhaps you just can't get much in the AIXI framework because there are no good loss bounds in that framework at all, as Vanessa mentions.) However, if so, this seems like a weakness of standard bounded-regret frameworks. Predicting the world seems to be a significant aspect of intelligence; we should be able to talk about this formally somehow.

Granted, it doesn't m...
A Critique of Functional Decision Theory
by wdmacaskill — 2019-09-13T19:23:22.532Z

A Critique of Functional Decision Theory

NB: My writing this note was prompted by Carl Shulman, who suggested we could try a low-time-commitment way of attempting to understanding the disagreement between some folks in the rationality community and academic decision theorists (including myself, though I’m not much of a decision theorist). Apologies that it’s sloppier than I’d usually aim for in a philosophy paper, and lacking in appropriate references. And, even though the paper is pretty negative about FDT, I want to emphasise that my writing this should be taken as a sign of respect for those involved in developing FDT. I’ll also caveat I’m unlikely to have time to engage in the comments; I thought it was better to get this out there all the same rather than delay publication further.

Introduction

There’s a long-running issue where many in the rationality community take functional decision theory (and its variants) very seriously, but the academic decision theory community does not. But there’s been little public discussion of FDT from academic decision theorists (one exception is

here

); this note attempts to partly address this gap.

So that there’s a clear object of discussion, I’m going to focus on Yudkowsky and Soares’ ‘

Functional Decision Theory

’ (which I’ll refer to as Y&S), though I also read a revised version of Soares and Levinstein’s

Cheating Death in Damascus

.

This note is structured as follows. Section II describes causal decision theory (CDT), evidential decision theory (EDT) and functional decision theory (FDT). Sections III-VI describe problems for FDT: (i) that it sometimes makes bizarre recommendations, recommending an option that is certainly lower-utility than another option; (ii) that it fails to one-box in most instances of Newcomb’s problem, even though the correctness of one-boxing is supposed to be one of the guiding motivations for the theory; (iii) that it results in implausible discontinuities, where what is rational to do can depend on arbitrarily small changes to the world; and (iv) that, because there’s no real fact of the matter about whether a particular physical process implements a particular algorithm, it’s deeply indeterminate what FDT’s implications are. In section VII I discuss the idea that FDT ‘does better at getting utility’ than EDT or CDT; I argue that Y&S’s claims to this effect are unhelpfully vague, and...
Theory of Ideal Agents, or of Existing Agents?
by johnswentworth — 2019-09-13T17:38:27.187Z

Within the context of AI safety, why do we want a

theory of agency

? I see two main reasons:

We expect AI have agenty properties, so we want to know how to design agenty systems which e.g. perform well in a maximally wide array of environments, and which reason about themselves and self-modify while maintaining their goals. The main use-case is to design an artificial agent.
We want AI to model humans as agenty, so we need a theory of e.g. how things made of atoms can “want” things, model the world, or have ontologies, and how to reliably back out those wants/models/ontologies from physical observables. The main use-case is to describe existing agenty systems.

The ideas involved overlap to a large extent, but I’ve noticed some major differences in what kinds of questions researchers ask, depending on which of these two goals they’re usually thinking about.

One type of difference seems particularly central: results which identify one tractable design within some class, vs characterizing all designs within a class.

If the goal is to design an artificial agenty system with ideal properties, then the focus is on existence-type proofs: given some properties, find a design which satisfies them.
If the goal is to describe agenty systems in the wild, then the focus is on characterizing all such systems: given some properties, show that any system which satisfies them must have some specific form or additional properties.

This difference suggests different trade-offs:

If the goal is to design one system with the best performance we can achieve in the widest variety of environments possible, then we’ll want the strongest properties we can get. On the other hand, it’s fine if there’s lots of possible agenty things which don’t satisfy our strong properties.
If the goal is to describe existing agents, then we’ll want to describe the widest variety of possible agenty things we can. On the other hand, it’s ok if the agents we describe don’t have very strong properties, as long as the properties they do have are realistic.

As an example, consider

logical induction

. Logical induction was a major step forward in designing agent systems with strong properties - e.g. eventually having sane beliefs over logic statements despite finite resources. On the other hand, for the most part it doesn’t help us describe existing agenty systems much - bacteria or cats or (more debatably) humans probab...