trentmkelly/LessWrong-43k huggingface dataset:
Confusions Concerning Pre-Rationality
by abramdemski — 2018-05-23T00:01:39.519Z

Robin Hanson's Uncommon Priors Require Origin Disputes is a short paper with, according to me, a surprisingly high ratio of does-something-interesting-there per character. It is not clearly right, but it merits some careful consideration. If it is right, it offers strong reason in support of the common prior assumption, which is a major crux of certain modest-epistemology flavored arguments.

Wei Dai wrote two posts reviewing the concepts in the paper and discussing problems/implications. I recommend reviewing those before reading the present post, and possibly the paper itself as well.

Robin Hanson's notion of pre-rationality is: an agent's counterfactual beliefs should treat the details of its creation process like an update. If the agent is a Bayesian robot with an explicitly programmed prior, then the agent's distribution after conditioning on any event "programmer implements prior p" should be exactly p.

These beliefs are "counterfactual" in that agents are typically assumed to know their priors already, so that the above conditional probability is not well-defined for any choice of p other than the agent's true prior. This fact leads to a major complication in the paper; the pre-rationality condition is instead stated in terms of hypothetical "pre-agents" which have "pre-priors" encoding the agent's counterfactual beliefs about what the world would have been like if the agent had had a different prior. (I'm curious what happens if we drop that assumption, so that we can represent pre-rationality within the agent's same prior.)

Wei Dai offers an example in which a programmer flips a coin to determine whether a robot believes coin-flips to have probability 2/3rds or 1/3rd. Pre-rationality seems like an implausible constraint to put on this robot, because the programmer's coin-flip is not good reason to form such expectations about other coins.

Wei Dai seems to be arguing against a position which Robin Hanson isn't quite advocating. Wei Dai's accusation is that pre-rationality implies a belief that the process which created you was itself a rational process, which is not always plausible. Indeed, it's easy to see this interpretation in the math. However, Robin Hanson's response indicates that he doesn't see it:

I just don't see pre-rationality bein

...
Sleeping Beauty Resolved?
by ksvanhorn — 2018-05-22T14:13:05.364Z

[This is Part 1. See also Part 2.]

Introduction

The Sleeping Beauty problem has been debated ad nauseum since Elga's original paper [Elga2000], yet no consensus has emerged on its solution. I believe this confusion is due to the following errors of analysis:

Failure to properly apply probability theory.
Failure to construct legitimate propositions for analysis.
Failure to include all relevant information.

The only analysis I have found that avoids all of these errors is in Radford Neal's underappreciated technical report on anthropic reasoning [Neal2007]. In this note I'll discuss how both “thirder” and “halfer” arguments exhibit one or more of the above errors, how Neal's analysis avoids them, and how the conclusions change when we alter the scenario in various ways.

As a reminder, this is the Sleeping Beauty problem:

On Sunday the steps of the experiment are explained to Beauty, and she is put to sleep.
On Monday Beauty is awakened. While awake she obtains no information that would help her infer the day of the week. Later in the day she is put to sleep again.
On Tuesday the experimenters flip a fair coin. If it lands Tails, Beauty is administered a drug that erases her memory of the Monday awakening, and step 2 is repeated.
On Wednesday Beauty is awakened once more and told that the experiment is over.

The question is this: when awakened during the experiment, what probability should Beauty give that the coin in step 3 lands Heads?

“Halfers” argue that the answer is 1/2, and “thirders” argue that the answer is 1/3. I will argue that any answer between 1/2 and 1/3 may be obtained, depending on details not specified in the problem description; but under reasonable assumptions the answer is slightly more than 1/3. Furthermore,

halfers employ valid reasoning but get the wrong answer because they omit some seemingly irrelevant information; and
thirders (except Neal) employ invalid reasoning that nonetheless arrives at (nearly) the right answer.

The standard framework for solving probability problems

There are actually three separate “Heads” probabilities that arise in this problem:

p1, the probability that Beauty should give for Heads on Sunday.
p2, the probability that Beauty should give for Heads on Monday/Tuesday.
p3, the probability that Beauty should give for Heads on Wednesday.

There is agreement that p1=p3=1/2, but disagreement as to whether p2=1/... ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Against Not Reading Math Books Problems-First (If You've Found It Helpful Before)
by Andrew Quinn — 2018-05-22T12:59:20.054Z

Key Insight: If you were the kid in math class who could usually just skip to the end of the chapter and start working out homework problems immediately, but then you lost that ability, it's still worth trying that quickly and failing before trying to read the chapter first, because you are more likely than average to have a problem-solving bent to your personality that makes this approach both more motivating and more enjoyable. It may even be faster than "front-loading" the knowledge, since now you have a map of what to work for.

Specifically, do not feel like this is somehow "cheating" or "not being rigorous enough", because it is a perfectly valid way to go about things.

------------

It's been said, in various guises, that mathematics is something you do, not something you read.

I generally agree with this advice, but my agreement might just be due to a lack of familiarity. I've only had a few upper level proofs-based math courses, and they've had a "problem-solving" timbre to them (combinatorics, graph theory, differential equations).

Still, I always feel like I have a much better grasp of how to actually work with the theorems and definitions once I've run through some calculations and proofs of my own, especially proofs chosen by the authors to highlight one specific insight at a time. (For calculations: This goes double when I have correct answers to check against.)

In fact, recently I've noticed that even with topics I feel a lot of intrinsic, theoretical motivation to learn more about, I always get a boost in motivation after skipping to the problems first and trying (usually failing) them after a minute or two of effort.

For some reason, this gives me a visceral, intellectual hunger to figure out just what on Earth these egghead authors are yammering about earlier in the chapter that I couldn't figure out on my own. What is it! I want to know now!

From there I usually piece together a "working knowledge" of just enough about the definitions and theorems to actually solve a few problems, and then, finally, I go back and read the chapter, and suddenly details I feel fairly confident I would not have paid attention to are seen in a much more stark and important light. (For a small, recent example of such a detail: That infinite unions, but not infinite intersections, are a...
Glide Meditations: A Statement of Intent
by musicmage4114 — 2018-05-22T07:48:08.648Z

Have you ever experienced that feeling where you finally got an opportunity to do something you think you wanted to do, but now that you have the opportunity to do that thing, you're actually super apprehensive about it?

The first time I experienced that feeling, it was caused by having the opportunity to order steak & eggs for breakfast. As a child, I was a huge fan of cartoon TV show Hey Arnold, and one particular episode featured characters on the show eating steak & eggs. For whatever reason, this struck me as really cool; I liked steak, and I liked eggs, so why not have them together? I would frequently mention to members of my family that I wanted to try the combination, but I was never in the position to do so.

Until one morning, when our family went out to breakfast at a restaurant that, coincidentally, had steak & eggs on the menu. It was my moment to finally try that odd-but-intriguing combination of foods... and I couldn't go through with it. Whether because of the anticipation of (loving and good-natured) teasing from those assembled, a fear of judgment from the (very sweet and motherly) waitress, or a sudden second-guessing of how good it might actually taste, I opted not to order the dish, and ended months' worth of posturing forever.

Now, here I am again, with the opportunity to do something I thought I wanted to do, but once again, I'm terrified.

I discovered Slate Star Codex by chance just a few months ago, and even more recently made my way to the old Less Wrong after the incredible Mr. Alexander mentioned the Sequences. I've been working through them slowly, and while I'm not sure they'll be quite as life-changing as I've seen some profess them to be, there are some cool ideas there, and my mind appreciates the brain food.

I've often felt like I have something to say, but a combination of severe (but finally being treated!) ADD and crippling fear of being ignored or disliked usually keep me from doing more than writing a short Reddit comment once every couple of weeks. But something about the way Mr. Alexander writes makes me feel like maybe it would be okay if I tried. I can't really put my finger on it, but perhaps it's because his site is populated by so many people who are pleasant and respectful that I can feel hopeful that I would receive the same treatment in a place like this.

So I'm going to give this a try, just for me. &qu...
Mini-review: The Book of Why
by Brian Slesinsky — 2018-05-21T23:11:28.317Z

Someone should probably write a real book review, but to make a brief recommendation: The Book of Why by Judea Pearl and Dana Mackenzie is probably the most interesting general-science book I've read since Thinking Fast and Slow.

Pearl's goal is to explain and promote causal inference, which you might think of as (allegedly) the next big thing after frequentist and Bayesian statistics. The introduction is probably skippable, since the authors make some rather grand claims that aren't backed up until later. I found myself thinking, "okay, maybe it's great, but explain what it is already".

Chapter 1 introduces the Ladder of Causation, the authors' way of distinguishing the correlations found via a model-free statistical summary of data (which is level 1) from deductions that require a causal model (levels 2 and 3).

Chapters 2 and 3 give a partial, "whiggish" history of statistics from a causal perspective, covering frequentist and Bayesian statistics and Pearl's AI work, when he invented Bayesian networks. At the end, he talks about the possible junctions in a Bayesian network: the chain, fork, and collider, and how they can easily cause confusion.

Chapter 4 uses causal reasoning to explain the logic behind randomized controlled trials and other ways of controlling for confounding variables.

Chapter 5 covers the scientific debate over cigarette smoking, and how lack of clarity about causation resulted in this debate taking years longer than it needed to.

Chapter 6 is a fun chapter showing how to use causal diagrams to shed new light on the Monty Hall problem and Simpson's paradox.

And that's as far as I've read, but it's enough to make a strong recommendation.

I did a quick search on Less Wrong and causality has been covered before, though not as clearly. In particular, see Yudkowsky's Causal Diagrams and Causal Models.

(I was confused about one bit, though: Yudkowsky writes that "Causal models (with specific probabilities attached) are sometimes known as 'Bayesian networks' or 'Bayes nets'." But in the book, the authors make a clear distinction: "Unlike the causal diagrams we will deal with throughout the book, a Bayesian network carries no assumption that the arrow has any causal meaning." Though later, they write, "These three junctions [...] are l...
Co-Proofs
by abramdemski — 2018-05-21T21:10:57.290Z

At the recommendation of Jacobian, I've been reading Too Like the Lightening. It is a thoughtful book which has several points of interest to rationalists (imho), but there is one concept which I think is nice enough to pluck out and discuss in itself, rather than being satisfied to suggest that people read the book. I also want to suggest a different name than the one from the book.

If you think discussion of a logical concept which is mentioned in a book is a spoiler, maybe stop here.

At one point, there is a discussion in which one character is explaining how much some other characters must already know. The term "anti-proof" is used to refer to failure to falsify a hypothesis. Having a short term for this concept seems like a really good idea. We have the phrase "absence of evidence is evidence of absence", but we don't have a word for the positive case, where absence of counter-evidence speaks in favor of a hypothesis.

Unfortunately, "anti-proof" sounds more like the former than the latter, even though it is being used for the latter in the book. A more appropriate term would be "co-proof", since it is the absence of a proof of the negation.

For example, an alibi would refute someone's involvement in a crime. The absence of an alibi, then, is a co-proof of their involvement: it does not prove involvement by any means, but it must constitute some supporting evidence, by conservation of expected evidence.

By "proof of H" I mean an observation which would make the probability of H very close to 1. (How close is "very close" depends on standards of proof in a context, with mathematics demanding the highest standards.) By "refutation" I mean a proof of the negation. So, a co-proof is an observation whose negation would have taken the probability of H to very near zero:

E is a co-proof of H := P(H|¬E)≈0

Why are co-proofs of interest? Popperian epistemology is the claim that scientific hypotheses can be supported only by co-proofs; we attempt to refute things, and if something has survived enough refutation attempts, it is considered to be a strong hypothesis. Bayesians are not Popperians, but Popper was still mostly right about this; so, having a short name for it seems useful.
The Alignment Newsletter #7: 05/21/18
by Rohin Shah — 2018-05-21T16:00:45.356Z

Highlights

Challenges to Christiano’s capability amplification proposal

(Eliezer Yudkowsky): A list of challenges faced by iterated distillation and amplification. First, a collection of aligned agents interacting does not necessarily lead to aligned behavior. (Paul's response: That's not the reason for optimism, it's more that there is no optimization pressure to be unaligned.) Second, it's unclear that even with high bandwidth oversight, that a collection of agents could reach arbitrary levels of capability. For example, how could agents with an understanding of arithmetic invent Hessian-free optimization? (Paul's response: This is an empirical disagreement, hopefully it can be resolved with experiments.) Third, while it is true that exact imitation of a human would avoid the issues of RL, it is harder to create exact imitation than to create superintelligence, and as soon as you have any imperfection in your imitation of a human, you very quickly get back the problems of RL. (Paul’s response: He's not aiming for exact imitation, he wants to deal with this problem by having a strong overseer aka informed oversight, and by having techniques that optimize worst-case performance.) Fourth, since Paul wants to use big unaligned neural nets to imitate humans, we have to worry about the possibility of adversarial behavior. He has suggested using large ensembles of agents and detecting and pruning the ones that are adversarial. However, this would require millions of samples per unaligned agent, which is prohibitively expensive. (Paul's response: He's no longer optimistic about ensembles and instead prefers the techniques in

this post

, but he could see ways of reducing the sample complexity further.)

My opinion: Of all of these, I'm most worried about the second and third problems. I definitely have a weak intuition that there are many important tasks that we care about that can't easily be decomposed, but I'm optimistic that we can find out with experiments. For the point about having to train a by-default unaligned neural net to imitate aligned agents, I'm somewhat optimistic about informed oversight with strong interpretability techniques, but I become a lot less optimistic if we think that won't be enough and need to use other techniques like verification, which seem unlikely to scale that far. In ...
Imitative Reinforcement Learning as an AGI Approach
by TIMUR ZEKI VURAL — 2018-05-21T14:47:26.796Z

I've been thinking that reinforcement-learning-driven imitation between agents may be an explanation of human intelligence, and is worth exploring more as an approach to AGI.

It's difficult to get agents to exhibit the complex behaviors humans do with most optimization functions, like "acquire food". But rewarding agents for imitating each other in addition to satisfying basic needs is an efficient method of building up complex survival strategies. Babies, for example, aren't aware how learning to speak will benefit them years later when they start doing it - they just observe adults talking a lot and find that they can get a dopamine hit from talking in response. The explanation for why humans have been so much more successful than chimpanzees is mostly that our dopamine system rewards imitation more strongly, not that we're better at discovering new things directly from the environment. After all, it would seem strange if a little bit of extra optimization of an already highly evolved system in the time between humans and chimpanzees resulted in a gigantic gap in the success of each species. Instead, two existing systems linked together - social behavior and the dopamine system.

Current machine learning focuses mostly on learning directly from an environment, and has found success in domains where humans do the same, like image processing, while lagging severely in areas like natural language, where humans learn imitatively. Deepmind was the pioneer in demonstrating that deep sensory neural nets and dopamine-like reinforcement learning could be combined, in their case to solve Atari games. However, Atari games only have a couple agents, so there was no incentive to use imitation learning.

It'a obvious that computers would be incentivized to learn imitatively if they want to tap into our existing knowledgebase rather than rediscover everything from scratch. But even if they do want to rediscover everything from scratch, or operate in a different environment than the real world, they still may be incentivized to run many agents that learn from each other. The strongest reason is that parallelism is where all the improvements in processors will happen in the future - circuits can't get any denser, but they can in theory get almost arbitrarily cheap. The secondary reason is that in the presence of high context switching costs, many slow agents that coll...
The Third Circle
by Zvi — 2018-05-21T12:10:00.168Z

Previously: The First Circle, The Second Circle

Epistemic Status: Having one’s fill

The third circle took place at Luna Labs. After the second circle, the decision was made to bring in a professional. The New York rationalist group, together with several others in the Luna orbit, gathered at quite the swanky little space to form another group of about twenty. This time, one of the most experienced out there would be leading us. She did not lack for confidence.

As an introduction, the rules are again explained and we went around saying what we were reading lately. Speak your personal truth, no speculation, ask if you’re curious, stay in the moment, everything for connection and all that.

We began with a series of paired exercises. We stare into each others’ eyes. We say things we are feeling or sensing, and what we feel the other person is feeling, and what we feel about that and how accurate it was. We share about what our biggest problems are, and how we feel about that.

It illustrates a different mode of thinking, of what to pay attention to. It was interesting, engaging and quite pleasant.

It also demonstrated how easy it is to fool your brain into thinking you’re making a deep connection with someone, that there’s suddenly definitely a thing there, simply by holding eye contact with someone and paying attention to each other. That doesn’t mean there wasn’t an actual connection with my partner. I think there was, she’s been to our Friday night dinners before, I like her a lot and I hope we get to be good friends. Despite that, it was obvious the circumstances were tricking my brain’s heuristics in ways I had to keep reminding myself to disregard.

Yet another way for saying the unfiltered thoughts on your mind is tricky, also impossible.

What was odd was that these exercises took up an hour and a half, leaving only half an hour for the actual circle. Seemed disproportionate. We’d come for the thing. Was it so far out of our reach we needed this much prep?

Finally, the main event began, under active facilitation. Everyone was tentative. People expressed things, and reactions, and so on, made good faith efforts. Many said nothing, doubtless feeling awkward, also there were twenty to thirty people. Too many.

On some levels, it was clearly working. It felt like insight was generated about people, their perspe...
Visions of Summer Solstice
by Raemon — 2018-05-21T05:50:55.736Z

Previously:

Winter Solstice
The Summer Solstice Paradox
The Steampunk Aesthetic

i. Off the beaten path

See the light as it shines on the sea? It's blinding
But no one knows how deep it goes

– How Far I'll Go, Moana

The experience begins with a journey.

This partly by design, but largely by necessity.

Winter Solstice is about the long arc of history. Summer Solstice is about the here and now, and why being human is something worth protecting. It's about freedom, fun, physicality. It's about figuring things out in Near Mode. It's about building things together.

Getting a particular kind of Special out of that physicality requires a particular kind of space.

It's not enough to find a small nearby park. Ideally, you want an outdoor space vast enough to feel in your bones that the sky is the limit. There is no one and nothing to help you build a tower to the stars, or to cross the ocean, or cartwheel forever in any direction. But neither is there anyone to stop you. There is only nature, and you, and your tribe, and whatever you choose to do.

If you live in a major city, this probably means you may need to undertake a nontrivial journey before finding such a place. The best places will be off the beaten path, and a bit hard to navigate to. If it were easy, humans would have already crowded around it. You might be able to have fun, but you wouldn't be able to carve out a spot for your people to invoke a Sacred Fun.

As you contemplate this from your comfortable couch and think about the journey, you may find it daunting. If you attempted it alone, you might find it frustrating and lonely.

So, don't attempt it alone.

Journey together. If you get lost along the way, getting un-lost is part of the fun. You may find something valuable in overcoming the obstacles. I do, anyway.

The best journey is one that borders on the mythologic – you pass through narrow passages, winding your way through wild undergrowth, slightly confused about where you are going but compelled onwards by curiosity. You cross a threshold into a fae-like enclave that clear communicates "you have left the default world behind."

And then suddenly find yourself at the top (or edge) of the world, slightly unclear how you got there.

Sometimes you are lucky, and such hidden enclaves exist right in your backyard. But the efficient Other World hypothesis says ...