trentmkelly/LessWrong-43k huggingface dataset:
A Qualitative and Intuitive Explanation of Expected Value
by Adam Zerner — 2021-08-10T03:31:13.314Z

Alice: Expected value is one of my favorite things in the world.

Bob: That involves math right?

Alice: Yeah.

Bob: And you use it in your daily life?

Alice: Yeah.

Bob: See I can't live life like that. I hate doing math. I prefer to just go with the flow.

Alice: Well, I take it back. I mean, it does involve math, but a) the math isn't that bad, and b) no one actually does the math for day-to-day stuff.

Bob: Huh?

Alice: I'll show you what I mean. I'm going to try to explain expected value in such a way where it is a) super intuitive, and b) qualitative instead of quantitative.

Bob: Ok.

Alice: There are three steps. 1) Consider both outcomes, 2) think about the probabilities, and 3) think about the payoffs. Let's start with step one.

Bob: Ok.

Alice: This is an extreme example, but consider a hypothetical base jumper named Carol. Base jumping is incredibly risky. People die. But Carol doesn't like to think about that possibility. She doesn't want to "let fear control her life". So she just "focuses on the positive". She thinks about what a life changing experience it will be, but not about the possibility that it will be a life ending experience.

Bob: I see. That's pretty crazy. You can't just focus on the positive. You have to consider the negative as well.

Alice: Yeah, for sure. And just to reiterate, this is exactly they type of response I want to evoke in you. That all of this is really just common sense.

Bob: Ok.

Alice: So that's step number one: don't blatently ignore one of the outcomes. Duh. You have to actually consider both of them. (Well, most of the time there are many potential outcomes, not just two. But we'll assume there are just two here to keep it simple.)

Bob: Makes sense so far.

Alice: Cool. Step number two is to consider probability.

Bob: Uh oh!

Alice: No, don't say that! The math isn't bad, I promise. And it's important that you go into this with an open mindset.

Bob: Ok, fine.

Alice: So imagine this scenario. You're walking down the street and see a food stand selling lentil tacos for a dollar. You've never really been a legume guy, so you think it's pretty unlikely that you'd like these lentil tacos. Let's say there's a 10% chance. And now as a second thing, imagine that it was a slice of pizza instead. Pizza is usually pretty good, so let's say there's a 90% chance you'd like the pizza. It's probably worth buying the slice of pizza, but not the lentil tacos. B...
Outline of Galef's "Scout Mindset"
by Rob Bensinger — 2021-08-10T00:16:59.050Z

Julia Galef's The Scout Mindset is superb.

For effective altruists, I think (based on the topic and execution) it's straightforwardly the #1 book you should use when you want to recruit new people to EA. It doesn't actually talk much about EA, but I think starting people on this book will result in an EA that's thriving more and doing more good five years from now, compared to the future EA that would exist if the top go-to resource were more obvious choices like The Precipice, Doing Good Better, the EA Handbook, etc.

For rationalists, I think the best intro resource is still HPMoR or R:AZ, but I think Scout Mindset is a great supplement to those, and probably a better starting point for people who prefer Julia's writing style over Eliezer's.

I've made an outline of the book below, for my own reference and for others who have read it. If you don't mind spoilers, you can also use this to help decide whether the book's worth reading for you, though my summary skips a lot and doesn't do justice to Julia's arguments.

Introduction

Scout mindset is "the motivation to see things as they are, not as you wish they were".
We aren't perfect scouts, but we can improve. "My approach has three prongs":

Realize that truth isn't in conflict with your other goals. People tend to overestimate how useful self-deception is for things like personal happiness and motivation, starting a company, being an activist, etc.
Learn tools that make it easier to see clearly. Use various kinds of thought experiments and probabilistic reasoning, and rethink how you go about listening to the "other side" of an issue.
Appreciate the emotional rewards of scout mindset. "It's empowering to be able to resist the temptation to self-deceive, and to know that you can face reality even when it's unpleasant. There's an equanimity that results from understanding risk and coming to terms with the odds you're facing. And there's a refreshing lightness in the feeling of being free to explore ideas and follow the evidence wherever it leads". Looking at lots of real-world examples of people who have exemplified scout mindset can make these positives more salient.

PART I: The Case for Scout Mindset

Chapter 1. Two Types of Thinking

"Can I believe it?" vs. "must I believe it?" In directionally motivated reasoning, often shortened to "motivated reasoning", we disproportionately put our effort into finding evide

...
Book Review: The Beginning of Infinity
by Sam Enright — 2021-08-09T21:53:03.962Z

This is a crosspost from my personal website. Inspired by: Naval, If Sapiens Were a Blogpost and Brett Hall’s podcast.

Many people have recommended the book The Beginning of Infinity: Explanations that Transform the World by David Deutsch to me. I don’t know how, because I can’t imagine any of them actually finished it. Previously on my blog I’ve reviewed books and been critical of aspects of them. But this post is more of a summary of The Beginning of Infinity. I decided to write it this way because this book is very complicated, reasonably long and frequently misunderstood. Deutsch is a physicist at Oxford and a pioneer of quantum computing, but his interests are wide-ranging.

All progress comes from good explanations

In this book I argue that all progress, both theoretical and practical, has resulted from a single human activity: the quest for what I call good explanations.

One of the key pieces of terminology in this book is the idea of a good explanation. In Deutsch’s formulation, a good explanation is one that accounts for observations while being hard to vary. If a theory can explain anything, it can explain nothing. Some people think that what makes a good explanation is testability. But this isn’t enough: some theories are perfectly testable but do not constitute good explanations. For example, consider the hypothesis “If you eat 1kg of grass, it will cure the common cold.” The problem with this statement isn’t that it’s not testable, it’s that no one should bother testing it. And the reason why no one should bother testing it is that it’s easy to vary: why 1kg, and not 2kg? What is the explanatory account of how eating grass could cure a cold? Bad explanations have more moving parts than there needs to be, and each of these parts could have been different.

This book has many different threads to it, but one of the most important is a kind of philosophical treatise about how good explanations come to be. One classical idea, which Deutsch rejects, is that we do so by induction, a doctrine known as inductivism. This is based on the idea that ‘the unseen resembles the seen’ or ‘the future resembles the past.’ We observe the sun rising day after day, and inductively reason that the sun will rise tomorrow. There are a few problems with this. One of them is that we do not, in fact, use induction to reason about most observations in the world. Consider someone ...
Eight Hundred Slightly Poisoned Word Games
by Scott Alexander — 2021-08-09T20:17:17.814Z

[cross-posted from my blog Astral Codex Ten]

In 2012, a Berkeley team found that indoor carbon dioxide had dramatic negative effects on cognition (paper, popular article). Subjects in poorly ventilated environments did up to 50% worse on a test of reasoning and decision-making. This is potentially pretty important, because lots of office buildings (and private houses) count as poorly-ventilated environments, so a lot of decision-making might be happening while severely impaired.

Since then people have debated this on and off, with some studies confirming the effect and others failing to find it. I personally am skeptical, partly because the effect is so big I would expect someone to have noticed, but also because submarines, spaceships, etc have orders of magnitude more carbon dioxide than any civilian environment, but people still seem to do pretty hard work in them pretty effectively.

As part of my continuing effort to test this theory in my own life, I played a word game eight hundred times under varying ventilation conditions.

…okay, fine, no, I admit it, I played a word game eight hundred times because I’m addicted to it. But since I was playing the word game eight hundred times anyway, I varied the ventilation conditions to see what would happen.

The game was WordTwist, which you can find here (warning: potentially addictive). You get a 5x5 square of letters and you have to find as many words as possible (of four letters or more) within three minutes. You can move up, down, right, left, or diagonal, and get more points for harder words. A typical board looks like this:

Did you spot “lace”? What about “intrapsychically”?

I played this game about 5-10x/day over three months. During this time, the carbon dioxide monitor in my room recorded levels between 445 ppm (with all windows open and the fan on) and 3208 ppm (with all windows closed and several people crammed into the room for several hours). I discounted a stray reading of 285 as an outlier, since this is climatologically impossible (I’m not claiming my monitor is perfectly calibrated, just that it clearly shows higher levels when my room is less well ventilated). CO2 445 is basically the same as outdoors; 3208 is considered extremely poor air quality, likely to cause headaches, nausea, and other minor ailments. The Berkeley study looked at levels between 600 and 2500, so my range was comparable to theirs....
Fermi Fingers
by Optimization Process — 2021-08-09T19:56:42.544Z

When doing Fermi estimates in my head, I sometimes lose track of powers of ten: ...that's three billion widgets, divided by fifty widgets per person is somewhat under a hundred million people, times $20 per person is somewhat under $2 billion... or, wait, did I drop a zero, should that have been somewhat under a billion people? Did I have three billion or thirty billion widgets? Shoot.

I find that it helps a lot to count on my fingers like a common third-grader! It's a lot harder to misplace fingers than to misplace the idea of a decimal point (probably because fingers... exist). Specifically, my calculations tend to go much more smoothly if I represent the log10 of the accumulator on my fingers:

(I use "palm up/down" to represent the sign of the log, since it often flip-flops during the calculation.)

A couple of worked examples:

Is it worth joining this social movement? How many hours does it take to save a QALY?

Well, if there are 1M people...
(1e6 people)

and each person spends 10hr/wk...
(1e7 hr/wk)

for 2 years = 100 wk...
(1e9 hr)

and thereby we make 100M people's lives...
(1e1 hr/life)

1% better...
(1e3 hr/QAL)

and a life is 100 years...
(1e1 hr/QALY)

then that averages to 10hr/QALY.

How many potential rationalists are there in Seattle?

Well, if 1/1000 people are receptive to the Message...
(1e-3/person)

and there are 2M people in King County...
(1e3.3 people)

and half of those are in Seattle...
(1e3 people)

then there are 1000 potential rationalists.

This will break down once in a while, when intermediate values wander outside the range of 10±10 (or ±35 if you're weird), but it's served me quite well for a couple years.
Implicature Conflation
by abramdemski — 2021-08-09T19:48:51.097Z

For some reason, I keep thinking about what General Semantics might be trying to get at. In my previous post on the subject, I identified equivocation as the main curse which General Semantics seeks to provide a counterspell for. (AKA conflation, identification, fusion, ...) However, equivocation has a lot of faces. It doesn't work to just say "try really hard not to equivocate" -- we need the details. We need tools to notice specific classes of equivocation and un-equivocate them.

Today, I want to talk about equivocation which conversational implicature brings about.

Conversational Implicature

In case you're not familiar with the term, "conversational implicature" refers to all the meaning that's not explicit. For example, if I say "you didn't do the laundry yet?" I might (depending on context) be implying "you should really do it now". This is implicature, because I didn't say it explicitly; it is inferred from context.

I've written before about Bayesianism and the explicit/implied distinction. I wrote that Bayesian signalling theory isn't adequate to understand the distinction: meaning is meaning, and in particular, the meaning of an utterance is what people understand from it. How could we make a Bayesian distinction between "explicit" and "implied" meaning?

(To put it a different way: from a map/territory standpoint, someone talking about "explicit" meaning might sound confused: the meaning of the map is precisely the way it corresponds to (IE statistically conveys information about) the territory. It's not like you actually hand over chunks of the territory! So what is this "explicit meaning"?)

I nonetheless do think there is such a thing as "explicit meaning". It may have gradations and nuances (many "explicit" phrasings have their roots in analogy, eg, "she's on time" doesn't make sense if you try to be very literal about what "on" means), but by and large, we can make a reasonable dividing line.

Policy Debates

In Policy Debates Should Not Appear One-Sided, Eliezer identifies a common failure mode, where listing cons is conflated with arguing against all the pros, (and listing pros is conflated with arguing against all the cons):

Robin Hanson proposed stores where banned products could be sold.1 There are a number of excellent arguments for such a policy—an inherent right of individual liberty, the career incentive of bureaucrats to prohibit everything, legis

...
Calculated Clairvoyance
by Space L Clottey — 2021-08-09T19:46:32.031Z

(Crosspost: spacelutt.com/cal)

In Steven Universe, there is a character called Sapphire who has the ability to see into the future. For the entirety of the show, it is presented to us as an innate ability which Sapphire can do naturally.

But in one of the final episodes of the show, Sapphire, while supposedly teaching a class on predictions, is seen to be explicitly discussing multiplications of probability to arrive at estimates of the future.

This reflects a larger problem I see with advice: people giving reasons and philosophies for things that can make you happy that are entirely separate from the reasons and philosophies they used themselves to become happy.

The post Life can be better than you think is a great example of this, where the author was severely depressed until they began taking anti-depressants, but a large part of the post is dedicated to explaining his life philosophy of happiness.

But which came first: the life philosophy, or the happiness?

Battling an energy disorder, my own life provides frequent examples of this: whenever I am tired, my thoughts tend towards pessimism and misery, but whenever I have energy they tend towards optimism and joy. It becomes second nature to list out long stretches of philosophical reasoning that justify being happy, but in truth I know the reasoning isn't the reason why I feel happy. I feel happy because I have energy.

Any other reason is a type of fake justification; it may be a way to become happy, but is certainty not what caused me be to be happy in the first place.

Sure, maybe it's possible to become happy just from integrating these long chains of reasoning. This may have been what Naval Ravikant did (or, probably more likely, his low-level interventions — like sleep, working out, and improving relationships — led to better high-level realisations.)

Maybe it's useless and unhelpful to encourage scepticism on people's advice on being happy. Ideas should be discussed on their own first, after all, before you make a single statement about the states of mind that brought about the idea.

But I think, if you're really trying to be happy, what caused the state of mind to bring about any optimistic philosophy is the most important factor, and the thing to try first. Be it better sleep, antidepressants if you need them, better friends, or more meaningful work.

Whatever brings about the states of mind that bring about th...
Combining two posts: Blankface (Scott Aa) and Is Science Slowing Down? (Scott Al)
by William Gasarch — 2021-08-09T17:37:44.713Z

(First time poster, hope this works. Will also be posting to complexityblog as well.)

Some papers result from taking two papers and combining them. Perhaps nobody else had read both of them so you can say something new! Or (looking over this post) it may guide people to two really good papers, or in this case two really good posts.

This blog will draw from two excellent blog posts.

Scott Aaronson blogged on his website on Aug 2, 2021 about blankfaces, people who let stupid or undefined rules dictate what you can do without apology (see his post for a better explanation). One example that struck me I quote

No, I never applied for that grant. I spend two hours struggling to log in to a web portal designed by the world's top blankfaces until I finally gave up in despair.

Scott Alexander blogged on Less Wrong on Nov 26, 2018 about Is science slowing down? which answers with an emphatic yes. His point is science-per-researcher is much less than it used to be, and he has graphs and stats to prove it (see his post for the evidence and some speculation as to why this is) One of the reasons he gave struck me which I quote

Certain features of the modern academic system like undepaid PhD's, interminably long postdocs, endless grant writing drudgery, and clueless funders have lowered productivity. The 1930's academic system was ineed 25x more effective at getting researchers to actually do good research.

Scott Alexander later downplays this reason, though I think there is still something to it.

(I note that he gives other reasons as well, most notably for our field that the low hanging fruit is gone. Our lack of progress on P vs NP is likely that its a hard problem, rather than the reason above. Of course, if its solved tomorrow by an outsider without funding, I will happily be proven wrong.)

Scott Alexander hits upon two types of blankfaces (without using the term).

Grant writing drudgery: the rules for how to submit get more and more detailed an onerous. This is what Scott Aaronson was alluding to. There are other ways its drudgery as well.

Clueless Funders: the people deciding who gets funded might not know the area (actually in my experience the grant I've reviews have been quite good and the problem is more not enough money to award all that are deserving.)

SO I pose the following non-r...
Framing Practicum: Stable Equilibrium
by johnswentworth — 2021-08-09T17:28:48.338Z

This is a

framing practicum

post. We’ll talk about what a stable equilibrium is, how to recognize stable equilibria in the wild, and what questions to ask when you find one. Then, we’ll have a challenge to apply the idea.

Today’s challenge: come up with 3 examples of stable equilibrium which do not resemble any you’ve seen before. They don’t need to be good, they don’t need to be useful, they just need to be novel (to you).

Expected time: ~15-30 minutes at most, including the Bonus Exercise.

What’s a Stable Equilibrium?

Put a marble at the bottom of a round bowl, and it will just sit there without moving. Put it in the bowl but not quite at the bottom, and it will roll around a bit, but eventually settle at the bottom, and sit there without moving. Give it a poke, and it will roll around some more, but eventually it will again sit at the bottom without moving.

This is stable equilibrium: the system may start in different states, or it may be “perturbed” into different states by some external force, but eventually it settles back to the same state (assuming it isn’t pushed too far away…).

A marble in a bowl will eventually sit stationary at the bottom of the bowl, and stay there.

What To Look For

Stable equilibrium should spring to mind whenever a system tends to return to the same state. If you could “poke” it somehow, and the system would go back to normal eventually, that’s probably a stable equilibrium. If a system tends to stay suspiciously the same over the long run, despite lots of short-run noise, that’s probably a stable equilibrium.

Useful Questions To Ask

The marble always returns to the bottom of the bowl. If we push the marble away from the bottom, that’s only a short-term change - it will roll back down eventually. So, if we’re mainly interested in the long run behavior of the marble, then we can ignore such little pushes.

On the other hand, there may also be ways to change the equilibrium state itself. For instance, if we tip the bowl to the side slightly, then the equilibrium position of the marble will change. If we deform the bowl, that could change equilibrium position. If we charge the marble with a little static electricity, then place another charged object near the bowl, that could also change the equilibrium. Finally, very large changes to the system state could push it out of the bowl entirely.

Tipping the bowl changes the equilibrium.

When ...
When Most VNM-Coherent Preference Orderings Have Convergent Instrumental Incentives
by TurnTrout — 2021-08-09T17:22:24.056Z

This post explains a formal link between "what kinds of instrumental convergence exists?" and "what does VNM-coherence tell us about goal-directedness?". It turns out that VNM coherent preference orderings have the same statistical incentives as utility functions; most such orderings will incentivize power-seeking in the settings covered by the power-seeking theorems.

In certain contexts, coherence theorems can have non-trivial implications, in that they provide Bayesian evidence about what the coherent agent will probably do. In the situations where the power-seeking theorems apply, coherent preferences do suggest some degree of goal-directedness. Somewhat more precisely, VNM-coherence is Bayesian evidence that the agent prefers to stay alive, keep its options open, etc.

However, VNM-coherence over action-observation histories tells you nothing about what behavior to expect from the coherent agent, because there is no instrumental convergence for generic utility functions over action-observation histories!

Intuition

The result follows because the VNM utility theorem lets you consider VNM-coherent preference orderings to be isomorphic to their induced utility functions (with equivalence up to positive affine transformation), and so these preference orderings will have the same generic incentives as the utility functions themselves.

Formalism

Let o1,...,on be outcomes, in a sense which depends on the context; outcomes could be world-states, universe-histories, or one of several fruits. Outcome lotteries are probability distributions over outcomes, and can be represented as elements of the n-dimensional probability simplex (ie as element-wise non-negative unit vectors).

A preference ordering ≺ is a binary relation on lotteries; it need not be eg complete (defined for all pairs of lotteries). VNM-coherent preference orderings are those which obey the VNM axioms. By the VNM utility theorem, coherent preference orderings induce consistent utility functions over outcomes, and consistent utility functions conversely imply a coherent preference ordering.

Definition 1: Permuted preference ordering. Let ϕ∈Sn be an outcome permutation, and let ≺ be a preference ordering. ≺ϕ is the preference ordering such that for any lotteries L,M: L≺ϕM if and only if ϕ(L)≺ϕ(M).

EDIT: Thanks to Edouard...