trentmkelly/LessWrong-43k huggingface dataset:
How does The Circle grow, exactly?
by Raemon — 2021-10-01T07:53:16.178Z

Epistemic status: still in the middle of working through a thing, off the cuff thoughts. Wears my half-baked current-generation morality on my sleave.

There's an EA song, adapted for Solstice, which presents an optimistic view of the expanding circle of concern:

CALLER: Raise a song, and so commence
ALL: Circle, grow and grow.
in praise of all Benevolence!
Circle, grow and grow.
Once a cold and silent night
did the loveless stars pervade;
yet we here, of star-stuff made,
cast a circle of warmer Light!

Circle, circle, grow and grow!

...

So will we bring our families in,
Circle, grow and grow.
those whom Nature made our kin?
Circle, grow and grow.

Countless likenesses we find,
by our common blood bestowed.
What a debt of care is owed;
what a blesséd tie that binds!

Circle, circle, grow and grow!

...

And will we bring our neighbors in,
Circle, grow and grow.
our expansion to begin?
Circle, grow and grow.

Bounty of the harvest sun,
shelter from all hazards dire,
share with each, as each require,
doing as you would be done.

Circle, circle, grow and grow!

...

And will we bring the stranger in,
Circle, grow and grow.
every state and speech and skin?
Circle, grow and grow.

Think upon the mystery:
how alike is Humankind!
Tho' manifold in face and mind,
conspecific sisters we!

Circle, circle, grow and grow!

...

And will we bring the far ones in,
Circle, grow and grow.
all who distant-born have been?
Circle, grow and grow.

For the hands you'll never hold,
for the names you'll never learn,
for all far-off hearts that yearn,
let compassion boundless roll!

Circle, circle, grow and grow!

...

And will we bring all creatures in,
Circle, grow and grow.
feather, fur, or silicon?
Circle, grow and grow.

Though their unseen thought beguile —
strange the substrate they employ —
all who suffer or enjoy
are brother soul, in body wild.

Circle, circle, grow and grow!

...

And will we bring the future in?
Circle, grow and grow.
All of time is ours to win!
Circle, grow and grow.

Will our children rise in power?
overwhelm the starry deep?
Lights unborn, for you we keep
will and hope, though dark the hour.

Circle, circle, grow and grow!

This song makes me feel warm and fuzzy. But as a steward of Rationalist Solstice, I also have vague concerns about the degree of philosophical commitment the song brings along with it. With Solstice songs, my epistemic concern is not only "does this song sound true and important by my current understanding of the world?"...
SIA > SSA, part 4: In defense of the presumptuous philosopher
by Joe Carlsmith — 2021-10-01T05:59:51.534Z

(Cross-posted from Hands and Cities. Previously in sequence: Part 1: Learning from the fact that you exist; Part 2: Telekinesis, reference classes, and other scandals; Part 3: An aside on betting in anthropics.)

This post is the last in a four-part sequence, explaining why I think that one prominent approach to anthropic reasoning (the “Self-Indication Assumption” or “SIA”) is better than another (the “Self-Sampling Assumption” or “SSA”). This part discusses prominent objections to SIA. In particular:

I suggest that the Presumptuous Philosopher is a strong candidate for what I call a “good bullet”: that is, a counterintuitive result, acceptance of which resolves a lot of gnarly issues into a simple and pretty satisfying theory, and rejection of which invites endless complication and counterexample (fans of the repugnant conclusion, take note). This doesn’t mean we should actually bite. But we shouldn’t die on the hill of non-biting.
Pascal’s muggings and infinities are problems for SIA, but this puts SIA in pretty respectable company (expected utility theory, population ethics) — and in particular, company that still seems to make itself useful.
SIA has problems with “counting observers.” I haven’t thought that much about this one, but I have some feeling like: don’t we all?
Given some values and decision theories, SIA (like SSA) suffers from inconsistencies between “the policy you’d want to commit to, from some ‘prior’ epistemic perspective” and “how you behave ex post.” But these inconsistencies are common in many other contexts, too; and if you’re worried about them, you can use an “updateless” decision theory. But I suggest not throwing out the concept of epistemology along the way.
We can maybe do a bit to make SIA more intuitive (though not as much as I’d like).

That said, even if SSA is worse than SIA, it’s not like SIA is sitting pretty (I especially don’t like how it breaks in infinite cases, and there are presumably many other objections I’m not considering). I briefly discuss whether we should expect to find a better alternative — the “Anthropic Theory X” above. My current answer is: maybe (and maybe it’s already out there), but Anthropic Theory X should probably keep SIA’s good implications (like “thirding” in Sleeping Beauty). And the good implications seem closely tied to (some of) the bad.

I close by quickly mentioning some of SIA’s p...
SIA > SSA, part 3: An aside on betting in anthropics
by Joe Carlsmith — 2021-10-01T05:52:02.111Z

(Cross-posted from Hands and Cities. Previously in sequence: Part 1: Learning from the fact that you exist; Part 2: Telekinesis, reference classes, and other scandals.)

This post is the third in a four-part sequence, explaining why I think that one prominent approach to anthropic reasoning (the “Self-Indication Assumption” or “SIA”) is better than another (the “Self-Sampling Assumption” or “SSA”). This part briefly discusses betting in anthropics. In particular: why it’s so gnarly, why I’m not focusing on it, and why I don’t think it’s the only desiderata. If you’re not interested in betting-type arguments, feel free to skip to part 4.

XII. An aside on betting in anthropics

I’ve now covered my main objections to SSA. In part 4, I say more in defense of SIA in particular. Before doing so, though, I want to mention a whole category of argument that I’ve generally avoided in this post: that is, arguments about what sorts of anthropic theories will lead to the right patterns of betting behavior.

I expect to some readers, this will seem a glaring omission. What’s the use of talking about credences, if we’re not talking about betting? What are credences, even, if not “the things you bet with”? Indeed, for some people, the question of “which anthropic theory will get me the most utility when applied” will seem the only question worth asking in this context, and they will have developed their own views about anthropics centrally with this consideration in mind. Why, then, aren’t I putting it front and center?

Basically, because questions about betting in anthropics get gnarly really fast. I’m hoping to write about them at some point, but this series of posts in particular is already super long. That said, I also don’t think that questions about betting are the only desiderata. Let me explain.

Why is betting in anthropics gnarly? At a high-level, it’s because how you should bet, in a given case, isn’t just a function of your credences. It’s also a function of things like whether you’re EDT-ish or CDT-ish, your level of altruism towards copies of yourself/other people in your epistemic position, how that altruism expresses itself (average vs. total, bounded vs. unbounded), and the degree to which you go in for various “act as you would’ve pre-committed to acting from some prior epistemic position” type moves (e.g. “updatelessness”) — either at the level of choice...
SIA > SSA, part 2: Telekinesis, reference classes, and other scandals
by Joe Carlsmith — 2021-10-01T05:49:19.901Z

(Cross-posted from Hands and Cities. Previously in sequence: SIA > SSA, part 1: Learning from the fact that you exist.)

This post is the second in a four-part series, explaining why I think that one prominent approach to anthropic reasoning (the “Self-Indication Assumption” or “SIA”) is better than another (the “Self-Sampling Assumption” or “SSA”). This part focuses on objections to SSA. In particular, SSA implies:

scientific presumptuousness comparable to SIA's;
~certainty that fair coins that haven’t yet been flipped will come up heads;
expecting to be able to move boulders with your mind;
a made-up and indeterminate ontology of reference classes that really seems like it shouldn’t be a thing, and which also shouldn’t be contorted willy-nilly to get whatever results you want (and also, this doesn’t work anyway);
sensitivity to differences that don’t matter (like whether a definitely-not-you observer is killed and/or given evidence you don’t get, vs. never created; or whether there are ten vs. twenty chimps in the jungle surrounding your thought experiment);
a “special snowflake” metaphysics on which, because you actually exist, you had to exist in any world still compatible with your evidence (though I think that SSA has some replies here);
intense updates towards solipsism.

IV. The inevitability of presumptuousness

To get an initial flavor of some basic trade-offs between SIA and SSA, let’s look at the basic dialectic surrounding two versions of God’s extreme coin toss:

God’s extreme coin toss with jackets: God flips a fair coin. If heads, he creates one person with a red jacket. If tails, he creates one person with a red jacket, and a million people with blue jackets.

Darkness: God keeps the lights in all the rooms off. You wake up in darkness and can’t see your jacket. What should your credence be on heads?
Light+Red: God keeps the lights in all the rooms on. You wake up and see that you have a red jacket. What should your credence be on heads?

(I’ll assume, for simplicity, that the SSA reference class here is “people,” and excludes God. I talk about fancier reference-class footwork below.)

In Darkness, SIA is extremely confident that the coin landed tails, because waking up at all is a million-to-one update towards tails. SSA, by contrast, is 50-50: you’re the same fraction of the reference class either way. In Light+Red, by contrast, SIA is 50-50: there’s only on...
SIA > SSA, part 1: Learning from the fact that you exist
by Joe Carlsmith — 2021-10-01T05:43:25.247Z

(Cross-posted from Hands and Cities)

This post is the first in a four-part sequence explaining why I think that one prominent approach to anthropic reasoning is better than another. Consider:

God’s extreme coin toss: You wake up alone in a white room. There’s a message written on the wall: “I, God, tossed a fair coin. If it came up heads, I created one person in a room like this. If it came up tails, I created a million people, also in rooms like this.” What should your credence be that the coin landed heads?

The approach I like better — the “Self Indication Assumption” (SIA) — says: ~one in a million. SIA thinks you’re more likely to exist in worlds with more people in your epistemic situation. Here, this is the tails-world by far.

The approach I like worse — the “Self-Sampling Assumption” (SSA) — says: one half. SSA thinks you’re more likely to exist in worlds where the people in your epistemic situation are a larger fraction of the people in your “reference class.” Don’t ask me what a reference class is, but in this case, let’s assume that the people in your epistemic situation are 100% of it either way. So SSA sticks with the one half prior.

I open with this case because it’s one of the worst for SIA, the approach I favor. In particular, we can construct scientific analogs, in which SIA becomes ludicrously confident in a given cosmology, simply in virtue of that cosmology positing more people in our epistemic situation. For many, this implication (known as the “Presumptuous Philosopher”) is a ~decisive objection to SIA.

But I think that the objections to SSA are stronger, and that in the absence of an alternative approach superior to both SSA and SIA (“Anthropic Theory X”), the Presumptuous Philosopher is a bullet we should consider biting.

I proceed as follows. The first part of the sequence (“Learning from the fact that you exist”) describes SIA and SSA. In particular, I emphasize that pace some presentations in the literature, SIA should not be seen as an additional assumption you add to SSA — one that “cancels out” SSA’s bad implications, but accepts SSA’s worldview. Rather, SIA is a different (and more attractive) picture altogether.

The second part (“Telekinesis, reference classes, and other scandals”) lays out the bulk of my case against SSA. In particular, SSA implies:

comparable scientific presumptuousness;
~certainty that fair coi

...
Evaluating terminal values
by kiwi0fruit — 2021-10-01T04:54:55.866Z

How to evaluate terminal values of humans (defined like in here)? Values are subjective but the question asks for some objective perspective. This question is of interest as “Humans' terminal values are often mutually contradictory, inconsistent, and changeable”.

Obviousness of natural selection (NS) can pose some constraints, albeit weak ones, as all known systems with sentient agents abide NS. But weak constraints are still better than no constraints at all.

Terminal goals are being split by natural selection into ones that fail to reproduce / maintain themselves and ones that survive (together with their bearers of cource). And sometimes we can even predict whether some terminal goals would go extinct or at least range their probability of survival (we already had put aside instrumental goals that “die” when they lose their purpose.).

So that's it. That's the only way to objectively judge terminal values I'm aware of. And judgment part comes from a feeling that I don't want to be invested in terminal goals that would most likely go extinct. At least they should be “mutated” in way to balance minimization of their change and maximization of their survival probability to be appealing.

Are you aware of any other ways to evaluate terminal values?
Consider Taking Zinc Every Time You Travel
by Elizabeth — 2021-10-01T03:50:06.702Z

Edit (November 2022): comments raised some probably valid concerns about my sources. I also don't currently believe it's possible to buy the kind of zinc lozenges studied; they're too unpleasant, what you buy in a drug store has been changed for palatability.

Zinc lozenges are pretty well established to prevent or shorten the duration of colds. People are more likely to get colds while travelling, especially if doing so by plane and/or to a destination full of other people who also travelled by plane. I have a vague sense you shouldn’t take zinc 100% of the time, but given the risks it might make sense to take zinc prophylactically while travelling.

https://commons.wikimedia.org/wiki/File:Zinc_pills.jpg

How much does zinc help? A meta-analysis I didn't look at very closely says it shortens colds by 33%, and that’s implied to be for people who waited until they were symptomatic to take it: taken preemptively I’m going to ballpark it at 50% shorter (including some colds never coming into existence at all). This is about 4 days, depending on which study you ask.

[Note: only a few forms of Zinc work for this. You want acetate if possible, gluconate if not, and it needs to be a lozenge, not something you swallow. Zinc works by physically coating your throat to prevent infection, it’s not a nutrient in this case. You need much more than you think to achieve the effect, the brand I use barely fits in my tiny mouth.]

Some risk factors for illness in general are “being around a lot of people”, “poor sleep” and “poor diet”. These factors compound: being around people who have been around a lot of people, or who have poor sleep or diet, is worse than being around a lot of well-rested, well-fed hermits. Travel often involves all of these things, especially by air and especially for large gatherings like conferences and weddings (people driving to camp in the wilderness: you are off the hook).

I struggled to find hard numbers for risk of infection during travel. It’s going to vary a lot by season, and of course covid has confused everything. Hocking and Foster gives a 20% chance of catching a cold after a flight during flu season, which seems high to me, but multiple friends reported a 50% chance of illness after travel, so fill in your own number here. Mine is probably 10%.

If my overall risk of a cold is 10%, and I lower the duration by 50%/4 days, I’ve in expectati...
The LessWrong Team is now Lightcone Infrastructure, come work with us!
by habryka — 2021-10-01T01:20:33.411Z

tl;dr: The LessWrong team is re-organizing as

Lightcone Infrastructure

. LessWrong is one of several projects we are working on to ensure the future of humanity goes well. We are looking to hire software engineers as well as generalist entrepreneurs in Berkeley who are excited to build infrastructure to ensure a good future.

I founded the LessWrong 2.0 team in 2017, with the goal of reviving LessWrong.com and reinvigorating the intellectual culture of the rationality community. I believed the community had great potential for affecting the long term future, but that the failing website was a key bottleneck to community health and growth.

Four years later, the website still seems very important. But when I step back and ask “what are the key bottlenecks for improving the longterm future?”, just ensuring the website is going well no longer seems sufficient.

For the past year, I’ve been re-organizing the LessWrong team into something with a larger scope. As I’ve learned from talking to over a thousand of you over the last 4 years, for most of you the rationality community is much larger than just this website, and your contributions to the future of humanity more frequently than not route through many disparate parts of our sprawling diaspora. Many more of those parts deserve attention and optimization than just LessWrong, and we seem to be the best positioned organization to make sure that happens.

I want to make sure that that whole ecosystem is successfully steering humanity towards safer and better futures, and more and more this has meant working on projects that weren't directly related to LessWrong.com:

A bit over a year ago we started building grant-making software for Jaan Tallinn and the Survival and Flourishing Fund, helping distribute over 30 million dollars to projects that I think have the potential to have a substantial effect on ensuring a flourishing future for humanity.
We helped run dozens of online meetups and events during the pandemic, and hundreds of in-person events for both this year and 2019s ACX Meetups everywhere
We helped build and run the EA Forum and the AI Alignment Forum,
We recently ran a 5-day retreat for 60-70 people whose work we think is highly impactful in reducing the likelihood of humanity's extinction,
We opened an in-person office space in the Bay Area for organizations that are working towards improving the long-term futu

...
Covid 9/30: People Respond to Incentives
by Zvi — 2021-09-30T18:40:03.862Z

Another week, more mandates, more numbers that look mostly how one would expect, more nonsensical policies. Not much unexpected to report this week.

Executive Summary

About what you’d expect. No key takeaways this week.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 750k cases (-7%) and 13,800 deaths (unchanged).

Results: 703k cases (-14%) and 13,818 deaths (unchanged)

Prediction for next week: 630k cases (-10%) and 13,700 deaths (-1%).

We’ve clearly peaked on cases and are rapidly headed downward. It should still take a few more weeks before deaths start their decline, and they’re now reflecting a period where cases didn’t change much, so I expect cases to remain stable for another week or two before they start dropping.

I mentioned it last week and now I found a link to the NPR story I heard on the radio about modelers predicting things to improve steadily throughout the winter, featuring this graphical projection that I hope is essentially correct. I do agree that something like this is a baseline scenario, but it is not the mean result, because it’s highly unlikely we do much better and there are various ways things could go much worse.

Deaths

A worrisome number in the Midwest, which reflects that its cases had a jump four weeks ago. Overall this is exactly what one would have expected, and the numbers are often not smooth across regions.

Cases

This was an even more rapid drop than expected, concentrated in a dramatic drop in the South, but with all regions falling. I don’t expect to keep up that pace, especially since the distributions have changed, but this is an excellent set of numbers.

Vaccinations

It’s interesting to think about the interaction between the vaccine mandates and new vaccinations, with new vaccinations remaining static over time. My read is that this mostly reflects that the mandates have lacked teeth, as discussed below.

Vaccine Effectiveness

We will of course never know for sure, because this information is far too important to know. A recent thread is willing to state the situation bluntly, for the general case not only for the current example. I’m going to quote extensively from it, if that’s not relevant to your interests you can skip ahead.

No one wants to do effectiveness studies. Once you know something is effective, ho...
Some Existing Selection Theorems
by johnswentworth — 2021-09-30T16:13:17.879Z

This post illustrates how various existing

Selection Theorems

and related results fit the general framework - what they tell us about agent type signatures, and how they (usually implicitly) tell us what agent types will be selected. I invite others to leave comments about any other Selection Theorems you know of - for instance,

Radical Probabilism

and

Logical Inductors

are results/frameworks which can be viewed as Selection Theorems but which I haven’t included below.

This post assumes you have read the

intro post on the Selection Theorem program

. The intended audience is people who might work on the Selection Theorem program, so these blurbs are intended to be link-heavy hooks and idea generators rather than self-contained explanations.

The Gooder Regulator Theorem

The

Gooder Regulator Theorem

talks about the optimal design of a “regulator” (i.e. agent) in an environment like this:

When viewed as a Selection Theorem, the outer optimization process selects for high values of Z and low-information models M (i.e. models which don’t take up much space). Assuming that Z is a “sufficiently flexible” function of Y, the theorem says that the optimal “model” M is isomorphic to the Bayesian posterior distribution (s↦P[S=s|X]). In other words, the system’s internal structure includes an explicit Bayesian world model.

Representation: world model represented by probability distribution
Interfaces: “inputs” X induce Bayesian updates of the world model. The distribution is over “system states” S.
Embedding: Agent subsystems are assumed to follow the causal structure in the diagram; the theorem then justifies labelling one of them the “model”.

Coherence Theorems

This cluster of theorems is the most common foundation for agent models today. It includes things like

Dutch Book Theorems

,

Complete Class Theorem

,

Savage’s Theorem

,

Fundamental Theorem of Asset Pricing

, variations of these, and probably others as well. These theorems provide many paths to the same agent type signature: Bayesian expected utility maximization.

Representation: utility function and probability distribution.
Interfaces: both the utility function and distribution take in “bet outcomes”, assumed to be specified as part of the environment. The outputs of the agent are “actions” which maximize expected utility under the distribution; the inputs are “ob

...
How can one train philosophical skill?
by abramdemski — 2021-09-30T14:56:35.313Z

I'm not sure whether this is better-framed as a babble question or a "serious" question, so, treat it as you will.

If you could sit down and train in philosophy the way you train in arts or sports or maths, how would you do it? What's the Deliberate Practice of philosophy look like? What would a philosophy coach do? Would there be philosophy competitions? What could they look like?

Or, if you think philosophy as it is currently practiced is useless/suboptimal, what might you put in its place?