trentmkelly/LessWrong-43k huggingface dataset:
How counterfactual are logical counterfactuals?
by Donald Hobson — 2024-12-15T21:16:40.515Z

Logical counterfactuals are when you say something like "Suppose π=3 , what would that imply?"

They play an important role in logical decision theory.

Suppose you take a false proposition P and then take a logical counterfactual in which P is true. I am imagining this counterfactual as a function f(logical propositions)→[0,1]⊂R that sends counterfactually true statements to 1 and false statements to 0.

Suppose P is " not Fermat's last theorem". In the counterfactual where Fermat's last theorem is false, I would still expect 2+2=4. Perhaps not with measure 1, but close. So f("2+2=4")>0.9

On the other hand, I would expect trivial rephrasing of Fermat's last theorem to be false, or at least mostly false.

But does this counterfactual produce a specific counter example? Does it think that 143+263=293? Or does it do something where the counterfactual insists a counter-example exists, but spreads probability over many possible counter-examples. Or does it act as if there is a non-standard number counterexample?

How would I compute the value of f in general?

Suppose you are a LDT agent trying to work out whether to cooperate or defect in a prisoners dilemma.

What does the defect counterfactual look like? Is it basically the same as reality except you in particular defect. (So exact clones of you defect, and any agent that knows your exact source-code and is running detailed simulations will defect.)

Or is it broader than that, is this a counterfactual world in which all LDT agents defect in prisoners dilemma situations in general. Is this a counterfactual world in which a bunch of homo-erectus defected on each other, and then all went extinct, leaving a world without humans?

All of the thought about logical counterfactuals I have seen so far is on toy problems that divide the world into Exact-simulations-of-you and Totally-different-from-you.

I can't see any clear idea about what to do with the vaguely similar but not identical to you agents.
Introducing Avatarism: A Rational Framework for Building actual Heaven
by ratiba ro — 2024-12-15T17:17:45.440Z

People of LessWrong share a common thread: a drive to understand reality as it is, coupled with a willingness to tackle ambitious projects that push beyond current human limitations. Today, I’d like to introduce a nascent idea—“Avatarism”—that attempts to meld rationality, scientific rigor, and technological ambition into a guiding vision.
The short version: Avatarism proposes that instead of relying on ancient metaphysical narratives to provide meaning or solace from death and injustice, we should consciously work to create a future state (a “True Heaven”) in our own actual universe—a realm of well-being and freedom from suffering, and even more outrageously resurrect past individuals into this improved state of existence. The claim is not that we know how to achieve this right now, but that we can at least start charting a course, applying the same tools of rationality, epistemic humility, and technological foresight that the LessWrong community already values.

A Brief Sketch of Avatarism

Acknowledging the Void: Traditional religions often fill existential dread with unfalsifiable promises. Avatarism starts by acknowledging that no external authority is guaranteed to deliver utopia or salvation from death. We have no inherent “purpose” gifted to us; we have to craft it. Treat the most ambitious goal we can think of as a generational engineering challenge.
Goal-Setting Under Uncertainty: The aim is to create a drastically better future environment, effectively a heaven on naturalistic terms. We don’t know the exact methods: maybe advanced neural engineering, virtual reality civilizations, or large-scale energy harnessing. The point is to keep the goal open-ended, allowing for rigorous updates as knowledge grows.
Rational Approach to Grand Goals: Avatarism does not claim certainty. It does not hinge on miracles or suspend natural laws. Instead, it approaches the question “Can we create the future we want?” with a physics, engineering mindset. We treat it like a frontier research question. Recreating individuals from the past into an artificial heaven, extremely hard, perhaps improbable, but not obviously impossible given millions of years of scientific advancement, a blink of an eye in the mind of a dead human.
Incremental and Iterative: Avatarism encourages continuous refinement. If tomorrow’s physics breakthroughs challenge a current approach, we pivot. If

...
A Public Choice Take on Effective Altruism
by vaishnav92 — 2024-12-15T16:58:50.683Z
Link: https://www.optimaloutliers.com/p/effective-altruism-neglectedness

Effective Altruism presents itself as an objective arbiter of where marginal resources can do the most good. Its framework emphasizes cause-neutrality and cost-effectiveness, aiming to direct support to whatever opportunities are most impactful at the current margin. Yet EA is also a movement with its own institutions, career paths, and organizational interests. This dual identity—as both impartial judge and institutional player—creates a fundamental tension that grows more acute as the movement expands.

This is best observed by zooming in on neglectedness. When EA is small, thinking on the margin poses no problem—individual EAs can freely direct their resources to whatever cause seems most compelling at the current margin. But as EA grows, any cause it champions becomes significantly less neglected. In theory, this is not a problem but a sign of success—solving problems quickly and effectively frees up bandwidth to look for the next problem to solve.. But as the movement grows larger, it will likely stand in the way of its own stated objectives.

Consider the fact that EA jobs are currently significantly oversubscribed.. Operations roles at EA organizations that pay well under $100,000 receive several hundreds, if not thousands of applications. EA orgs can afford long drawn-out evaluation processes spanning 3-6 months. Some in EA defend this oversubscription with arguments about power laws - claiming these roles are so much higher impact that the marginal value of an additional applicant doesn't diminish even with thousands of applicants.

But this seems implausible for most roles with bounded autonomy, even in exceptionally impactful organizations. The exceptions are high leverage roles like leading organizations or specialized technical positions. For a marketing manager or operations coordinator, it's hard to make the case that from a pool of 2000 qualified applicants, the delta between the best and second-best candidate justifies this insistence on working for an EA organization.

More importantly, this rationalization illustrates precisely how institutional forces can warp even well-intentioned movements. Even if we grant the power law argument, we should be wary of how conveniently it justifies the existing institutional structure. It's a perfect example of how EA organizations, despite their commitment to cause neutrality, can develop self-perpetuating logi...
Dress Up For Secular Solstice
by Gordon H.S. — 2024-12-15T16:28:24.607Z

I argue below that 1) fashion norms are good, even purely within the rationalist community and 2) Secular Solstice should have an unofficial attire norm: a slightly dressy black base layer with shiny accents (shiny jewelry, reflective material, metallic colors, etc; see examples in the main text).

Others have written general defenses of “dressing up” in day-to-day life. These defenses typically make the very safe argument that “how you dress conveys information about you to other people, and you should take that into account”. Fundamentally, these are arguments for rationalists to dress based on the perceptions of those outside of the rationalist community. I think it is far more interesting (and contentious) to consider how the rationalist community should approach dress internally. In settings where rationalists set the norms, what should those norms be?

One tempting response is that rationalists should not care about fashion. Why devote mental energy to your own presentation when it could be better spent on other things? And, with that in mind, why should you evaluate other’s presentations of themselves when you could get to know them or their ideas, and evaluate them on that basis alone? Perhaps the Bayesian framing of this is that clothing is either totally uninformative (at least conditioning on rationalism) or a signal of information you would rather not use (maybe that’s wealth, maybe that’s country of origin, whatever).[1]

I think these positions are hard to defend. Absent any internal rationalist coordination, clothing is already informative. Many people wear shirts that explicitly tell you things about them, like where they went to college, where they work, what bands they like, what they think is funny, etc. And maybe you don’t want to use all of the available information, but why wouldn’t you want to use this information when deciding who to talk to, who to avoid, who to crack jokes with, who to be professional around, etc? So, I think even in rationalist spaces clothing contains information that is worth caring about.

But there is additional power in coordination within a group. Many a college dance has created meaning from nothing by adopting a “stoplight” convention: wearing red means you’re taken, green means you’re single, and yellow means you want to advertise your drama. Once a dance has a “stoplight” theme there is no avoid...
Remap your caps lock key
by bilalchughtai — 2024-12-15T14:03:33.623Z

When was the last time you (intentionally) used your caps lock key?

No, seriously.

Here is a typical US-layout qwerty (mac) keyboard. Notice:

Caps lock is conveniently located only one key away from A, which is where your left pinky should rest on the home row by default.
Caps lock is absolutely massive.
How far various other keys you might want use often are from the home row.

Remap your caps lock key.

I have mine mapped to escape.

Modifier keys such as control or command are also good options (you could then map control/command to escape).

How do I do this you ask?

On Mac, system settings > keyboard > keyboard shortcuts > modifier keys.
On Windows, Microsoft PowerToys' Keyboard Manager is one solution.
If you use Linux, I trust you can manage on your own.

Thanks to Rudolf for introducing me to this idea.
Effective Evil's AI Misalignment Plan
by lsusr — 2024-12-15T07:39:34.046Z

Doctor Susan Connor loved working for Effective Evil. Her job provided autonomy, mastery and purpose. There were always new mysteries to uncover.

Some of Effective Evil's activities, like closed borders and artificial pandemics, were carried out with the sanction and funding of major world governments. Others, like assassination markets, had to be carried out in secret. This created liquidity problems for the prediction markets, but that's another story.

As Doctor Connor rose through the ranks, many doors were opened to her. She learned how to spike a well in Sierra Leone with cholera, and how to push a highway expansion through a city planning meeting. But it wasn't what was said that caught her inquiry. Rather, what was unsaid. The deeper she got the less she heard mention of AI, even as AI expanded its tendrils into every other aspect of society.

Eventually Doctor Connor couldn't stand the charade anymore. She burst into Division Chief Douglas Morbus's office.

"I'm confused," said Doctor Susan Connor.

"About what?" Division Chief Douglas Morbus.

"What is Effective Evil's AI alignment policy?" asked Doctor Connor.

"I've told you before, we must solve the alignment problem. Otherwise a rogue Superintelligence will turn the universe into paperclips when it could instead turn the universe into something much worse," said Morbus. He returned to his paperwork.

"Yes, that's our official story. But we don't have any of our own scientists working on this problem. Instead, we're…donating money to notkilleveryoneist organizations? Did I read that right?" said Doctor Connor.

"There are many fates worse than death," said Morbus.

"That's not my point," said Doctor Connor, "If we were actually trying to solve the alignment problem then we'd have in-house alignment engineers attempting to build an evil ASI. But that's not what's happening. Instead, all of our funds go to external think tanks advocating for safe AI. And we're not even funding engineering teams. We're funding notkilleveryoneist political advocacy groups. It makes no sense. It's neither evil nor effective."

Division Chief Morbus didn't even bother looking up. "I am aware of our initiatives in this domain."

"Then why don't you put a stop to it?" asked Doctor Connor.

"Because everything is going according to plan," said Morbus.

"What plan?" asked Doctor Connor.

Morbus rolled his eyes.

Doctor Connor noticed that she had left ...
Write Good Enough Code, Quickly
by Oliver Daniels — 2024-12-15T04:45:56.797Z

At the start of my Ph.D. 6 months ago, I was generally wedded to writing "good code". The kind of "good code" you learn in school and standard software engineering these days: object oriented, DRY, extensible, well-commented, and unit tested. I like writing "good code" - in undergrad I spent hours on relatively trivial assignments, continually refactoring to construct clean and intuitive abstractions. Part of programming's appeal is this kind of aesthetic sensibility - there's a deep pleasure in constructing a pseudo-platonic system of types, objects, and functions that all fit together to serve some end. More freedom than math, but more pragmatism than philosophy. This appreciation for the "art" of programming can align with more practical ends - beauty often coincides with utility. Expert programmers are attracted to Stripe because of their culture of craftsmanship, and Stripe promotes a culture of craftsmanship (in part, presumably) because "building multi-decadal abstractions" (in the words of Patrick Collison) is useful for the bottom line.

And this is all well and good, if (as is more often the case than not) you expect your code to be used in 1, 2, 10 years. But in research, this is often not the case! Projects typically last on the order of weeks to months, not years to decades. Moreover, projects typically involve a small number (often 1 or 0) of highly involved collaborators, as opposed to the large, fragmented teams typical of industry. Moreover, speed is paramount. Research is a series of bets, and you want to discover the outcome of the bet as fast as possible. Messy code might incur technical debt, but you don't have to pay if you scrap the entire project.

I had heard advice like this going into my PhD, both in the context of research and product development generally (MVP, Ballmer Peak, etc). It took me a while to internalize it though, in part, I suspect, because there's an art to writing "messy" code too. Writing error-prone spaghetti code is not the answer - you need stuff to work quickly to get results quickly. The goal is to write good enough code, efficiently, but learning what good enough means is a skill unto itself.

Principles for Good-Enough Code

Below is a first pass at some guiding principles. I focused on ML research in Python, but I suspect the lessons are generalizable

Future-t you is the target user

where t i

...
How to Edit an Essay into a Solstice Speech?
by Czynski — 2024-12-15T04:30:50.545Z
Link: https://thepdv.wordpress.com/2024/12/14/how-to-edit-an-essay-into-a-solstice-speech/

By request, my notes on how I adapted I Have Seen the Tops of Clouds into a speech for Solstice and best-effort advice on how to do the same yourself.
How Your Physiology Affects the Mind's Projection Fallacy
by YanLyutnev — 2024-12-14T21:10:23.240Z

Key Definitions from This Article:

Mind's Projection Fallacy

(Edward T. Jaynes)

(A) (My imagination) → (The actual property of nature)

(B) (My ignorance) → (Nature is undefined)

Sense of Pleasure

(Yan Lyutnev)

A first-person, trackable sharp increase in pleasant sensation (a hormone release)

***

On this particular day, I hadn’t slept well and was experiencing an unpleasant, suboptimal sensation in my head due to sleep deprivation. When I’ve had enough sleep, this feeling is absent. I was standing at the self-checkout in the supermarket and thought: "This checkout machine is totally non-pleasurable, but surely someone put a lot of effort into making it, and someone must find it pleasurable. When I compare it to the fact that someone finds it pleasurable, it gives a kind of surprise, a dissonance, and what’s called an 'insight.' Right now, I’m blocking myself from calling objects 'non-pleasurable,' but the state of sleep deprivation made me more stressed about controlling my phrasing, and old cognitive algorithms resurfaced.

My current explanation of the mechanism behind calling the machine 'non-pleasurable' is: from the inside, it seems like the machine has a certain property that can be verbalized as 'non-pleasurable.' It seems constant, but if that’s the case, it also seems like others will notice this property too. After all, when you check if it’s true or not, you look at your own feelings—and I was sleep-deprived, and my limbic system wasn’t giving any anticipation for interacting with the checkout machine.

If the checkout machine is a barrier between you and the end of the unpleasant payment process (which, on another day, might be pleasant), the categorization 'machine' can get associated with the categorization 'non-pleasurable.' Categorizing things has consequences, in that the brain saves and compresses all internal categories into something that seems constant and similar.

This apparent property of the machine, 'non-pleasurable,' can be remembered, and people do this for many things. Physiological memories of such properties are efficient for many tasks: if, instead of the machine, there was an instrument or strategy that was suboptimal in terms of 'limbic system comfort for future interaction,' and the brain later sought to avoid low stimulation (with hormone influxes that feel pleasant), the brain would recall ...
Introducing the Evidence Color Wheel
by Larry Lee — 2024-12-14T16:08:38.744Z

Version 5.1.0
December 12, 2024

Introduction

Weighing evidence is hard. Intuition is liable to a myriad of biases and relies on flawed heuristics. Bayes theory provides a rigorous mathematical basis, but applying Bayes theory in practice is too difficult for most people. Yet, what is more central to thinking critically than being able to weigh evidence correctly?

The Evidence Color Wheel is a tool for weighing evidence. When presented with evidence for a claim, simply move your finger from one color region to the next to determine how likely the claim is given the evidence. The probability that a claim is true is represented by a segment with three colors. The outer color gives the maximum possible probability that the claim is true. The central color gives the probability that the claim is true when the colors values equal those in the legend. The inner color gives the smallest possible probability that the claim is true. These bounds are necessary because each color represents a range of possible values.

A poster describing the Evidence Color Wheel. Feel free to download an print out a copy. The Color Wheel is under an Attribution Creative Commons license!

Using the Evidence Color Wheel

To use the Evidence color wheel you need to have a claim. A claim is some statement that may or may not be true. For instance, imagine that you are a doctor diagnosing a patient who may or may not have a rare disease. A claim may be that the patient has the given disease.

Next, you must have some new piece of evidence. For example, you may send the patient in for a diagnostic test and get back a positive test result. Before you see this new evidence, you must have some estimate of how likely the claim was to be true. To continue our running example, you may know that the disease is rare. The color wheel tells you how likely the claim is to be true given the new evidence.

Prior Probability

Start by placing your finger in a colored segment in the outermost ring. If the claim was very unlikely to be true (<= 5% probability) before you saw the new evidence, place your finger in the red segment. If the claim was unlikely to be true (> 5% and < 20% probability) place your finger in the yellow segment. The following table shows how each color segment maps onto probability ranges:

Color	Probability	Description
Red	≤5%	Very Unlikely
Orange	>5% −≤20%	Unlikely
Yellow	>20% −&

...
An Illustrated Summary of "Robust Agents Learn Causal World Model"
by Dalcy — 2024-12-14T15:02:44.828Z

This post was written during Alex Altair's agent foundations fellowship program, funded by LTFF. Thank you Alex Altair, Alfred Harwood, Daniel C for feedback and comments.

Introduction

The selection theorems agenda aims to prove statements of the following form: "agents selected under criteria X has property Y," where Y are things such as world models, general purpose search, modularity, etc. We're going to focus on world models.

But what is the intuition that makes us expect to be able to prove such things in the first place? Why expect world models?

Because: assuming the world is a Causal Bayesian Network with the agent's actions corresponding to the D (decision) node, if its actions can robustly control the U (utility) node despite various "perturbations" in the world, then intuitively it must have learned the causal structure of how U's parents influence U in order to take them into account in its actions.

And the same for the causal structure of how U's parents' parents influence U's parents ... and by induction, it must have further learned the causal structure of the entire world upstream of the utility variable.

This is the intuitive argument that the paper Robust Agents Learn Causal World Model by Jonathan Richens and Tom Everitt formalizes.

Informally, its main theorem can be translated as: if an agent responds to various environment interventions by prescribing policies that overall yield low regret, then it's possible to appropriately query the agent to reconstruct an implicit world model that matches up with the ground truth causal structure.

I will refer to this result as the "Causal Good Regulator Theorem". This sequence Thoughts on the Causal Good Regulator Theorem will do the following:

[1] a precise statement of the theorem alongside a high-level argument of its proof and discussions of the paper's results in the context of the selection theorem agenda - basically a self-complete distillation of the paper. The rest of the posts in this sequence are optional. (this post)
[2] detailed proof of the paper's theorems, presented in a way that focuses on how one could've discovered the paper's results by themselves, with an emphasis on intuition.
[3] extension of the paper's theorem by showing how one of the main assumptions can be relaxed without much trouble.

Basic Setup

World

The world is ...