trentmkelly/LessWrong-43k huggingface dataset:
Boltzmann brain decision theory
by Stuart_Armstrong — 2018-09-11T13:24:30.016Z

Suppose I told you that you had many Boltzmann brain copies of you. Is it then your duty to be as happy as possible, so that these copies were also happy?

(Now some people might argue that you can't really make yourself happy through mental effort; but you can certainly make yourself sad, so avoiding doing that also counts.)

So, I told you that some proportion of your Boltzmann brain copies were happy and some were sad, it seems that the best thing you could do, to increase the proportion of happy ones, is to be happy all the time - after all, who knows when in your life a Boltzmann brain might "happen"?

But that reasoning is wrong, a standard error of evidential decision theories. Being happy doesn't make your Boltzmann brain copies happy; instead, it ensures that among all the existing Boltzmann brains, only the happy ones may be copies of you.

This is similar to the XOR blackmail problem in functional decision theory. If you pay Omega when they send the blackmail letter ("You currently have a termite problem in your house iff you won't sent me £1000"), you're not protecting your house; instead, you're determining whether you live in a world where Omega will send the letter.

On the other hand, if there were long-lived identical copies of you scattered around the place, and you cared about their happiness in a total utilitarian style way, then it seems there is a strong argument you should make yourself happy. So, somewhere between instantaneous Boltzmann brains and perfect copies, your decision process changes. What forms the boundaries between the categories?

Duration and causality

If a Boltzmann brain only has time to make a single decision, then immediately vanishes, then that decision is irrelevant. So we have to have long-lived Boltzmann brains, where long-lived means a second or more.

Similarly, the decision has to be causally connected to the Boltzmann brain's subsequent experience. It makes no sense if you decide to be happy, and then your brain gets immediately flooded with pain immediately after - or the converse. Your decision only matters if your view of causality is somewhat accurate. Therefore you require long-lived Boltzmann brains who respect causality.

In a previous post, I showed that the evidence seems to suggest that Boltzmann brains caused by quantum fluctuations are generally very short-lived (...
Track-Back Meditation
by abramdemski — 2018-09-11T10:31:53.354Z

Intro -- Looking Back on Looking Back

At a CFAR event, Anna once (iirc) put forward the following idea: when you notice that you are distracted, try to think back to when you were last paying attention, recalling as much as you can of the sequence of where your attention went as it wandered. At first you might only be able to think back a couple of steps: "I was thinking about trains just now; why was that? Ah, it was because I was thinking about how I have to catch the train later, and started thinking about trains in general. I don't remember what got me on that topic." However, if you keep trying the exercise, you may eventually be able to recall all the way back to what got you off-track in the first place. Once you have a habit of recalling what got you distracted, you can start to see patterns. "Ah, I was hungry, which made me start thinking about food. Being hungry seems to distract me a lot." Then you can keep peanuts at your desk, or whatever.

Another story:

At one point a couple of years ago, I noticed that I was using a particular visual analogy to think about something, which didn't seem like a very good analogy for what I was thinking about. I don't recall the example, but, let's say I was using a mental image of trees when thinking about matrix operations. I got annoyed at the useless imaginary trees, and wondered why I was imagining them. Then, I noticed that I was physically looking at a tree! This was fairly surprising to me. Some of the surprise was that I took a random object in my visual field to use for thinking about something unrelated, but more of the surprise was that I didn't immediately know this to be the case, even when I wondered why I was imagining trees.

After I noticed this once, I started to notice it again and again: objects from my visual field end up in my imagination, and I often try to use them as visual analogies whether they're appropriate or not. It quickly became a familiar, rather than surprising, event. More interestingly, though, after a while it started to seem like a conscious event, rather than an automatic and uncontrollable one: I've become aware of the whole process from start to finish, and can intervene at any point if I wish.

This idea of making mental processes conscious rather than unconscious, simply by paying attention to them over time, seems interes...
The Scent of Bad Psychology
by Jacob Falkovich — 2018-09-10T18:42:21.682Z
Link: https://putanumonit.com/2018/09/07/the-scent-of-bad-psychology/

Inspired by the psychology replication quiz, I've come up with four rules to tell bullshit studies in psychology from real ones:

1. The rule of antisignificance

2. The rule of Taleb's grandma

3. The rule of multiplicity

4. The rule of silicone boobs

And finally, a reason why this could mean that there's a brighter future for good psychology research.
A Dialogue on Rationalist Activism
by moridinamael — 2018-09-10T18:30:01.130Z

You're walking home through a cornfield late one night when a streak of light splits the sky like a sheet. When your vision clears, you see a golden saucer hovering before you, which deposits a human figure onto the now-flattened cornstalks right in front of you, and then vanishes in a flash.

Visitor: Greetings, Earthling.

You: Welcome to Earth!

Visitor: Thank you. May I introduce myself? I am a construct of the galactic civilization, created to begin the process of uplifting your species through ethical means. You have been selected, among all humanity, as one of a small few individuals amenable to contact, due to your passion for science fiction and rationalism. I am aware that your current epistemic state is one of suspicion that this is a dream or hallucination, in no small part due to the resemblance of this scenario to some of your most fervent fantasies and wishes. Would it be acceptable to proceed as if this interaction were really happening, for the time being, and address the improbability of this situation at a later point?

You: As long as you don't start offering me deals that involve up-arrow notation, sure.

Visitor: Fair. Before I say my piece, I'll give you the opportunity to ask any questions which might otherwise serve as distractions for you, if left unasked.

You: First question: Why are you making contact now?

Visitor: Your species is within a few years of discovering the true physical laws, which will enable something like what you would think of as FTL travel. Although, the truth of physical reality is so radically different from your current conception, that the term "FTL travel" is misleading, to the degree that many of your philosophers' conclusions about the nature of ethics involving large timescales and populations are distorted. Before we get too far down that rabbit hole, let me clarify that there's an empirically correct order to these type of proceedings, and explaining the true physics doesn't come till much later.

You: You mentioned uplifting the human race through ethical means. Could you clarify that?

Visitor: It is within our capability to rapidly and unilaterally enhance the intelligence and reflectively-correct moral quality of all members of a species. It is much preferable, ethically, to hold the hand of that species and guide it past a sequence of well-worn moral and ethical guideposts, culminating in allowing individual membe...
Alignment Newsletter #23
by Rohin Shah — 2018-09-10T17:10:01.228Z

Highlights

Visual Reinforcement Learning with Imagined Goals (Vitchyr Pong and Ashvin Nair): This is a blog post explaining a paper by the same name that I covered in AN #16. It's particularly clear and well-explained, and I continue to think the idea is cool and interesting. I've recopied my summary and opinion here, but you should read the blog post, it explains it very well.

Hindsight Experience Replay (HER) introduced the idea of accelerating learning with sparse rewards, by taking trajectories where you fail to achieve the goal (and so get no reward, and thus no learning signal) and replacing the actual goal with an "imagined" goal chosen in hindsight such that you actually achieved that goal, which means you get reward and can learn. This requires that you have a space of goals such that for any trajectory, you can come up with a goal such that the trajectory achieves that goal. In practice, this means that you are limited to tasks where the goals are of the form "reach this goal state". However, if your goal state is an image, it is very hard to learn how to act in order to reach any possible image goal state (even if you restrict to realistic ones), since the space is so large and unstructured. The authors propose to first learn a structured latent representation of the space of images using a variational autoencoder (VAE), and then use that structured latent space as the space of goals which can be achieved. They also use Q-learning instead of DDPG (which is what HER used), so that they can imagine any goal with a minibatch (s, a, s') and learn from it (whereas HER/DDPG is limited to states on the trajectory).

My opinion: This is a cool example of a relatively simple yet powerful idea -- instead of having a goal space over all states, learn a good latent representation and use that as your goal space. This enables unsupervised learning in order to figure out how to use a robot to generally affect the world, probably similarly to how babies explore and learn.

Impact Measure Desiderata (TurnTrout): This post gives a long list of desiderata that we might want an impact measure to satisfy. It considers the case where the impact measure is a second level of safety, that is supposed to protect us if we don't succeed at value alignment. This means that we want our impact measure to be agnostic to human values. We'd ...
Disagreement with Paul: alignment induction
by Stuart_Armstrong — 2018-09-10T13:54:09.844Z

I had a discussion with Paul Christiano, about his Iterated Amplification and Distillation scheme. We had a disagreement, a disagreement that I believe points to something interesting, so I'm posting this here.

It's a disagreement about the value of the concept of "preserving alignment". To vastly oversimplify Paul's idea, the AI A[n] will check that A[n+1] is still aligned with human preferences; meanwhile, A[n-1] will be checking that A[n] is still aligned with human preferences, all the way down to A[0] and an initial human H that checks on it.

Intuitively, this seems doable - A[n] is "nice", so it seems that it can reasonably check that A[n+1] is also nice, and so on.

But, as I pointed out in this post, it's very possible that A[n] is "nice" only because it lacks power/can't do certain things/hasn't thought of certain policies. So niceness - in the sense of behaving sensibly as an autonomous agent - does not go through the inductive step in this argument.

Instead, Paul confirmed that "alignment" means "won't take unaligned actions, and will assess the decisions of a higher agent in a way that preserves alignment (and preserves the preservation of alignment, and so on)".

This concept does induct properly, but seems far less intuitive to me. It relies on humans, for example, being able to ensure that A[0] will be aligned, that any more powerful copies it assesses will be aligned, that any more powerful copies those copies assess are also aligned, and so on.

Intuitively, for any concept C of alignment for H and A[0], I expect one of four things will happen, with the first three being more likely:

The C does not induct.
The C already contains all of the friendly utility function; induction works, but does nothing.
The C does induct non-trivially, but is incomplete: it's very narrow, and doesn't define a good candidate for a friendly utility function.
The C does induct in a non-trivial way, the result is friendly, but only one or two steps of the induction are actually needed.

Hopefully, further research should clarify if my intuitions are correct.
Good Citizenship Is Out Of Date
by Ben_LandauTaylor — 2018-09-10T00:15:24.456Z
Link: https://benlandautaylor.com/2018/09/03/good-citizenship-is-out-of-date/

Norms of good citizenship have been declining. These norms are a crucial piece of social technology vital to the health of local communities and institutions. While good citizenship norms are certainly still present in America today, they are substantially weaker than they were in the 1930s-1950s. This is not because of contemporary people’s personal failings; rather, it’s because we’re still operating from a foundation of norms that were built for the New Deal era, and so are not adapted to today’s conditions.

A society’s norms lead to better or worse outcomes depending on how well they fit the circumstances. For example, in a small town, politeness norms often involve greeting everyone you pass and sometimes chatting a bit; this functions well because there are few people and they mostly know and care about each other. In New York City, this would be utterly impractical, so instead politeness norms demand ignoring passersby. Less adaptive norms will naturally lose force as people notice that they don’t lead to good outcomes. Norms can be adapted to physical characteristics (like population), to the landscape of institutions (contrast American vs Mexican norms of bribing police officers, which are adapted to the local police institutions), or even to other norms (contrast American vs Japanese norms of public cleanliness, which are adapted to local levels of conscientiousness and trust).

In the mid-1900s, the norms of good citizenship were richer and more powerful than today. There was a shared idea that the good citizen was an active and integral part of his or her (norms differed somewhat by gender, but there was more similarity than difference) local community, as captured by arch-Americanist Norman Rockwell in his iconic Freedom of Speech. The good citizen was supposed to be involved with organizing at least one local civic organization, perhaps a church, or a local relief society, or a fraternal club like the Shriners. My grandfather made a point of serving on the board of the St. Louis chapter of the ACLU and writing incessant letters about local issues to the St. Louis Post-Dispatch, while his wife was heavily involved with the St. Louis planetarium and science center. For them, these things were part of a larger project through which engaged citizens would do their part to bring about a better world.

(I don’t mean to imply that everyone was always, or e...
Comment on decision theory
by Rob Bensinger — 2018-09-09T20:13:09.543Z

A comment I made on social media last year about why MIRI cares about making progress on decision theory:

We aren't working on decision theory in order to make sure that AGI systems are decision-theoretic, whatever that would involve. We're working on decision theory because there's a cluster of confusing issues here (e.g., counterfactuals, updatelessness, coordination) that represent a lot of holes or anomalies in our current best understanding of what high-quality reasoning is and how it works.

As an analogy: it might be possible to build a probabilistic reasoner without having a working understanding of classical probability theory, through sufficient trial and error. (Evolution "built" humans without understanding probability theory.) But you'd fundamentally be flying blind when it comes to designing the system — to a large extent, you couldn't predict in advance which classes of design were likely to be most promising to consider, couldn't look at particular proposed designs and make good advance predictions about safety/capability properties of the corresponding system, couldn't identify and address the root causes of problems that crop up, etc.

The idea behind looking at (e.g.) counterfactual reasoning is that counterfactual reasoning is central to what we're talking about when we talk about "AGI," and going into the development process without a decent understanding of what counterfactual reasoning is and how it works means you'll to a significantly greater extent be flying blind when it comes to designing, inspecting, repairing, etc. your system. The goal is to be able to put AGI developers in a position where they can make advance plans and predictions, shoot for narrow design targets, and understand what they're doing well enough to avoid the kinds of kludgey, opaque, non-modular, etc. approaches that aren't really compatible with how secure or robust software is developed.

Nate's way of articulating it:

The reason why I care about logical uncertainty and decision theory problems is something more like this: The whole AI problem can be thought of as a particular logical uncertainty problem, namely, the problem of taking a certain function f : Q → R and finding an input that makes the output large. To see this, let f be the function that takes the AI agent’s next action (enc

...
Advances in Baby Formula
by Laura B — 2018-09-09T11:50:09.463Z

Someone is finally trying.

Four years ago I was shopping for baby formula. I was horrified to discover that most formulas were mostly fructose and soybean oil. Including the fancy ‘organic’ ones. Humans digest fructose hepatically, and overconsumption of fructose has been implicated in a variety of modern health problems including diabetes and atherosclerosis. Why would we feed it to our babies as their primary sugar?

Oh right. It’s super cheap.

Soybean oil is a plant oil high in polyunsaturated fats and low in saturated fats. It and other high omega-6 oils have been implicated in numerous health problems. It is notoriously the only fat found in the tpn (intravenous) formulation given to infants, which causes liver failure and death. The FDA will not approve a tpn with the appropriate fat contents to keep infants alive, sparking outrage. Why would we feed our babies soybean oil instead of milk fat? It was pointed out to me that this was probably related to shelf-stability of the fat in question. Milk fat likely spoils faster.

Plus, soybean oil is cheap. As is fructose. And made with 'organic' vegetables…

So, I sprang for the premium formula which contained lactose instead of fructose, but still was mostly soybean oil.

Four years later, I am pleasantly surprised by the rise of much more sophisticated baby formulas. Enfamil Enspire, and it looks like some other formulas, have added a more complex mix of sugars and fats. The primary sugar is still lactose, but in addition there are galactooligosaccharides and polydextrose. These prebiotic sugars pass through the stomach undigested, but serve as a food source for bacteria in the gut and slowly release sugar as they are digested there. They are thought to lead to healthier gut bacteria and more stable blood sugar.

The primary fat is still a vegetable oil blend (still high in omega 6’s), but it now contains coconut oil which is high in saturated fat. Some milk fat has been added and supplemented with omega 3s DHA and ARA. One of the newest advancements is the addition of MFGM (milkfat globule membrane), which is a mix of gangliosides and phospholipids thought to play an important part in cognitive development. If you’ve spent any time studying or researching weird Ashkenazi recessive mutations, you’ll know that many of them are mutations in brain lipid metaboloism including: Nieman-Pick’s, Fa...
An Ontology of Systemic Failures: Dragons, Bullshit Mountain, and the Cloud of Doom
by ialdabaoth — 2018-09-08T18:19:26.887Z

Core Claim

I assert that a lot of value can be achieved by categorizing systemic failures into four broad categories.

For the sake of pithiness, I will name them "bugs", "dragons", "bullshit mountain", and "the cloud of doom".

A Bug is the simplest kind of failure: you have a single cause, and a single symptom. Fixing a bug is pretty easy, compared to other modes - you just fix the cause, and the symptom goes away. Bugs aren't even really systemic failures, since they don't involve any cross-talk between multiple causes or effects, but they are included here for completeness.

A Dragon is like a Bug, except that instead of a single symptom, there are multiple seemingly independent symptoms. Because the symptoms seem large and diverse, a Dragon will often seem far more daunting than it actually is - although most Dragons are still pretty large and significant causes. Dealing with a Dragon simply involves some particularly Heroic-type identifying that there's a Dragon, hunting it down, and slaying it.

Bullshit Mountain is basically the opposite of the Dragon. There is one huge, unbearable, painful symptom, that everyone knows about and everyone wishes would just GO AWAY. But no one can get any traction on it. This is because that symptom is actually being contributed to by a thousand little causes, all of which only contribute a little - so making progress on any one of them feels like it doesn't help much, if at all. The only way to solve Bulshit Mountain is for everyone in the org to roll up their sleeves, get a shovel, pick some little corner of Bullshit Mountain to work on, and start shoveling - and not stop until the problem gets noticeably better, even if their work doesn't seem to be doing much to contribute to the improvement.

Finally, we have the Cloud of Doom. The Cloud of Doom is where you have a thousand tiny causes, each of which meaningfully contribute to each of a thousand little symptoms, which together make the whole system feel unworkable. The only way to fix the Cloud of Doom is to pour an ungodly amount of Slack into the system, and hope the cloud shakes loose and blows away - otherwise, everyone needs to just throw up their hands and abandon the whole mess.

A good question to ask yourself, when trying to tackle a seemingly insurmountable mess of problems, is: is this a Dragon, Bulls...