trentmkelly/LessWrong-43k huggingface dataset:
Forcing yourself to keep your identity small is self-harm
by Gordon Seidoh Worley — 2021-04-03T14:03:06.469Z

The title lays out my thesis in perhaps the maximally provocative way: if you put direct effort into keeping your identity small (à la Graham), you are doing harm to yourself akin to the kind of harm you do to yourself by suppressing emotions.

I was inspired to write this after reading some of the recent discussion on the EA Forum about why people do or do not identity as an EA. Responses that mention not wanting to identity as EA in the name of keeping one's identity small got me thinking about this topic and made me realize that, despite likely being somewhat less attached to various identities than average, I don't eschew being identified with things or using identities as a means of communicating with others. And I immediately had a strong sense of why I don't do that: taking action to directly avoid having a large identity would be trying to control myself in a way that leads to cognitive dissonance and dissociation at best and cognitive fusion at worst.

Here's what I see going on. A person realizes they are identifying as or with something, let's say effective altruism since that was the motivating example. They have a commitment to keeping their identity small, so give themselves feedback that they are doing something wrong when they identity as an effective altruist and should stop doing it. Negative reinforcement like this, though, rarely makes the underlying reason for wanting to do something go away, like perhaps wanting to signal that you are the kind of person who cares about everyone equally and thinks rationally about things. Instead it creates a secondary desire to not do the thing that lives alongside the desire to do it. So now you both want to identity as an EA and don't want to, and whether or not you do it depends on which desire is stronger.

A good model of what happens next is offered by Internal Family Systems: competing parts of the brain are now engaged in a battle of trying to protect you from what other parts of your brain want you to do. Aside from causing a lot of suffering as a result of being the battleground on which this fight occurs, it can eventually lead to isolation (exile) of those parts of the self that seek to identify with something. They're still there trying to do their thing, of course, but they are so well suppressed that they become hidden from self-awareness.

I see this as analogous to the effects of suppressing em...
Moral Golems
by Erich_Grunewald — 2021-04-03T10:12:59.152Z
Link: https://www.erichgrunewald.com/posts/moral-golems/

Stop thinking of the project of ethics as "figure out which simple theory is True".

Start instead thinking of ethics as a project of trying to piece together psychological models of this insanely complicated and messy thing, "human morality".

Binding exceptionless commitments matter to understanding this complicated thing; folk concepts like courage and honesty and generosity matter; taboo tradeoffs and difficult attempts to quantify, aggregate, and weigh relative well-being matter.

Stop picking a "side" and then losing all interest in the parts of human morality that aren't associated with your "side": these are all just parts of the stew, and we need to work hard to understand them and reconcile them just right, not sort ourselves into Team Virtue vs. Team Utility vs. Team Duty.[1]

– Rob Bensinger

Rob Bensinger writes that the search for true moral systems is quixotic, that ethics is so complicated that we cannot hope to capture it in a neat, rule-based system & that our project ought instead to be to construct, starting from observations about what humans think, do & feel, a conglomerate of systems, or maybe an amalgam, that together capture these observations as well as possible, though necessarily imperfectly.

He writes this in a post about things he would like to change about contemporary philosophy, which seems a little off, in a way, because my impression is that philosophers have been mostly anti-realist during the past century. That is to say, they don't think moral claims (e.g. "it is wrong to murder") are things that can be true or false, or that moral claims don't even pretend to be such things. Though I suppose it is possible to be both anti-realist & non-syncretic. Anyway, Rob would probably say that it doesn't matter, because even if moral claims can be true, there is no way for us to find out which of them are true – morality is just too complicated. The point is that we should not get tunnel vision on seeking the one true moral system, because no moral system could possibly describe the complex nature of human morality.

This is similar to the sentiment expressed here by Bernard Williams:

Now, I don't think, when we talk about utilitarianism, that we should be thinking of it as if we were discussing simply the timeless truth of some moral system, as if it were somehow written in the galaxy what the true moral system was and we were

...
People are gathering 2 million USD to save a kid with a rare disease. I feel weird about it. Why?
by hookdump — 2021-04-02T23:00:47.345Z

If someone organizes a charity to gather 50,000 USD to save a kid from a rare disease, I'm all for it. Even though that same amount could help or save a few people if invested more effectively. Screw effectiveness! A life is priceless!

I am not a fan of utilitarism at all. I don't care about arithmetically maximizing a well-being formula. These are human beings we're talking about.
But when the amount is as huge as 2 million USD... I start feeling like a threshold has been crossed. I feel like instead of saving a single kid, that money could save hundreds or thousands of lives, and I wonder why that kid is worth so many deaths.

It's like the trolley problem with 1 rare-disease-kid in one track, and hundreds of innocents in the other track. And everybody around me is choosing to save the child. And I can empathize with their choice, but it makes me feel uneasy.

Of course I don't oppose this specific charity. If saving my son's life costed 2 million, I would disregard these feelings and start campaining right away.

But from an outsider's perspective, I feel weird about it. And I have a hard time articulating why. Can someone help me understand what I'm feeling?
Predictive Coding has been Unified with Backpropagation
by lsusr — 2021-04-02T21:42:12.937Z

Artificial Neural Networks (ANNs) are based around the backpropagation algorithm. The backpropagation algorithm allows you to perform gradient descent on a network of neurons. When we feed training data through an ANNs, we use the backpropagation algorithm to tell us how the weights should change.

ANNs are good at inference problems. Biological Neural Networks (BNNs) are good at inference too. ANNs are built out of neurons. BNNs are built out of neurons too. It makes intuitive sense that ANNs and BNNs might be running similar algorithms.

There is just one problem: BNNs are physically incapable of running the backpropagation algorithm.

We do not know quite enough about biology to say it is impossible for BNNs to run the backpropagation algorithm. However, "a consensus has emerged that the brain cannot directly implement backprop, since to do so would require biologically implausible connection rules"[1].

The backpropagation algorithm has three steps.

Flow information forward through a network to compute a prediction.

Compute an error by comparing the prediction to a target value.

Flow the error backward through the network to update the weights.

The backpropagation algorithm requires information to flow forward and backward along the network. But biological neurons are one-directional. An action potential goes from the cell body down the axon to the axon terminals to another cell's dendrites. An axon potential never travels backward from a cell's terminals to its body.

Hebbian theory

Predictive coding is the idea that BNNs generate a mental model of their environment and then transmit only the information that deviates from this model. Predictive coding considers error and surprise to be the same thing. Hebbian theory is specific mathematical formulation of predictive coding.

Predictive coding is biologically plausible. It operates locally. There are no separate prediction and training phases which must be synchronized. Most importantly, it lets you train a neural network without sending axon potentials backwards.

Predictive coding is easier to implement in hardware. It is locally-defined; it parallelizes better than backpropagation; it continues to function when you cut its substrate in half. (Corpus callosotomy is used to treat epilepsy.) Digital computers break when you cut them in half. Predictive coding is something evolution could plausibly invent.

Unificat

...
In plain English - in what ways are Bayes' Rule and Popperian falsificationism conflicting epistemologies?
by Sandro P. — 2021-04-02T21:21:23.865Z

Bayes' Rule dictates how much credence you should put in a given proposition in light of prior conditions/evidence. It answers the question How probable is this proposition?

Popperian falsificationism dictates whether a given proposition, construed as a theory, is epistemically justifiable, if only tentatively. But it doesn't say anything about how much credence you should put in an unfalsified theory (right?). It answers the question Is this proposition demonstrably false (and if not, lets hold on to it, for now)?

I gather that the tension has something to do with inductive reasoning/generalizing, which Popperians reject as not only false, but imaginary. But I don't see where inductive reasoning even comes in to Bayes' Rule. In Arbital's waterfall example, it just is the case that "the bottom pool has 3 parts of red water to 4 parts of blue water" - which means that there just is a roughly 43% probability that a randomly sampled water molecule from that pool is red. How could a Popperian disagree?

What am I missing?

Thanks!
Phylactery Decision Theory
by Bunthut — 2021-04-02T20:55:45.786Z

This is definitely a hack, but it seems to solve many problems around Cartesian Boundaries. Much of this is development of earlier ideas about the Predict-O-Matic, see there if something is unclear.

Phylactery Decision Theory takes a Base Decision Theory (BDT) as an input and builds something around it, creating a new modified decision theory. Its purpose it to give its base the ability to """learn""" its position in the world.

I'll start by explaining a model of it in a Cartesian context. Lets say we have an agent, with a set of designated input and output channels. Then it makes its "decisions" like this: First, it has a probability distribution over everything, including the values of the output channels in the future, and updates it based on the input. There is then an automated mechanism which assigns the output bits, and it gives each output with exactly the probability that the agent assigned to it. The agents prior includes something like the following: "The output bits will be like BDT(MyProbabilityDistribution, MyUtilityFunction, PossibleOutputs)" Then it is easy to see that this belief is stable: Since the agent believes it, the automatism will set the output bits to be that way, and then the agent will observe it, and notice that its beliefs were right. So far this is just a more complicated way to make a BDT agent. Its like a daemon inside an oracle, but on purpose.

Descending to Possibility

Now consider a case where we don't know the set of output channels ahead of time. Instead start with a set of things OP that we think might be output channels (which, if we are very uncertain that can just mean all events). Then the agents prior will consist of many versions of the one previously suggested, one for each subset of OP. We will start by assigning 1−ϵ probability that all potential outputs are like BDT(MyProbabilityDistribution, MyUtilityFunction, OP). Then we assign 1/|OP|(1−ϵ)ϵ probability to each theory that all but one potential output are like BDT(MyProbabilityDistribution, MyUtilityFunction, OP\{O_i}). Similarly the hypothesis that all but two are according to BDT gets probabilities on the order of (1−ϵ)ϵ2, and so on. Then for ϵ sufficiently close to 0, the agent will eventually learn the "right" set of outputs, because for the others it sees that it can't controll them.

For the input channels, we can't entire...
Affordances
by abramdemski — 2021-04-02T20:53:35.639Z

I noticed recently that, although the term "affordance" seems to be in the LessWrong lexicon, there isn't any article about it. I think it might be one of those concepts which you currently have to pick up from CFAR, or learn through osmosis. This seems like an important concept, so I'm going to attempt to do it justice in this essay.

The easiest way to define the term is with examples: a button affords pushing, a lever affords pulling, etc. An affordance is a vividly present option for action. Note that affordances vary from person to person: one person may have the affordance to dance at a party, while another person lacks that affordance.

When I google "affordances", I mainly get results about UX design and human-computer interaction. This makes sense: a big part of designing products (be they software or hardware) is about making sure the user has all the right affordances. If you design a new kitchen implement which requires a pumping action, you want the user to immediately have a "pumping action" affordance when they see it.

The biggest hurdle to having an affordance is raising the possibility to attention. But that's not quite all there is to it. For example, if you're struggling to ask someone out on a date, but you just can't, I would say you lack the affordance. You explicitly know what you want to do, but you "don't have that menu option".

A Toy Model

I'm going to use a system 1 / system 2 model to explain what might be going on.

System 1 has a vast array of possible actions it could take at any time; at least, all possible combinations of muscle tensions across time. To simplify planning, system 1 abstracts a small number of possibilities for our attention. We usually don't even consider options that aren't on this list, because this list just is the action-space to us, at least when we're not thinking hard about it.

On the other hand, system 2 has its own abstract, simplified view of the world, and its own role to play in planning. Sometimes, then, system 2's idea of the action-space will get out of sync with system 1's. Thus, a shy person finds themselves tongue-tied when they try to ask an attractive person out. System 1 doesn't present that menu option: we lack that affordance. If system 2 can manage to override system 1 at all, we're turning off the auto-pilot, and steering manually. Forming coherent sentences suddenly seems difficult. We do...
My take on Michael Littman on "The HCI of HAI"
by Alex Flint — 2021-04-02T19:51:44.327Z

This is independent research. To make it possible for me to continue writing posts like this, please consider supporting me.

Many thanks to Professor Littman for reviewing a draft of this post.

Yesterday, at a seminar organized by The Center for Human-compatible AI (CHAI), Professor Michael Littman gave a presentation entitled "The HCI of HAI'', or "The Human Computer Interaction of Human-compatible Artificial Intelligence". Professor Littman is a computer science professor at Brown who has done foundational work in reinforcement learning as well as many other areas of computer science. It was a very interesting presentation and I would like to reflect a little on what was said.

The basic question Michael addressed was: "how do we get machines to do what we want?" and his talk was structured around the various modalities that we’ve developed to convey our intentions to machines, starting from direct programming, through various forms of machine learning, and on to some new approaches that his lab is developing. I found his taxonomy helpful, so I’ll paraphrase some of it below and then add my own thoughts afterwards.

One way we can get machines to do what we want is by directly programming them to do what we want. But, the problem here is that direct programming is challenging for most people, and Michael talked through some studies that asked non-programmers to build simple if-then rulesets to implement programs such as "if it’s raining and the dog is outside then do such-and-such". Most participants had difficulty reasoning about the difference between "if the dog goes outside while it is raining" versus "if it starts raining while the dog is outside".

I’m not sure that this is the main reason to look beyond direct programming, but I do agree that we need ways to instruct machines that are at least partially example-driven rather than entirely rule-driven, so I’m happy to accept the need for something beyond direct programming.

Next, Michael discussed reinforcement learning, in which the human provides a reward function, and it is the machine’s job to find a program that maximizes rewards. We might say that this allows the human to work at the level of states rather than behavior, since the reward function can evaluate the desirability of states of the world and leave it up to the machine to construct a set of rules for navigating towards desirable states:

But, ...
Analyzing Multiplayer Games using IMPACT
by midco — 2021-04-02T19:10:06.381Z

This post is an informal writeup of an idea discovered while investigating POWER. It offers an additional tool for framing multi-agent games in terms of attainable utility, with the hope that data from both IMPACT and POWER can offer a more complete view of POWER-scarcity dynamics in multi-agent games.

Note on writing style: I use royal "we" for explaining math; the results are essentially my own unreviewed work.

Overview

The purpose of this post is to define and give intuition for a measure of a player's impact on some real-valued outcome variable in an arbitrary multiplayer game; we call this measure "IMPACT". We present motivating examples of multiplayer games, then define IMPACT in the more general context of arbitrary Bayesian networks. We then explore some basic connections between IMPACT and standard multiplayer game theory and present some conjectures to motivate further research.

Motivating Examples - Understanding Multiplayer Games

One of the difficulties in understanding multiplayer games is the principle that in general, a player's optimal action is dependent on the actions of other players. One way around this problem is to restrict consideration to Nash Equilibria, which fully specify optimal actions for us. However, this loses the power to describe sub-optimal actions, which are relevant in real-world examples involving imperfect humans and intractably large action spaces.

While Nash Equilibria "work" by first conditioning on optimal actions and then considering probabilistic strategies, we go the other way around: fix some arbitrary mixed strategy profile, then analyze expected utility. In the single-player setting, this is analogous to a value function and can be constructed straightforwardly. In the multiplayer setting, we're forced to deal with dependencies on other players' strategies that complicate the notion of "expected utility". To handle this dependency, our framework makes the following assumptions motivated by Bayesian games:

Each player's posterior distribution over other players' actions is optimal given that player's information ("common prior assumption" )
We consider interim expected utility (IEU), which takes an expectation across the player's posterior beliefs of other players' actions

Borrowing notation from Bayesian games, we now have enough machinery to consider various games in terms of IEU ui:Ai→R. In each example, we fix ...
Learning Russian Roulette
by Bunthut — 2021-04-02T18:56:26.582Z

Surprisingly, our current theories of anthropics don't seem to cover this.

You have a revolver with six chambers. One of them has a bullet in it. You are offered $1 for spinning the barrel, pointing at your head, and pulling the trigger. You remember doing this many times, and surviving each time. You also remember many other people doing this many times, and dying about 1/6th of the time. Should you play another round?

It seems to me that the answer is no, but existing formal theories disagree. Consider two hypothesis: A says that everyone has a 1/6 chance of dying. B says that everyone else has a 1/6 chance of dying, but I survive for sure. Now A has a lot more prior probability, but the likelihood ratio is 5:6 for every time I played. So if I played often enough, I will have updated to mostly believing B. Neither Self Indication Assumption nor Self Selection Assumption update this any further. SIA, because theres one of me in both worlds. SSA, because that one me is also 100% of my reference class. UDT-like approaches reason that in the A world, you want to never play, and in the B world you want to always play. Further, if I remember playing enough rounds, almost all my remaining measure will be in the B world, and so I should play, imitating the simple bayesian answer.

I'm not sure how we got to this point. It seems like most of the initial anthropics-problems were about birth-related uncertainty, and this stuck pretty well.

Problems for any future solution

Now one obvious way to fix this is to introduce a [death] outcome, which you can predict but which doesn't count towards the normalization factor when updating. Trying to connect this [death] with the rest of your epistemology would require some solution to embedding.

Worse than that however, this would only stop you from updating on your survival. I think the bigger problem here is that we aren't learning anything (in the long term) from the arbitrarily large control group. After all even if we don't update on our survival, that only means our odds ratio between A and B stays fixed. Its hardly a solution to the problem if "having the right prior" is doing all the work.

Learning from the control group has its own problems however. Consider for example the most obvious way of doing so: we observe that most things work out similarly for them as they do for us, and so we generalize this to playing russia...