trentmkelly/LessWrong-43k huggingface dataset:
Who are you?
by KatjaGrace — 2010-01-31T00:26:01.000Z

There are two things that people debate with regards to continuation of personhood. One is whether edge cases to our intuitions of what ‘me’ refers to are really me. For instance if a simulation of me is run on a computer, is it me? If it is definitely conscious? What if the fleshy bloody one is still alive? What if I’m copied atom for atom?

The other question is whether there is some kind of thread that holds together me at one point and some particular next me. This needn’t be an actual entity, but just there being a correct answer to the question of who the current you becomes. The opposite is a bullet that Eliezer Yudkowsky does not bite:

…to reject the idea of the personal future – … that there’s any meaningful sense in which I can anticipate being myself in five seconds, rather than Britney Spears. In five seconds there will be an Eliezer Yudkowsky, and there will be a Britney Spears, but it is meaningless to speak of the currentEliezer “continuing on” as Eliezer+5 rather than Britney+5; these are simply three different people we are talking about.

The two questions are closely related. If there’s such a thread, the first question is just about where it goes. If there’s not, the first question is often thought meaningless.

I see no reason to suppose there is such a thread. Which lump of flesh is you is a matter of definition choice as open as that of which lumps of material you want to call the same mountain. But this doesn’t mean we should give up labeling mountains at all. Let me explain.

Why would one think there is a thread holding us together? Here are the reasons I can think of:

1.	It feels like there is.

2.	We remember it always happened that way in the past. There was a me who wondered if I might just as well experience being Britney next, then later there was a me looking back thinking ‘nope, still Katja’ or some such thing.

3. We expect the me looking back is singular even if you were copied. You wouldn’t feel like two people suddenly. So you would feel like one or the other.

4.	Consciousness seems like a dimensionless thing, so it’s hard to imagine it branching, as if it could be closer or further from another consciousness. As far as our intuitions go, even if two consciousnesses are identical they might be in a way infinitely distant. What happens at that moment between there being one and there being two? ...
Deontology for Consequentialists
by Alicorn — 2010-01-30T17:58:43.881Z

Consequentialists see morality through consequence-colored lenses. I attempt to prise apart the two concepts to help consequentialists understand what deontologists are talking about.

Consequentialism1 is built around a group of variations on the following basic assumption:

The rightness of something depends on what happens subsequently.

It's a very diverse family of theories; see the Stanford Encyclopedia of Philosophy article. "Classic utilitarianism" could go by the longer, more descriptive name "actual direct maximizing aggregative total universal equal-consideration agent-neutral hedonic act2 consequentialism". I could even mention less frequently contested features, like the fact that this type of consequentialism doesn't have a temporal priority feature or side constraints. All of this is is a very complicated bag of tricks for a theory whose proponents sometimes claim to like it because it's sleek and pretty and "simple". But the bottom line is, to get a consequentialist theory, something that happens after the act you judge is the basis of your judgment.

To understand deontology as anything but a twisted, inexplicable mockery of consequentialism, you must discard this assumption.

Deontology relies on things that do not happen after the act judged to judge the act. This leaves facts about times prior to and the time during the act to determine whether the act is right or wrong. This may include, but is not limited to:

The agent's epistemic state, either actual or ideal (e.g. thinking that some act would have a certain result, or being in a position such that it would be reasonable to think that the act would have that result)

The reference class of the act (e.g. it being an act of murder, theft, lying, etc.)

Historical facts (e.g. having made a promise, sworn a vow)

Counterfactuals (e.g. what would happen if others performed similar acts more frequently than they actually do)

Features of the people affected by the act (e.g. moral rights, preferences, relationship to the agent)

The agent's intentions (e.g. meaning well or maliciously, or acting deliberately or accidentally)

Individual deontological theories will have different profiles, just like different consequentialist theories. And some of the theories you can generate using the criteria above have overlap with some consequentialist theories3. ...
Complexity of Value ≠ Complexity of Outcome
by Wei Dai — 2010-01-30T02:50:49.369Z

Complexity of value is the thesis that our preferences, the things we care about, don't compress down to one simple rule, or a few simple rules. To review why it's important (by quoting from the wiki):

Caricatures of rationalists often have them moved by artificially simplified values - for example, only caring about personal pleasure. This becomes a template for arguing against rationality: X is valuable, but rationality says to only care about Y, in which case we could not value X, therefore do not be rational.

Underestimating the complexity of value leads to underestimating the difficulty of Friendly AI; and there are notable cognitive biases and fallacies which lead people to underestimate this complexity.

I certainly agree with both of these points. But I worry that we (at Less Wrong) might have swung a bit too far in the other direction. No, I don't think that we overestimate the complexity of our values, but rather there's a tendency to assume that complexity of value must lead to complexity of outcome, that is, agents who faithfully inherit the full complexity of human values will necessarily create a future that reflects that complexity. I will argue that it is possible for complex values to lead to simple futures, and explain the relevance of this possibility to the project of Friendly AI.

The easiest way to make my argument is to start by considering a hypothetical alien with all of the values of a typical human being, but also an extra one. His fondest desire is to fill the universe with orgasmium, which he considers to have orders of magnitude more utility than realizing any of his other goals. As long as his dominant goal remains infeasible, he's largely indistinguishable from a normal human being. But if he happens to pass his values on to a superintelligent AI, the future of the universe will turn out to be rather simple, despite those values being no less complex than any human's.

The above possibility is easy to reason about, but perhaps does not appear very relevant to our actual situation. I think that it may be, and here's why. All of us have many different values that do not reduce to each other, but most of those values do not appear to scale very well with available resources. In other words, among our manifold desires, there may only be a few that are not easily satiated when we have access to the resources of an entire galaxy or u...
Logical Rudeness
by Eliezer Yudkowsky — 2010-01-29T06:48:27.969Z

The concept of "logical rudeness" (which I'm pretty sure I first found here, HT) is one that I should write more about, one of these days. One develops a sense of the flow of discourse, the give and take of argument. It's possible to do things that completely derail that flow of discourse without shouting or swearing. These may not be considered offenses against politeness, as our so-called "civilization" defines that term. But they are offenses against the cooperative exchange of arguments, or even the rules of engagement with the loyal opposition. They are logically rude.

Suppose, for example, that you're defending X by appealing to Y, and when I seem to be making headway on arguing against Y, you suddenly switch (without having made any concessions) to arguing that it doesn't matter if ~Y because Z still supports X; and when I seem to be making headway on arguing against Z, you suddenly switch to saying that it doesn't matter if ~Z because Y still supports X. This is an example from an actual conversation, with X = "It's okay for me to claim that I'm going to build AGI in five years yet not put any effort into Friendly AI", Y = "All AIs are automatically ethical", and Z = "Friendly AI is clearly too hard since SIAI hasn't solved it yet".

Even if you never scream or shout, this kind of behavior is rather frustrating for the one who has to talk to you. If we are ever to perform the nigh-impossible task of actually updating on the evidence, we ought to acknowledge when we take a hit; the loyal opposition has earned that much from us, surely, even if we haven't yet conceded. If the one is reluctant to take a single hit, let them further defend the point. Swapping in a new argument? That's frustrating. Swapping back and forth? That's downright logically rude, even if you never raise your voice or interrupt.

The key metaphor is flow. Consider the notion of "semantic stopsigns", words that halt thought. A stop sign is something that happens within the flow of traffic. Swapping back and forth between arguments might seem merely frustrating, or rude, if you take the arguments at face value - if you stay on the object level. If you jump back a level of abstraction and try to sense the flow of traffic, and imagine what sort of traffic signal this corresponds to... well, you ...
Play for a Cause
by  — 2010-01-28T20:52:24.091Z

Some of you have been trying to raise money for the Singularity Institute, and I have an idea that may help.

The idea is to hold public competitions on LessWrong with money going to charity. Agree to a game and an amount of money, then have each player designate a charity. After the game, each player gives the agreed upon amount to the charity designated by the winner.1 It’s a bit like celebrity Jeopardy.2

Play the game here on LessWrong or post a record of it.3 That will spread awareness of the charities and encourage others to emulate you.

The game can be as simple as a wager or something more involved:

Notations exist for go and chess.

The AI Box Experiment is essentially an arguing game.

Play anything else you can agree on.4

We’ve already played few games on LessWrong, more on those in a moment.

In most ways the AI problem is enormously more demanding than the personal art of rationality, but in some ways it is actually easier. In the martial art of mind, we need to acquire the realtime procedural skill of pulling the right levers at the right time on a large, pre-existing thinking machine whose innards are not end-user-modifiable.

—The Martial Art of Rationality

First, I have a confession to make: I don’t really care how much money gets donated to the Singularity Institute, nor am I trying to drum up money for some other cause. I mainly want you all playing games.

Not just playing them, of course. Playing them here in front of the rest of LessWrong and analyzing the moves in terms of the “personal art of rationality.”

We need more approaches to improvement. Even in Eliezer_Yudkowsky’s Bayesian Conspiracy fictional series/manifesto, many other schools of thought (called, for dramatic effect, “conspiracies”) were present. As I recall, the “Competitive Conspiracy” was mentioned frequently, but there are other reasons for choosing to start with games.

Games are fun, of course. They also deal with the “personal art” of getting familiar with your own brain, which I think has been underrepresented on LW. I do believe there are certain important things we can’t learn properly just by reading, arguing, and doing math (valuable as those techniques are). Games are an easy, intuitive first step to filling that gap.

We’ve already had a few games here. Warrigal held an Aumann’...
Bizarre Illusions
by MrHen — 2010-01-27T18:25:54.834Z

Illusions are cool. They make me think something is happening when it isn't. When offered the classic illusion pictured to the right, I wonder at the color of A and B. How weird, bizarre, and incredible.

Today I looked at the above illusion and thought, "Why do I keep thinking A and B are different colors? Obviously, something is wrong with how I am thinking about colors." I am being stupid when my I look at this illusion and I interpret the data in such a way to determine distinct colors. My expectations of reality and the information being transmitted and received are not lining up. If they were, the illusion wouldn't be an illusion.

The number 2 is prime; the number 6 is not. What about the number 1? Prime is defined as a natural number with exactly two divisors. 1 is an illusionary prime if you use a poor definition such as, "Prime is a number that is only divisible by itself and 1." Building on these bad assumptions could result in all sorts of weird results much like dividing by 0 can make it look like 2 = 1. What a tricky illusion!

An optical illusion is only bizarre if you are making a bad assumption about how your visual system is supposed to be working. It is a flaw in the Map, not the Territory. I should stop thinking that the visual system is reporting RGB style colors. It isn't. And, now that I know this, I am suddenly curious about what it is reporting. I have dropped a bad belief and am looking for a replacement. In this case, my visual system is distinguishing between something else entirely. Now that I have the right answer, this optical illusion should become as uninteresting as questioning whether 1 is prime. It should stop being weird, bizarre, and incredible. It merely highlights an obvious reality.

Addendum: This post was edited to fix a few problems and errors. If you are at all interested in more details behind the illusion presented here, there are a handful of excellent comments below.
You cannot be mistaken about (not) wanting to wirehead
by Kaj_Sotala — 2010-01-26T12:06:40.664Z

In the comments of Welcome to Heaven, Wei Dai brings up the argument that even though we may not want to be wireheaded now, our wireheaded selves would probably prefer to be wireheaded. Therefore we might be mistaken about what we really want. (Correction: what Wei actually said was that an FAI might tell us that we would prefer to be wireheaded if we knew what it felt like, not that our wireheaded selves would prefer to be wireheaded.)
This is an argument I've heard frequently, one which I've even used myself. But I don't think it holds up. More generally, I don't think any argument that says one is wrong about what they want holds up.
To take the example of wireheading. It is not an inherent property of minds that they'll become desperately addicted to anything that feels sufficiently good. Even from our own experience, we know that there are plenty of things that feel really good, but we don't immediately crave for more afterwards. Sex might be great, but you can still afterwards get fatigued enough that you want to rest; eating good food might be enjoyable, but at some point you get full. The classic counter-example is that of the rats who could pull a lever stimulating a part of their brain, and ended up compulsively pulling it, to the exclusion of all else. People thought this to mean they were caught in a loop of stimulating their "pleasure center", but it later turned out that wasn't the case. Instead, the rats were stimulating their "wants to seek out things -center".
The systems for experiencing pleasure and for wanting to seek out pleasure are separate ones. One can find something pleasurable, but still not develop a desire to seek it out. I'm sure all of you have had times when you haven't felt the urge to participate in a particular activity, even though you knew you'd enjoy the activity in question if you just got around doing it. Conversly, one can also have a desire to seek out something, but still not find it pleasurable when it's achieved.
Therefore, it is not an inherent property of wireheading that we'd automatically end up wanting it. Sure, you could wirehead someone in such a way that the person stopped wanting anything else, but you could also wirehead them in such a way that they were indifferent to whether or not it continued. You could even wirehead them in such a way that they enjoyed every minute of it, but at the same time want...
Welcome to Heaven
by denisbider — 2010-01-25T23:22:45.169Z

I can conceive of the following 3 main types of meaning we can pursue in life.

1. Exploring existing complexity: the natural complexity of the universe, or complexities that others created for us to explore.

2. Creating new complexity for others and ourselves to explore.

3. Hedonic pleasure: more or less direct stimulation of our pleasure centers, with wire-heading as the ultimate form.

What I'm observing in the various FAI debates is a tendency of people to shy away from wire-heading as something the FAI should do. This reluctance is generally not substantiated or clarified with anything other than "clearly, this isn't what we want". This is not, however, clear to me at all.

The utility we get from exploration and creation is an enjoyable mental process that comes with these activities. Once an FAI can rewire our brains at will, we do not need to perform actual exploration or creation to experience this enjoyment. Instead, the enjoyment we get from exploration and creation becomes just another form of pleasure that can be stimulated directly.

If you are a utilitarian, and you believe in shut-up-and-multiply, then the correct thing for the FAI to do is to use up all available resources so as to maximize the number of beings, and then induce a state of permanent and ultimate enjoyment in every one of them. This enjoyment could be of any type - it could be explorative or creative or hedonic enjoyment as we know it. The most energy efficient way to create any kind of enjoyment, however, is to stimulate the brain-equivalent directly. Therefore, the greatest utility will be achieved by wire-heading. Everything else falls short of that.

What I don't quite understand is why everyone thinks that this would be such a horrible outcome. As far as I can tell, these seem to be cached emotions that are suitable for our world, but not for the world of FAI. In our world, we truly do need to constantly explore and create, or else we will suffer the consequences of not mastering our environment. In a world where FAI exists, there is no longer a point, nor even a possibility, of mastering our environment. The FAI masters our environment for us, and there is no longer a reason to avoid hedonic pleasure. It is no longer a trap.

Since the FAI can sustain us in safety until the universe goes poof, there is no reason for...
Adaptive bias
by Morendil — 2010-01-25T17:45:16.884Z

If we want to apply our brains more effectively to the pursuit of our chosen objectives, we must commit to the hard work of understanding how brains implement cognition. Is it enough to strive to "overcome bias"? I've come across an interesting tidbit of research (which I'll introduce in a moment) on "perceptual pop-out", that hints it is not enough.

"Cognition" is a broad notion; we can dissect it into awareness, perception, reasoning, judgment, feeling... Broad enough to encompass what I'm coming to call "pure reason": our shared toolkit of normative frameworks for assessing probability, evaluating utility, guiding decision, and so on. Pure reason is one of the components of rationality, as this term is used here, but it does not encompass all of rationality, and we should beware the many Myths of Pure Reason. The Spock caricature is one; by itself enough cause to use the word "rational" sparingly, if at all.

Or the idea that all bias is bad.

It turns out, for instance, that a familiar bugaboo, confirmation bias, might play an important role in perception. Matt Davis at Cambridge Medical School has crafted a really neat three-part audio sample based showcasing one of his research topics. The first and last part of the sample are exactly the same. If you are at all like me, however, you will perceive them quite differently.

Here is the audio sample (mp3). Please listen to it now.

Notice the difference? Matt Davis, who has researched these effects extensively, refers to them as "perceptual pop-out". The link with confirmation bias is suggested by Jim Carnicelli: " Once you have an expectation of what to look for in the data, you quickly find it."

In Probability Theory, E.T. Jaynes notes that perception is "inference from incomplete information"; and elsewhere adds:

Kahneman & Tversky claimed that we are not Bayesians, because in psychological tests people often commit violations of Bayesian principles. [...] People are reasoning to a more sophisticated version of Bayesian inference than [Kahneman and Tversky] had in mind. [...] We would expect Natural Selection to produce such a result: after all, any reasoning format whose results conflict with Bayesian inference will place a creature at a decided survival disadvantage.

There is an apparent paradox between our susceptibility to various biases, and the fact that these biases are prevalent precisely because t...
Simon Conway Morris: "Aliens are likely to look and behave like us".
by Roko — 2010-01-25T14:16:18.752Z

Professor Simon Conway Morris at Cambridge University will tell a conference on alien life that extraterrestrials will most likely have evolved just like "earthlings" and so resemble us to a degree with heads, limbs and bodies.

[link]

Unfortunately they will have also evolved our foibles and faults which could make them dangerous if they ever did visit us on Earth.

The evolutionary paleobiologist's beliefs mean that science fiction films such as Star Wars and Star Trek could be more accurate than they ever imagined in depicting alien life.

Prof Conway Morris believes that extraterrestrial life is most likely to occur on a planet similar to our own, with organisms made from the same biochemicals. The process of evolution will even shape alien life in a similar way, he added.

“It is difficult to imagine evolution in alien planets operating in any manner other than Darwinian," he said.

"In the end the number of options is remarkably restrictive. I don't think an alien will be a blob. If aliens are out there they should have evolved just like us. They should have eyes and be walking on two legs.

"In short if there is any life out there then it is likely to be very similar to us."

Extra-terrestrials might not only resemble us but have our foibles, such as greed, violence and a tendency to exploit others' resources, claims Professor Conway Morris.

They could come in peace but also be searching for somewhere to live, and to help themselves to water, minerals and fuel he is due to tell a conference at the Royal Society, in London.

However he also thinks that because much of the Universe is older than us they would have evolved further down the line and we should have heard from them by now.

He believes it is increasingly looking like they may not be out there at all.

Thoughts on this?

Conway Morris is a big hitter in the scientific establishment. He is, however, a theist, and has "argued against materialism" according to wikipedia. But what are his arguments? Alas the press piece doesn't say. Kudos to anyone who finds the conference and posts the arguments.

“It is difficult to imagine evolution in alien planets operating in any manner other than Darwinian,"

Is this just an instance of a slightly woo scientist-theist failing to take into account that nature might be more imaginative than him?