trentmkelly/LessWrong-43k huggingface dataset:
Subjective Anticipation and Death
by LucasSloan — 2010-03-17T01:14:24.994Z

tldr; It is incoherent to talk about a "you" which stretches through time. Instead, we should think of a series of similar mind-moments.

Once upon a time, there was a little boy, who answered to the name Lucas Sloan and was scared of dying. I too answer to the name Lucas Sloan, and I remember being afraid of dying. Little Lucas wasn't scared of the present state of affairs, but it is fairly obvious that Little Lucas isn't around anymore. By any practical definition, Little Lucas is dead, he only exists as a memory in my mind and more indirectly in the minds of others. Little Lucas did not care that other people remembered him, he cared that he did not die. So what is this death thing, if Little Lucas was scared of it, but was not scared of the present situation?

I would now like to introduce the term mind-moment. I'm not sure quite how to define that, it may have to mean a single plank-time snapshot of a mind, or it might be as much as a couple weeks. I doubt that what I mean by this is anywhere near the upper bound I just gave, a more likely upper bound might be about a second - about the time it takes to notice something and realize something is going on.

Confusion about the exact definition of mind-moment aside, I think it is obvious that Little Lucas and I are separate mind-moments. And the fact of the matter is that the mind moment that was Little Lucas no longer exists, that mind-moment is definitely dead, gone, kaput. In fact, if I'm right about what I mean by mind-moment, many mind-moments have ceased exist since I started writing this. And frankly, both of those things are, in fact, good. What would the point be of constantly re-running the same mind-moment over and over again? I certainly don´t want to be caught in a time loop till the end of time, even if I couldn't tell that I was – that would be as much a waste of the future as converting the stars into orgasmium. I want to experience new things, I want to grow and learn. But the problem is that my use of the word I is incoherent - the "me" that experiences those new things, that knows more than I do, is not me. My time is passing, soon enough, I will be the Little Lucas remembered only because he had some interesting ideas.

It's not that I don't want to die, it's that I want there to, in the future, exist happy, fulfilled mind-moments that remember being me. This model neatly solv...
Overcoming the mind-killer
by woozle — 2010-03-17T00:56:01.710Z

I've been asked to start a thread in order to continue a debate I started in the comments of an otherwise-unrelated post. I started to write a post on that topic, found myself introducing my work by way of explanation, and then realized that this was a sub-topic all its own which is of substantial relevance to at least one of the replies to my comments in that post -- and a much better topic for a first-ever post/thread .

So I'm going to write that introductory post first, and then start another thread specifically on the topic under debate.

I run issuepedia.org, a wiki site largely dedicated to the rational analysis of politics.

As part of that analysis, it covers areas such as logical fallacies (and the larger domain of what I call "rhetorical deceptions" and which LessWrong calls "the dark arts"), history, people, organizations, and any other topics necessary to understand an issue. Coverage of each area generally includes collecting sources (such as news articles, editorials, and blog posts), essential details to provide a quick overview, and usually an attempt to draw some kind of conclusion1 about the topic's ethical significance based, as much as possible, on the sources collected. (Readers are, of course, free to use the wiki format to offer counterarguments and additional facts/sources.)

I started Issuepedia in 2005, largely in an attempt to understand how Bush could possibly have been re-elected (am I deluded, or is half the country deluded? if the latter, how did this happen?). Most of the content is my writing, as I am better at writing than at community-building, but it is all freely copyable under a CC license. I do not receive any money for my work on the site; it does accept donations, but this fact is not heavily advertised and so far there have been no donors. It does not display advertisements, nor have I advertised it (other than linking to it in contexts where its content seems relevant, such as comments on blog entries). I am considering doing the latter at some point when I have sufficient time to give the project the focus it will need in order to grow successfully.

Rationality and Politics

My main hypothesis2 in starting Issuepedia is that it is, in fact, possible to be rational about politics, to overcome its "mind-killing" qualities -- if given sufficient "thinking room" in which to record and work through all the relevant (and often m...
Omega's subcontracting to Alpha
by Stuart_Armstrong — 2010-03-16T18:52:42.058Z

This is a variant built on Gary Drescher's xor problem for timeless decision theory.

You get an envelope from your good friend Alpha, and are about to open it, when Omega appears in a puff of logic.

Being completely trustworthy as usual (don't you just hate that?), he explains that Alpha flipped a coin (or looked at the parity of a sufficiently high digit of pi), to decide whether to put £1000 000 in your envelope, or put nothing.

He, Omega, knows what Alpha decided, has also predicted your own actions, and you know these facts. He hands you a £10 note and says:

"(I predicted that you will refuse this £10) if and only if (there is £1000 000 in Alpha's envelope)."

What to do?

EDIT: to clarify, Alpha will send you the envelope anyway, and Omega may choose to appear or not appear as he and his logic deem fit. Nor is Omega stating a mathematical theorem: that one can deduce from the first premise the truth of the second. He is using XNOR, but using 'if and only if' seems a more understandable formulation. You get to keep the envelope whatever happens, in case that wasn't clear.
The problem of pseudofriendliness
by  — 2010-03-15T14:17:00.528Z

The Friendly AI problem is complicated enough that it can be divided into a large number of subproblems. Two such subproblems could be:

The problem of goal interpretation – This means that the human expectation of the results of an AI implementing a goal differ from the results that the AI actually works toward.

The problem of innate drives (see Steve Omohundro’s ‘Basic Drives’ paper for more detail) – This is when either specific goals, or goal based reasoning in general, creates subgoals that humans do not anticipate.

Let's call an AI which does not suffer from these problem a pseudofriendly AI. Would this be a useful type of AI to produce? Well, maybe or maybe not. But even if it fails to be useful in and of itself, solving the pseudofriendly AI problem may be a helpful step toward developing the mode of thinking needed to solve the Friendly AI problem.

It's also possible that pseudofriendliness might be able to interact usefully with Eliezer's Coherent Extrapoltated Volition (CEV - see here for more details). Eliezer has expressed CEV as follows:

In poetic terms, our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted.

However, an FAI is not to be given the CEV as it's goal but rather a superintelligence is to use our CEV to determine what goals an FAI should be given. What this means though is that there will be a point where a superintelligence exists that is not friendly. Could a pseudofriendly AI fill a gap here? Probably not - pseudofriendliness is not friendliness, nor should it be confused with it. However, it might be part of a solution that help the CEV approach to be safely implemented.

Why all this hassle though? We seem to have exchanged one very important problem for two less important ones. Well, part of the benefit of pseudofriendliness is that it seems like it should be easier to formalise. First, let us introduce the concept of an interpretation system.

An interpretation system takes a partially specified world state (called a goal) and outputs a triple (Wx, Sx, Cx) where Wx is a partially specified world state, Sx is a set contain...
Undiscriminating Skepticism
by Eliezer Yudkowsky — 2010-03-14T23:23:25.539Z

Tl;dr: Since it can be cheap and easy to attack everything your tribe doesn't believe, you shouldn't trust the rationality of just anyone who slams astrology and creationism; these beliefs aren't just false, they're also non-tribal among educated audiences. Test what happens when a "skeptic" argues for a non-tribal belief, or argues against a tribal belief, before you decide they're good general rationalists. This post is intended to be reasonably accessible to outside audiences.

I don't believe in UFOs. I don't believe in astrology. I don't believe in homeopathy. I don't believe in creationism. I don't believe there were explosives planted in the World Trade Center. I don't believe in haunted houses. I don't believe in perpetual motion machines. I believe that all these beliefs are not only wrong but visibly insane.

If you know nothing else about me but this, how much credit should you give me for general rationality?

Certainly anyone who was skillful at adding up evidence, considering alternative explanations, and assessing prior probabilities, would end up disbelieving in all of these.

But there would also be a simpler explanation for my views, a less rare factor that could explain it: I could just be anti-non-mainstream. I could be in the habit of hanging out in moderately educated circles, and know that astrology and homeopathy are not accepted beliefs of my tribe. Or just perceptually recognize them, on a wordless level, as "sounding weird". And I could mock anything that sounds weird and that my fellow tribesfolk don't believe, much as creationists who hang out with fellow creationists mock evolution for its ludicrous assertion that apes give birth to human beings.

You can get cheap credit for rationality by mocking wrong beliefs that everyone in your social circle already believes to be wrong. It wouldn't mean that I have any ability at all to notice a wrong belief that the people around me believe to be right, or vice versa - to further discriminate truth from falsity, beyond the fact that my social circle doesn't already believe in something.

Back in the good old days, there was a simple test for this syndrome that would get quite a lot of mileage: You could just ask me what I thought about God. If I treated the idea with deeper respect than I treated as...
Musings on probability
by bogdanb — 2010-03-14T23:17:10.250Z

I read this comment, and after a bit of rambling I realized I was as confused as the poster. A bit more thinking later I ended up with the “definition” of probability under the next heading. It’s not anything groundbreaking, just a distillation (specifically, mine) of things discussed here over the time. It’s just what my brain thinks when I hear the word.

But I was surprised and intrigued when I actually put it in writing and read it back and thought about it. I don’t remember seeing it stated like that (but I probably read some similar things).

It probably won’t teach anyone anything, but it might trigger a similar “distillation” of “mind pictures” in others, and I’m curious to see that.

What “probability” is...

Or, more exactly, what is the answer to “what’s the probability of X?”

Well, I don’t actually know, and it probably depends on who asks. But here’s the skeleton of the answer procedure:

Take the set of all (logically) possible universes. Assign to each universe a finite, real value m (see below).

Eliminate from the set those those that are inconsistent with your experiences. Call the remaining set E.

Construct T, the subset of E where X (happens, or happened, or is true).

Assign to each universe u in set E a value p, such that p(u) is inversely proportional to m(u), and the integral of p over set E is 1.

Calculate the integral of p over the set T. The result is called “the probability of X”, and is the answer to the question.

I’m aware that this isn’t quite a definition; in fact, it leaves more unsaid (undefined) than it explains. Nevertheless, to me it seems that the structure itself is right: people might have different interpretations for the details (and, like me, be uncertain about them), but those differences would still be mentally structured like above.

In the next section I explain a bit where each piece comes from and what it means, and in the one after I’m going to ramble a bit.

Clarifications

About (logically possible) universes: We don’t actually know what our universe is; as such, other possible universes isn’t quite a well-defined concept. For generality, the only constraint I put above is that they be logically possible, for the only reason that the description is (vaguely) mathematical ...
Crunchcourse - a tool for combating learning akrasia
by  — 2010-03-14T22:53:59.998Z

Crunchcourse is a free website that might be of use to people trying to learn things outside the normal classroom setting. It aims to get together groups of people interested in the same topic and use our social instincts to motivate us to do the work.

It is in its early stages. If it proves useful, it might be useful to standardize on it as the place to learn the various prerequisites that lesswrong has.
Reasoning isn't about logic (it's about arguing)
by Morendil — 2010-03-14T04:42:12.048Z

"Why do humans reason" (PDF), a paper by Hugo Mercier and Dan Sperber, reviewing an impressive amount of research with a lot of overlap with themes previously explored on Less Wrong, suggests that our collective efforts in "refining the art of human rationality" may ultimately be more successful than most individual efforts to become stronger. The paper sort of turns the "fifth virtue" on its head; rather than argue in order to reason (as perhaps we should), in practice, we reason in order to argue, and that should change our views quite a bit.

I summarize Mercier and Sperber's "argumentative theory of reasoning" below and point out what I believe its implications are to the mission of a site such as Less Wrong.

Human reasoning is one mechanism of inference among others (for instance, the unconscious inference involved in perception). It is distinct in being a) conscious, b) cross-domain, c) used prominently in human communication. Mercier and Sperber make much of this last aspect, taking it as a huge hint to seek an adaptive explanation in the fashion of evolutionary psychology, which may provide better answers than previous attempts at explanations of the evolution of reasoning.

The paper defends reasoning as serving argumentation, in line with evolutionary theories of communication and signaling. In rich human communication there is little opportunity for "costly signaling", that is, signals that are taken as honest because too expensive to fake. In other words, it's easy to lie.

To defend ourselves against liars, we practice "epistemic vigilance"; we check the communications we receive for attributes such as a trustworthy or authoritative source; we also evaluate the coherence of the content. If the message contains symbols that matches our existing beliefs, and packages its conclusions as an inference from these beliefs, we are more likely to accept it, and thus our interlocutors have an interest in constructing good arguments. Epistemic vigilance and argumentative reasoning are thus involved in an arms race, which we should expect to result in good argumentative skills.

What of all the research suggesting that humans are in fact very poor at logical reasoning? Well, if in fact "we reason in order to argue", when the subjects are studied in non-argumentative situations this is precisely what we should expect.

Mercier and Sperber argue that, when you look at r...
The Importance of Goodhart's Law
by blogospheroid — 2010-03-13T08:19:29.974Z

This article introduces Goodhart's law, provides a few examples, tries to explain an origin for the law and lists out a few general mitigations.

Goodhart's law states that once a social or economic measure is turned into a target for policy, it will lose any information content that had qualified it to play such a role in the first place. wikipedia The law was named for its developer, Charles Goodhart, a chief economic advisor to the Bank of England.

The much more famous Lucas critique is a relatively specific formulation of the same.

The most famous examples of Goodhart's law should be the soviet factories which when given targets on the basis of numbers of nails produced many tiny useless nails and when given targets on basis of weight produced a few giant nails. Numbers and weight both correlated well in a pre-central plan scenario. After they are made targets (in different times and periods), they lose that value.

We laugh at such ridiculous stories, because our societies are generally much better run than Soviet Russia. But the key with Goodhart's law is that it is applicable at every level. The japanese countryside is apparently full of constructions that are going on because constructions once started in recession era are getting to be almost impossible to stop. Our society centres around money, which is supposed to be a relatively good measure of reified human effort. But many unscruplous institutions have got rich by pursuing money in many ways that people would find extremely difficult to place as value-adding.

Recently GDP Fetishism by David henderson is another good article on how Goodhart's law is affecting societies.

The way I look at Goodhart's law is Guess the teacher's password writ large. People and instituitions try to achieve their explicitly stated targets in the easiest way possible, often obeying the letter of the law.

A speculative origin of Goodhart's law

The way I see Goodhart's law work, or a target's utility break down, is the following.

Superiors want an undefined goal G.

They formulate G* which is not G, but until now in usual practice, G and G* have correlated.

Subordinates are given the target G*.

The well-intentioned subordinate may recognise G and suggest G** as a substitute, but such people are relatively few and far inbetween. Most people try to achieve G*.

As time goes on, every means of achieving G

...
Rational feelings: a crucial disambiguation
by Academian — 2010-03-13T00:48:20.576Z

Ever wonder something like, "I know it's bad for me that I lost my job, but I actually feel happy about it... is that rational?"

What could a question like that mean? There is a divisive ambiguity here that really messes people up. A feeling as an experience is neither rational nor irrational. It's like asking how ethical a shade of purple is. The point is that a feeling must be framed as a behavior or a statement to ask whether it is rational, and which one matters heaps and loads to the answer.

If you think of the happiness as a behavior, something that you're doing, then the question is secretly asking about
instrumental rationality: whether you're applying your
beliefs correctly to attain your values. In our opening example, the question becomes "Does feeling happy serve my values?", or simply "Do I value feeling happy?". If you're almost anyone, the answer is probably "yes".

If you think of the happiness as a statement or instruction that says "Your values are being served", which can be true/false and justified/unjustified, then the question is really about
epistemic rationality, and asks: "Am I justified to believe my values are being served?". If "it's bad for me" means "no", then "no".

Because of this ambiguity, although it can make sense to say "I'm happy" to indicate "my values are being served", I propose that in the interest of epistemic hygiene it's worth being more specific. Conflating feelings-as-behaviors with feelings-as-statements inflicts a great deal of pondering and confusion about
whether feelings are rational (also precipitated by
Hollywood), and to make matters worse, each of these similes has only limited validity:

1) A feeling is a behavior only insofar as you have control over it. This is something perhaps to strive for, but which certainly varies in feasibility. If someone carefully injects you with dopamine at a funeral, you might feel happy. That doesn't mean you've made an instrumentally irrational choice. 2) A feeling is a statement only insofar as it has a given interpretation. In my opening example, the happiness might rather signify "There's nothing you can do about this so you can relax and move on." This might be clarified on reflection, and if the statement isn't a rational one, you might consider retraining your feeling to offer more rational suggestions. But then, as with any statement, your epistemic rati...
Addiction
by KatjaGrace — 2010-03-11T18:51:43.000Z

Imagine a strange genie offers you the opportunity to no longer crave food, drink, sleep, warmth or sex. Are you be interested? I wouldn’t be – getting things I need and crave seems to be more fun on the whole than getting things I just casually enjoy. There’s a pretty steep diminishing return to the things I listed though, and like many people I have about as much as I want of most of them. So how to make my life much better?

An obvious strategy is to acquire more such strong desires. This is an unpopular path though. If you want to enjoy life more it is common to look for new casually pleasurable activities – make new friends, try a new sport, take your partner on an unusual romantic excursion. If you become too attached to an activity, you are deemed ‘addicted’ though, which is a bad thing. There are activities that are known to make people particularly addicted, and beginning them is seen as a stupid move. But why is addiction so bad?

There is an argument that addictions don’t make you happy – they entrap you in a cycle of endless obsession with no real satisfaction. Remember enjoyment and desire don’t always coincide. But while that’s surely true for some things people become addicted to, if addiction merely entails strong ongoing craving, why should the subjects of such cravings tend to be unpleasant more so than less serious desires?

For these reasons I recently set out to acquire an addiction to coffee, a not particularly dangerous drug. It’s not very strong yet; I do alright without coffee for long periods, but feel relieved and invigorated for having it. So far this seems a great benefit, compared both to casually and unaffectedly drinking coffee and to not drinking coffee at all. However if I say to people that I’m somewhat addicted to coffee they usually express pity. If I say I think it’s a fine arrangement they seem to think I’m silly.

Why is addiction bad, beyond the few specific addictions that aren’t pleasurable or to lead to externalities such as weaponed thievery?