trentmkelly/LessWrong-43k huggingface dataset:
Re-formalizing PD
by cousin_it — 2009-04-28T12:10:30.070Z

The Prisoner's Dilemma has been discussed to death here on OB/LW, right? Well, here's a couple new twists to somewhat... uh... expand the discussion.

Warning: programming and math ahead.

Scenario 1

Imagine a PD tournament between programs that can read each other's source code. In every match, player A receives the source code of player B as an argument, and vice versa. Matches are one-shot, not iterated.

In this situation it's possible to write a program that's much better than "always defect". Yes, in an ordinary programming language like C or Python, no futuristic superintelligent oracles required. No, Rice's theorem doesn't cause any problems.

Here's an outline of the program:

// begin PREFIX
Strategy main(SourceCode other)
{
// get source code of this program from "begin PREFIX" to "end PREFIX",
// using ordinary quine (self-printing program) techniques
String PREFIX = "..."

if (other.beginsWith(PREFIX))
return Strategy.COOPERATE;
else
return anythingElse(other);
}
// end PREFIX

// from this point you can write anything you wish
Strategy anythingElse(SourceCode other)
{
return Strategy.DEFECT;
}

Some features of this program:

It cooperates with itself.

It cooperates with any other program that begins with PREFIX.

It's impossible to cheat, because opponents that begin with PREFIX can't not cooperate with this program.

Other authors now have an incentive to include PREFIX in their programs, moving their original logic into the "anythingElse" subroutine. This modification has no downside.

So, introducing such a program into the tournament should lead to a chain reaction until everyone cooperates. Unless I've missed something. What say ye?

Edit: the last point and the conclusion were wrong. Thanks to Warrigal for pointing this out.

Scenario 2

Now imagine another tournament where programs can't read each other's source code, but are instead given access to a perfect simulator. So programs now look like this:

Strategy main(ObjectCode self, ObjectCode other, Simulator simulator) {...}

and can call simulator.simulate(ObjectCode a, ObjectCode b) arbitrarily many times with any arguments. To give ...
What is control theory, and why do you need to know about it?
by Richard_Kennaway — 2009-04-28T09:25:48.139Z

This is long, but it's the shortest length I could cut from the material and have a complete thought.

1. Alien Space Bats have abducted you.

In the spirit of this posting, I shall describe a magical power that some devices have. They have an intention, and certain means available to achieve that intention. They succeed in doing so, despite knowing almost nothing about the world outside. If you push on them, they push back. Their magic is not invincible: if you push hard enough, you may overwhelm them. But within their limits, they will push back against anything that would deflect them from their goal. And yet, they are not even aware that anything is opposing them. Nor do they act passively, like a nail holding something down, but instead they draw upon energy sources to actively apply whatever force is required. They do not know you are there, but they will struggle against you with all of their strength, precisely countering whatever you do. It seems that they have a sliver of that Ultimate Power of shaping reality, despite their almost complete ignorance of that reality. Just a sliver, not a whole beam, for their goals are generally simple and limited ones. But they pursue them relentlessly, and they absolutely will not stop until they are dead.

You look inside one of these devices to see how it works, and imagine yourself doing the same task...

Alien Space Bats have abducted you. You find yourself in a sealed cell, featureless but for two devices on the wall. One seems to be some sort of meter with an unbreakable cover, the needle of which wanders over a scale marked off in units, but without any indication of what, if anything, it is measuring. There is a red blob at one point on the scale. The other device is a knob next to the meter, that you can turn. If you twiddle the knob at random, it seems to have some effect on the needle, but there is no fixed relationship. As you play with it, you realise that you very much want the needle to point to the red dot. Nothing else matters to you. Probably the ASBs' doing. But you do not know what moves the needle, and you do not know what turning the knob actually does. You know nothing of what lies outside the cell. There is only the needle, the red dot, and the knob. To make matters worse, the red dot also jumps along the scale from time to time, in no particular pattern, and nothing you do seems to have any eff

...
Epistemic vs. Instrumental Rationality: Approximations
by Peter_de_Blanc — 2009-04-28T03:12:55.675Z

What is the probability that my apartment will be struck by a meteorite tomorrow? Based on the information I have, I might say something like 10-18. Now suppose I wanted to approximate that probability with a different number. Which is a better approximation: 0 or 1/2?

The answer depends on what we mean by "better," and this is a situation where epistemic (truthseeking) and instrumental (useful) rationality will disagree.

As an epistemic rationalist, I would say that 1/2 is a better approximation than 0, because the Kullback-Leibler Divergence is (about) 1 bit for the former, and infinity for the latter. This means that my expected Bayes Score drops by one bit if I use 1/2 instead of 10-18, but it drops to minus infinity if I use 0, and any probability conditional on a meteorite striking my apartment would be undefined; if a meteorite did indeed strike, I would instantly fall to the lowest layer of Bayesian hell. This is too horrible a fate to imagine, so I would have to go with a probability of 1/2.

As an instrumental rationalist, I would say that 0 is a better approximation than 1/2. Even if a meteorite does strike my apartment, I will suffer only a finite amount of harm. If I'm still alive, I won't lose all of my powers as a predictor, even if I assigned a probability of 0; I will simply rationalize some other explanation for the destruction of my apartment. Assigning a probability of 1/2 would force me to actually plan for the meteorite strike, perhaps by moving all of my stuff out of the apartment. This is a totally unreasonable price to pay, so I would have to go with a probability of 0.

I hope this can be a simple and uncontroversial example of the difference between epistemic and instrumental rationality. While the normative theory of probabilities is the same for any rationalist, the sorts of approximations a bounded rationalist would prefer can differ very much.
How Not to be Stupid: Know What You Want, What You Really Really Want
by Psy-Kosh — 2009-04-28T01:11:46.326Z

Previously: Starting Up

So, you want to be rational, huh? You want to be Less Wrong than you were before, hrmmm? First you must pass through the posting titles of a thousand groans. Muhahahahaha!

Let's start with the idea of preference rankings. If you prefer A to B, well, given the choice between A and B, you'd choose A.

For example, if you face a choice between a random child being tortured to death vs them leading a happy and healthy life, all else being equal and the choice costing you nothing, which do you choose?

This isn't a trick question. If you're a perfectly ordinary human, you presumably prefer the latter to the former.

Therefore you choose it. That's what it means to prefer something. That if you prefer A over B, you'd give up situation B to gain situation A. You want situation A more than you want situation B.

Now, if there're many possibilities, you may ask... "But, what if I prefer B to A, C to B, and A to C?"

The answer, of course, is that you're a bit confused about what you actually prefer. I mean, all that ranking would do is just keep you switching between those, looping around.

And if thinking in terms of resources, the universe or an opponent or whatever could, for a small price, sell each of those to you in sequence, draining you of the resource (time, money, whatever) as you go around the vortex of confused desires.

This, of course, translates more precisely into a sequence of states, Ai, Bi, Ci, and preferences of the form A0 < B1 < C2 < A3 < B4 ... where each one of those is the same as the original name except you also have a drop less of the relevant resource as you did before. ie, indicating a willingness to pay the price. If the sequence keeps going all the way, then you'll be drained, and that's a rather inefficient way of going about it if you just want to give the relevant resource up, no? ;)

Still, a strict loop, A > B, B > C, C > A really is an indication that you just don't know what you want. I'll just dismiss that at this point as "not really what I'd call preferences" as such.

Note, however, that it's perfectly okay to have some states of reality, histories of the entire universe, whatever, such that A, B, and C are all ranked equally in your preferences.

If you, however, say something like "I don't prefer A less than B, nor more than B, nor equally to B", I'm just going to give you a very stern look unt...
How Not to be Stupid: Starting Up
by Psy-Kosh — 2009-04-28T00:23:24.171Z

First, don't stand up. ;)

Okay. So what I'm hoping to do in this mini sequence is to introduce a basic argument for Bayesian Decision Theory and epistemic probabilities. I'm going to be basing it on dutch book arguments and Dr. Omohundro's vulnerability based argument, however with various details filled in because, well... I myself had to sit and think about those things, so maybe it would be useful to others too. For that matter, actually writing this up will hopefully sort out my thoughts on this.

Also, I want to try to generalize it a bit to remove the explicit dependancy of the arguments on resources. (Though I may include arguments from that to illustrate some of the ideas.)

Anyways, the spirit of the idea is "don't be stupid." "Don't AUTOMATICALLY lose when there's a better alternative that doesn't risk you losing even worse."

More to the point, repeated application of that idea is going to let us build up the mathematics of decision theory. My plan right now is for each of the posts in this sequence to be relatively short, discussing and deriving one principle or (a couple of related principles) of decision theory and bayesian probability at a time from the above. The math should be pretty simple, with the very worst being potentially a tiny bit of linear algebra. I expect the nastiest bit of math will be one instance of matrix reduction down the line. Everything else ought to be rather straightforward, showing the mathematics of decision theory to be a matter of, as Mr. Smith would say, "inevitability."

Consider this whole sequence a work in progress. If anyone thinks any partcular bits of it could be rewritted more clearly, please speak up! Or at least type up. (But of course, don't stand up. ;))
Verbal Overshadowing and The Art of Rationality
by pangloss — 2009-04-27T23:39:34.940Z

To begin, here are some Fun Psychology Facts:

People who were asked to describe a face after seeing it are worse at recognizing the same face later.

People who are asked to describe a wine after drinking it are worse at recognizing the same wine later.

People who are asked to give reasons for their preferences among a collection of jellies are worse at identifying their own preferences among those jellies.

This effect, known as Verbal Overshadowing, occurs primarily when a principally non-verbal process is disrupted by a task which involves verbalization. The above generalizations (and Verbal Overshadowing effects more generally), do not occur among what we can term "Verbal Experts": individuals who are as good at verbalizing the relevant process as they are at doing it implicitly or automatically. This seems like it will be very important to keep in mind when cultivating our own Rationality.

Here's an oversimplified picture of what this means: We've got an implicit facial recognition process, IFRP, which is pretty good. We've also got a generalized explicit verbal thinking process, GEVTP, which is good for lots of things, but isn't especially good at recognizing faces. Normally, IFRP is in charge of facial recognition, but there are some things we can do, like, trying to put a face into words, that wakes up GEVTP, which then muscles IFRP out of the way, and all of a sudden, we are a lot worse at recognizing faces.

The good news is that GEVTP can be trained. To take the wine case, people who put in the time and effort can become verbal experts about wine. This isn't to say they automatically have better judgments about wine. Rather, it means that their GEVTP is on par with their implicit wine recognition, because it has been trained to do the same quality job as the the implicit process.

As a crude metaphor, imagine the difference between the natural process by which you go about walking, versus having to keep track of each and every instruction that needs to be sent to different joints and muscles if you had to consciously issue each one.

Now, obviously the specific studies mentioned are important for wine tasting, eye-witness identification, or determining one's own jelly preferences, but the phenomenon of Verbal Overshadowing has a much larger, more systematic importance for th Art of Rationality.

Let's ...
Bayesian Cabaret
by Scott Alexander — 2009-04-27T22:29:36.931Z

I'd heard rumors that some leading Bayesians had achieved rank in the Bardic Conspiracy. But I wasn't aware that every two years, some of the world's top statisticians hold a Bayesian Cabaret, full of songs, dances, and skits about Bayesian probability theory.
...no, really. Really. I think my favorite has got to be this one.
YouTube seems to be full of this stuff, including What A Bayesian World and We Didn't Start The Prior. Be sure to also check out some of the recordings and the Bayesian Songbook.
Eliezer's finished his sequences, and it's a new era for Less Wrong. Let's celebrate...Bayesian style!
Final Words
by Eliezer Yudkowsky — 2009-04-27T21:12:02.966Z

Sunlight enriched air already alive with curiosity, as dawn rose on Brennan and his fellow students in the place to which Jeffreyssai had summoned them.

They sat there and waited, the five, at the top of the great glassy crag that was sometimes called Mount Mirror, and more often simply left unnamed. The high top and peak of the mountain, from which you could see all the lands below and seas beyond.

(Well, not all the lands below, nor seas beyond. So far as anyone knew, there was no place in the world from which all the world was visible; nor, equivalently, any kind of vision that would see through all obstacle-horizons. In the end it was the top only of one particular mountain: there were other peaks, and from their tops you would see other lands below; even though, in the end, it was all a single world.)

"What do you think comes next?" said Hiriwa. Her eyes were bright, and she gazed to the far horizons like a lord.

Taji shrugged, though his own eyes were alive with anticipation. "Jeffreyssai's last lesson doesn't have any obvious sequel that I can think of. In fact, I think we've learned just about everything that I knew the beisutsukai masters know. What's left, then -"

"Are the real secrets," Yin completed the thought.

Hiriwa and Taji and Yin shared a grin, among themselves.

Styrlyn wasn't smiling. Brennan suspected rather strongly that Styrlyn was older than he had admitted.

Brennan wasn't smiling either. He might be young, but he kept high company, and had witnesssed some of what went on behind the curtains of the world. Secrets had their price, always, that was the barrier that made them secrets; and Brennan thought he had a good idea of what this price might be.

There was a cough from behind them, at a moment when they had all happened to be looking in any other direction but that one.

As one, their heads turned.

Jeffreyssai stood there, in a casual robe that looked more like glass than any proper sort of mirrorweave.

Jeffreyssai stood there and looked at them, a strange abiding sorrow in those inscrutable eyes.

"Sen...sei," Taji started, faltering as that bright anticipation stumbled over Jeffreyssai's return look. "What's next?"

"Nothing," Jeffreyssai said abruptly. "You're finished. It's done."

Hiriwa, Taji, and Yin all blinked, a perfect synchronized gesture of shock. Then, bef...
The End (of Sequences)
by Eliezer Yudkowsky — 2009-04-27T21:07:52.368Z

This concludes the final sequence on Overcoming Bias / Less Wrong. I have not said everything I wanted to say, but I hope I have said (almost) everything I needed to say. (Such that I actually could say it in these twenty-one months of daily posting, August 2007 through April 2009.)

The project to which Less Wrong is devoted - the art and science and craft of human rationality - is, indeed, important. But the calculus of choosing among altruistic efforts is, in some ways, a calculus of who can take your place. I am more easily replaced here, than elsewhere. And so it has come time for me to begin pulling my focus away from Less Wrong, and turning toward other matters, where I am less easily replaced.

But I do need replacing - or rather, the work that I was doing needs replacing, whether by one person or by many people or by e.g. a karma system.

And so my final sequence was my letter that describes the work that I can already see remaining to be done, gives some advice on how to configure the effort, and warns direly against standard failure modes.

Any idea that can produce great enthusiasm is a dangerous idea. It may be a necessary idea, but that does not make it any less dangerous. I do fear, to a certain extent, that I will turn my focus away, and then find out that someone has picked up the ideas and run with them and gotten it all wrong...

But you can only devote your whole life to one thing at a time. In those ways I have thought to anticipate, at least, I have placed a blocking Go stone or two, and you have been warned.

I am not going to turn my attention away entirely and all at once. My initial plan is to cut back my posting to no more than one post per week.

At some future point, though, there must come a time when I turn my attention entirely away from building rationalism, and focus only on that other task.

So, yes, just to belabor the point - if there's going to be a lasting community, and not just a body of online writing that people occasionally stumble across, it needs to set itself up to run without me.

The last explicit dependency left on me is promoting posts, and I've been mostly doing that based on user voting (though not entirely; my activation threshold is lower for posts I perceive as higher-quality). I plan to start trying to delegate that power to co-editors shortly.

For myself...&nbs...
Theism, Wednesday, and Not Being Adopted
by Alicorn — 2009-04-27T16:49:32.087Z

(Disclaimer: This post is sympathetic to a certain subset of theists. I am not myself a theist, nor have I ever been one. I do not intend to justify all varieties of theism, nor do I intend to justify much in the way of common theistic behavior.)

I'm not adopted. You all believe me, right? How do you think I came by this information, that you're confident in my statement? The obvious and correct answer is that my parents told me so1. Why do I believe them? Well, they would be in a position to know the answer, and they have been generally honest and sincere in their statements to me. A false belief on the subject could be hazardous to me, if I report inaccurate family history to physicians, and I believe that my parents have my safety in mind. I know of the existence of adopted people; the possibility isn't completely absent from my mind - but I believe quite confidently that I am not among those people, because my parents say otherwise.

Now let's consider another example. I have a friend who plans to name her first daughter Wednesday. Wednesday will also not be adopted, but that isn't the part of the example that is important: Wednesday will grow up in Provo, Utah, in a Mormon family in a Mormon community with Mormon friends, classmates, and neighbors, attending an LDS church every week and reading scripture and participating in church activities. It is overwhelmingly likely that she will believe the doctrines of the LDS church, because not only her parents, but virtually everyone she knows will reinforce these beliefs in her. Given the particular nuances of Mormonism as opposed to other forms of Christianity, Wednesday will also be regularly informed that several of these people are in a position to have special knowledge on the subject via direct prayer-derived evidence2 - in much the same way that her parents will have special knowledge of her non-adopted status via direct experience when she wasn't in a state suitable to notice or remember the events. Also, a false belief on the subject could have all kinds of bad consequences - if the Muslims are right, for instance, no doubt Hell awaits Wednesday and her family - so if she also correctly assumes that her parents have her best interests at heart, she'll assume they would do their best to give her accurate information.

Athei...
Excuse me, would you like to take a survey?
by Scott Alexander — 2009-04-26T21:23:53.642Z

Related to: Practical Rationality Questionnaire

Here among this community of prior-using, Aumann-believing rationalists, it is a bit strange that we don't have any good measure of what the community thinks about certain things.
I no longer place much credence in raw majoritarianism: the majority is too uneducated, too susceptible to the Dark Arts, and too vulnerable to cognitive biases. If I had to choose the people whose mean opinion I trusted most, it would be - all of you.
So, at the risk of people getting surveyed-out, I'd like to run a survey on the stuff Anna Salamon didn't. Part on demographics, part on opinions, and part on the interactions between the two.
I've already put up an incomplete rough draft of the survey I'd like to use, but I'll post it here again. Remember, this is an incomplete rough draft survey. DO NOT FILL IT OUT YET. YOUR SURVEY WILL NOT BE COUNTED.
Incomplete rough draft of survey
Right now what I want from people is more interesting questions that you want asked. Any question that you want to know the Less Wrong consensus on. Please post each question as a separate comment, and upvote any question that you're also interested in. I'll include as many of the top-scoring questions as I think people can be bothered to answer.
No need to include questions already on the survey, although if you really hate them you can suggest their un-inclusion or re-phrasing.

Also important: how concerned are you about privacy? I was thinking about releasing the raw data later in case other people wanted to perform their own analyses, but it might be possible to identify specific people if you knew enough about them. Are there any people who would be comfortable giving such data if only one person were to see the data, but uncomfortable with it if the data were publically accessible?