trentmkelly/LessWrong-43k huggingface dataset:
AI indifference through utility manipulation
by Stuart_Armstrong — 2010-09-02T17:06:12.574Z

Indifference is a precious and rare commodity for complex systems. The most likely effect of making a change in an intricate apparatus is a whole slew of knock-on effects crowned with unintended consequences. It would be ideal if one could make a change and be sure that the effects would remain isolated - that the rest of the system would be indifferent to the change.

For instance, it might be a sensible early-AI precaution to have an extra observer somewhere, sitting with his hand upon a button, ready to detonate explosives should the AI make a visible power grab. Except, of course, the AI will become aware of this situation, and will factor it in in any plans it makes, either by increasing its deception or by grabbing control of the detonation system as a top priority. We would be a lot safer if the AI were somehow completely indifferent to the observer and the explosives. That is a complex wish that we don't really know how to phrase; let's make it simpler, and make it happen.

Assume that there is an AI that is implementing a utility function U, and that is faces the choice between cooperating with humans or defecting and taking over. There is a single human observer H that will press a button in one hour's time if it perceives the AI defecting. The button will activate a quantum observation X that will output 1 with probability p and 0 with probability (1-p). If X outputs 1, explosives will shatter the AI (this p is important, and we'll see later why we don't simply make the explosives go off with certainty).

Now the AI has the choice between cooperating, defecting while taking control of human H (by physical or psychological means, or simply by distracting vim), or defecting without taking control of H. Call these C, D+H and D-H. There are four possible outcomes: AI in control (A), Business as usual (B) with the AI cooperating, Explosion (E), and AI Failure to take over for non-explosive reasons (F).

Give the AI a 1% chance of F if it implements D-H, and a 2% chance of F if it implements D+H (extra restrictions increase the risk of failure). So the table of possible outcomes looks like this:

Decision:
Outcomes
Probability

C

B

1

D+H

F

0.02

A
0.98

D-H
F
0.01

E
0.99(p)

A
0.99(1-p)

Is there a way to make the AI indifferent to the explosives - which we can interpret as being indifferent to the value of p? If U(E)=U(A), then this would be the case...
Rationality quotes: September 2010
by Morendil — 2010-09-01T06:53:51.432Z

This is our monthly thread for collecting these little gems and pearls of wisdom, rationality-related quotes you've seen recently, or had stored in your quotesfile for ages, and which might be handy to link to in one of our discussions.

Please post all quotes separately, so that they can be voted up/down separately. (If they are strongly related, reply to your own comments. If strongly ordered, then go ahead and post them together.)

Do not quote yourself.

Do not quote comments/posts on LW/OB.

No more than 5 quotes per person per monthly thread, please.
Berkeley LW Meet-up Sunday September 5
by LucasSloan — 2010-09-01T02:47:23.293Z

So it has come to my attention that there are a lot of LWers in and around Berkeley, so I thought it might be nice if we all got together and shot the breeze for a couple hours. I think it would be good to meet at the Starbucks at 2224 Shattuck Avenue at 7 o'clock on September 5th.

ETA: We will be meeting tonight at 7, disregard all comments suggesting a date change.
Dreams of AIXI
by jacob_cannell — 2010-08-30T22:15:04.520Z

Implications of the Theory of Universal Intelligence

If you hold the AIXI theory for universal intelligence to be correct; that it is a useful model for general intelligence at the quantitative limits, then you should take the Simulation Argument seriously.

AIXI shows us the structure of universal intelligence as computation approaches infinity. Imagine that we had an infinite or near-infinite Turing Machine. There then exists a relatively simple 'brute force' optimal algorithm for universal intelligence.

Armed with such massive computation, we could just take all of our current observational data and then use a particular weighted search through the subspace of all possible programs that correctly predict this sequence (in this case all the data we have accumulated to date about our small observable slice of the universe). AIXI in raw form is not computable (because of the halting problem), but the slightly modified time limited version is, and this is still universal and optimal.

The philosophical implication is that actually running such an algorithm on an infinite Turing Machine would have the interesting side effect of actually creating all such universes.

AIXI’s mechanics, based on Solomonoff Induction, bias against complex programs with an exponential falloff ( 2^-l(p) ), a mechanism similar to the principle of Occam’s Razor. The bias against longer (and thus more complex) programs, lends a strong support to the goal of String Theorists, who are attempting to find a simple, shorter program that can unify all current physical theories into a single compact description of our universe. We must note that to date, efforts towards this admirable (and well-justified) goal have not born fruit. We may actually find that the simplest algorithm that explains our universe is more ad-hoc and complex than we would desire it to be. But leaving that aside, imagine that there is some relatively simple program that concisely explains our universe.

If we look at the history of the universe to date, from the Big Bang to our current moment in time, there appears to be a clear local telic evolutionary arrow towards greater X, where X is sometimes described as or associated with: extropy, complexity, life, intelligence, computation, etc etc. Its also fairly clear that X (however quantified) is an exponential...
Morality as Parfitian-filtered Decision Theory?
by SilasBarta — 2010-08-30T21:37:22.051Z

Non-political follow-up to: Ungrateful Hitchhikers (offsite)

Related to: Prices or Bindings?, The True Prisoner's Dilemma

Summary: Situations like the Parfit's Hitchhiker problem select for a certain kind of mind: specifically, one that recognizes that an action can be optimal, in a self-interested sense, even if it can no longer cause any future benefit. A mind that can identify such actions might put them in a different category which enables it to perform them, in defiance of the (futureward) consequentialist concerns that normally need to motivate it. Our evolutionary history has put us through such "Parfitian filters", and the corresponding actions, viewed from the inside, feel like "something we should do", even if we don’t do it, and even if we recognize the lack of a future benefit. Therein lies the origin of our moral intuitions, as well as the basis for creating the category "morality" in the first place.

Introduction: What kind of mind survives Parfit's Dilemma?

Parfit's Dilemma – my version – goes like this: You are lost in the desert and near death. A superbeing known as Omega finds you and considers whether to take you back to civilization and stabilize you. It is a perfect predictor of what you will do, and only plans to rescue you if it predicts that you will, upon recovering, give it $0.01 from your bank account. If it doesn’t predict you’ll pay, you’re left in the desert to die. [1]

So what kind of mind wakes up from this? One that would give Omega the money. Most importantly, the mind is not convinced to withhold payment on the basis that the benefit was received only in the past. Even if it recognizes that no future benefit will result from this decision -- and only future costs will result -- it decides to make the payment anyway.

If a mind is likely to encounter such dilemmas, it would be an advantage to have a decision theory capable of making this kind of "un-consequentialist" decision. And if a decision theory passes through time by being lossily stored by a self-replicating gene (and some decompressing apparatus), then only those that shift to encoding this kind of mentality will be capable of propagating themselves through Parfit's Hitchhiker-like scenarios (call these scenarios "Parfitian filte...
Harry Potter and the Methods of Rationality discussion thread, part 3
by Unnamed — 2010-08-30T05:37:32.615Z

Update: This post has also been superseded - new comments belong in the latest thread.

The second thread has now also exceeded 500 comments, so after 42 chapters of MoR it's time for a new thread.

From the first thread:

Spoiler Warning: this thread contains unrot13'd spoilers for Harry Potter and the Methods of Rationality up to the current chapter and for the original Harry Potter series. Please continue to use rot13 for spoilers to other works of fiction, or if you have insider knowledge of future chapters of Harry Potter and the Methods of Rationality.
A suggestion: mention at the top of your comment which chapter you're commenting on, or what chapter you're up to, so that people can understand the context of your comment even after more chapters have been posted. This can also help people avoid reading spoilers for a new chapter before they realize that there is a new chapter.
Exploitation and cooperation in ecology, government, business, and AI
by PhilGoetz — 2010-08-27T14:27:16.995Z

Ecology

An article in a recent issue of Science (Elisa Thebault & Colin Fontaine, "Stability of ecological communities and the architecture of mutualistic and trophic networks", Science 329, Aug 13 2010, p. 853-856; free summary here) studies 2 kinds of ecological networks: trophic (predator-prey) and mutualistic (in this case, pollinators and flowers). They looked at the effects of 2 properties of networks: modularity (meaning the presence of small, highly-connected subsets that have few external connections) and nestedness (meaning the likelihood that species X has the same sort of interaction with multiple other species). (It's unfortunate that they never define modularity or nestedness formally; but this informal definition is still useful. I'm going to call nestedness "sharing", since they do not state that their definition implies nesting one network inside another.) They looked at the impact of different degrees of modularity and nestedness, in trophic vs. mutualistic networks, on persistence (fraction of species still alive at equilibrium) and resilience (1/time to return to equilibrium after a perturbation). They used both simulated networks, and data from real-world ecological networks.

What they found is that, in trophic networks, modularity is good (increases persistence and resilience) and sharing is bad; while in mutualistic networks, modularity is bad and sharing is good. Also, in trophic networks, species go extinct so as to make the network more modular and less sharing; in mutualistic networks, the opposite occurs.

The commonsense explanation is that, if species X is exploiting species Y (trophic), the interaction decreases the health of species Y; and so having more exploiters of Y is bad for both X and Y. OTOH, if species X benefits from species Y, X will get a secondhand benefit from any mutually-beneficial relationships that Y has; if Y also benefits from X (mutualistic), then neither X nor Y will adapt to prevent Z from also having a mutualistic relationship with Y. (The theory does not address a mixture of trophic and mutualistic interactions in a single network.)

The effect is strong - see this figure:

This shows that, when nodes have exploitative (trophic) relationships, and you simulate evolution starting from a random network, the network almost always becomes more modular and less s...
Cryonics Questions
by James_Miller — 2010-08-26T23:19:43.399Z

Cryonics fills many with disgust, a cognitively dangerous emotion. To test whether a few of your possible cryonics objections are reason or disgust based, I list six non-cryonics questions. Answering yes to any one question indicates that rationally you shouldn’t have the corresponding cryonics objections.

1. You have a disease and will soon die unless you get an operation. With the operation you have a non-trivial but far from certain chance of living a long, healthy life. By some crazy coincidence the operation costs exactly as much as cryonics does and the only hospitals capable of performing the operation are next to cryonics facilities. Do you get the operation?

Answering yes to (1) means you shouldn’t object to cryonics because of costs or logistics.

2. You have the same disease as in (1), but now the operation costs far more than you could ever obtain. Fortunately, you have exactly the right qualifications NASA is looking for in a space ship commander. NASA will pay for the operation if in return you captain the ship should you survive the operation. The ship will travel close to the speed of light. The trip will subjectively take you a year, but when you return one hundred years will have passed on Earth. Do you get the operation?

Answering yes to (2) means you shouldn't object to cryonics because of the possibility of waking up in the far future.

3. Were you alive 20 years ago?

Answering yes to (3) means you have a relatively loose definition of what constitutes “you” and so you shouldn’t object to cryonics because you fear that the thing that would be revived wouldn’t be you.

4. Do you believe that there is a reasonable chance that a friendly singularity will occur this century?

Answering yes to (4) means you should think it possible that someone cryogenically preserved would be revived this century. A friendly singularity would likely produce an AI that in one second could think all the thoughts that would take a billion scientists a billion years to contemplate. Given that bacteria seem to have mastered nanotechnology, it’s hard to imagine that a billion scientists working for a billion years wouldn’t have a reasonable chance of mastering it. Also, a friendly post-singularity AI woul...
Self-fulfilling correlations
by PhilGoetz — 2010-08-26T21:07:28.112Z

Correlation does not imply causation. Sometimes corr(X,Y) means X=>Y; sometimes it means Y=>X; sometimes it means W=>X, W=>Y. And sometimes it's an artifact of people's beliefs about corr(X, Y). With intelligent agents, perceived causation causes correlation.

Volvos are believed by many people to be safe. Volvo has an excellent record of being concerned with safety; they introduced 3-point seat belts, crumple zones, laminated windshields, and safety cages, among other things. But how would you evaluate the claim that Volvos are safer than other cars?

Presumably, you'd look at the accident rate for Volvos compared to the accident rate for similar cars driven by a similar demographic, as reflected, for instance in insurance rates. (My google-fu did not find accident rates posted on the internet, but insurance rates don't come out especially pro-Volvo.) But suppose the results showed that Volvos had only 3/4 as many accidents as similar cars driven by similar people. Would that prove Volvos are safer?

Perceived causation causes correlation

No. Besides having a reputation for safety, Volvos also have a reputation for being overpriced and ugly. Mostly people who are concerned about safety buy Volvos. Once the reputation exists, even if it's not true, a cycle begins that feeds on itself: Cautious drivers buy Volvos, have fewer accidents, resulting in better statistics, leading more cautious drivers to buy Volvos.

Do Montessori schools or home-schooling result in better scores on standardized tests? I'd bet that they do. Again, my google-fu is not strong enough to find any actual reports on, say, average SAT-score increases for students in Montessori schools vs. public schools. But the largest observable factor determining student test scores, last I heard, is participation by the parents. Any new education method will show increases in student test scores if people believe it results in increases in student test scores, because only interested parents will sign up for that method. The crazier, more-expensive, and more-difficult the method is, the more improvement it should show; craziness should filter out less-committed parents.

Are vegetarian diets or yoga healthy for you? Does using the phone while driving increase accident rates? Yes, proba...
Criteria for Rational Political Conversation
by woozle — 2010-08-26T15:53:19.223Z

Query: by what objective criteria do we determine whether a political decision is rational?

I propose that the key elements -- necessary but not sufficient -- are (where "you" refers collectively to everyone involved in the decisionmaking process):

you must use only documented reasoning processes:

use the best known process(es) for a given class of problem

state clearly which particular process(es) you use

document any new processes you use

you must make every reasonable effort to verify that:

your inputs are reasonably accurate, and

there are no other reasoning processes which might be better suited to this class of problem, and

there are no significant flaws in in your application of the reasoning processes you are using, and

there are no significant inputs you are ignoring

If an argument satisfies all of these requirements, it is at least provisionally rational. If it fails any one of them, then it's not rational and needs to be corrected or discarded.
This is not a circular definition (defining "rationality" by referring to "reasonable" things, where "reasonable" depends on people being "rational"); it is more like a recursive algorithm, where large ambiguous problems are split up into smaller and smaller sub-problems until we get to a size where the ambiguity is negligible.
This is not one great moral principle; it is more like a self-modifying working process (subject to rational criticism and therefore improvable over time -- optimization by successive approximation). It is an attempt to apply the processes of science (or at least the same reasoning which arrived at those processes) to political discourse.
So... can we agree on this?

This is a hugely, vastly, mindbogglingly trimmed-down version of what I originally posted. All comments prior to 2010-08-26 20:52 (EDT) refer to that version, which I have reposted here for comparison purposes and for the morbidly curious. (It got voted down to negative 6. Twice.)
The prior of a hypothesis does not depend on its complexity
by cousin_it — 2010-08-26T13:20:05.054Z

Many thanks to Unknowns for inventing the scenario that led to this post, and to Wei Dai for helpful discussion.

Imagine you subscribe to the universal prior. Roughly, this means you assign credence 2^-k to each program of length k whose output matches your sensory inputs so far, and 0 to all programs that failed to match. Does this imply you should assign credence 2^-m to any statement about the universe ("hypothesis") that has length m? or maybe Kolmogorov complexity m?

The answer is no. Consider the following examples:

1. The complexity of "A and B and C and D" is roughly equal to the complexity of "A or B or C or D", but we know for certain that the former hypothesis can never be more probable than the latter, no matter what A, B, C and D are.

2. The hypothesis "the correct theory of everything is the lexicographically least algorithm with K-complexity 3^^^^3" is quite short, but the universal prior for it is astronomically low.

3. The hypothesis "if my brother's wife's first son's best friend flips a coin, it will fall heads" has quite high complexity, but should be assigned credence 0.5, just like its negation.

Instead, the right way to derive a prior over hypotheses from a prior over predictors should be to construct the set of all predictors (world-algorithms) that "match" the hypothesis, and see how "wide" or "narrow" that set is. There's no connection to the complexity of the hypothesis itself.

An exception is if the hypothesis gives an explicit way to construct a predictor that satisfies it. In that case the correct prior for the hypothesis is bounded from below by the "naive" prior implied by length, so it can't be too low. This isn't true for many interesting hypotheses, though. For example the words "Islam is true", even expanded into the complete meanings of these words as encoded in human minds, don't offer you a way to implement or predict an omnipotent Allah, so the correct prior value for the Islam hypothesis is not obvious.

This idea may or may not defuse Pascal's Mugging - I'm not sure yet. Sorry, I was wrong about that, see Spurlock's comment and my reply.
Luminosity (Twilight fanfic) discussion thread
by FAWS — 2010-08-25T08:49:22.156Z

In the vein of the Harry Potter and the Methods of Rationality discussion threads this is the place to discuss anything relating to Alicorn's Twilight fanfic Luminosity. The fanfic is also archived on Alicorn's own website <strike>(warning: white text on black background)</strike>.

Previous discussion is hidden so deeply within the first Methods of Rationality thread that it's difficult to find even if you already know it exists.

Similar to how Eliezer's fanfic popularizes material from his sequences Alicorn is using the insights from her Luminosity sequence.

Spoilers for the fanfic itself as well as the original novels need and should not be hidden, but spoiler protection still applies for any other works of fiction, except for Harry Potter and the Methods of Rationality chapters more than a week old so we can freely discuss similarities and differences.

EDIT: Post-ginormous-spoiler discussion should go to the second thread. (If you have any doubt on whether you have reached the spoiler in question you have not.)
Burning Man Meetup: Bayes Camp
by Larks — 2010-08-25T06:14:54.005Z

In celebration of the virtues of applied rationality, Less Wrong is going to Burning Man! And because Heinlein rationalists should win, Bayes Camp is going to be the most awesome place there.

A bunch of people from SingInst/Less Wrong will be descending upon the desert, bedecked as the members of the Bayesian Conspiracy. Kevin, Jasen, JustinShovelain, Peter de Blanc, Michael Vassar and Nick Tarleton, among others, will be there. If you'd like to stop by, say so in the comments!

We'll be at 6:50, F, and should be there from Monday 30th.

Please note: Burning Man is serious stuff, and if you don’t think you’re up to the desert, you shouldn’t come. Either way, read the survival guide.

EDIT: updated location