trentmkelly/LessWrong-43k huggingface dataset:
Localized theories and conditional complexity
by jimmy — 2009-10-19T07:29:34.468Z

Suppose I hand you a series of data points without providing the context. Consider the theory v = a*t for t<<1, v = b for t>>1. Without knowing anything a priori about the shapes of the curves, one must have enough data to make sure that v follows the right lines at the two limits since there is complexity that must be justified. Here we have two one-parameter curves, so we need at least two data points to pick the right slope and offset, as well as at least a couple more to make sure it follows the right shape.

This is what I’ll call a completely local theory – see data, fit curve. Dealing with problems at this level does not leave much room for human bias or error, but it also does not allow for improvement by including background knowledge.

Now consider the case where v = velocity of a rocket sled, a = thrust/mass of sled, and b = sqrt(thrust/(1/2*rho*Cd*Af)). If you have a theory explaining rockets and aerodynamics, the equation v=a*t and v = b are just the limiting cases for small and large t. In this case, you only need two data points to find a and b since you already know the shape (over the full range) from solving the differential equations. If you understand the aerodynamics well enough, and know the shape and mass of the sled, you don’t even need to do the experiment! The “conditional complexity” is 0 since it is directly predicted from what we already know. This is the magic of keeping track of the dependencies between theories.

We can take this a step further and derive a theory of aerodynamics from a theory of air molecules- and so on until we have one massively connected TOE.

Now step back to the beginning. If all I tell you is that when t = 1e-5, v = 2e-5 and when t = 1e-3, v = 2e-3, you’re going to come up with the equationv = 2*t. If someone, with no further information, suggested that v = 2*t was only a small t approximation, and that for large t, v = 5.32, you’d think that he’s nutso (and rightfully so), with all that unnecessary complexity.

As a wannabe Bayesian, you need to update on all evidence, so we're almost never trying to fit data without knowing what it means. We prefer globally simple theories, not theories where each local section is simple but they don't want to fit together.

I suspect that one of the main reasons people fai...
Why will we be extra wrong about AI values?
by KatjaGrace — 2009-10-18T23:12:53.000Z

I recently discussed the unlikelihood of an AI taking off and leaving the rest of society behind. The other part I mentioned of Singularitarian concern is that powerful AIs will be programmed with the wrong values. This would be bad even if the AIs did not take over the world entirely, but just became a powerful influence. Is that likely to happen?

Don’t get confused by talk of ‘values’. When people hear this they often think an AI could fail to have values at all, or that we would need to work out how to give an AI values. ‘Values’ just means what the AI does. In the same sense your refrigerator might value making things inside it cold (or for that matter making things behind it warm). Every program you write has values in this sense. It might value outputting ‘#t’ if and only if it’s given a prime number for instance.

The fear then is that a super-AI will do something other than what we want. We are unfortunately picky, and most things other than what we want, we really don’t want. Situations such as being enslaved by an army of giant killer robots, or having your job taken by a simulated mind are really incredibly close to what you do want compared to situations such as your universe being efficiently remodeled into stationery. If you have a machine with random values and the ability to manipulate everything in the universe, the chance of it’s final product having humans and tea and crumpets in it is unfathomably unlikely. Some SIAI members seem to believe that almost anyone who manages to make a powerful general AI will be so incapable of giving it suitable values as to approximate a random selection from mind design space.

The fear is not that whoever picks the AI’s goals will do so at random, but rather that they won’t forsee the extent of the AI’s influence, and will pick narrow goals that may as well be random when they act on the world outside the realm they were intended. For instance an AI programmed to like finding really big prime numbers might find methods that are outside the box, such as hacking computers to covertly divert others’ computing power to the task. If it improves its own intelligence immensely and copies itself we might quickly find ourselves amongst a race of superintelligent creatures whose only value is to find prime numbers. The first thing they would presumably do...
How does raising awareness stop prejudice?
by KatjaGrace — 2009-10-17T06:00:46.000Z

Imagine you are in the habit of treating people who have lesser social status as if they are below you. One day you hear an advertisement talking about a group of people you know nothing about. It’s main thrust is that these people are as good as everyone else, or perhaps even special in some ways which the advertisement informs you are good, and that therefore you should respect them.

ANTaR informs us that Aboriginals do not get enough respect

What do you infer?

These people are totally normal except for being special in various exciting ways, and you should respect them.

These people are so poorly respected by others that somebody feels the need to buy advertising to rectify the situation.

What about the next day when you hear that other employers are going to court for failing to employ these people enough?

I can’t think of any better way to stop people wanting to associate with someone than by suggesting to them that nobody else wants to. Low social status seems like the last thing you can solve by raising awareness.
How far can AI jump?
by KatjaGrace — 2009-10-16T17:39:47.000Z

I went to the Singularity Summit recently, organized by the Singularity Institute for Artificial Intelligence (SIAI). SIAI’s main interest is in the prospect of a superintelligence quickly emerging and destroying everything we care about in the reachable universe. This concern has two components. One is that any AI above ‘human level’ will improve its intelligence further until it takes over the world from all other entities. The other is that when the intelligence that takes off is created it will accidentally have the wrong values, and because it is smart and thus very good at bringing about what it wants, it will destroy all that humans value. I disagree that either part is likely. Here I’ll summarize why I find the first part implausible, and there I discuss the second part.

The reason that an AI – or a group of them – is a contender for gaining existentially risky amounts of power is that it could trigger an intelligence explosion which happens so fast that everyone else is left behind. An intelligence explosion is a positive feedback where more intelligent creatures are better at improving their intelligence further.

Such a feedback seems likely. Even now as we gain more concepts and tools that allow us to think well we use them to make more such understanding. AIs fiddling with their architecture don’t seem fundamentally different. But feedback effects are easy to come by. The question is how big this feedback effect will become. Will it be big enough for one machine to permanently overtake the rest of the world economy in accumulating capability?

In order to grow more powerful than everyone else you need to get significantly ahead at some point. You can imagine this could happen either by having one big jump in progress or by having slightly more growth over a long period of time. Having slightly more growth over a long period is staggeringly unlikely to happen by chance, so it needs to share some cause too. Anything that will give you higher growth for long enough to take over the world is a pretty neat innovation, and for you to take over the world everyone else has to not have anything close. So again, this is a big jump in progress. So for AI to help a small group take over the world, it needs to be a big jump.

Notice that no jumps have been big enough before in human invention. Some species, such as humans, have mostly ...
Waterloo, ON, Canada Meetup: 6pm Sun Oct 18 '09!
by Eliezer Yudkowsky — 2009-10-15T04:30:49.778Z

Michael Vassar and I will be attending the Quantum to Cosmos Festival of the Perimeter Institute. Is anyone interested in meeting up at the Symposium Cafe on 4 King St N, Waterloo, ON, Canada, on Sunday at 6pm on October 18th 2009? I might duck out at around 8pm, but Michael Vassar seems more likely to stick around. If we get at least two more positive reply comments (plus the one person who suggested the meetup) then it'll be on and I'll take the question mark off the title. If that time doesn't work for you but you're in the area, feel free to email me about meeting informally.

Result: Okay, we have exactly two people RSVPing (I was hoping for three). We'll show up at the Symposium Cafe at 6pm, possibly walk around or head out to elsewhere by 6:30pm (i.e., join the meetup by 6:30pm if you want to be part of the walking-around group), and we've both got a talk we want to attend at 8:00pm. Either stuff will happen, or not.
Information theory and FOOM
by PhilGoetz — 2009-10-14T16:52:35.107Z

Information is power. But how much power? This question is vital when considering the speed and the limits of post-singularity development. To address this question, consider 2 other domains in which information accumulates, and is translated into an ability to solve problems: Evolution, and science.

DNA Evolution

Genes code for proteins. Proteins are composed of modules called "domains"; a protein contains from 1 to dozens of domains. We classify genes into gene "families", which can be loosely defined as sets of genes that on average share >25% of their amino acid sequence and have a good alignment for >75% of their length. The number of genes and gene families known doubles every 28 months; but most "new" genes code for proteins that recombine previously-known domains in different orders.

Almost all of the information content of a genome resides in the amino-acid sequence of its domains; the rest mostly indicates what order to use domains in individual genes, and how genes regulate other genes. About 64% of domains (and 84% of those found in eukaryotes) evolved before eukaryotes split from prokaryotes about 2 billion years ago. (Michael Levitt, PNAS July 7 2009, "Nature of the protein universe"; D. Yooseph et al. "The Sorcerer II global ocean sampling expedition", PLoS Bio 5:e16.) (Prokaryotes are single-celled organisms lacking a nucleus, mitochondria, or gene introns. All multicellular organisms are eukaryotes.)

It's therefore accurate to say that most of the information generated by evolution was produced in the first one or two billion years; the development of more-complex organisms seems to have nearly stopped evolution of protein domains. (Multi-cellular organisms are much larger and live much longer; therefore there are many orders of magnitude fewer opportunities for selection in a given time period.) Similarly, most evolution within eukaryotes seems to have occurred during a period of about 50 million years leading up to the Cambrian explosion, half a billion years ago.

My first observation is that evolution has been slowing down in information-theoretic terms, while speeding up in terms of the intelligence produced. This means that adding information to the gene pool increases the effective intelligence that can be produced using that information by a more-than-lin...
The Shadow Question
by Alicorn — 2009-10-14T01:40:56.490Z

This is part 2 of a sequence on problem solving. Here's part 1, which introduces the vocabulary of "problems" versus "tasks". This post's title is a reference1 worth 15 geek points if you get it without Googling, and 20 if you can also get it without reading the rest of the post.

You have to be careful what you wish for. You can't just look at a problem, say "That's not okay," and set about changing the world to contain something, anything, other than that. The easiest way to change things is usually to make them worse. If I owe the library fifty cents that I don't have lying around, I can't go, "That's not okay! I don't want to owe the library fifty cents!" and consider my problem solved when I set the tardy book on fire and now owe them, not money, but a new copy of the book. Or you could make things, not worse in the specific domain of your original problem, but bad in some tangentially related department: I could solve my library fine problem by stealing fifty cents from my roommate and giving it to the library. I'd no longer be indebted to the library. But then I'd be a thief, and my roommate might find out and be mad at me. Calling that a solution to the library fine problem would be, if not an outright abuse of the word "solution", at least a bit misleading.

So what kind of solutions are we looking for? How do we answer the Shadow Question? It's hard to turn a complex problem into doable tasks without some idea of what you want the world to look like when you've completed those tasks. You could just say that you want to optimize according to your utility function, but that's a little like saying that your goal is to achieve your goals: no duh, but now what? You probably don't even know what your utility function is; it's not a luminous feature of your mind.

For little problems, the answer to the Shadow Question may not be complete. For instance, I have never before thought to mentally specify, when making a peanut butter sandwich, that I'd prefer that my act of sandwich-making not lead to the destruction of the Everglades. But it's complete enough. The Everglades aren't close enough to my sandwich for me to think they're worth explicitly acting to protect, even now that Everglades-destruction has occurred to me as an undesirable potential side effect.&nb...
PredictionBook.com - Track your calibration
by Eliezer Yudkowsky — 2009-10-14T00:08:43.863Z

Our hosts at Tricycle Developments have created PredictionBook.com, which lets you make predictions and then track your calibration - see whether things you assigned a 70% probability happen 7 times out of 10.

The major challenge with a tool like this is (a) coming up with good short-term predictions to track (b) maintaining your will to keep on tracking yourself even if the results are discouraging, as they probably will be.

I think the main motivation to actually use it, would be rationalists challenging each other to put a prediction on the record and track the results - I'm going to try to remember to do this the next time Michael Vassar says "X%" and I assign a different probability. (Vassar would have won quite a few points for his superior predictions of Singularity Summit 2009 attendance - I was pessimistic, Vassar was accurate.)
We're in danger. I must tell the others...
by AllanCrossman — 2009-10-13T23:06:46.947Z

... Oh, no! I've been shot!

— C3PO

A strange sort of paralysis can occur when risk-averse people (like me) decide that we're going to play it safe. We imagine the worst thing that could happen if we go ahead with our slightly risky plan, and this stops us from carrying it out.

One possible way of overcoming such paralysis is to remind yourself just how much danger you're actually in.

Humanity could be mutilated by nuclear war, biotechnology disasters, societal meltdown, environmental collapse, oppressive governments, disagreeable AI, or other horrors. On an individual level, anybody's life could turn sour for more mundane reasons, from disease to bereavement to divorce to unemployment to depression. The terrifying scenarios depend on your values, and differ from person to person. Those here who hope to live forever may die of old age, and then cryonics turns out not to work.

There must be some number X which is the probability of Really Bad Things happening to you. X is probably not a tiny figure, but instead significantly above zero, which encourages you to go ahead with whatever slightly risky plan you were contemplating, as long as it only nudges X upwards a little.

Admittedly, this tactic seems like a cheap hack that relies on an error in human reasoning - is nudging your danger level from .2 to .201 actually more acceptable than nudging it from 0 to .001? Perhaps not. Needless to say, a real rationalist ought to ignore all this and take the action with the highest expected value.
The power of information?
by PhilGoetz — 2009-10-13T01:07:35.908Z

I'm thinking about how to model an ecosystem of recursively self-improving computer programs. The model I have in mind assumes finite CPU cycles/second and finite memory as resources, and that these resources are already allocated at time zero. It models the rate of production of new information by a program given its current resources of information, CPU cycles, and memory; the conversion of information into power to take resources from other programs; and a decision rule by which a program chooses which other program to take resources from. The objective is to study the system dynamics, in particular looking for attractors and bifurcations/catastrophes, and to see what range of initial conditions don't lead to a singleton.

(A more elaborate model would also represent the fraction of ownership one program had of another program, that being a weight to use to blend the decision rules of the owning programs with the decision rule of the owned program. It may also be desirable to model trade of information. I think that modeling Moore's law wrt CPU speed and memory size would make little difference, if we assume the technologies developed would be equally available to all agents. I'm interested in the shapes of the attractors, not the rate of convergence.)

Problem: I don't know how to model power as a function of information.

I have a rough model of how information grows over time; so I can estimate the relative amounts of information in a single real historical society at two points in time. If I can say that society X had tech level T at time A, and society Y had tech level T at time B, I can use this model to estimate what tech level society Y had at time A.

Therefore, I can gather historical data about military conflicts between societies at different tech levels, estimate the information ratio between those societies, and relate it to the manpower ratios between the armies involved and the outcome of the conflict, giving a system of inequalities.

You can help me in 3 ways:

Tell me why this is a bad idea.

Tell me some better way to relate information to power.

Gather historical data of the type I just described.

If you choose the last option, choose a historical conflict between sides of uneven tech level, and post here as many as you can find of the following details:

Identifying details: Year, opponents involved, n

...
Anticipation vs. Faith: At What Cost Rationality?
by Wei Dai — 2009-10-13T00:10:47.818Z

Anticipation and faith are both aspects of the human decision process, in a sense just subroutines of a larger program, but they also generate subjective experiences (qualia) that we value for their own sake. Suppose you ask a religious friend why he doesn’t give up religion, he might say something like “Having faith in God comforts me and I think it is a central part of the human experience. Intellectually I know it’s irrational, but I want to keep my faith anyway. My friends and the government will protect me from making any truly serious mistakes as a result of having too much faith (like falling into dangerous cults or refusing to give medical treatment to my children)."

Personally I've never been religious, so this is just a guess of what someone might say. But these are the kinds of thoughts I have when faced with the prospect of giving up the anticipation of future experiences (after being prompted by Dan Armak). We don't know for sure yet that anticipation is irrational, but it's hard to see how it can be patched up to work in an environment where mind copying and merging are possible, and in the mean time, we have a decision theory (UDT) that seems to work fine, but does not involve any notion of anticipation.

What would you do if true rationality requires giving up something even more fundamental to the human experience than faith? I wonder if anyone is actually willing to take this step, or is this the limit of human rationality, the end of a short journey across the space of possible minds?
Do the 'unlucky' systematically underestimate high-variance strategies?
by MBlume — 2009-10-12T22:27:37.461Z

From the UK Telegraph:

A decade ago, I set out to investigate luck. I wanted to examine the impact on people's lives of chance opportunities, lucky breaks and being in the right place at the right time. After many experiments, I believe that I now understand why some people are luckier than others and that it is possible to become luckier.

To launch my study, I placed advertisements in national newspapers and magazines, asking for people who felt consistently lucky or unlucky to contact me. Over the years, 400 extraordinary men and women volunteered for my research from all walks of life: the youngest is an 18-year-old student, the oldest an 84-year-old retired accountant.

Be lucky -- it's an easy skill to learn

On reading the article, the takeaway message seems to be that the 'unlucky' systematically fail to take advantage of high-expected-but-low-median value opportunities.
What Program Are You?
by RobinHanson — 2009-10-12T00:29:19.218Z

I've been trying for a while to make sense of the various alternate decision theories discussed here at LW, and have kept quiet until I thought I understood something well enough to make a clear contribution. Here goes.

You simply cannot reason about what to do by referring to what program you run, and considering the other instances of that program, for the simple reason that: there is no unique program that corresponds to any physical object.

Yes, you can think of many physical objects O as running a program P on data D, but there are many many ways to decompose an object into program and data, as in O = <P,D>. At one extreme you can think of every physical object as running exactly the same program, i.e., the laws of physics, with its data being its particular arrangements of particles and fields. At the other extreme, one can think of each distinct physical state as a distinct program, with an empty unused data structure. Inbetween there are an astronomical range of other ways to break you into your program P and your data D.

Eliezer's descriptions of his "Timeless Decision Theory", however refer often to "the computation" as distinguished from "its input" in this "instantiation" as if there was some unique way to divide a physical state into these two components. For example:

The one-sentence version is: Choose as though controlling the logical output of the abstract computation you implement, including the output of all other instantiations and simulations of that computation.
The three-sentence version is: Factor your uncertainty over (impossible) possible worlds into a causal graph that includes nodes corresponding to the unknown outputs of known computations; condition on the known initial conditions of your decision computation to screen off factors influencing the decision-setup; compute the counterfactuals in your expected utility formula by surgery on the node representing the logical output of that computation.

And also:

Timeless decision theory, in which the (Godelian diagonal) expected utility formula is written as follows: Argmax[A in Actions] in Sum[O in Outcomes](Utility(O)*P(this computation yields A []-> O|rest of universe)) ... which is why TDT one-boxes on Newcomb's Problem - both your current self's physical act, and Omega's physical act in the past, are logical-causal descend...