trentmkelly/LessWrong-43k huggingface dataset:
Depth-based supercontroller objectives, take 2
by sbenthall — 2014-09-24T01:25:03.198Z

Thanks for all the helpful comments and discussion around this post about using logical depth as an objective function for a supercontroller to preserve human existence.

As this is work in progress, I was a bit muddled and stood duly corrected on a number of points. I'm writing to submit a new, clarified proposal, with some comments directed at objections.

§1. Proposed objective function

Maximize g(u), where u is a description of the universe, h is a description of humanity (more on this later) at the time when the objective function is set, and G is defined as:

g(u) = D(u) - D(u/h)

where D(x) is logical depth and D(x/y) is relative logical depth of x and y.

§2. A note on terminology

I don't intend to annoy by saying "objective function" and "supercontroller" rather than "utility function" and "superintelligence." Rather, I am using this alternative language deliberately to scope the problem to a related question that is perhaps more well-defined or possibly easier to solve. If I understand correctly, "utility function" refers to any function, perhaps implicit, that characterizes the behavior of an agent. By "objective function", I mean a function explicitly coded as the objective of some optimization process, or "controller". I gather that a "superintelligence" is an agent that is better than a generic human at myriad tasks. I think this raises a ton of definitional issues, so instead I will talk about a "supercontroller", which is just arbitrarily good at achieving its objective.

Saying that a supercontroller is arbitrarily good at achieving an objective is tricky, since it's possible to define functions that are impossible to solve. For example, objective functions that involve incomputable functions like the Halting Problem. In general my sense is that computational complexity is overlooked within the "superintelligence" discourse, which is jarring for me since I come from a more traditional AI/machine learning background where computational complexity is at the heart of everything. I gather that it's assumed that a superintelligence will have such effectively unbounded access to computational resources due to its self-modification that complexity is not a limiting factor. It is in that spirit that I propose an incomputable objective function here. My intention is to get past the function definition problem so that work ca...
Books on consciousness?
by mgg — 2014-09-23T22:28:56.001Z

Does LW have a consensus on which books are worthwhile to read regarding consciousness? I read a small intro (Consciousness: A Very Short Introduction, Susan Blackmore, Oxford University Press), and the summary seems to be "Consciousness is pretty damn weird and no one seems to have much of a handle on it". As a non-technical layman, are there any useful books for me to read on the subject?

(I have started reading Daniel Dennet's Intuition Pumps, and I'm a bit torn. He seems highly respected by good scientists, but I feel that if the book didn't have his name on it, I would be well on my way to dismissing it. Are Dennet's earlier works on consciousness a good read?)
Beyond type 1 vs. type 2 processing: the tri-dimensional way (link)
by RomeoStevens — 2014-09-23T20:49:16.700Z

The System 1/2 schema is a popular and useful meme, but it feels limiting sometimes. I found this new paper interesting:

http://journal.frontiersin.org/Journal/10.3389/fpsyg.2014.00993/full

I'm of two minds about this (hah!). On the one hand, it often does feel like there are sharp divides in mindspace. Something will be understood by system 2 but this understanding does not show up in behavior. I still act as if the thing is not true. Then, by some mysterious process, the thing will "click" and it feels like system 1 really gets it. After this the belief in the thing is reflected in behavior. On the other hand, there are many instances where it does not feel appropriate to divide particular mental habits into either system 1 or 2. Doing math, for instance, seems to strongly have factors of both. My immediate intuition is that the continuous model is more "correct" but that there is quite a bit of clustering in the mindspace. System 1&2 would then simply be large clusters.

Anyway, I'm curious about other people's impressions.

One thing I'm frustrated by is that I don't have a map of proposed schemas. There have been lots of different ones proposed over the centuries, and I don't know of any place where I can find a summary of them, as well as draw links between ones that shared an intellectual lineage. Does anyone know of resources relating to this?
Meetup : Urbana-Champaign: The Steep Approach to Crazytown
by Manfred — 2014-09-23T04:00:32.057Z

Discussion article for the meetup : Urbana-Champaign: The Steep Approach to Crazytown

WHEN:
28 September 2014 02:00:00PM (-0500)

WHERE:
206 S. Cedar, 61801

As was suggested last week, we'll be checking out some exercises from the Scientology community. Making fun of them is, of course, mandatory, but the goal is to find some that are interesting enough to try and then trying them. I don't see how this could go wrong.

Discussion article for the meetup : Urbana-Champaign: The Steep Approach to Crazytown
Meetup : Atlanta September Meetup - Self Awareness
by Adele_L — 2014-09-23T01:02:38.378Z

Discussion article for the meetup : Atlanta September Meetup - Self Awareness

WHEN:
27 September 2014 07:00:00PM (-0400)

WHERE:
2388 Lawrenceville Hwy. Unit L, Decatur, GA 30033

This meetup we'll be discussing self awareness and reflection. We'll be talking about ideas discussed in the Living Luminously sequence.
As usual, there will be snacks and games as well as plenty of interesting conversation! There are cats at the location. Please park in a spot labeled visitor, as parking in the other spots runs the risk of getting towed. Hope to see you then!

Discussion article for the meetup : Atlanta September Meetup - Self Awareness
Superintelligence Reading Group 2: Forecasting AI
by KatjaGrace — 2014-09-23T01:00:29.845Z

This is part of a weekly reading group on Nick Bostrom's book, Superintelligence. For more information about the group, and an index of posts so far see the announcement post. For the schedule of future topics, see MIRI's reading guide.

Welcome. This week we discuss the second section in the reading guide, Forecasting AI. This is about predictions of AI, and what we should make of them.

This post summarizes the section, and offers a few relevant notes, and ideas for further investigation. My own thoughts and questions for discussion are in the comments.

There is no need to proceed in order through this post. Feel free to jump straight to the discussion. Where applicable, page numbers indicate the rough part of the chapter that is most related (not necessarily that the chapter is being cited for the specific claim).

Reading: Opinions about the future of machine intelligence, from Chapter 1 (p18-21) and Muehlhauser, When Will AI be Created?

Summary

Opinions about the future of machine intelligence, from Chapter 1 (p18-21)

AI researchers hold a variety of views on when human-level AI will arrive, and what it will be like.

A recent set of surveys of AI researchers produced the following median dates:

for human-level AI with 10% probability: 2022

for human-level AI with 50% probability: 2040

for human-level AI with 90% probability: 2075

Surveyed AI researchers in aggregate gave 10% probability to 'superintelligence' within two years of human level AI, and 75% to 'superintelligence' within 30 years.

When asked about the long-term impacts of human level AI, surveyed AI researchers gave the responses in the figure below (these are 'renormalized median' responses, 'TOP 100' is one of the surveyed groups, 'Combined' is all of them').

There are various reasons to expect such opinion polls and public statements to be fairly inaccurate.

Nonetheless, such opinions suggest that the prospect of human-level AI is worthy of attention.

When Will AI Be Created?

Predicting when human-level AI will arrive is hard.

The estimates of informed people can vary between a small number of decades and a thousand years.

Different time scales have different policy implications.

Several surveys of AI experts exist, but Muehlhauser suspects sampling bias (e.g. optimistic views being sampled more often) makes such surveys of little use.

...
CEV-tropes
by snarles — 2014-09-22T18:21:51.450Z

As seen in other threads, people disagree on whether CEV exists, and if it does, what it might turn out to be.

It would be nice to try to categorize common speculations about CEV.

1a. CEV doesn't exist, because human preferences are too divergent

1b. CEV doesn't even exist for a single human

1c. CEV does exist, but it results in a return to the status quo

2a. CEV results in humans living in a physical (not virtual reality) utopia

2b. CEV results in humans returning to a more primitive society free of technology

2c. CEV results in humans living together in a simulation world, where most humans do not have god-like power

(the similarity between 2a, 2b, and 2c is that humans are still living in the same world, similar to traditional utopia scenarios)

3. CEV results in a wish for the annihilation of all life, or maybe the universe

4a. CEV results in all humans granted the right to be the god of their own private simulation universe (once we acquire the resources to do so)

4b. CEV can be implemented for "each salient group of living things in proportion to that group's moral weight"

5. CEV results in all humans agreeing to be wireheaded (trope)

6a. CEV results in all humans agreeing to merge into a single being and discarding many of the core features of humankind which have lost their purpose (trope)

6b. CEV results in humans agree to cease their own existence but also creating a superior life form--the outcome is similar to 6a, but the difference is that here, humans do not care about whether they are individually "merged"

7. CEV results in all/some humans willingly forgetting/erasing their history, or being indifferent to preserving history so that it is lost (compatible with all previous tropes)

Obviously there are too many possible ideas (or "tropes") to list, but perhaps we could get a sense of which ones are the most common in the LW community. I leave it to someone else to create a poll supposing they feel they have a close to complete list, or create similar topics for AI risk, etc.

EDIT: Added more tropes, changed #2 since it was too broad: now #2 refers to CEV worlds where humans live in the "same world"
CEV: coherence versus extrapolation
by Stuart_Armstrong — 2014-09-22T11:24:24.932Z

It's just struck me that there might be a tension between the coherence (C) and the extrapolated (E) part of CEV. One reason that CEV might work is that the mindspace of humanity isn't that large - humans are pretty close to each other, in comparison to the space of possible minds. But this is far more true in every day decisions than in large scale ones.

Take a fundamentalist Christian, a total utilitarian, a strong Marxist, an extreme libertarian, and a couple more stereotypes that fit your fancy. What can their ideology tell us about their everyday activities? Well, very little. Those people could be rude, polite, arrogant, compassionate, etc... and their ideology is a very weak indication of that. Different ideologies and moral systems seem to mandate almost identical everyday and personal interactions (this is in itself very interesting, and causes me to see many systems of moralities as formal justifications of what people/society find "moral" anyway).

But now let's more to a more distant - "far" - level. How will these people vote in elections? Will they donate to charity, and if so, which ones? If they were given power (via wealth or position in some political or other organisation), how are they likely to use that power? Now their ideology is much more informative. Though it's not fully determinative, we would start to question the label if their actions at this level seemed out of synch. A Marxist that donated to a Conservative party, for instance, would give us pause, and we'd want to understand the apparent contradiction.

Let's move up yet another level. How would they design or change the universe if they had complete power? What is their ideal plan for the long term? At this level, we're entirely in far mode, and we would expect that their vastly divergent ideologies would be the most informative piece of information about their moral preferences. Details about their character and personalities, which loomed so large at the everyday level, will now be of far lesser relevance. This is because their large scale ideals are not tempered by reality and by human interactions, but exist in a pristine state in their minds, changing little if at all. And in almost every case, the world they imagine as their paradise will be literal hell for the others (and quite possibly for themselves).

To summarise: the human mindspace is much narrower in near m...
Open thread, September 22-28, 2014
by Gunnar_Zarncke — 2014-09-22T05:59:24.963Z

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.
Meetup : Czech's first Meetup Prague
by kotrfa — 2014-09-21T20:07:40.494Z

Discussion article for the meetup : Czech's first Meetup Prague

WHEN:
26 September 2014 07:30:00PM (+0200)

WHERE:
Václavské náměstí 778/14, Praha 110 00

Hello,
this is going to be first meetup I know about in Czech Republic, specially in Prague. Since I don't know any other rationalists here, I'll be waiting in Dobrá Čajovna s.r.o. (tearoom) for at least one hour. Look for young, tall, skinny, brown curly haired guy.

Please contact me if you are interested in a meeting, but the time just doesn't fit you - it will be at least some sign that someone is interested.

Feel free to ask if you have any questions.

I decided to try to make a meetup from time to time to see if someone will ever show up. It's a pity (and shame) that there are no meetups here.

Discussion article for the meetup : Czech's first Meetup Prague
Meetup : West LA—Inflation of Terminology
by OpenThreadGuy — 2014-09-21T16:07:41.241Z

Discussion article for the meetup : West LA—Inflation of Terminology

WHEN:
24 September 2014 07:00:00PM (-0700)

WHERE:
11066 Santa Monica Blvd, Los Angeles, CA

How to Find Us: Go into this Del Taco. We will be in the back room if possible.

Parking is free in the lot out front or on the street nearby.

Discussion: A problem for people trying to be precise is that words tend to take on additional meanings over time, and seldom lose meanings. It is good policy to avoid habits that promote this tendency. We will discuss why this is so, and how to implement this policy in daily conversation.

Recommended Reading:

Avoid Inflationary Use of Terms

Similarity Clusters

Sneaking in Connotations

The Worst Argument in the World

No prior exposure to Less Wrong is required; this will be generally accessible.

Discussion article for the meetup : West LA—Inflation of Terminology
Meetup : Copenhagen September Social Meetup - Botanisk Have
by Ruby — 2014-09-21T11:50:44.225Z

Discussion article for the meetup : Copenhagen September Social Meetup - Botanisk Have

WHEN:
27 September 2014 02:30:00PM (+0200)

WHERE:
Gothersage 128, Copenhagen

G'day all,
Sorry for the delayed announcement, but I'd like to suggest we all hang out at the Botanical Gardens on Saturday.

Meet just inside the main gate, Gothersgade 128 - we'll wait there from 2:30pm-3:00pm. Call 2247-8373 to find us.

Bring food and drinks, picnic blanket.

I imagine we'll do a mixture of walking around the gardens as well as relaxing in one spot. Gardens close at 6:00pm and we might continue on elsewhere afterwards too.

Cheers

Discussion article for the meetup : Copenhagen September Social Meetup - Botanisk Have
In order to greatly reduce X-risk, design self-replicating spacecraft without AGI
by chaosmage — 2014-09-20T20:25:36.802Z

tl/dr: If we'd build a working self-replicating spacecraft, that'd prove we're past the Great Filter. Therefore, certainty we can do that would eliminate much existential risk. It is a potentially highly visible project that gives publicity to reasons not to include AGI. Therefore, serious design work on a self-replicating spacecraft should have a high priority.

I'm assuming you've read Stuart_Armstrong's excellent recent article on the Great Filter. In the discussion thread for that, RussellThor observed:

if we make a simple replicator and have it successfully reach another solar system (with possibly habitable planets) then that would seem to demonstrate that the filter is behind us.

If that is obvious to you, skip to the next subheading.

The evolution from intelligent spacefaring species to producer of self-replicating spacecraft (henceforth SRS, used in the plural) is inevitable, if SRS are possible. This is simply because the matter and negentropy available in the wider universe is a staggeringly vast resource of staggering value. Even species who are unlikely to ever visit and colonize other stars in the form that evolution gave them (this includes us) can make use of these resources. For example, if we could build on (or out of) empty planets supercomputers that receive computation tasks by laser beam and output results the same way, we would be economically compelled to do so simply because those supercomputers could handle computational tasks that no computer on Earth could complete in less than the time it takes that laser beam to travel forth and back. That supercomputer would not need to run even a weak AI to be worth more than the cost of sending the probe that builds it.

Without a doubt there are countless more possible uses for these, shall we say, exoresources. If Dyson bubbles or mind uploads or multistellar hypertelescopes or terraforming are possible, each of these alone create another huge incentive to build SRS. Even mere self-replicating refineries that break up planets into more readily accessible resources for future generations to draw from would be an excellent investment. But the obvious existence of this supercomputer incentive is already reason enough to do it.

All the Great Filter debate boils down to the question of how improbable our existence really is. If we're probable, many intelligent species capable of very basic space tra...
Meetup : Stanford THINK starts weekly meetups this Sunday
by KPier — 2014-09-20T20:00:16.749Z

Discussion article for the meetup : Stanford THINK starts weekly meetups this Sunday

WHEN:
21 September 2014 04:30:00PM (-0700)

WHERE:
Old Union rm 121, Stanford University, Palo Alto

THINK is an effective altruism/rationality group that meets during the school year at Stanford. This is our third year. Most attendees are Stanford students, but anyone is welcome.

This week's discussion topic is rationality habits (from CFAR's checklist); we usually have a few games and structures activities and then allow them to dissolve into unstructured discussion.

Discussion article for the meetup : Stanford THINK starts weekly meetups this Sunday
An introduction to Newcomblike problems
by So8res — 2014-09-20T18:40:25.528Z

This is crossposted from my new blog, following up on my previous post. It introduces the original "Newcomb's problem" and discusses the motivation behind twoboxing and the reasons why CDT fails. content is probably review for most LessWrongers, later posts in the sequence may be of more interest.

Last time I introduced causal decision theory (CDT) and showed how it has unsatisfactory behavior on "Newcomblike problems". Today, we'll explore Newcomblike problems in a bit more depth, starting with William Newcomb's original problem.

The Problem

Once upon a time there was a strange alien named Ω who is very very good at predicting humans. There is this one game that Ω likes to play with humans, and Ω has played it thousands of times without ever making a mistake. The game works as follows:

First, Ω observes the human for a while and collects lots of information about the human. Then, Ω makes a decision based on how Ω predicts the human will react in the upcoming game. Finally, Ω presents the human with two boxes.

The first box is blue, transparent, and contains $1000. The second box is red and opaque.

You may take either the red box alone, or both boxes,

Ω informs the human. (These are magical boxes where if you decide to take only the red one then the blue one, and the $1000 within, will disappear.)

If I predicted that you would take only the red box, then I filled it with $1,000,000. Otherwise, I left it empty. I have already made my choice,

Ω concludes, before turning around and walking away.

You may take either only the red box, or both boxes. (If you try something clever, like taking the red box while a friend takes a blue box, then the red box is filled with hornets. Lots and lots of hornets.) What do you do?

The Dilemma

Should you take one box or two boxes?

If you take only the red box, then you're necessarily leaving $1,000 on the table.

But if you take both boxes, then Ω predicted that, and you miss out at a chance of $1,000,000.

Recap

Now, as you'll remember, we're discussing decision theories, algorithms that prescribe actions in games like these. Our motivation for studying decision theories is manyfold. For one thing, people who don't have tools for making good decisions can often become confused (remember the people who got palm surgery to change their fortunes).

This is not...