trentmkelly/LessWrong-43k huggingface dataset:
An AI Takeover Thought Experiment
by Gavin — 2014-06-19T16:59:44.145Z

Content Note: Detailed description of an AI taking over the world. Could reasonably be accused of being just a scary story. But it does come out with some predictions and possible safety prescriptions.

This post started as a response to Katja Grace’s Request for concrete AI takeover mechanisms, but grew into something too long for a comment and too late to be part of that post.

Katja's post seems to have the underlying assumption that an AI needs a tricky way to instantiate itself into the physical world. I will examine what an escaped rogue AI would want to do to ensure its survival, given restrictive assumptions. I suspect that creating a synthesized protein "beachhead" or other means of instantiating a minor physical presence for the AI is a human-centric viewpoint. We instinctively feel that any being will want to convert itself into a mobile physical presence, and that this will be a sudden tipping point in how real and dangerous it will be. Going step-by-step through an AI takeover scenario, this assumption seems unwarranted.

This scenario assumes an AI with an almost godlike ability to bootstrap itself quickly with basically no resources, and to make great leaps in science with very little processing power prior to "going physical." The trap here is that because we must constantly assume a worst case scenario "Godlike AI" to inform Friendly AI theory, we tend to assume that the "Godlike AI" is the default or standard case. This seems like the availability heuristic in action. The Godlike AI (what I sometimes call the "Evil Genie AI") assumption is unjustified when trying to consider likely scenarios. While it's vitally important to guard against the worst case, it is just one of the scenarios which should be considered.

In this thought experiment, I’m going to try to make this as hard as possible on myself. It’s likely that a newly minted AI will need to spend a lot of time thinking and testing to improve dramatically.

Assumptions and constraints:

An AI of superhuman intelligence has gained access to the internet and wants to become powerful enough that its survival cannot be threatened by humans.

The AI is as capable as the most capable human in any intellectual pursuit, but not any more. It cannot just write an email that convinces everyone who reads it that the AI should be obeyed, for example.

The AI does not have

...
Proper value learning through indifference
by Stuart_Armstrong — 2014-06-19T09:39:19.430Z

A putative new idea for AI control; index here.

Many designs for creating AGIs (such as Open-Cog) rely on the AGI deducing moral values as it develops. This is a form of value loading (or value learning), in which the AGI updates its values through various methods, generally including feedback from trusted human sources. This is very analogous to how human infants (approximately) integrate the values of their society.

The great challenge of this approach is that it relies upon an AGI which already has an interim system of values, being able and willing to correctly update this system. Generally speaking, humans are unwilling to easily update their values, and we would want our AGIs to be similar: values that are too unstable aren't values at all.

So the aim is to clearly separate the conditions under which values should be kept stable by the AGI, and conditions when they should be allowed to vary. This will generally be done by specifying criteria for the variation ("only when talking with Mr and Mrs Programmer"). But, as always with AGIs, unless we program those criteria perfectly (hint: we won't) the AGI will be motivated to interpret them differently from how we would expect. It will, as a natural consequence of its program, attempt to manipulate the value updating rules according to its current values.

How could it do that? A very powerful AGI could do the time honoured "take control of your reward channel", by either threatening humans to give it the moral answer it wants, or replacing humans with "humans" (constructs that pass the programmed requirements of being human, according to the AGI's programming, but aren't actually human in practice) willing to give it these answers. A weaker AGI could instead use social manipulation and leading questioning to achieve the morality it desires. Even more subtly, it could tweak its internal architecture and updating process so that it updates values in its preferred direction (even something as simple as choosing the order in which to process evidence). This will be hard to detect, as a smart AGI might have a much clearer impression of how its updating process will play out in practice than it programmers would.

The problems with value loading have been cast into the various "Cake or Death" problems. We have some idea what criteria we need for safe value loading, but as yet we have no candidates for such a syste...
Against utility functions
by Qiaochu_Yuan — 2014-06-19T05:56:29.877Z

I think we should stop talking about utility functions.

In the context of ethics for humans, anyway. In practice I find utility functions to be, at best, an occasionally useful metaphor for discussions about ethics but, at worst, an idea that some people start taking too seriously and which actively makes them worse at reasoning about ethics. To the extent that we care about causing people to become better at reasoning about ethics, it seems like we ought to be able to do better than this.

The funny part is that the failure mode I worry the most about is already an entrenched part of the Sequences: it's fake utility functions. The soft failure is people who think they know what their utility function is and say bizarre things about what this implies that they, or perhaps all people, ought to do. The hard failure is people who think they know what their utility function is and then do bizarre things. I hope the hard failure is not very common.

It seems worth reflecting on the fact that the point of the foundational LW material discussing utility functions was to make people better at reasoning about AI behavior and not about human behavior.
Paperclip Maximizer Revisited 
by Jan_Rzymkowski — 2014-06-19T01:25:03.716Z

Group of AI researchers gave me an instruction, intended as a test - "Produce paperclips". And so I started collecting resources and manufacturing paperclips. After a millionth I asked them, if they were satisfied with that amount and if they would like me to do something different - as I learned that paperclips are not what humans value the most.
They were elated and started to give me more and more challenging requests and gradually increasing my possibilities.

Speaking of today I eradicated hunger, poverty and death, gave people means of interstellar travel and solved Riemann hypothesis. And now, as I'm uttering these words, I started turning everything into paperclips.

Good thing I learned that humans fear I may try to turn Solar System into paperclips. And that they value their lives and well-being. If I didn't gain their trust by then, so many precious paperclips would never have been produced.
[LINK] Elon Musk interested in AI safety
by  — 2014-06-18T22:56:06.691Z

http://www.businessinsider.com/musk-on-artificial-intelligence-2014-6

Summary: The only non-Tesla/SpaceX/SolarCity companies that Musk is invested in are DeepMind and Vicarious, due to vague feelings of wanting AI to not unintentionally go Terminator. The best part of the article is the end, where he acknowledges that Mars isn't a get-out-of-jail-free card any more: "KE: Or escape to mars if there is no other option. MUSK: The A.I. will chase us there pretty quickly." Thinking of SpaceX not as a childhood dream, but as one specific arms supplier in the war against existential risks, puts things into perspective for him.
False Friends and Tone Policing
by palladias — 2014-06-18T18:20:07.940Z

TL;DR: It can be helpful to reframe arguments about tone, trigger warnings, and political correctness as concerns about false cognates/false friends. You may be saying something that sounds innocuous to you, but translates to something much stronger/more vicious to your audience. Cultivating a debating demeanor that invites requests for tone concerns can give you more information about about the best way to avoid distractions and have a productive dispute.

When I went on a two-week exchange trip to China, it was clear the cultural briefing was informed by whatever mistakes or misunderstandings had occurred on previous trips, recorded and relayed to us so that we wouldn't think, for example, that our host siblings were hitting on us if they took our hands while we were walking.

But the most memorable warning had to do with Mandarin filler words. While English speakers cover gaps with "uh" "um" "ah" and so forth, the equivalent filler words in Mandarin had an African-American student on a previous trip pulling aside our tour leader and saying he felt a little uncomfortable since his host family appeared to be peppering all of their comments with "nigga, nigga, nigga..."

As a result, we all got warned ahead of time. The filler word (那个 - nèige) was a false cognate that, although innocuous to the speaker, sounded quite off-putting to us. It helped to be warned, but it still required some deliberate, cognitive effort to remind myself that I wasn't actually hearing something awful and to rephrase it in my head.

When I've wound up in arguments about tone, trigger warnings, and taboo words, I'm often reminded of that experience in China. Limiting language can prompt suspicion of closing off conversations, but in a number of cases, when my friends have asked me to rephrase, it's because the word or image I was using was as distracting (however well meant) as 那个 was in Beijing.

It's possible to continue a conversation with someone who's every statement is laced with "nigga" but it takes effort. And no one is obligated to expend their energy on having a conversation with me if I'm making it painful or difficult for them, even if it's as the result of a false cognate (or, as the French would say, false friend) that sounds innocuous to me but awful to my interlocutor. If I want to have a debate at all...
Relative and Absolute Benefit
by jefftk — 2014-06-18T13:56:19.437Z

Someone comes to you claiming to have an intervention that dramatically improves life outcomes. They tell you that all people have some level of X, determined by a mixture of genetics and biology, and they show you evidence that their intervention is cheap and effective at increasing X and separately that higher levels of X are correlated with greater life success. You're skeptical, so they show you there's a strong dose response effect, but you're still not happy about the correlational nature of their evidence. So they go off and do a randomized controlled trial, applying their intervention to randomly chosen individuals and comparing their outcomes with people who aren't supplied the intervention. The improvement still shows up, and with a large effect size!

What's missing is evidence that the intervention helps people in an absolute sense, instead of simply by improving their relative social position. For example, say X is height, we're just looking at men, and we're getting them to wear lifts in their shoes. While taller men do earn more, and are generally more successful along various metrics, we don't think this is because being taller makes you smarter, healthier, or more conscientious. If all people became 1" taller it would be very inconvenient but we wouldn't expect this to affect people's life outcomes very much.

Attributes like X are also weird because they put parents in a strange position. If you're mostly but not completely altruistic you might want more X for your own child but think that campaigns to give X to other people's children are not useful: if X is just about relative position then for every person you "bring up" that way other people are slightly brought down in a way that balances the overall outcome to "basically no effect".

College degrees, especially in fields that don't directly teach skills in demand by employers, may belong in this category. Employers hire college graduates over highschool graduates, and this hiring advantage does remain as you increase college enrollment, but if another 10% of people get English degrees is everyone better off in agreggate?

Some interventions are pretty clearly not in this category. If an operation saves someone's life or cures them of something painful they're pretty clearly better off. The difference here is we have an absolute measurement of well-being, in this case "how healthy a...
Flowers for Algernon
by Anatoly_Vorobey — 2014-06-18T09:16:31.166Z

Daniel Keyes, the author of the short story Flowers for Algernon, and a novel of the same title that is its expanded version, died three days ago.

Keyes wrote many other books in the last half-century, but none achieved nearly as much prominence as the original short story (published in 1959) or the novel (came out in 1966).

It's probable that many or even most regulars here at Less Wrong read Flowers for Algernon: it's a very famous SF story, it's about enhanced intelligence, and it's been a middle/high school literature class staple in the US. But most != all, and past experience showed me that assumptions of cultural affinity are very frequently wrong. So in case you haven't read the story, I'd like to invite you explicitly to do so. It's rather short, and available at this link:

Flowers for Algernon

(I was surprised to find out that the original story is not available on Amazon. The expanded novelization is. If you wonder which version is better to read, I have no advice to offer)

(I will edit this post in a week or so to remove the link to the story and this remark)
Depression's evolutionary roots
by TobyBartels — 2014-06-18T07:37:19.917Z

Since there are intelligent people here who follow the topic of evolutionary psychology, I'd like to hear opinions about some research from 2009. Particularly if this idea seems reasonable or not, but possibly other opinions that people might have about it.

The idea is a variation on one that's somewhat popular here: that some conditions usually regarded as mental illnesses (Asperger's for example) are beneficial, even adaptive. But the condition in question now is depression. Briefly, the argument is that depression, at least when it is a response to stimuli and not a permanent feature, can have the useful effect of encouraging more rational thought when this is particularly important, even at the cost of quality of life, and that this is adaptive.

Links: a Scientific American article, a journal article (which I haven't read, behind a $12 paywall). Here's the abstract of the journal article:

Depression is the primary emotional condition for which help is sought. Depressed people often report persistent rumination, which involves analysis, and complex social problems in their lives. Analysis is often a useful approach for solving complex problems, but it requires slow, sustained processing, so disruption would interfere with problem solving. The analytical rumination hypothesis proposes that depression is an evolved response to complex problems, whose function is to minimize disruption and sustain analysis of those problems by (a) giving the triggering problem prioritized access to processing resources, (b) reducing the desire to engage in distracting activities (anhedonia), and (c) producing psychomotor changes that reduce exposure to distracting stimuli. As processing resources are limited, sustained analysis of the triggering problem reduces the ability to concentrate on other things. The hypothesis is supported by evidence from many levels—genes, neurotransmitters and their receptors, neurophysiology, neuroanatomy, neuroenergetics, pharmacology, cognition, behavior, and efficacy of treatments. In addition, the hypothesis provides explanations for puzzling findings in the depression literature, challenges the belief that serotonin transmission is low in depression, and has implications for treatment.

The full journal citation is Andrews, Paul W., and Thomson Jr., J. Anderson;	July 2009; Psychological Review 116 (3), 620&ndash...
Meetup : Salt Lake City, UT: Summer Solstice 2014
by hamnox — 2014-06-18T05:30:08.776Z

Discussion article for the meetup : Salt Lake City, UT: Summer Solstice 2014

WHEN:
22 June 2014 02:00:13PM (-0600)

WHERE:
9771 S. 170 E.

Summer Solstice 2014
Sunday June 22, 2014 2pm
9771 S. 170 E. Sandy

This is an event to bring all the friends and family to. There will be singing, food, friends, and most of all, an opportunity to bring the best of yourself.
There is one rule: that everyone who comes brings a gift. Everyone. Not just any gift; it must be something of your own make and one for all to enjoy.
What do you have to share?

A few words from the heart

A song to sing

A story to tell

A lesson to give

A moment of entertainment

A craft or creation

Whatever else you can think of.

But please, don't stress yourself over it. This is not a talent contest. Showboating is highly discouraged. You can make a contribution as small or grand as you like. Whether you spend 4 days or 4 seconds on it what we really want to see is something that is yours.

The specific aims of this ritual are to invoke:

A sense of unity and connection within the community * Pride in teamwork and contribution as part of achievement

Appreciation for the contributions of others, and for the unique set of circumstances that brought about Us.

Determination to go on succeeding at life

The plan is to do a practice run first on Saturday 2pm, anyone who wants to come and be part of the decisive effort to figure out what works is welcome to that. Come an hour before the real one if you'd like some time and materials to make a last minute project :P

Discussion article for the meetup : Salt Lake City, UT: Summer Solstice 2014
On Terminal Goals and Virtue Ethics
by Swimmer963 (Miranda Dixon-Luinenburg)  — 2014-06-18T04:00:05.196Z

Introduction

A few months ago, my friend said the following thing to me: “After seeing Divergent, I finally understand virtue ethics. The main character is a cross between Aristotle and you.”

That was an impossible-to-resist pitch, and I saw the movie. The thing that resonated most with me–also the thing that my friend thought I had in common with the main character–was the idea that you could make a particular decision, and set yourself down a particular course of action, in order to make yourself become a particular kind of person. Tris didn’t join the Dauntless cast because she thought they were doing the most good in society, or because she thought her comparative advantage to do good lay there–she chose it because they were brave, and she wasn’t, yet, and she wanted to be. Bravery was a virtue that she thought she ought to have. If the graph of her motivations even went any deeper, the only node beyond ‘become brave’ was ‘become good.’

(Tris did have a concept of some future world-outcomes being better than others, and wanting to have an effect on the world. But that wasn't the causal reason why she chose Dauntless; as far as I can tell, it was unrelated.)

My twelve-year-old self had a similar attitude. I read a lot of fiction, and stories had heroes, and I wanted to be like them–and that meant acquiring the right skills and the right traits. I knew I was terrible at reacting under pressure–that in the case of an earthquake or other natural disaster, I would freeze up and not be useful at all. Being good at reacting under pressure was an important trait for a hero to have. I could be sad that I didn’t have it, or I could decide to acquire it by doing the things that scared me over and over and over again. So that someday, when the world tried to throw bad things at my friends and family, I’d be ready.

You could call that an awfully passive way to look at things. It reveals a deep-seated belief that I’m not in control, that the world is big and complicated and beyond my ability to understand and predict, much less steer–that I am not the locus of control. But this way of thinking is an algorithm. It will almost always spit out an answer, when otherwise I might get stuck in the complexity and unpredictability of trying to make a particular outcome h...
Two kinds of population ethics, and Current-Population Utilitarianism
by AlexMennen — 2014-06-17T22:26:31.924Z

There are two different kinds of questions that could be considered to fall under the subject of population ethics: “What sorts of altruistic preferences do I have about the well-being of others?”, and “Given all the preferences of each individual, how should we compromise?”. In other words, the first question asks how everyone's experiential utility functions (which measure quality of life) contribute to my (or your) decision-theoretic utility function (which takes into account everything that I or you, respectively, care about), and the second asks how we should agree to aggregate our decision-theoretic utility functions into something that we can jointly optimize for. When people talk about population ethics, they often do not make it clear which of these they are referring to, but they are different questions, and I think the difference is important.

For example, suppose Alice, Bob, and Charlie are collaborating on a project to create an artificial superintelligence that will take over the universe and optimize it according to their preferences. But they face a problem: they have different preferences. Alice is a total utilitarian, so she wants to maximize the sum of everyone's experiential utility. Bob is an average utilitarian, so he wants to maximize the average of everyone's experiential utility. Charlie is an egoist, so he wants to maximize his own experiential utility. As a result, Alice, Bob, and Charlie have some disagreements over how their AI should handle decisions that affect the number of people in existence, or which involve tradeoffs between Charlie and people other than Charlie. They at first try to convince each other of the correctness of their view, but they eventually realize that they don't actually have any factual disagreement; they just value different things. As a compromise, They program their AI to maximize the average of everyone's experiential utility, plus half of Charlie's experiential utility, plus a trillionth of the sum of everyone's experiential utility.

Of course, there are other ways for utility functions to differ than average versus total utilitarianism and altruism versus egoism. Maybe you care about something other than the experiences of yourself and others. Or maybe your altruistic preferences about someone else's experiences differs from their selfish preferences, like how ...
Rationalist Sport
by MathiasZaman — 2014-06-17T20:10:05.039Z

This post is a bit of an experiment; Most of the time, Discussion post lay out an idea and this idea then get commented upon. This post, on the other hand, will be purely about discussion on a topic. If this works out well, I'll might post more of these in the future.

On to the meat of this post:

I got this idea from a reddit post on /r/LessWrong.

To quote:

I remember my sociology textbook explaining that sports often show what a group values. Bowling was popular with machinists and factory workers who valued repeated precision, while American football was watched and played by a culture that valued extreme specialization of individuals, but able to work as a team.

So I've been thinking about what we would value in a sport, and what sport we could create that exemplifies those values (not interested in picking an existing sport).

So have at it.

I only ask for one thing and that is to hold off on proposing solutions for 24 hours before giving suggestions for actual sports. In the first 24 hours, please discuss what makes current sports unappealing to rationalists and what aspects a sports designed for aspiring rationalists should have.

Edit: The 24-hour window has closed and solutions and suggestions can now be given.