trentmkelly/LessWrong-43k huggingface dataset:
An overall schema for the friendly AI problems: self-referential convergence criteria
by Stuart_Armstrong — 2015-07-13T15:40:10.000Z

A putative new idea for AI control; index here.

After working for some time on the Friendly AI problem, it's occurred to me that a lot of the issues seem related. Specifically, all the following seem to have commonalities:

Nasty extrapolation of concepts (though badly implemented value learning or badly coded base concepts).

AI's making themselves into nasty expected utility maximisers.

AI's hacking themselves to maximum reward.

AI's creating successor agents that differ from them in dangerous ways.

People hacking themselves to maximum apparent happiness.

Problems with Coherent Extrapolated Volition.

Problems with unrestricted search.
Some issues I have with some of Paul Christiano's designs.

Reflective equilibrium itself.

Speaking very broadly, there are two features all them share:

The convergence criteria are self-referential.

Errors in the setup are likely to cause false convergence.

What do I mean by that? Well, imagine you're trying to reach reflective equilibrium in your morality. You do this by using good meta-ethical rules, zooming up and down at various moral levels, making decisions on how to resolve inconsistencies, etc... But how do you know when to stop? Well, you stop when your morality is perfectly self-consistent, when you no longer have any urge to change your moral or meta-moral setup. In other words, the stopping point (and the the convergence to the stopping point) is entirely self-referentially defined: the morality judges itself. It does not include any other moral considerations. You input your initial moral intuitions and values, and you hope this will cause the end result to be "nice", but the definition of the end result does not include your initial moral intuitions (note that some moral realists could see this process dependence as a positive - except for the fact that these processes have many convergent states, not just one or a small grouping).

So when the process goes nasty, you're pretty sure to have achieved something self-referentially stable, but not nice. Similarly, a nasty CEV will be coherent and have no desire to further extrapolate... but that's all we know about it.

The second feature is that any process has errors - computing errors, conceptual errors, errors due to the weakness of human brains, etc... If you visualise this as noise, you can see that noise in a convergent process is more likely to cause premature

...
The AI, the best human advisor
by Stuart_Armstrong — 2015-07-13T15:39:58.000Z

A putative new idea for AI control; index here.

All my ideas on reduced impact AI follow the same format: some restriction to ensure a reduced impact (I generally use something like "could a future AI detect this one existed"?) and some "trick" to allow a way of still achieving a larger impact. Further more, there is always a "nice" way of achieving this larger impact (do the thing we really want you to do, and only that!), so that we hope the AI can be forced into that behaviour. Examples of "tricks" are:

It could be an output channel which does not count against reduced impact.

It could involve several AIs that combine to have larger impact, with a conditional motivation to "fool" the reduced impact requirement.

It could be a variant of the above where we attempt to generalise the conditional motivation in a sensible manner.

In this post, I'll present another variant - one that I think is the most interesting so far. I won't change the reduced impact condition, but will present a different "trick".

AI pretends to be human

As in this post, imagine that we have a reduced-impact AI in a box that conceals it from us. The AI has some goal, and will output some data to further that goal, then turn itself off. And then we'll shake the box around or something, so we can't be sure that the AI was turned on.

There are two issues to deal with, here: a) The message itself is strong evidence that the AI was turned on, and b) if we exclude the message from the reduced impact, the message becomes dangerous.

To deal with a), first of all. This approach gets much easier if we have a whole brain emulation (WBE). Then we can house the WBE in the box with the AI. If the AI doesn't send out a message, then the WBE sends out is own message instead. There are variants we can try here if we don't have WBE - maybe some human volunteer willing to either die or take amnesia pills at the end of the experiment.

So, receiving a coherent message is not a surprising thing to come out the output channel. However, an AI motivated strongly by reduced impact, will simply seek to imitate as precisely as possible what the WBE would send. We still need a trick to get something useful.

AI pretends to be BEST human

Imagine that we wanted the AI to give us investment advice to make us rich. Say, a binary task: get us ten million dollars over the course of the year.

Now, this is the kind of thing that the

...
An overall schema for the friendly AI problems: self-referential convergence criteria
by Stuart_Armstrong — 2015-07-13T15:34:01.379Z

A putative new idea for AI control; index here.

After working for some time on the Friendly AI problem, it's occurred to me that a lot of the issues seem related. Specifically, all the following seem to have commonalities:

Nasty extrapolation of concepts (though badly implemented value learning or badly coded base concepts).

AI's making themselves into nasty expected utility maximisers.

AI's hacking themselves to maximum reward.

AI's creating successor agents that differ from them in dangerous ways.

People hacking themselves to maximum apparent happiness.

Problems with Coherent Extrapolated Volition.

Problems with unrestricted search.

Some issues I have with some of Paul Christiano's designs.

Reflective equilibrium itself.

Speaking very broadly, there are two features all them share:

The convergence criteria are self-referential.

Errors in the setup are likely to cause false convergence.

What do I mean by that? Well, imagine you're trying to reach reflective equilibrium in your morality. You do this by using good meta-ethical rules, zooming up and down at various moral levels, making decisions on how to resolve inconsistencies, etc... But how do you know when to stop? Well, you stop when your morality is perfectly self-consistent, when you no longer have any urge to change your moral or meta-moral setup. In other words, the stopping point (and the the convergence to the stopping point) is entirely self-referentially defined: the morality judges itself. It does not include any other moral considerations. You input your initial moral intuitions and values, and you hope this will cause the end result to be "nice", but the definition of the end result does not include your initial moral intuitions (note that some moral realists could see this process dependence as a positive - except for the fact that these processes have many convergent states, not just one or a small grouping).

So when the process goes nasty, you're pretty sure to have achieved something self-referentially stable, but not nice. Similarly, a nasty CEV will be coherent and have no desire to further extrapolate... but that's all we know about it.

The second feature is that any process has errors - computing errors, conceptual errors, errors due to the weakness of human brains, etc... If you visualise this as noise, you can see that noise in a convergent process is more likely to cause ...
The AI, the best human advisor
by Stuart_Armstrong — 2015-07-13T15:33:22.960Z

A putative new idea for AI control; index here.

All my ideas on reduced impact AI follow the same format: some restriction to ensure a reduced impact (I generally use something like "could a future AI detect this one existed"?) and some "trick" to allow a way of still achieving a larger impact. Further more, there is always a "nice" way of achieving this larger impact (do the thing we really want you to do, and only that!), so that we hope the AI can be forced into that behaviour. Examples of "tricks" are:

It could be an output channel which does not count against reduced impact.

It could involve several AIs that combine to have larger impact, with a conditional motivation to "fool" the reduced impact requirement.

It could be a variant of the above where we attempt to generalise the conditional motivation in a sensible manner.

In this post, I'll present another variant - one that I think is the most interesting so far. I won't change the reduced impact condition, but will present a different "trick".

AI pretends to be human

As in this post, imagine that we have a reduced-impact AI in a box that conceals it from us. The AI has some goal, and will output some data to further that goal, then turn itself off. And then we'll shake the box around or something, so we can't be sure that the AI was turned on.

There are two issues to deal with, here: a) The message itself is strong evidence that the AI was turned on, and b) if we exclude the message from the reduced impact, the message becomes dangerous.

To deal with a), first of all. This approach gets much easier if we have a whole brain emulation (WBE). Then we can house the WBE in the box with the AI. If the AI doesn't send out a message, then the WBE sends out is own message instead. There are variants we can try here if we don't have WBE - maybe some human volunteer willing to either die or take amnesia pills at the end of the experiment.

So, receiving a coherent message is not a surprising thing to come out the output channel. However, an AI motivated strongly by reduced impact, will simply seek to imitate as precisely as possible what the WBE would send. We still need a trick to get something useful.

AI pretends to be BEST human

Imagine that we wanted the AI to give us investment advice to make us rich. Say, a binary task: get us ten million dollars over the course of the year.

Now, this is the k...
Open Thread, Jul. 13 - Jul. 19, 2015
by MrMind — 2015-07-13T06:55:08.524Z

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.
Meetup : Canberra: The Efficient Market Hypothesis
by DanielFilan — 2015-07-13T04:01:59.618Z

Discussion article for the meetup : Canberra: The Efficient Market Hypothesis

WHEN:
24 July 2015 06:00:00PM (-0700)

WHERE:
57 McKinlay Place, Narrabundah

Note the new location! Also, date and time for this might be off since I am posting from San Francisco: it should be at 6:00 pm on Friday the 24th of July.

What is the Efficient Market Hypothesis, is it supported by market data, and what are its implications for ordinary people looking to invest their spare cash? Rémy Hamilton-smith will tell us as we eat vegan snacks at his place.

For more detail on the talk:
"Since the 80s the growing consensus among academics (though still hotly debated among practitioners) is that the major stock and security markets in developed countries are relatively efficient. This is because huge multinational investment banks can detect and take advantage of the slightest mispricings extremely quickly, making them disappear. They achieve this speed using automated trading programs and have been known to physically move their office next door to the stock exchange in order to get the lowest possible ping.

Thus from the point of view of everyone else this leads to the Efficient Market Hypothesis, which states that it is not possible to predictably ‘beat the market’. In this talk I'll discuss the EMH more formally as well as where it might suggest we should be investing our spare cash."

General meetup info:

If you use Facebook, please join our group.

Structured meetups are (usually) held on the second Saturday and fourth Friday of each month from 6 pm until late.

Discussion article for the meetup : Canberra: The Efficient Market Hypothesis
Entrepreneurial autopsies
by  — 2015-07-13T03:27:58.928Z

Entrepreneurial ideas come and go. Some I don't give a second thought to. Others I commence market research for, examine the competitive landscape and explore the feasibility for development. This can be time consuming, and has yet to have produced any tangible, commercialized product.

I figure it's about time I devote the time I would spend to exploiting my existing repertoire of knowledge to develop an idea, to exploring parsimonious, efficient techniques for assessing viability.

In my search I found [Autopsy.io], a startup graveyard. Founders describe why their startups failed, concisely. It made me think about my past startup ideas and why they haven't flied.

I'm going to work that out, put it in a spreadsheet and regress to whatever problem keeps popping up - then, I'll work on improving my subject matter knowledge in that domain - for example, if its the feasibility of implementing with existing technology - I might learn more about the current technological landscape in general. Or, more about existing services for investors, if my product is a service for investors, like my last startup idea, which I have autopsied in detail here
I just thought I'd share my general strategy for anyone who'd want to copy this procedure for startup autopsy. Please use this space to suggest other appropriate diagnostic methods.

edit 1: Thanks for pointing out the typos :)

'Charge for something and make more than you spend' - Marco Arment, Founder of Instapaper
Meetup : Less Wrong NH Inaugural Meet-up
by Mollie — 2015-07-12T16:34:31.103Z

Discussion article for the meetup : Less Wrong NH Inaugural Meet-up

WHEN:
14 July 2015 07:00:00PM (-0400)

WHERE:
269 Pearl St., Manchester, NH

(Posting this for a friend because she still can't comment)

The first NH meet-up is this Tuesday, 7/14, in Manchester, NH at 7 pm at a private residence. Light refreshments will be provided.

Discussion article for the meetup : Less Wrong NH Inaugural Meet-up
The Waker - new mode of existence
by Jan_Rzymkowski — 2015-07-12T14:00:29.535Z

This short text describes the idea of a Waker - a new way of experiencing reality / consciousness / subjectivity / mode of existence. Sadly, it cannot be attained without advanced uploading technology, that is one which allows far-fetched manipulation of mind. Despite that, the author doesn't find it premature to start planning a retirement as a posthuman.

A Waker is based on the experience of waking up from a dream - slowly we realize unreality of world we just were in, we realize discrepancies between dreamscape and "the real world", like that we no longer attend high school, one of our grandparents has had passed away few years ago, we work at a different place, etc. Despite the fact the world we wake up in is new and different, we quickly remember who we are, what we do, who are our friends, how does that world look like and in few seconds we have a perfect knowledge of that world and find it a real world, place, we have been living in since our birth. Meanwhile, dream world becomes a weird story and we typically feel some kind of a sentiment for it. Sometimes we're glad to escape that reality, sometimes we're sad - nevertheless we mostly treat it as something of little importance. Not a real world we lost forever, but rather a silly, made-up world.

A Waker's subjective experience would differ from ours in that way, she would always have the choice of waking up from current reality. As she would do that, she would find herself in a bed, or a chair, or laying on the grass, just having woken up. She would remember the world, she was just in, probably better then we usually remember our dream, nevertheless she would see it as a dream - she wouldn't feel strong connection to that reality. In the same time, she would start "remembering" the world she just woken up in. Somehow different then in our case, this would be a world she never had actually lived in, however she would acquire full knowledge of it and a sense of having spent all her life in that world. Despite all that, she would have full awareness of her being a Waker. She would find connection to the world she lives in different then we do and at first glance somehow paradoxical. She would feel how real it is, she would find it more real then any of the "dreams" she had, she would have investment in life goals, relationships with other people, she'll be capable of real love. And yet, she will be fully ...
Don't steer with guilt
by So8res — 2015-07-12T07:00:00.000Z

I've spoken at length about shifting guilt or dispelling guilt. What I haven't talked about, yet, is guilt itself.

So let's talk about guilt.

Guilt is one of those strange tools that works by not occurring. You place guilt on the branches of possibility that you don't want to happen, and then, if all goes well, those futures don't occur. Guilt is supposed to steer the future towards non-guilty futures; it's never supposed to be instantiated in reality.

Guilt works by the same mechanism as threats: imagine the tribesperson who precommits to breaking the legs of anyone who steals their food. If this precommitment works, then it never needs to be carried out: violence is a dangerous business, and the tribesperson would much rather that they never need to break legs at all. The threat is something that the tribesperson places on possibilities that they disprefer, in attempts to ensure that they never come to be.

Imagine, by contrast, the tribesperson who threatens to breaking the legs of anyone who looks at them funny: they might find themselves attempting violence every single day, and this likely makes their life unpleasant, to say the least. In this case, I would argue that they're using their threats poorly. I would say that, if you keep finding yourself carrying out a threat, then you really need to consider whether or not your threats are really capable of steering the future in the way you hoped.

Guilt is the same way: if you find yourself regularly experiencing guilt, then you're using guilt incorrectly.

Guilt works only when you wield it in such a way that it doesn't happen.

Guilt is costly when deployed. Once activated, it's usually strongly demotivating, and can easily lead to failure spirals or vicious cycles of depression.

As far as I can tell, the way that guilt-motivated people tend to operate is by working fervently in attempts to avoid the scourge of guilt. This may be effective when it works, but as soon as it starts to fail, the failure often cascades into a full-blown failure spiral (you're guilty that you're not working, which makes you feel bad, which makes it hard to work, which makes you guiltier, which you feel worse, which makes it harder to work, …). As a result, guilt motivation often results in a boom/bust productivity/depression cycle that, as far as I can tell, results in people feeling quite bad about themselves and being much less effect...
Inaugral bump thread (12th July to 19th July)
by  — 2015-07-12T06:49:04.510Z

Recently I came across the Akrasia Tactics Review article, then a 'bump' thread which spread the relevant content over multiple places, making tracking the content harder. It's apparent that some people believe some Lesswrong articles may be undervalued by the community (after factoring in karma as an indication of the community's appraisal)

Bump threads crowd out new articles and may annoy the more comprehensive or more experienced readers. This article is prototype for a regular (or whenever anyone else wants to take the initiative to start one) discussion board thread where people can lobby for increased visitation to articles of their interest. Don't make the threads run for too long - 1 week as a guide. Tag them with bump_thread

Future thread starters should not suggest an article in the initial discussion post, as I have. Although, this is useful as a guide for what I have in mind.
I need a protocol for dangerous or disconcerting ideas.
by Fivehundred — 2015-07-12T01:58:51.257Z

I have a talent for reasoning my way into terrifying and harmful conclusions. The first was modal realism as a fourteen-year-old. Of course I did not understand most of its consequences, but I disliked the fact that existence was infinite. It mildly depressed me for a few days. The next mistake was opening the door to solipsism and Brain-in-a-Vat arguments. This was so traumatic to me that I spent years in a manic depression. I could have been healed in a matter of minutes if I had talked to the right person or read the right arguments during that period, but I didn't.

Lesswrong has been a breeding ground of existential crisis for me. The Doomsday argument (which I thought up independently), ideas based on acausal trade (one example was already well known; one I invented myself), quantum immortality, the simulation argument, and finally my latest and worst epiphany: the potential horrible consequences of losing awareness of your reality under Dust Theory. I don't know that that's an accurate term for the problem, but it's the best I can think of.

This isn't to say that my problems were never solved; I often worked through them myself, always by refuting the horrible consequences of them to my own satisfaction and never through any sort of 'acceptance.' I don't think that my reactions are a consequence of an already depressed mind-state (which I certainly have anyway) because the moment I refute them I feel emotionally as if it never happened. It no longer wears on me. I have OCD, but if it's what's causing me to ruminate than I think I prefer having it as opposed to irrational suppression of a rational problem. Finding solutions would have taken much longer if I hadn't been thinking about them constantly.

I've come to realize that this site, due to perhaps a confluence of problems, was extremely unhelpful in working through any of my issues, even when they were brought about of Lesswrong ideas and premises. My acausal problem [1] I sent to about five or six people, and none of them had anything conclusive to say but simply referred me to Eliezer. Who didn't respond, even though this sort of thing is apparently important to him. This whole reaction struck me as disproportionate to the severity of the problem, but that was the best response I've had so far.

The next big failure was my resolution to the Doomsday argument. [2] I'm not very good yet at conveying thes...
Meetup : West LA: Practical Rationality Sources
by abramdemski — 2015-07-11T22:21:49.095Z

Discussion article for the meetup : West LA: Practical Rationality Sources

WHEN:
15 July 2015 07:00:00PM (-0700)

WHERE:
11066 santa monica avenue, LA, CA

How to Find Us: Go into this Del Taco. We will be in the back room if possible.

Parking is free in the lot out front or on the street nearby.

Discussion: If we wish to make a concerted effort to learn pragmatic rationality, as opposed to the most intellectually shiny concepts in rationality, what should we be reading? We will discuss what's out there and brainstorm other things to look for. In addition to LessWrong, we'd like to draw from other communities that give practical advice; Mister Money Mustache appears to be one of these.

Recommended Reading:

Scientific Self-Help: The State of Our Knowledge

Mister Money Mustache: Is it Convenient?

Mister Money Mustache: Hedonic Adaptation

No prior exposure to Less Wrong is required; this will be generally accessible.

Discussion article for the meetup : West LA: Practical Rationality Sources