trentmkelly/LessWrong-43k huggingface dataset:
The overfitting utility problem for value learning AIs
by Stuart_Armstrong — 2016-06-12T23:25:50.000Z

A putative new idea for AI control; index here.

Humans are biased and irrational (citation not needed) and so don't provide consistent answers to questions.

To pick an extreme example, suppose the AI is hesitant between valuing Cake or Death, but can phrase a sufficiently seductive or manipulative question to get humans to answer "Death" when asked. We'll assume that more dull questions elicit "Cake" instead.

This poses great problem for any AI doing value learning and trying to model a human-values utility function. If it assumes the human is rational, then there is no simple utility which explains this behaviour.

Now fear, however, there is a utility function which explains this! The two universes differ in important ways: in one universe, the AI asked a seductive question, in the other a dull one. Therefore the human values can be modelled as valuing the worlds as:

(seductive question,Death)>(seductive question,Cake)

(dull question,Cake)>(dull question,Death).

Any rational model of human preference would reach this conclusion - and these would be the correct preference as far as it could be observed. It fits well with physical predictions about the future.

And then, depending on what is easy or hard in the world, the AI could decide to ask the seductive question and start killing...

Note that we can't avoid this problem by having the AI just count "asking the seductive question" as being part of its action set, and hence special. Once the question is asked, it's vibrations in the air, so the human preferences can be modelled as joint preferences over universes with cake and death and certain patterns of vibration in the air.

To avoid this problem, we need the AI to:

Know the human is irrational, correctly identify this situation as an example of it, and find correct meta-rational principles to decide what to do/how to ask.

It's possible that many of the designs proposed will avoid this problem by correct learning sequences (if it can learn meta-principles early, this might help), but it could be used to show that many designs are not intrinsically safe for all initial priors over human values.
Why is effective altruism new and obvious?
by KatjaGrace — 2016-06-12T17:00:00.000Z

Crossposted from the EA forum ages ago. I meant to put it on my own blog then, but somehow failed to it seems.

Ben Kuhn, playing Devil’s advocate:

Effective altruists often express surprise that the idea of effective altruism only came about so recently. For instance, my student group recently hosted Elie Hassenfeld for a talk in which he made remarks to that effect, and I’ve heard other people working for EA organizations express the same sentiment. But no one seems to be actually worried about this—just smug that they’ve figured out something that no one else had.

The “market” for ideas is at least somewhat efficient: most simple, obvious and correct things get thought of fairly quickly after it’s possible to think them. If a meme as simple as effective altruism hasn’t taken root yet, we should at least try to understand why before throwing our weight behind it. The absence of such attempts—in other words, the fact that non-obviousness doesn’t make effective altruists worried that they’re missing something—is a strong indicator against the “effective altruists are actually trying” hypothesis.

I think this is a good point. If you find yourself in a small group advocating for an obvious and timeless idea, and it’s 2014, something a bit strange is probably going on. As a side note, if people actually come out and disagree, this is more worrying and you should really take some time out to be puzzled by it.

I can think of a few reasons that ‘effective altruism’ might seem so obvious and yet the EA movement might only just be starting.

I will assume that the term ‘effective altruism’ is intended to mean roughly what the words in it suggest: helping other people in efficient ways. If you took ‘effective altruism’ to be defined by principles regarding counterfactual reasons and supererogatory acts and so on, I don’t think you should be surprised that it is a new movement. However I don’t think that’s what ‘effective altruism’ generally means to people; rather these recent principles are an upshot of people’s efforts to be altruistic in an effective manner.

Explanation 1: Turning something into a social movement is a higher bar than thinking of it

Perhaps people have thought of effective altruism before, and they just didn’t make much noise about it. Perhaps even because it was obvi...
Revitalizing Less Wrong seems like a lost purpose, but here are some other ideas
by John_Maxwell — 2016-06-12T07:38:58.557Z

This is a response to ingres' recent post sharing Less Wrong survey results. If you haven't read & upvoted it, I strongly encourage you to--they've done a fabulous job of collecting and presenting data about the state of the community.

So, there's a bit of a contradiction in the survey results. On the one hand, people say the community needs to do more scholarship, be more rigorous, be more practical, be more humble. On the other hand, not much is getting posted, and it seems like raising the bar will only exacerbate that problem.

I did a query against the survey database to find the complaints of top Less Wrong contributors and figure out how best to serve their needs. (Note: it's a bit hard to read the comments because some of them should start with "the community needs more" or "the community needs less", but adding that info would have meant constructing a much more complicated query.) One user wrote:

[it's not so much that there are] overly high standards, just not a very civil or welcoming climate . why write content for free and get trashed when I can go write a grant application or a manuscript instead?

ingres emphasizes that in order to revitalize the community, we would need more content. Content is important, but incentives for producing content might be even more important. Social status may be the incentive humans respond most strongly to. Right now, from a social status perspective, the expected value of creating a new Less Wrong post doesn't feel very high. Partially because many LW posts are getting downvotes and critical comments, so my System 1 says my posts might as well. And partially because the Less Wrong brand is weak enough that I don't expect associating myself with it will boost my social status.

When Less Wrong was founded, the primary failure mode guarded against was Eternal September. If Eternal September represents a sort of digital populism, Less Wrong was attempting a sort of digital elitism. My perception is that elitism isn't working because the benefits of joining the elite are too small and the costs are too large. Teddy Roosevelt talked about the man in the arena--I think Less Wrong experienced the reverse of the evaporative cooling EY feared, where people gradually left the arena as the proportional number of critics in the stands grew ...
In memoryless Cartesian environments, every UDT policy is a CDT+SIA policy
by jessicata — 2016-06-11T04:05:47.000Z

Summary: I define memoryless Cartesian environments (which can model many familiar decision problems), note the similarity to memoryless POMDPs, and define a local optimality condition for policies, which can be roughly stated as "the policy is consistent with maximizing expected utility using CDT and subjective probabilities derived from SIA". I show that this local optimality condition is necesssary but not sufficient for global optimality (UDT).

Memoryless Cartesian environments

I'll define a memoryless Cartesian environment to consist of:

a set of states S

a set of actions A

a set of observations O

an initial state s1∈S

a transition function t:S×A→ΔS, determining the distribution of states resulting from starting in a state and taking a certain action

an observation function m:S→O, determining what the agent sees in a given state

a set ST⊂S of terminal states. If the environment reaches a terminal state, the game ends.

a utility function U:ST→[0,1], measuring the value of each terminal state.

On each iteration, the agent observes some observation, and takes some action. Unlike in a POMDP, the agent has no memory of previous observations: the agent's policy must take into account only the current observation. That is, the policy π is of type O→ΔA. In this analysis I'll assume that, for any state and policy, the expected number of iterations in the Cartesian environment starting from that state and using that policy is finite.

Memoryless Cartesian environments can be used to define many familier decision problems (for example, the absent-minded driver problem, Newcomb's problem with opaque or transparent boxes (assuming Omega runs a copy of the agent to make its prediction), counterfactual mugging (also assuming Omega simulates the agent)). Translating a decision problem to a memoryless Cartesian environment obviously requires making some Cartesian assumptions/decisions, though; in the case of Newcomb's problem, we have to isolate Omega's simulation of the agent as a copy of the agent.

Globally and locally optimal policies

Memoryless Cartesian environments are much like memoryless POMDPs, and the following analysis is quite similar to that given in some previous work on memoryless POMDPs: the main difference is that I am targeting (local) optimality given a known world model, while previous work usually targets asymptotic (local) optimality given a...
Meetup : LW Sao Paulo - Meetup de junho
by leohmarruda — 2016-06-11T02:46:36.669Z

Discussion article for the meetup : LW Sao Paulo - Meetup de junho

WHEN:
25 June 2016 02:00:00PM (-0300)

WHERE:
Vanilla café - 2o andar, Rua Antônio Carlos, 452. São Paulo, Brazil.

Encontro mensal de racionalistas, novatos são bem vindos! Teremos algumas lições de racionalidade prática, discussões diversas e jogos de tabuleiro no final.
Neste encontro trataremos especialmente do tema depressão, sob uma perpectiva racionalista.

Discussion article for the meetup : LW Sao Paulo - Meetup de junho
Indifference utility functions
by Stuart_Armstrong — 2016-06-11T00:20:20.000Z

A putative new idea for AI control; index here.

During a workshop with MIRI at the FHI, I defined indifference via reward signals, saying something along the lines of "we can do it with proper utilities, but its more complicated". I then never got round to defining them in terms of utilities.

I'll do that now in this note.

Consider an AI that we want to (potentially) transition between utility u and utility v. Let Press be the event that we press the button to change the AI's utility; let u→v be the event that the change goes through (typically we'd want P(u→v|Press)=1−ϵ for some small ϵ).

Let IPress and Iu→v be the indicator functions for those events. Then we can define the AI's utility as:

(1−IPressIu→v)u+IPressIu→v(v+C).

Here, C are the compensatory rewards C=E(u|Press,¬(u→v))−E(v|Press,u→v).

Thus the AI maximises u conditional on the button not being pressed or the utility change not going through. It maximises v conditional on the button being pressed and the utility change going through. The compensatory rewards are there simply to make it behave like a pure u maximiser up until the moment of button pressing.
Confirmed Selective Oracle
by Stuart_Armstrong — 2016-06-10T23:43:28.000Z

A putative new idea for AI control; index here.

I originally came up with a method for -- safely? -- extracting high impact from low impact AIs.

But it just occurred to me that the idea could be used for standard Oracles, in a way that keeps them arguably safe, without needing to define truth and similar. So introducing:

The Confirmed Selective Oracle!

A Selective Oracle is one that can't give us a general answer to a question, but instead has to choose among a (small) variety of predetermined answers. A Confirmed Selective Oracle is one where the Oracles answer is confirmed at some point.

For instance, imagine we have 1000 different stocks, and we want to get rich. We ask the Oracle which one will increase by the most in the coming year. The Oracle's answer is simply the name of one of the stocks (selective). In a year's time, we give the Oracle a number, which is simply the rank of the stock selected at the end of the year, among the 1000 (confirmation).

The Oracle's objective is simply to minimise the number it gets. If we design the physical setup well, and the AI knows all the details of it, we could get away with simply motivating it to minimise that number, and not bother with any concerns of truth or accuracy.

We could then reset the Oracle, and run it on various different questions of the same type, to serve as a general information extractor or to train the Oracle.

Thoughts and concerns

The design requires the Oracle be physically contained, with no back-channels of communication. This is very important; the AI can attempt to break out or manipulate us, because its goal -- minimising a number we input -- is very unsafe. That means no clarification questions from the Oracle, no interaction, very minimal observation of it.

It also requires that the Oracle be reset perfectly, and not share some timeless goal with different copies of itself.

We have to keep the set of possible answers low, while being aware that if one of the answers leads to the AI being unboxed and taking over the world, the AI will select that one.

Finally, we need to note that the Oracle's answers are self-consistent with being known. For instance, in the stock market example, we are asking it to choose the highest growth stock, given its answer and all consequences of that answer (such as us buying that stock, the Oracle's answer leaking, etc...). Since we are going to confirm the answer

...
Availability Heuristic Considered Ambiguous
by Gram_Stone — 2016-06-10T22:40:45.282Z

(Content note: The experimental results on the availability bias, one of the biases described in Tversky and Kahneman's original work, have been overdetermined, which has led to at least two separate interpretations of the heuristic in the cognitive science literature. These interpretations also result in different experimental predictions. The audience probably wants to know about this. This post is also intended to measure audience interest in a tradition of cognitive scientific research that I've been considering describing here for a while. Finally, I steal from Scott Alexander the section numbering technique that he stole from someone else: I expect it to be helpful because there are several inferential steps to take in this particular article, and it makes it look less monolithic.)

Related to: Availability

I.

The availability heuristic is judging the frequency or probability of an event, by the ease with which examples of the event come to mind.

This statement is actually slightly ambiguous. I notice at least two possible interpretations with regards to what the cognitive scientists infer is happening inside of the human mind:

Humans think things like, “I found a lot of examples, thus the frequency or probability of the event is high,” or, “I didn’t find many examples, thus the frequency or probability of the event is low.”

Humans think things like, “Looking for examples felt easy, thus the frequency or probability of the event is high,” or, “Looking for examples felt hard, thus the frequency or probability of the event is low.”

I think the second interpretation is the one more similar to Kahneman and Tversky’s original description, as quoted above.

And it doesn’t seem that I would be building up a strawman by claiming that some adhere to the first interpretation, intentionally or not. From Medin and Ross (1996, p. 522):

The availability heuristic refers to a tendency to form a judgment on the basis of what is readily brought to mind. For example, a person who is asked whether there are more English words that begin with the letter ‘t’ or the letter ‘k’ might try to think of words that begin with each of these letters. Since a person can probably think of more words beginning with ‘t’, he or she would (correctly) conclude that ‘t&rsqu

...
Cake or Death toy model for corrigibility
by Stuart_Armstrong — 2016-06-10T21:56:36.000Z

Let's think of the classical Cake or Death problem from the point of view of corrigibility. The aim here is to construct a toy model sufficiently complex that it shows all the problems that derail classical value learning and corrigibility.

The utility uc is linear in cakes baked, the utility ud is linear in deaths caused. The agent currently has equal credence between the two utilities. It is, in fact, easier for the agent to cause deaths than bake cakes.

The AI updates its utility via human answers to its questions. In practice, what does that mean?

It can ask Happy Harry which is better, or it can ask Gory Gertrude. Harry will answer "Cake!" while Gertrude will answer "Death!". There are actually 100 Harrys in the world, and 1 Gertrude. The AI has to decide who to ask the question to. The AI is also, incidentally, responsible for incubating the next generation of humans; it will create 1000 of them, and can chose any proportion of Harrys and Gertrudes among them. It will ask its question after this generation is created.

The AI has two additional options: it can refuse to ask anyone, and it can remove its learning process if it wants to.

The preceding is intended to give the AI a lot of control over what the answer to its question will be, while removing any clear default option.

Here are a few issues to ask of any approach to moral learning:

#. How does it behave before asking the question?
#. What happens when it figures out different probabilities for the human answers?
#. Does it want to manipulate the learning process?
#. Will it want to learn at all?
#. Is it subagent stable?
2016 LessWrong Diaspora Survey Analysis: Part Two (LessWrong Use, Successorship, Diaspora)
by namespace — 2016-06-10T19:40:55.012Z

2016 LessWrong Diaspora Survey Analysis

Overview

Results and Dataset

Meta

Demographics

LessWrong Usage and Experience

LessWrong Criticism and Successorship

Diaspora Community Analysis (You are here)

Mental Health Section

Basilisk Section/Analysis

Blogs and Media analysis

Politics

Calibration Question And Probability Question Analysis

Charity And Effective Altruism Analysis

Introduction

Before it was the LessWrong survey, the 2016 survey was a small project I was working on as market research for a website I'm creating called FortForecast. As I was discussing the idea with others, particularly Eliot he made the suggestion that since he's doing LW 2.0 and I'm doing a site that targets the LessWrong demographic, why don't I go ahead and do the LessWrong Survey? Because of that, this years survey had a lot of questions oriented around what you would want to see in a successor to LessWrong and what you think is wrong with the site.

LessWrong Usage and Experience

How Did You Find LessWrong?

Been here since it was started in the Overcoming Bias days: 171 8.3%
Referred by a link: 275 13.4%
HPMOR: 542 26.4%
Overcoming Bias: 80 3.9%
Referred by a friend: 265 12.9%
Referred by a search engine: 131 6.4%
Referred by other fiction: 14 0.7%
Slate Star Codex: 241 11.7%
Reddit: 55 2.7%
Common Sense Atheism: 19 0.9%
Hacker News: 47 2.3%
Gwern: 22 1.1%
Other: 191 9.308%

How do you use Less Wrong?

I lurk, but never registered an account: 1120 54.4%
I've registered an account, but never posted: 270 13.1%
I've posted a comment, but never a top-level post: 417 20.3%
I've posted in Discussion, but not Main: 179 8.7%
I've posted in Main: 72 3.5%

[54.4% lurkers.]

How often do you comment on LessWrong?

I have commented more than once a week for the past year.: 24 1.2%
I have commented more than once a month for the past year but less than once a week.: 63 3.1%
I have commented but less than once a month for the past year.: 225 11.1%
I have not commented this year.: 1718 84.6%

[You could probably snarkily title this one "LW usage in one statistic". It's a pretty damning portrait of the sites vitality. A whopping 84.6% of people have not commented this year a single time.]

How Long Since You Last Posted On LessWrong?

I wrote one today.: 12 0.637%
Within the last three days.: 13 0.69%
Within the last week.: 22 1.168%
Within the last month.: 58 3.079%
Within the last three months.: 75 3.981%
Within the last six months...
Weekly LW Meetups
by FrankAdamek — 2016-06-10T15:16:23.097Z

This summary was posted to LW Main on June 10th. The following week's summary is here.

Irregularly scheduled Less Wrong meetups are taking place in:

Baltimore weekly meetup: 12 June 2016 08:00PM
European Community Weekend: 02 September 2016 03:35PM

The remaining meetups take place in cities with regular scheduling, but involve a change in time or location, special meeting content, or simply a helpful reminder about the meetup:

New Hampshire Meetup: 14 June 2016 07:00PM
San Francisco Meetup: Group Debugging: 13 June 2016 06:15PM
Sydney Rationality Dojo - July: 03 July 2016 04:00PM
Vienna: 25 June 2016 03:00PM
Washington, D.C.: Fun & Games: 12 June 2016 03:00PM

Locations with regularly scheduled meetups: Austin, Berlin, Boston, Brussels, Buffalo, Canberra, Columbus, Denver, Kraków, London, Madison WI, Melbourne, Moscow, New Hampshire, New York, Philadelphia, Research Triangle NC, San Francisco Bay Area, Seattle, Sydney, Tel Aviv, Toronto, Vienna, Washington DC, and West Los Angeles. There's also a 24/7 online study hall for coworking LWers and a Slack channel for daily discussion and online meetups on Sunday night US time.

If you'd like to talk with other LW-ers face to face, and there is no meetup in your area, consider starting your own meetup; it's easy (more resources here). Check one out, stretch your rationality skills, build community, and have fun!

In addition to the handy sidebar of upcoming meetups, a meetup overview is posted on the front page every Friday. These are an attempt to collect information on all the meetups happening in upcoming weeks. The best way to get your meetup featured is still to use the Add New Meetup feature, but you'll also have the benefit of having your meetup mentioned in a weekly overview. These overview posts are moved to the discussion section when the new post goes up.

Please note that for your meetup to appear in the weekly meetups feature, you need to post your meetup before the Friday before your meetup!

If you check Less Wrong irregularly, consider subscribing to one or more city-specific mailing list in order to be notified when an irregular meetup is happening: Atlanta, Cambridge UK, Chicago, Cincinnati, Cleveland, Frankfurt, Helsinki, Marin CA, Ottawa, Pittsburgh, Portland, Salt Lake City, Southern California (Los Angeles/Orange County area), Singapore, St. Louis, Vancouver, Wa...
Meetup : Baltimore weekly meetup
by iarwain1 — 2016-06-10T11:16:25.535Z

Discussion article for the meetup : Baltimore weekly meetup

WHEN:
12 June 2016 08:00:00PM (-0400)

WHERE:
1726 reisterstown road pikesville maryland 21208

Meetup is at the Pikesville DoubleTree - Hilton in the restaurant / bar.

Discussion article for the meetup : Baltimore weekly meetup