trentmkelly/LessWrong-43k huggingface dataset:
One week left for CSER researcher applications
by RyanCarey — 2015-04-17T00:40:00.901Z

This is the last week to apply for one of four postdoctoral research positions at the Centre for the Study of Existential Risk. We are seeking researchers in disciplines including: economics, science and technology studies, science policy, arms control policy, expert elicitation and aggregation, conservation studies and philosophy.

The application requires a research proposal of no more than 1500 words from an individual with a relevant doctorate.

"We are looking for outstanding and highly-committed researchers, interested in working as part of growing research community, with research projects relevant to any aspect of the project. We invite applicants to explain their project to us, and to demonstrate their commitment to the study of extreme technological risks.

We have several shovel-ready projects for which we are looking for suitable postdoctoral researchers. These include:

1. Ethics and evaluation of extreme technological risk (ETR) (with Sir Partha Dasgupta;

2. Horizon-scanning and foresight for extreme technological risks (with Professor William Sutherland);

3. Responsible innovation and extreme technological risk (with Dr Robert Doubleday and the Centre for Science and Policy).

However, recruitment will not necessarily be limited to these subprojects, and our main selection criterion is suitability of candidates and their proposed research projects to CSER’s broad aims.

More details are available here. Applications close on April 24th.

- Sean OH and Ryan
Modal Bargaining Agents
by orthonormal — 2015-04-16T22:19:03.000Z

Summary: Bargaining problems are interesting in the case of Löbian cooperation; Eliezer suggested a geometric algorithm for resolving bargaining conflicts by leaving the Pareto frontier, and this algorithm can be made into a modal agent, given an additional suggestion by Benja.

Bargaining and Fairness

When two agents can read each others' source code before playing a game, Löbian cooperation can transform many games into pure bargaining problems. Explicitly, the algorithm of "if I can prove you choose X, then I choose Y, otherwise I choose Z" makes it possible for an agent to offer a deal better for both parties than a Nash equilibrium, then default to that Nash equilibrium if the deal isn't verifiably taken. In the simple case where there is a unique Nash equilibrium, two agents with that sort of algorithm can effectively reduce the decision theory problem to a negotiation over which point on the Pareto frontier they will select.

However, if two agents insist on different deals (e.g. X insists on point D, Y insists on point A), then it falls through and we're back to the Nash equilibrium O. And if you try and patch this by offering several deals in sequence along the Pareto frontier, then a savvy opponent is just going to grab whichever of those is best for them. So we'd like both a notion of the fair outcome, and a way to still make deals with agents who disagree with us on fairness (without falling back to the Nash equilibrium, and without incentivizing them to play hardball with us).

Note that utility functions are only defined up to affine transformations, so any solution should be invariant under independent rescalings of the players' utilities. This requirement, plus a few others (winding up on the Pareto frontier, independence of irrelevant alternatives, and symmetry) are all satisfied by the Nash solution to the bargaining problem: choose the point on the Pareto frontier so that the area of the rectangle from O to that point is maximized.

This gives us a pretty plausible answer to the first question, but leaves us with the second: are we simply at war with agents that have other ideas of what's fair? (Let's say that X thinks that the Nash solution N is fair, but Y thinks that B is fair.) Other theorists have come up with other definitions of the fair solution to a bargaining problem, so this is a live question!

And the question of incentives makes it ev

...
Meetup : Sydney Rationality Dojo - Planning and Debugging
by luminosity — 2015-04-16T21:57:18.623Z

Discussion article for the meetup : Sydney Rationality Dojo - Planning and Debugging

WHEN:
03 May 2015 04:00:00PM (+1000)

WHERE:
Humanist House, 10 Shepherd St Chippendale

Our next dojo will focus on how to successfully make plans to achieve our goals, and to debug potential problems with those plans until we end up with one we are happy with.

Discussion article for the meetup : Sydney Rationality Dojo - Planning and Debugging
Can an AI suffer?
by kingmaker — 2015-04-16T15:44:41.618Z

As I have said before in my previous posts, I do not think Friendly-AI is a good idea. Being on this site has me more convinced than before of the feasibility of FAI, but I still do not think that the creation of an FAI, or any other AI for that matter, should be attempted.

We have directives in much the same way that we would give an FAI directives. Our directives are not immediately clear, since if they were clear then producing an FAI would be simple, but we know that they are achieved through our emotions and desires, and that they were produced by our evolution. We have no reason to believe our emotions are the most effective way we could be programmed to achieve (arguably) our primary goal; to further the species, but we know that they are effective.

If we accept the material atheistic position as true, then we must accept that all of our emotions and experiences are the product of nothing more than interactions of matter and energy. With this in mind, we do not know if consciousness in an AI can be avoided, or if it is an emergent property of all complex systems, and thus we may be creating a being capable of immense suffering.

To argue that we could simply give an FAI super-human intelligence without any consciousness is not an escape, as the only intelligence we have yet seen to exhibit full general intelligence is our own, and some select non-human persons. This means that every general intelligence encountered is conscious (but that is not to say that every AGI must be). Although companies such as Google's DeepMind have shown huge success with their software, we cannot be certain that their software does not possess some level of consciousness as we have not yet created a reliable test for machine consciousness. Simply put, if we do not know the nature of our own consciousness and all other potential forms of consciousness then we cannot prevent it, especially as the most promising AI research is concerned with imitating certain structures of the human brain.

To argue that we could create an FAI that doesn't feel pain or emotions is also not an escape. Pain evolved to alert us to damage to our bodies, so the AI would quickly develop an equivalent to pain to protect its resources from damage.However, since the FAI may be able to ensure its resources' security, this may not be necessary. Emotions also serve a purpose;...
Can an AI suffer?
by kingmaker — 2015-04-16T15:27:13.333Z

As I have said before in my previous posts, I do not think Friendly-AI is a good idea. Being on this site has me more convinced than before of the feasibility of FAI, but I still do not think that the creation of an FAI, or any other AI for that matter, should be attempted.

We have directives in much the same way that we would give an FAI directives. Our directives are not immediately clear, since if they were clear then producing an FAI would be simple, but we know that they are achieved through our emotions and desires, and that they were produced by our evolution. We have no reason to believe that our emotions are the most effective way we could be programmed to achieve (arguably) our primary goal; to further the species, but we know that they are effective (although I criticized their effectiveness in my previous post).

If we accept the material atheistic position as true, then we must accept that all of our emotions and experiences are the product of nothing more than interactions of matter and energy. With this in mind, we do not know if consciousness in an AI can be avoided, or if it is an emergent property of all complex systems, and thus we may be creating a being capable of immense suffering.

To argue that we could simply give an FAI super-human intelligence without any consciousness is not an escape, as the only intelligence we have yet seen to exhibit full general intelligence is our own, and some select non-human persons. This means that every general intelligence encountered is conscious (but that is not to say that every AGI must be). Although companies such as Google's DeepMind have shown huge success with their software, we cannot be certain that their software does not possess some level of consciousness as we have not yet created a reliable test for machine consciousness. Simply put, if we do not know the nature of our own consciousness and all other potential forms of consciousness then we cannot prevent it, especially as the most promising AI research is concerned with imitating certain structures of the human brain.

To argue that we could create an FAI that doesn't feel pain or emotions is also not an escape. Pain evolved to alert us to damage to our bodies, so the AI would quickly develop an equivalent to pain to protect its resources from damage. However, since the FAI may be able to ensure its resource...
Meetup : West LA: Improv & Rationality
by abramdemski — 2015-04-16T08:47:04.742Z

Discussion article for the meetup : West LA: Improv & Rationality

WHEN:
22 April 2015 07:00:00PM (-0700)

WHERE:
11066 Santa Monica Blvd, Los Angeles, CA

How to Find Us: Go into this Del Taco. We will be in the back room if possible.

Parking is free in the lot out front or on the street nearby.

Discussion: Do improvisational acting and rationality go together? I think this link should be explored. Improvisation involves a special kind of relationship with System 1, which most people need to train in order to pull off well. As such, learning improv skills may improve fast reactions, particularly in social settings. Improv games are also good group bonding activities. We will play some improv games geared toward rationality skills, and discuss possible relationships between improv and rationality.

Recommended Reading:

Improv tips: shorter, longer.

CFAR exercise brainstorming threads: consequentialism, be specific, motivated cognition

No prior exposure to Less Wrong is required; this will be generally accessible.

Discussion article for the meetup : West LA: Improv & Rationality
Meetup : West LA: Improv & Rationality
by abramdemski — 2015-04-16T08:46:24.828Z

Discussion article for the meetup : West LA: Improv & Rationality

WHEN:
22 April 2015 07:00:00PM (-0700)

WHERE:
11066 Santa Monica Blvd, Los Angeles, CA

[deleted -- accidental duplicate]

Discussion article for the meetup : West LA: Improv & Rationality
Why isn't the following decision theory optimal?
by Elliot_Olds — 2015-04-16T01:38:13.423Z

I've recently read the decision theory FAQ, as well as Eliezer's TDT paper. When reading the TDT paper, a simple decision procedure occurred to me which as far as I can tell gets the correct answer to every tricky decision problem I've seen. As discussed in the FAQ above, evidential decision theory get's the chewing gum problem wrong, causal decision theory gets Newcomb's problem wrong, and TDT gets counterfactual mugging wrong.

In the TDT paper, Eliezer postulates an agent named Gloria (page 29), who is defined as an agent who maximizes decision-determined problems. He describes how a CDT-agent named Reena would want to transform herself into Gloria. Eliezer writes

By Gloria’s nature, she always already has the decision-type causal agents wish they had, without need of precommitment.

Eliezer then later goes on the develop TDT, which is supposed to construct Gloria as a byproduct.

Gloria, as we have defined her, is defined only over completely decision-determined problems of which she has full knowledge. However, the agenda of this manuscript is to introduce a formal, general decision theory which reduces to Gloria as a special case.

Why can't we instead construct Gloria directly, using the idea of the thing that CDT agents wished they were? Obviously we can't just postulate a decision algorithm that we don't know how to execute, and then note that a CDT agent would wish they had that decision algorithm, and pretend we had solved the problem. We need to be able to describe the ideal decision algorithm to a level of detail that we could theoretically program into an AI.

Consider this decision algorithm, which I'll temporarily call Nameless Decision Theory (NDT) until I get feedback about whether it deserves a name: you should always make the decision that a CDT-agent would have wished he had pre-committed to, if he had previously known he'd be in his current situation and had the opportunity to precommit to a decision.

In effect, you are making an general precommittment to behave as if you made all specific precommitments that would ever be advantageous to you.

NDT is so simple, and Eliezer comes so close to stating it in his discussion of Gloria, that I assume there is some flaw with it that I'm not seeing. Perhaps NDT does not count as a "real"/"well defined" decision procedure, or can't be formalized for some reason? Even so, it does seem...
Cooperative conversational threading
by philh — 2015-04-15T18:40:50.820Z

(Cross-posted from my blog.)

Sometimes at LW meetups, I'll want to raise a topic for discussion. But we're currently already talking about something, so I'll wait for a lull in the current conversation. But it feels like the duration of lull needed before I can bring up something totally unrelated, is longer than the duration of lull before someone else will bring up something marginally related. And so we can go for a long time, with the topic frequently changing incidentally, but without me ever having a chance to change it deliberately.

Which is fine. I shouldn't expect people to want to talk about something just because I want to talk about it, and it's not as if I find the actual conversation boring. But it's not necessarily optimal. People might in fact want to talk about the same thing as me, and following the path of least resistance in a conversation is unlikely to result in the best possible conversation.

At the last meetup I had two topics that I wanted to raise, and realized that I had no way of raising them, which was a third topic worth raising. So when an interruption occured in the middle of someone's thought - a new person arrived, and we did the "hi, welcome, join us" thing - I jumped in. "Before you start again, I have three things I'd like to talk about at some point, but not now. Carry on." Then he started again, and when that topic was reasonably well-trodden, he prompted me to transition.

Then someone else said that he also had two things he wanted to talk about, and could I just list my topics and then he'd list his? (It turns out that no I couldn't. You can't dangle an interesting train of thought in front of the London LW group and expect them not to follow it. But we did manage to initially discuss them only briefly.)

This worked pretty well. Someone more conversationally assertive than me might have been able to take advantage of a less solid interruption than the one I used. Someone less assertive might not have been able to use that one.

What else could we do to solve this problem?

Someone suggested a hand signal: if you think of something that you'd like to raise for discussion later, make the signal. I don't think this is ideal, because it's not continuous. You make it once, and then it would be easy for people to forget, or just to not notice.

I think what I'm going to do is bring some poker chips to the next meetup. I'll put a bunch...
Meetup : Washington, D.C.: Sword of Good
by RobinZ — 2015-04-15T17:18:31.578Z

Discussion article for the meetup : Washington, D.C.: Sword of Good

WHEN:
19 April 2015 03:00:00PM (-0400)

WHERE:
Reynolds Center

We will be meeting in the Kogod Courtyard of the Donald W. Reynolds Center for American Art and Portraiture (8th and F Sts or 8th and G Sts NW, go straight past the information desk from either entrance) to talk about the LessWrongian short story "The Sword of Good". We will congregate between 3:00 and 3:30 p.m., and begin the discussion at 3:30.

"The Sword of Good" is a short story by Eliezer Yudkowsky in the form of an excerpt from a swords-and-sorcery fantasy novel. I won't give any major spoilers here, but it's a good read, and deconstructs some of the tropes of the subgenre. Discussion will continue as long as people are interested; as always, side conversations are permitted and encouraged.

The WMATA major track work schedule is clear for April 19; the weekend service adjustments chiefly affect the Red Line, which should run every 15 minutes.

Upcoming meetups:

Apr. 26: (Outdoors) Fun & Games (bring games, play games, converse, socialize, or any combination thereof)

May 3: Sharing Book Excerpts (bring books to share and discuss favorite passages)

Discussion article for the meetup : Washington, D.C.: Sword of Good
Concept Safety: Producing similar AI-human concept spaces
by Kaj_Sotala — 2015-04-14T20:39:31.321Z

A frequently-raised worry about AI is that it may reason in ways which are very different from us, and understand the world in a very alien manner. For example, Armstrong, Sandberg & Bostrom (2012) consider the possibility of restricting an AI via "rule-based motivational control" and programming it to follow restrictions like "stay within this lead box here", but they raise worries about the difficulty of rigorously defining "this lead box here". To address this, they go on to consider the possibility of making an AI internalize human concepts via feedback, with the AI being told whether or not some behavior is good or bad and then constructing a corresponding world-model based on that. The authors are however worried that this may fail, because

Humans seem quite adept at constructing the correct generalisations – most of us have correctly deduced what we should/should not be doing in general situations (whether or not we follow those rules). But humans share a common of genetic design, which the OAI would likely not have. Sharing, for instance, derives partially from genetic predisposition to reciprocal altruism: the OAI may not integrate the same concept as a human child would. Though reinforcement learning has a good track record, it is neither a panacea nor a guarantee that the OAIs generalisations agree with ours.

Addressing this, a possibility that I raised in Sotala (2015) was that possibly the concept-learning mechanisms in the human brain are actually relatively simple, and that we could replicate the human concept learning process by replicating those rules. I'll start this post by discussing a closely related hypothesis: that given a specific learning or reasoning task and a certain kind of data, there is an optimal way to organize the data that will naturally emerge. If this were the case, then AI and human reasoning might naturally tend to learn the same kinds of concepts, even if they were using very different mechanisms. Later on the post, I will discuss how one might try to verify that similar representations had in fact been learned, and how to set up a system to make them even more similar.

Word embedding

A particularly fascinating branch of recent research relates to the learning of word embeddings, which are mappings of words to very high-dimensional vectors. It turns out that if you train a system on one of several kinds of tasks, ...
Anti-Pascaline satisficer
by Stuart_Armstrong — 2015-04-14T19:13:55.000Z

It occurred to me that the anti-Pascaline agent design could be used as part of a satisficer approach.

The obvious thing to reduce dangerous optimisation pressure is to make a bounded utility function, with an easily achievable bound. Such as giving them a utility linear in paperclips that maxs out at 10.

The problem with this is that, if the entity is a maximiser (which it might become), it can never be sure that it's achieved its goals. Even after building 10 paperclips, and an extra 2 to be sure, and an extra 20 to be really sure, and an extra 3^^^3 to be really really sure, and extra cameras to count them, with redundant robots patrolling the cameras to make sure that they're all behaving well, etc... There's still an ε chance that it might have just dreamed this, say, or that its memory is faulty. So it has a current utility of (1-ε)10, and can increase this by reducing ε - hence building even more paperclips.

Hum... ε, you say? This seems a place where the anti-Pascaline design could help. Here we would use it at the lower bound of utility. It currently has probability ε of having utility < 10 (ie it has not built 10 paperclips) and (1-ε) of having utility = 10. Therefore and anti-Pascaline agent with ε lower bound would round this off to 10, discounting the unlikely event that it has been deluded, and thus it has no need to build more paperclips or paperclip counting devices.

Note that this is an un-optimising approach, not an anti-optimising one, so the agent may still build more paperclips anyway - it just has no pressure to do so.
Anti-Pascaline satisficer
by Stuart_Armstrong — 2015-04-14T18:49:23.787Z

A putative new idea for AI control; index here.

It occurred to me that the anti-Pascaline agent design could be used as part of a satisficer approach.

The obvious thing to reduce dangerous optimisation pressure is to make a bounded utility function, with an easily achievable bound. Such as giving them a utility linear in paperclips that maxs out at 10.

The problem with this is that, if the entity is a maximiser (which it might become), it can never be sure that it's achieved its goals. Even after building 10 paperclips, and an extra 2 to be sure, and an extra 20 to be really sure, and an extra 3^^^3 to be really really sure, and extra cameras to count them, with redundant robots patrolling the cameras to make sure that they're all behaving well, etc... There's still an ε chance that it might have just dreamed this, say, or that its memory is faulty. So it has a current utility of (1-ε)10, and can increase this by reducing ε - hence by building even more paperclips.

Hum... ε, you say? This seems a place where the anti-Pascaline design could help. Here we would use it at the lower bound of utility. It currently has probability ε of having utility < 10 (ie it has not built 10 paperclips) and (1-ε) of having utility = 10. Therefore and anti-Pascaline agent with ε lower bound would round this off to 10, discounting the unlikely event that it has been deluded, and thus it has no need to build more paperclips or paperclip counting devices.

Note that this is an un-optimising approach, not an anti-optimising one, so the agent may still build more paperclips anyway - it just has no pressure to do so.