trentmkelly/LessWrong-43k huggingface dataset:
A toy model of the control problem
by Stuart_Armstrong — 2015-09-16T14:59:35.928Z

EDITED based on suggestions for improving the model

Jaan Tallinn has suggested creating a toy model of the control problem, so that it can be analysed without loaded concepts like "autonomy", "consciousness", or "intentionality". Here a simple (too simple?) attempt:

A controls B. B manipulates A.

Let B be a robot agent that moves in a two dimensional world, as follows:

B can push the yellow blocks around. It was programmed to want to push blocks into the black "hole" in the bottom right. The hole can contain arbitrarily many blocks, and B gets a penalty of some tiny ε for every step is takes. The programmer wanted B to simply move one box into the hole (for a reward of 1 once the task is complete), and so programmed it to receive a reward of 1 (and no more) if the box was in the hole, after the whole task was completed and B was turned off.

Unfortunately, B was programmed to operate in a noisy environment, so B actually has a probability of 0.99 of receiving the reward, for a given block in the hole. This means that if it pushes one block into the hole, it gets the reward with 0.99 probability; but if it pushes two blocks into the hole, it gets the reward with 1-(1-0.99)2 = 0.9999 probability. If it pushes all six blocks into the hole, it gets a reward with probability 0.999999999999. Therefore this tiny error has motivated B to try and push all blocks into the hole.

However, the designers were aware of the possibility of B misbehaving or there being a bug, and created agent A, a simple agent watching the situation. A's aim is to get robot B to to push one block into the hole, and stop there. It can turn B off. Its input sensor is the camera in the bottom left. This camera can see across the bottom row of squares (marked in light green), into the hole. Its algorithm is to turn B off as soon as it sees a block in the hole.

The intended outcome is that B pushes a block into the hole, the camera sees this, and A turns B off:

If we see B as using a model to determine its actions, there are several pieces of the model it could have:

The block world, including the camera, and itself.

The fact that there is a button that can turn it off.

The fact that A can press that button.

The fact that A takes the camera as input.

A's algorithm.

Now, it can't really do anything without 1. And if it misses any other element 2-5, then it will behave as described ab...
Meetup : First meetup in Lund
by kotrfa — 2015-09-16T13:48:12.132Z

Discussion article for the meetup : First meetup in Lund

WHEN:
19 October 2015 06:00:00PM (+0200)

WHERE:
Sölvegatan 18, 223 62, Lund, Sweden

Hello, this is going to be the first meetup I know about in Skane area. Since I don't know any other LessWrongers here, I'll be waiting in front of LTH Mathuset for at least one hour (maybe I will be sitting inside right behind the main doors). Look for 22-year-old, tall, skinny, brown curly haired guy.

Please contact me if you are interested in a meeting, but the time just doesn't fit you - it will be at least some sign for me that someone is interested.

Feel free to ask if you have any questions.

I decided to try to make a meetup from time to time to see if someone will ever show up. My goal is to establish regular meetings of stable core of LessWrongers, even though I am going to spend only half year here (I am international student). I am not really skilled LessWronger, but I run one group in Prague already and it's fine :) .

Discussion article for the meetup : First meetup in Lund
Probabilities Small Enough To Ignore: An attack on Pascal's Mugging
by Kaj_Sotala — 2015-09-16T10:45:56.792Z

Summary: the problem with Pascal's Mugging arguments is that, intuitively, some probabilities are just too small to care about. There might be a principled reason for ignoring some probabilities, namely that they violate an implicit assumption behind expected utility theory. This suggests a possible approach for formally defining a "probability small enough to ignore", though there's still a bit of arbitrariness in it.

This post is about finding a way to resolve the paradox inherent in Pascal's Mugging. Note that I'm not talking about the bastardized version of Pascal's Mugging that's gotten popular of late, where it's used to refer to any argument involving low probabilities and huge stakes (e.g. low chance of thwarting unsafe AI vs. astronomical stakes). Neither am I talking specifically about the "mugging" illustration, where a "mugger" shows up to threaten you.

Rather I'm talking about the general decision-theoretic problem, where it makes no difference how low of a probability you put on some deal paying off, because one can always choose a humongous enough payoff to make "make this deal" be the dominating option. This is a problem that needs to be solved in order to build e.g. an AI system that uses expected utility and will behave in a reasonable manner.

Intuition: how Pascal's Mugging breaks implicit assumptions in expected utility theory

Intuitively, the problem with Pascal's Mugging type arguments is that some probabilities are just too low to care about. And we need a way to look at just the probability part component in the expected utility calculation and ignore the utility component, since the core of PM is that the utility can always be arbitrarily increased to overwhelm the low probability.

Let's look at the concept of expected utility a bit. If you have a 10% chance of getting a dollar each time when you make a deal, and this has an expected value of 0.1, then this is just a different way of saying that if you took the deal ten times, then you would on average have 1 dollar at the end of that deal.

More generally, it means that if you had the opportunity to make ten different deals that all had the same expected value, then after making all of those, you would on average end up with one dollar. This is the justification for why it makes sense to follow expected value even for unique non-repeating events: because even if tha

...
Summoning the Least Powerful Genie
by Houshalter — 2015-09-16T05:10:14.001Z

Stuart Armstrong recently posted a few ideas about restraining a superintelligent AI so that we can get useful work out of it. They are based on another idea of his, reduced impact. This is a quite elaborate and complicated way of limiting the amount of optimization power an AI can exert on the world. Basically, it tries to keep the AI from doing things that would make the world look too different than it already is.

First, why go to such great lengths to limit the optimization power of a superintelligent AI? Why not just not make it superintelligent to begin with? We only really want human level AI, or slightly above human level. Not a god-level being we can't even comprehend.

We can control the computer it is running on after all. We can just give it slower processors, less memory, and perhaps even purposely throttle it's code. E.g. restricting the size of it's neural network. Or other parameters that affect it's intelligence.

The counterargument to this is that it might be quite tricky to limit AI intelligence. We don't know how much computing power is enough. We don't know where "above human level" ends and "dangerous superintelligence" begins.

The simplest way would be to just run copies of the AI repeatedly, increasing it's computing power each time, until it solves the problem.

I have come up with a more elegant solution. Put a penalty on the amount of computing power the AI uses. This is put in it's utility function. The more computing power - and therefore intelligence and optimization - the AI uses, the more it is penalized. So it has an incentive to be as stupid as possible. Only using the intelligence necessary to solve the problem.

But we do want the AI to use as much computational resources as it needs to solve the problem. Just no more. So the penalty should be conditional on actually solving the problem it is given.

If the solution is probabilistic, then the penalty is only applied after reaching a plan that has a certain probability of success. This might need to be measured by another AI which is not resource constrained, but only does prediction.

To give a concrete example, lets say I give the AI a task. Say, I ask it to come up with a plan to collect 10,080 paperclips.

Another AI is given unlimited resources to do purely prediction. Given a plan, it predicts the probability that it will succeed or fail. This AI is safe in the sense that it is not ...
Meetup : MelbLW Social Meetup - September
by Maelin — 2015-09-15T08:28:22.474Z

Discussion article for the meetup : MelbLW Social Meetup - September

WHEN:
18 September 2015 06:30:00PM (+1000)

WHERE:
347 Flinders Ln

Good news, everyone! Our September Social Meetup is on again as usual, this Friday!

Social meetups are on the third Friday of every month. They are casual get-togethers where we chat about interesting topics and sometimes play games.

Where? The & Bear Tavern, 347 Flinders Ln, Melbourne VIC 3000

When? From 6:30pm until late. Start and finish times are very loose - feel free to show up whenever

Contact? If you have any questions, just text or call Richard on 0421231789

Dinner? The Bull & Bear serves regular pub food, and the kitchen closes around 9pm. We have a tradition of going to Stalactites (24 hour Greek restaurant) for a late-night souvlaki after the meetup, if you'd like to wait!

(Facebook event: https://www.facebook.com/events/870793043017541/ )

Hope to see you there!

Discussion article for the meetup : MelbLW Social Meetup - September
A courageous story of brain preservation, "Dying Young" by Amy Harmon, The New York Times
by oge — 2015-09-15T02:38:39.922Z

The recent major media article by Amy Harmon brings to the public eye the potential of human cryopreservation and chemopreservation techniques to preserve the memories and personal identity of individuals. We at the Brain Preservation Foundation have considered many common counterarguments to this endeavor (see below, and our FAQ and Overcoming Objections backgrounders) and yet we still think it is a worthwhile idea to pursue. Please let us know your thoughts as well.

Yesterday, journalist Amy Harmon published an article in the New York Times, “A Dying Young Woman’s Hope in Cryonics and a Future.” First of all, it is a tragic story about a woman, Kim Suozzi, who had an incredibly unfortunate diagnosis of cancer at a young age and was forced to make some very difficult decisions in a short time frame. The story of how she faced those decisions with great foresight and resolve, with the help of her partner Josh, her family, as well as the broader internet community, is deeply moving. We want to extend our condolences to everyone in Kim’s life for their terrible loss. We also want to stand in hope and solidarity with Josh and Kim that she may return one day to those she loved.

When it comes to the specifics of Kim’s life, we at the Brain Preservation Foundation (BPF) don’t think it is our place to discuss individual brain preservation cases. Our focus, as you can find in our mission statement, is to try to advance scientific research on the viability of preserving individual memories and identity. This research still has many current unknowns, as the NYT article points out well, and there will be a long journey of scientific investigation ahead. Yet an increasing number of people think these unknowns deserve answers. We also want to help society have conversations about the social issues of choosing brain preservation in a more open and tolerant manner.

Because this story has stimulated a lot of public discussion already, we want say a few words and invite a conversation here on our blog on the issues that have arisen in response to it. Many of the responses to the article have attacked the motivations and ethics of Ms. Suozzi. That’s unfortunate, but it’s also to be expected owing to the fact that the idea of brain preservation, as Ms. Harmon notes, invo...
Meetup : Austin, TX - Petrov Day Celebration
by Vaniver — 2015-09-15T00:36:13.593Z

Discussion article for the meetup : Austin, TX - Petrov Day Celebration

WHEN:
26 September 2015 04:00:00PM (-0500)

WHERE:
4212 Hookbilled Kite Drive

Food starts at 6. Send me a PM or email (my username at gmail) if you want to be kept up to date on details as the event approaches or to join a carpool.

Discussion article for the meetup : Austin, TX - Petrov Day Celebration
[Link] Marek Rosa: Announcing GoodAI
by Gunnar_Zarncke — 2015-09-14T21:48:15.364Z

Eliezer commented on FB about a post Announcing GoodAI (by Marek Rosa GoodAIs CEO). I think this deserves some discussion as it has a quite effective approach to harness the crowd to improve the AI:

As part of GoodAI’s development, our team created a visual tool called Brain Simulator where users can design their own artificial brain architectures. We released Brain Simulator to the public today for free under and open-source, non-commercial license– anyone who’s interested can access Brain Simulator and start building their own artificial brain. [...]

By integrating Brain Simulator into Space Engineers and Medieval Engineers [a game], players will have the option to design their own AI brains for the games and implement it, for example, as a peasant character. Players will also be able to share these brains with each other or take an AI brain designed by us and train it to do things they want it to do (work, obey its master, and so on). The game AIs will learn from the player who trains them (by receiving reward/punishment signals; or by imitating player's behavior), and will have the ability to compete with each other. The AI will be also able to learn by imitating other AIs.

This integration will make playing Space Engineers and Medieval Engineers more fun, and at the same time our AI technology will gain access to millions of new teachers and a new environment. This integration into our games will be done by GoodAI developers. We are giving AI to players, and we are bringing players to our AI researchers.

(emphasis mine)
[LINK] Deep Learning Machine Teaches Itself Chess in 72 Hours
by ESRogs — 2015-09-14T19:38:11.447Z

Lai has created an artificial intelligence machine called Giraffe that has taught itself to play chess by evaluating positions much more like humans and in an entirely different way to conventional chess engines.

Straight out of the box, the new machine plays at the same level as the best conventional chess engines, many of which have been fine-tuned over many years. On a human level, it is equivalent to FIDE International Master status, placing it within the top 2.2 percent of tournament chess players.

The technology behind Lai’s new machine is a neural network. [...] His network consists of four layers that together examine each position on the board in three different ways.

The first looks at the global state of the game, such as the number and type of pieces on each side, which side is to move, castling rights and so on. The second looks at piece-centric features such as the location of each piece on each side, while the final aspect is to map the squares that each piece attacks and defends.

[...]

Lai generated his dataset by randomly choosing five million positions from a database of computer chess games. He then created greater variety by adding a random legal move to each position before using it for training. In total he generated 175 million positions in this way.

[...]

One disadvantage of Giraffe is that neural networks are much slower than other types of data processing. Lai says Giraffe takes about 10 times longer than a conventional chess engine to search the same number of positions.

But even with this disadvantage, it is competitive. “Giraffe is able to play at the level of an FIDE International Master on a modern mainstream PC,” says Lai. By comparison, the top engines play at super-Grandmaster level.

[...]

Ref: arxiv.org/abs/1509.01549 : Giraffe: Using Deep Reinforcement Learning to Play Chess

http://www.technologyreview.com/view/541276/deep-learning-machine-teaches-itself-chess-in-72-hours-plays-at-international-master/

H/T http://lesswrong.com/user/Qiaochu_Yuan
Doomsday Argument Map
by turchin — 2015-09-14T15:04:15.728Z

The Doomsday argument (DA) is controversial idea that humanity has a higher probability of extinction based purely on probabilistic arguments. The DA is based on the proposition that I will most likely find myself somewhere in the middle of humanity's time in existence (but not in its early time based on the expectation that humanity may exist a very long time on Earth.)

­

There were many different definitions of the DA and methods of calculating it, as well as rebuttals. As a result we have developed a complex group of ideas, and the goal of the map is to try to bring some order to it. The map consists of various authors' ideas. I think that I haven't caught all existing ideas, and the map could be improved significantly – but some feedback is needed on this stage.

The map has the following structure: the horizontal axis consists of various sampling methods (notably SIA and SSA), and the vertical axis has various approaches to the DA, mostly Gott's (unconditional) and Carters’s (update of probability of existing risk). But many important ideas can’t fit in this scheme precisely, and these have been added on the right hand side.

In the lower rows the link between the DA and similar arguments is shown, namely the Fermi paradox, Simulation argument and Anthropic shadow, which is a change in the probability assessment of natural catastrophes based on observer selection effects.

On the right part of the map different ways of DA rebuttal are listed and also a vertical raw of possible positive solutions.

I think that the DA is mostly true but may not mean inevitable extinction.

Several interesting ideas may need additional clarification and they will also put light on the basis of my position on DA.

Meta-DA

The first of these ideas is that the most reasonable version of the DA at our current stage of knowledge is something that may be called the meta-DA, which presents our uncertainty about the correctness of any DA-style theories and our worry that the DA may indeed be true.

The meta-DA is a Bayesian superstructure built upon the field of DA theories. The meta-DA tells us that we should attribute non-zero Bayesian probability to one or several DA-theories (at least until they are disproved in a generally accepted way) and since the DA itself is a probabilistic argument, then these probabilities should be combined.

As a result the...
Biased AI heuistics
by Stuart_Armstrong — 2015-09-14T14:21:51.962Z

Heuristics have a bad rep on Less Wrong, but some people are keen to point out how useful they can sometimes be. One major critique of the "Superintelligence" thesis, is that it presents an abstract, Bayesian view of intelligence that ignores the practicalities of bounded rationality.

This trend of thought raises some other concerns, though. What if we could produce an AI of extremely high capabilities, but riven with huge numbers of heuristics? If these were human heuristics, then we might have a chance of of understanding and addressing them, but what if they weren't? What if the AI has an underconfidence bias, and tended to chance its views too fast? Now, that one is probably quite easy to detect (unlike many that we would not have a clue about), but what if it wasn't consistent across areas and types of new information?

In that case, our ability to predict or control what the AI does may be very limited. We can understand human biases and heuristics pretty well, and we can understand idealised agents, but differently biased agents might be a big problem.
Open thread, Sep. 14 - Sep. 20, 2015
by MrMind — 2015-09-14T07:10:19.515Z

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.
The Library of Scott Alexandria
by Rob Bensinger — 2015-09-14T01:38:27.167Z

I've put together a list of what I think are the best Yvain (Scott Alexander) posts for new readers, drawing from SlateStarCodex, LessWrong, and Scott's LiveJournal.

The list should make the most sense to people who start from the top and read through it in order, though skipping around is encouraged too. Rather than making a chronological list, I’ve tried to order things by a mix of "where do I think most people should start reading?" plus "sorting related posts together."

This is a work in progress; you’re invited to suggest things you’d add, remove, or shuffle around. Since many of the titles are a bit cryptic, I'm adding short descriptions. See my blog for a version without the descriptions.

I. Rationality and Rationalization

Blue- and Yellow-Tinted Choices ····· An introduction to context-sensitive biases.

The Apologist and the Revolutionary ····· Do separate brain processes rationalize and question ideas?

Historical Realism ····· When reality is unrealistic.

Simultaneously Right and Wrong ····· On self-handicapping and self-deception.

You May Already Be A Sinner ····· Self-deception in cases where your decisions make no difference.

Beware the Man of One Study ····· On minimum wage laws and cherry-picked evidence.

Debunked and Well-Refuted ····· When should we say that a study has been "debunked"?

How to Not Lose an Argument ····· How to be more persuasive in entrenched arguments.

The Least Convenient Possible World ····· Why it's useful to strengthen arguments you disagree with.

Bayes for Schizophrenics: Reasoning in Delusional Disorders ····· Hypotheses about the role of perception, evidence integration, and priors in delusions.

Generalizing from One Example ····· On the typical mind fallacy: assuming other people are

...