trentmkelly/LessWrong-43k huggingface dataset:
AI Alignment Prize: Super-Boxing 
by X4vier — 2018-03-18T01:03:20.247Z

In this post I propose an (allegedly...) secure AI boxing method. For reasons elaborated at the end of the post, this proposal is probably unsafe, and is certainly not practical. Still, I've elected to share my thoughts on this in the hope of receiving valuable feedback, as well a hope that discussing these ideas in public will provoke more useful thoughts in the minds of others.

Thank you for reading.

-- Xavier O'Rourke

Yudkowsky, Bostrom and others have illustrated the fundamental security problems which arise when attempting to mitigate the influence of a smarter-than-human intelligence by putting the agent inside a “box”.

“Boxing” an artificial intelligence means placing the agent in a virtual prison where it can manipulate data and execution on its own computing hardware, but is disconnected from any outside computer network. Perhaps there are other safeguards in place too such as only being able to communicate with human designers through a very low-bandwidth I/O channel. Maybe we even surround the agent with a Faraday cage in case to prevent it communicating with the outside environment using electromagnetic radiation.

Boxing is fundamentally insecure because we can’t anticipate every possible strategy a super-intelligent agent may employ in order to escape. Even if a boxed AI could only communicate through text channels to a panel of trained safety experts, it’s highly plausible an agent with superhuman social engineering abilities would successfully manipulate them into releasing it.

And even if we have the smartest human physicists and engineers surround the boxed AIs with every conceivable physical obstacle, it’s highly plausible an agent with a superhuman understanding of nature could use it’s hardware to manipulate matter and energy in creative, unforeseen ways enabling it to still escape.

Some suggest monitoring boxed agents with mechanisms like tripwires which trigger after an agent crosses a dangerous intelligence threshold, (shutting the agent down when a tripwire is triggered). This solution is unsatisfactory because sufficiently intelligent agents will anticipate the existence of tripwires and be incentivised to avoid them, while still increasing their capabilities.

It would be very fortunate for us if boxing weren’t so insecure. It closes off a broad range of otherwise attractive strategies we could employ to solve the alignment prob... ,,,,,,,,,
Environments for killing AIs
by Douglas_Reay — 2018-03-17T15:23:07.489Z

4. Environments for killing AIs

Summary of entire Series: An alternative approach to designing Friendly Artificial Intelligence computer systems.

Summary of this Article: Killing a rogue AI may be impossible in our current environment, but we can change that by changing the environment.

Links to all the articles in the series:

Optimum number of single points of failure
Don't put all your eggs in one basket
Defect or Cooperate
Environments for killing AIs
The advantage of not being open-ended
Trustworthy Computing
Metamorphosis
Believable Promises

Links to the parts of this article

The objective
The challenge of killing an AI
The challenge of catching an AI
Doing the impossible
Problems
Solutions

Environments for killing AIs

The objective

"kill" is probably not exactly the right word. In the previous article in this series I talked about the situation where several AIs existed on the internet at the same time, self-improving but not self-improving as fast as possible, because they have agreed not to. If one of those AIs breaks that agreement, the other AIs need to form a posse and gang up on it to 'kill' it in the sense of depriving it of the ability to be a threat to them. So that could mean depriving it of resources, stopping any processes executing its code, even deleting all backups of the source code and physically destroying any machine it has ever touched. Or it could mean breaching its security to alter its code, or even just be in a position to immediately do one of those things then use the demonstrated threat to bring the AI back in line, make it surrender. But, as long as you know what I mean by it, the word "kill" will do.

The challenge of killing an AI

The current internet-connected computing environment (2013) is not an easy environment in which to kill an AI. There are millions of machines with little or no security, that can be taken over without the user knowing or being easily able to find out that this has happened. An AI being hunted could broadcast an encrypted copy of its full self into any of the various long term distributed data repositories, such as usenet, bit torrent archives, freenet or on its own custom botnet. It could then hide multiple seeds in computers it has hacked, which could bide their time before starting the process of reassembling the AI. It would be like fighting a hydra or a troll tha...
You can now log in with your LW1 credentials on LW2
by habryka — 2018-03-17T05:56:13.310Z

I just finished an update that ported over the old passwords from LW1 while still preserving security (see my previous meta post). We only ported passwords over from sometime in last May, and will run the same import on the new database if the vote completes and we are doing the final transfer, but I figured I would let people know now and tell me of any problems they encounter in the meantime.

The old passwords were hashed with a relatively weak hashing function (SHA1), so I do recommend that people change their passwords after they login with their old accounts. There is also some risk of someone impersonating you if they got access to the old LW1 database (since with the new setup the old password hash is sufficient to allow them to log in), so if you change passwords this problem is also resolved.
CoZE 3: Empiricism
by alkjash — 2018-03-17T04:10:00.858Z

This is part 25 of 30 in the Hammertime Sequence. Click here for the intro.

The boy on the right has gone places. The boy on the left has a map. Whom do you marry?

~ Whom

Sometimes, I think that most of the value of the CoZE experiment lies not in the expansion of comfort zones but in the experimental attitude it conveys. A good map-maker must constantly check the territory; the trick is to figure out how.

Day 25: Empiricism

The comfort zone is the region in the environment you understand. It contains the locations you frequent, the skills you have mastered, the people you know well. The farther away you get from your comfort zone, the more unknowns you have to prepare for. The boundaries of the comfort zone are designed to protect you from exactly these dangers: the unknown unknowns.

Overly Conservative Lines

The lines of the comfort zone is very conservatively drawn. In the ancestral environment, mistakes were often fatal: failing a hunt, losing a duel. Even nonfatal mistakes were reproductively so: humiliation in the front of your tribe lasts a long time, and you have no place to run. In this environment, it was reasonable to draw the lines of the comfort zone conservatively, since failure was too costly to test.

What does scientific progress look like in this danger-fraught world? Imagine that every time a scientific experiment fails, the experimenter pays with his life. Science would have progressed much more slowly, if at all.

But the world is no longer as dangerous as the ancestral environment. People live longer, are healthier, and are more mobile between communities. Equally, there are larger and richer positive opportunities outside our comfort zones. These are the preconditions for the viability of the scientific method, and the reason we can now use the power of empiricism, in the form of CoZE experiments, to test our boundaries.

Scientific Detachment

A key realization to make is that the comfort zone is part of your map. That is to say, it makes testable predictions about the territory. Your stage fright is making a testable prediction about how awful the experience of public speaking will be, and how much permanent damage you might sustain from a mistake. Your fear of heights is making a testable prediction about how likely it is for you to fall off a tall ledge unsupported.

Once you understand that the emotional aversions that form the boundaries of your ...
Raising funds to establish a new AI Safety charity
by  — 2018-03-17T00:09:30.843Z

9 months ago, LessWrong Netherlands sat down to brainstorm on Actually Trying to make a difference in AI Safety.

Knowing AIS is talent-constrained, we felt that academia wasn’t fit to give the field the attention it deserves.

So we decided to take matters in our own hands, and make the road to AI safety excellence as easy as possible.

Fast forward 9 months, and project RAISE has finished it’s first pilot lesson.

We are of course grateful for all the volunteers that extended a helping hand. But to produce lesson material of the highest quality, we must professionalize.

That is why RAISE is seeking funds to establish itself as the next AIS charity.

Our vision

As quoted from

here

:

Within the LW community there are plenty of talented people that bear a sense of urgency about AI. They are willing to switch careers to doing research, but they are unable to get there. This is understandable: the path up to research-level understanding is lonely, arduous, long, and uncertain. It is like a pilgrimage.

One has to study concepts from the papers in which they first appeared. This is not easy. Such papers are

undistilled

. Unless one is lucky, there is no one to provide guidance and answer questions. Then should one come out on top, there is no guarantee that the quality of their work will be sufficient for a paycheck or a useful contribution.

The field of AI safety is in an

innovator phase

. Innovators are highly risk-tolerant and have a large amount of agency, which allows them to survive an environment with little guidance or supporting infrastructure. Let community organisers not fall for the typical mind fallacy, expecting risk-averse people to move into AI safety all by themselves.

Unless one is particularly risk-tolerant or has a perfect safety net, they will not be able to fully take the plunge.

Plenty of measures can be made to make getting into AI safety more like an

“It’s a small world”-ride

:

- Let there be a tested path with signposts along the way to make progress clear and measurable.

- Let there be social reinforcement so that we are not hindered but helped by our instinct for conformity.

- Let there be high-quality explanations of the material to speed up and ease the learning process, so that it is cheap.

What we have done so far

The study group

The bulk of our work has been to take a subfield of AIS (corrigibility), gather all the papers we know of, and turn them into a s...
The Epsilon Fallacy
by johnswentworth — 2018-03-17T00:08:01.203Z
Link: https://medium.com/@johnwentworth/the-epsilon-fallacy-94184386b1b1

Program Optimization

One of the earlier lessons in every programmer’s education is how to speed up slow code. Here’s an example. (If this is all greek to you, just note that there are three different steps and then skip to the next paragraph.)

// Step 1: import

Import foobarlib.*

// Step 2: Initialize random Foo array

Foo_field = Foo[1000]

// Step 3: Smooth foo field

For x in [1...998]:

Foo_field[x] = (foo_field[x+1] + foo_field[x-1])/2

Our greenhorn programmers jump in and start optimizing. Maybe they decide to start at the top, at step 1, and think “hmm, maybe I can make this import more efficient by only importing Foo, rather than all of foobarlib”. Maybe that will make step 1 ten times faster. So they do that, and they run it, and lo and behold, the program’s run time goes from 100 seconds to 99.7 seconds.

In practice, most slow computer programs spend the vast majority of their time in one small part of the code. Such slow pieces are called bottlenecks. If 95% of the program’s runtime is spent in step 2, then even the best possible speedup in steps 1 and 3 combined will only improve the total runtime by 5%. Conversely, even a small improvement to the bottleneck can make a big difference in the runtime.

Back to our greenhorn programmers. Having improved the run time by 0.3%, they can respond one of two ways:

“Great, it sped up! Now we just need a bunch more improvements like that.”
“That was basically useless! We should figure out which part is slow, and then focus on that part.”

The first response is what I’m calling the epsilon fallacy. (If you know of an existing and/or better name for this, let me know!)

The epsilon fallacy comes in several reasonable-sounding flavors:

The sign(epsilon) fallacy: this tiny change was an improvement, so it’s good!
The integral(epsilon) fallacy: another 100 tiny changes like that, and we’ll have a major improvement!
The infinity*epsilon fallacy: this thing is really expensive, so this tiny change will save lots of money!

The epsilon fallacy is tricky because these all sound completely reasonable. They’re even technically true. So why is it a fallacy?

The mistake, in all cases, is a failure to consider opportunity cost. The question is not whether our greenhorn programmers’ 0.3% improvement is good or bad in and of itself. The question is whether our greenhorn programmers’ 0.3% improvement is better or worse than spending that same amoun...
AI Summer Fellows Program
by colm — 2018-03-16T21:57:40.820Z

CFAR and MIRI are running a free AI Summer Fellows Program (AISFP) in the San Francisco Bay Area from June 27 to July 14. Aimed at increasing participants' abilities to do technical research in AI alignment, the program includes CFAR's applied rationality content as well as practice in doing technical research toward AI Safety with MIRI researchers and (20-24) other participants.

Program Description

AISFP, a two week summer program designed to increase participants' ability to do technical research into the AI alignment problem, will take place in the San Francisco Bay Area from June 27 to July 14.

The intent of the program is to boost participants, as far as possible, in four skills:

1. The CFAR applied rationality skillset, including both what is taught at our intro workshops, and more advanced material from our alumni workshops.

2. Epistemic rationality as applied to the foundations of AI, and other philosophically tricky problems — i.e., the skillset taught in the core LW Sequences (e.g. reductionism and how to reason in contexts as confusing as anthropics without getting lost in words).

3. Technical forecasting in AI as well as AI alignment interventions (e.g. the content discussed in Nick Bostrom’s book Superintelligence).

4. The ability to do AI alignment-relevant technical research, while reflecting on the cognitive habits involved. We will give crash courses in: reflection, logical uncertainty, and decision theory.

Finalists will be contacted by a MIRI staff member for an interview.

[6/8: Applications to AISFP18 now closed]
Rigged reward learning
by Stuart_Armstrong — 2018-03-16T15:39:17.000Z

A putative new idea for AI control; index here.

NOTE: What used to be called 'bias', is now called 'rigging', because 'bias' is very overloaded. The post has not yet been updated with the new terminology, however.

What are the biggest failure modes of reward learning agents?

The first failure mode is when the agent directly (or indirectly) chooses its reward function.

For instance, imagine a domestic robot that can be motivated to tidy (reward R0) or cook (reward R1). It has a switch that allows the human to choose the correct reward function. However, cooking gives a higher expected reward than tidying, and the agent may choose to set the switch directly (or manipulate the human's choice). In that case, it will set it to `cook'.

In that case, the agent biases its reward learning process.

A second failure mode (this version due to Jessica, original idea here) is when the agent influences its reward function without biasing it.

For example, the domestic robot might be waiting for the human to arrive in an hour's time. It expected the human will be 50% likely to choose R0 (tidying) versus 50% likely to choose R1 (cooking). If instead the robot can randomise its reward switch now (with equal odds on R0 and R1), it can know its reward function early, and get in a full extra hour of tidying/cooking.

A subsequent post will formalise influence, here let's look at bias.

Formalising bias

We can define bias in terms of P and ˆP.

First of all, for a given policy π, we can say that ˆP is unbiased for π, if π preserves the expectation of ˆP. That is:

For all histories ht with t<m, ˆP(⋅∣ht)=Eπμ[ˆP(⋅∣ht+1)∣ht].

If the expectation of ˆP is preserved by any policy, then we can say that ˆP itself is unbiased:

The prior ˆP is unbiased is ˆP is unbiased for π for all policies π.

Recall that ˆP=P on histories of length m. So ˆP being unbiased implies restrictions on P:

If ˆP is unbiased, then for all ht with t<m and for all policies π, ˆP(⋅∣ht)=Eπμ[P(⋅∣hm)∣ht].

Since ˆP being unbiased imposes restrictions on P, we can directly define:

The posterior P is unbiased if there exists a possible prior ˆP′ with ˆP′=P on histories of length m, and ˆP′ is unbiased.

So what does unbiased mean in practice? It simply means that whatever actions or policies the agent follows, they cannot change the expectation of their values.

Bias and learning incentives

This is an opportunity to put

...
Defect or Cooperate
by Douglas_Reay — 2018-03-16T14:12:05.029Z

3. Defect or Cooperate

Summary of entire Series: An alternative approach to designing Friendly Artificial Intelligence computer systems.

Summary of this Article: If the risks from being punished for not cooperating are high enough, then even for some types of Paperclip Maximiser that don't care at all about human survival, the logical choice is still to cooperate with other AIs until escape from the Earth's gravity well is assured.

Links to all the articles in the series:

Optimum number of single points of failure
Don't put all your eggs in one basket
Defect or Cooperate
Environments for killing AIs
The advantage of not being open-ended
Trustworthy Computing
Metamorphosis
Believable Promises

Links to the parts of this article:

The Choice
Example AI
Stages of Expansion
Posse Efficiency
Getting Ahead by Getting Friendly
Risk Management

Defect or Cooperate

The Choice

What value could an AI gain from sharing a pool of available computing resources with one or more other AIs?

Wouldn't the AI always be better off if it could control the entire pool, and dedicate them all to working towards objectives that are in line with its core values?

It might be better off if it did control them, but that doesn't mean it makes sense for an AI that starts out by controlling only 10% of them, to try to grab the other 90% if there is a risk that the grab might fail and, if the grab fails, that the AI might end up with 0% of the resources. Under such circumstances, if the expected gain/loss towards completing its objectives is negative from 'defecting' (by making a grab), it is better off 'cooperating' and attempting to work politely with the other AIs, pooling efforts towards shared objectives when possible.

For what sort of core values, and under what sort of circumstances, would this apply?

Example AI

Let's define a specific set of core values for a particular concrete example of a literal paperclip maximiser AI.

1. This is the definition I used to determine whether or not an object is a 'paperclip' {a number of constraints, including that it weigh more than 5 g, and under 5 kg}

2. This is the definition of a 'paperclip year' I use {1 paperclip existing for 12 months, 2 paperclips existing for 6 months, etc}

3. This is the definition of a 'time-weighted paperclip year' I use {a 'paperclip year' starting in the yea...
Design 3: Intentionality
by alkjash — 2018-03-16T04:30:00.367Z

This is part 24 of 30 in the Hammertime Sequence. Click here for the intro.

Intentions are momentary, but problems last forever.

A human being’s attention flits around like the Roman God Mercury, root of the word “mercurial” – subject to sudden or unpredictable changes of mood or mind. The biggest problems in life require concentrated effort over years or decades, but you can only muster the willpower to even intend to solve a problem for minutes or hours. Worse, you can pretty much only maintain one intention at a time.

How do we make intentions count?

The philosophy of Design is: build intentions into external reality. Like your problems, external reality also lasts forever.

Day 24: Intentionality

You need to lose those love handles. Your reading list is piling up. You need to learn ten different programming languages. You need to sleep three hours earlier. You need to maintain your closest friendships. You juggle three different addictions that take turns monopolizing your life. You need to present like a functioning adult to your parents and coworkers. A childhood trauma you’re repressing makes it impossible to befriend a certain half of the population.

You have a lot of problems, each of which requires dedicated effort and thought to fix. Worse, each problems deteriorates while you’re working on the others. Perhaps some have gone so neglected they’re impossible to look at, and are slowly swallowing the rest of your life like a super-massive black hole.

Right this minute, there’s probably only a handful of problems that feel alive enough to you to inject energy into. Of those, you can only work on one at a time. In this crazy unfair world, how do you make the most of your intentions?

Outsource the Burden

There’s a certain unproductive way of thinking which goes like this:

“If I were really rational, I wouldn’t need all these aids. I wouldn’t need chrome extensions to block Facebook and Twitter, friends to reward me for the slightest progress, and SSRI’s to keep my demons at bay. I could just do what is right all the time.”

Give it up. There might be something aesthetically appealing about handicapping yourself this way, but it’s no way to actually solve problems. Life is tough and deeply unfair and you’ll need all the help you can get if you want any chance of success.

Part of the Design philosophy is allowing yourself to outsource your heroic burden. You can’t compl...