trentmkelly/LessWrong-43k huggingface dataset:
A general model of safety-oriented AI development
by Wei Dai — 2018-06-11T21:00:02.670Z

This may be trivial or obvious for a lot of people, but it doesn't seem like anyone has bothered to write it down (or I haven't looked hard enough). It started out as a generalization of Paul Christiano's IDA, but also covers things like safe recursive self-improvement.

Start with a team of one or more humans (researchers, programmers, trainers, and/or overseers), with access to zero or more AIs (initially as assistants). The human/AI team in each round develops a new AI and adds it to the team, and repeats this until maturity in AI technology is achieved. Safety/alignment is ensured by having some set of safety/alignment properties on the team that is inductively maintained by the development process.

The reason I started thinking in this direction is that Paul's approach seemed very hard to knock down, because any time a flaw or difficulty is pointed out or someone expresses skepticism on some technique that it uses or the overall safety invariant, there's always a list of other techniques or invariants that could be substituted in for that part (sometimes in my own brain as I tried to criticize some part of it). Eventually I realized this shouldn't be surprising because IDA is an instance of this more general model of safety-oriented AI development, so there are bound to be many points near it in the space of possible safety-oriented AI development practices. (Again, this may already be obvious to others including Paul, and in their minds IDA is perhaps already a cluster of possible development practices consisting of the most promising safety techniques and invariants, rather than a single point.)

If this model turns out not to have been written down before, perhaps it should be assigned a name, like Iterated Safety-Invariant AI-Assisted AI Development, or something pithier?
Thoughts on the Inner Bruce
by LeoHolman — 2018-06-11T20:18:39.803Z

This is a response to an article recommended to me.

So Bruce, according to Rizzo, needs to fail. He will influence decisions so that the outcome is failure. But Rizzo also points to how people point outward for reasons. “I got mana screwed.” There seems to be two bits that produce one idea. “I did nothing wrong.” + “I lost.” = “I deserve to lose.” I think this is the essence of Bruce.

Rizzo offers Bruce as a failure engine, an agent in direct opposition to whichever agent is acting on behalf of The Goal. The Goal is to win the game of magic, Bruce, therefore, is looking for the out, the way to lose. But it’s more complicated than that. Bruce has rules. If Bruce were really at the wheel, you might declare you’re a failure by crying and running away from the match. But you don’t. You just accept a hand you know is going to leave you mana screwed. It seems Bruce is inherently a hidden agent. He acts such that what seems like the most likely reason for your failure is not “I was trying to lose.” Or “I made decisions that pointed me toward failure.”

I’m playing with the idea that Bruce is not just the inverse of success. I’m thinking that evolutionarily this trait should have extinguished itself in a big hurry. If The Goal is to eat, and Bruce lets you purposefully go to the part of the forest you know has less food, you’d die of starvation before you could evolve into modern humans. So how can Bruce exist? I think Bruce is a social failsafe. Consider the following example.

Your nephew just learned how to play chess. He knows how all the pieces move but he doesn’t know anything about strategy and constantly puts his pieces into situations that causes him to lose them because he hasn’t yet learned how to check for that. You’re decent at chess, certainly no grandmaster but your fundamental grasp of chess strategy is enough that you can defeat your nephew 100% of the time (assume your nephew remains at the same skill level). You lose to your nephew. Why? Rizzo might say, “Bruce, obviously. Because you’re so terrible at chess even a child could beat you and Bruce wants to make sure you know that.” I offer an alternative. There’s greater social value in losing to your nephew than the value of winning. What is there to be gained by winning? Proving you’re better than your nephew? You knew that before you started the match. Getting better at the game? Your nephew’s sk...
Announcing the second AI Safety Camp
by Lachouette — 2018-06-11T18:59:48.794Z

What is the AI Safety Camp?

Would you like to work on AI safety or strategy research and are looking for a concrete way to get started? We are organizing this camp for aspiring AI safety and strategy researchers. At the camp, you:

build connections with others in the field
build your research portfolio
receive feedback on your research ideas and help others with theirs
make concrete progress on open AI safety research questions

Read more about the last research camp here, including a summary of the produced research.

What’s the structure of the camp?

The camp is preceded by 7 weeks of preparation in form of an online study group of 3-5 people, followed by a 10-day intensive camp with the aim of creating and publishing a research paper, extensive blog post, or github repository.

What will attendants work on?

Participants will work in groups on tightly-defined research projects, for example in the following areas:

Strategy and Policy
Agent Foundations (decision theory, subsystem alignment, embedded world models, MIRI-style)
Value learning (IRL, approval-directed agents, wireheading, …)
Corrigibility / Interruptibility
Side Effects, Safe Exploration
Scalable & Informed Oversight
Robustness (distributional shift, adversarial examples)
Human Values (including philosophical and psychological approaches)

When and where?

4–14 October 2018, in Prague, Czech Republic.

Pricing

Attendance is free.

Apply

Applications and more information on aisafetycamp.com
The Alignment Newsletter #10: 06/11/18
by Rohin Shah — 2018-06-11T16:00:28.458Z

Now that we're at the tenth Alignment Newsletter, it seems like the right time for a

survey

! It's really short and should only take 1-5 minutes (depending on how much feedback you give), so please do take it :)

Highlights

Measuring and avoiding side effects using relative reachability

(

blog post

) (Victoria Krakovna et al): One classic description of the AI alignment problem, from Stuart Russell, is that if you optimize a function of n variables, where the objective depends on k < n of these variables, then the remaining variables will often be set to extreme values, which can be highly undesirable if we actually care about those variables. This can be thought of as a negative side effect. This work attacks the problem of preventing negative side effects in general, even if the reward function says nothing about the side effect. They show simple examples that motivate four properties that any solution should satisfy -- penalize side effects, not effects necessary for the objective; penalize agent-caused effects but not environment effects; penalize irreversible effects higher than reversible ones; and penalize multiple irreversible effects more than a single irreversible effect. They add a penalty term called relative reachability to the reward function to incentivize the agent not to cause side effects. Since we don't want to penalize environment effects (effects that would happen anyway), they compare against an "inaction baseline", where the agent does nothing (or follows some hardcoded safe policy). Since we want something more quantitative than "is this reversible", they create a numeric score of "coverage", which measures how easy it is to reach states from the current state, and penalize decreases in coverage relative to the baseline. This satisfies all of the properties we want -- it will still penalize irreversible actions that are necessary to achieve the objective, but as long as the penalty is small enough the reward for achieving the objective will dominate and the agent will take the action. It doesn't penalize environment effects because both the actual policy and the inaction baseline contain such effects. Clearly irreversible effects would lead to much lower coverage than reversible ones, and so irreversible effects are penalized more. Finally, multiple irreversible effects would lead to larg...
Front Row Center
by Zvi — 2018-06-11T13:50:00.237Z

Epistemic Status: Lightweight

Related: Choices are Bad, Choices Are Really Bad

Yesterday, my wife and I went out to see Ocean’s Eight (official review: as advertised). The first place we went was a massively overpriced theater (thanks MoviePass!) with assigned seating, but they were sold out (thanks MoviePass!) so we instead went to a different overpriced theater without assigned seating, and got tickets for a later show. We had some time, so we had a nice walk and came back for the show.

When we got back, there was nowhere for us to sit together outside of the first two rows. They’re too close, up where you have to strain your neck to see the screen. My wife took the last seat we could find a few rows behind that, and I got a seat in the second row. It was fine, but I’d have much preferred to sit together.

It was, of course, our fault for showing up on time rather than early to a sold out screening. I mention it because it’s a clean example of how offering less can provide more value.

The theater should, if they don’t want to do assigned seating, rip out the first two rows.

At first this seems crazy. Many people prefer sitting in the first two rows to being unable to attend the show, so the seats create value while increasing profits. What’s the harm?

The harm is introducing risk, and creating an expensive auction.

The risk is that if you go to the movies, especially the movies you most want to see, you’ll be stuck in the first two rows. So when you buy a ticket and go upstairs, you might get a bad experience. If the show is sold out, that might be better, as you can buy a different ticket or none at all.

The auction is worse. Seats are first come, first serve. So if it’s important to get served first, you need to come first. If it’s very important to not be last, to avoid awful seats, you need to come early, and so does everyone else, bidding up the price of not-last the same way you’d bid up being first.

With no awful seats, those who care a lot about better seats will still come early, but most people care a lot less. So everyone can come substantially earlier, and not feel pressure. Many will show at the last minute, and be totally fine.

The deadweight loss in time, of adding those forty extra seats, is massive, distributed throughout the theater. Everyone feels pressure to get there early even when they ...
A Loophole for Self-Applicative Soundness
by Diffractor — 2018-06-11T07:57:26.000Z

See comments. This is a rediscovery of a result from the 1980's that allow concluding ϕ from □n(ϕ) via a O(n)-length proof, and even the statement that the theory has no disproof of length n or less has a single O(n)-length proof. This is not vulnerable to Critch's Bounded Parametric Lob proof, and was created by looking for ways to make it fail.

This result is just an idea developed very recently, and I'd put ~2:1 odds on it having a fatal flaw, but it looks extremely promising if it works out. EDIT: It works. None of the proof theory checks have been done yet, but it does causes both Lob's theorem, and Critch's Bounded Parametric Lob result to fail.

So, to begin with, if a theory thinks it is sound, then it is inconsistent. Proof by Lob's theorem.

⊢□⊥→⊥⇒⊢⊥

Well that didn't work.

What if we give the theory a soundness schema over any proof which is of bounded length? Maybe the "there exists a proof" in the standard provability predicate is causing problems.

Well then Critch's Bounded Parametric Lob comes in to ruin our day. The entire proof will be reproduced below.

Let g(k), f(k), and h(k) be such that f(k)≥g(k)+h(k)+log(k), Eg(k)<h(k), and log(f(k))<g(k), asymptotically.

As a specific example, this can be done by g(k)=k, h(k)=k2, and f(k)=k3.

If it takes a constant number of steps to derive a specific proof regardless of k, the number on it will be suppressed for readability. Also, technically, the original proof has Olog(k) instead of log(k), but this change doesn't alter much.

G(⌈ψ⌉,k):=□g(k)ψ(k)→⊥
⊢∀k:ψ(k)↔G(⌈ψ⌉,k)
(Parametric Diagonal Lemma)
⊢□∀k:ψ(k)↔G(⌈ψ⌉,k)
(Bounded Necessitation)
⊢∀k:□log(k)(ψ(k)→G(⌈ψ⌉,k))
(Quantifier Distribution)
⊢∀k,a:□aψ(k)→□a+log(k)G(⌈ψ⌉,k)
(Implication Distribution)
⊢∀k,a,b:□aψ(k)→(□b□g(k)ψ(k)→□a+b+log(k)⊥)
(Implication Distribution)
Now specialize to a=g(k), b=h(k). Also, f(k)≥g(k)+h(k)+log(k) for sufficiently large k above k1.
⊢∀k>k1:□g(k)ψ(k)→(□h(k)□g(k)ψ(k)→□f(k)⊥)
⊢∀k,a:□aψ(k)→□Ea□aψ(k)
(Bounded Inner Necessitation)
Now, Specializing to a=g(k), we get
⊢∀k:□g(k)ψ(k)→□Eg(k)□g(k)ψ(k)
Now, since Eg(k)<h(k) after some time k2,
⊢∀k>k2:□g(k)ψ(k)→□f(k)⊥
Pick a specific value of k, k′, which is sufficiently large.
⊢□g(k′)ψ(k′)→□f(k′)⊥
By the soundness schema, □f(k′)⊥→⊥. The length of this proof isn't con

...
Oops on Commodity Prices
by sarahconstantin — 2018-06-10T15:40:00.499Z

Epistemic status: Casual

Some patient and thoughtful folks on LessWrong, and, apparently, some rather less patient folks on r/SneerClub, have pointed out that GDP-to-gold, or GDP-to-oil, are bad proxy measures for economic growth.

Ok, this is a counterargument I want to make sure I understand.

Is the following a good representation of what you believe?

When you divide GDP by a commodity price, when the commodity has a nearly-fixed supply (like gold or land) we’d expect the price of the commodity to go up over time in a society that’s getting richer — in other words, if you have better tech and better and more abundant goods, but not more gold or land, you’d expect that other goods would become cheaper relative to gold or land. Thus, a GDP/gold or GDP/land value that doesn’t increase over time is totally consistent with a society with increasing “true” wealth, and thus doesn’t indicate stagnation.

paulfchristiano:

Yes. The detailed dynamics depend a lot on the particular commodity, and how elastic we expect demand to be; for example, over the long run I expect GDP/oil to go way up as we move to better substitutes, but over a short period where there aren’t good substitutes it could stay flat.

Commenters on this blog have also pointed out that the Dow is a poor measure of the value of the stock market, since it’s small and unnormalized.

These criticisms weaken my previous claim about economic growth being stagnant.

Now, a little personal story time:

Nearly ten years ago (yikes!) in college, I had an econ blog. My big brush with fame was having a joke of mine hat-tipped by Megan McArdle once. I did most of the required courses for an econ major, before eventually settling on math. My blog, I realized with dismay when I pulled it up many years later, consisted almost entirely of me agreeing with other econ bloggers I encountered, and imitating buzzwords. I certainly sounded a lot more mainstream in those days, but I understood — if possible — less economics than I do now. I couldn’t use what I’d learned in school to reason about real-world questions.

I think I learn a heck of a lot more by throwing an idea out there and being corrected than I did back when I was not even asking questions. “A shy person cannot learn, an impatient person cannot teachR...
Resolving the Dr Evil Problem
by Chris_Leong — 2018-06-10T11:56:09.549Z

Deep in Dr Evil's impregnable fortress (paraphrased):

Dr Evil is just about to complete his evil plan of destroying the Earth, when he receives a message from the Philosophy Defence Force on Mars. They have created a clone in the exact same subjective situation Dr Evil now occupies; he believes he is Dr Evil and is currently in a recreation of the fortress. If the clone of Dr Evil tries to destroy the Earth, they will torture him, otherwise they will treat him well. Dr Evil wants to destroy the Earth, but he would prefer to avoid being tortured much, much more and he is now uncertain about whether he should surrender or not. Should Dr Evil surrender?

The paper then concludes:

I conclude that Dr. Evil ought to surrender. I am not entirely comfortable with that conclusion. For if INDIFFERENCE is right, then Dr. Evil could have protected himself against the PDF’s plan by (in advance) installing hundreds of brains in vats in his battlestation—each brain in a subjective state matching his own, and each subject to torture if it should ever surrender

This article will address two areas of this problem:

Firstly, it argue that Dr Evil should surrender, however focusing on a different path, particularly what it means to "know" and how this is a leaky abstraction
Secondly, it will argue that hundreds of brains in a vat would indeed secure him against this kind of blackmail

I'll note that this problem is closely related to The AI that Boxes You. Several people noted there that you could avoid blackmail by pre-committing to reboot the AI if it tried to threaten you, although my interest is in the rational behaviour of an agent who has failed to pre-commit.

Why Dr Evil Should Surrender

I think there's a framing effect in the question that is quite misleading. Regardless of whether we say, "You are Dr Evil. What do you do?" or "What should Dr Evil do?" we are assuming that you or a third-party Dr Evil knows that they are Dr Evil? However that's an assumption that needs to be questioned.

If you knew that you were Dr Evil, then knowing that a clone has been created and placed in a situation that appears similar wouldn't change what you should do if you don't care about the clone. However, you strictly can't ever actually know that you are Dr Evil. All you can know is that you have memories of being Dr Evil and you appear...
Simplified Poker Conclusions
by Zvi — 2018-06-09T21:50:00.400Z

Previously: Simplified Poker, Simplified Poker Strategy

Related (Eliezer Yudkowsky): Meta Honesty: Firming Honesty Around Its Edge Cases

About forty people submitted programs that used randomization. Several of those random programs correctly solved for the Nash equilibrium, which did well.

I submitted the only deterministic program.

I won going away.

I broke even against the Nash programs, utterly crushed vulnerable programs, and lost a non-trivial amount to only one program, a resounding heads-up defeat handed to me by the only other top-level gamer in the room, fellow Magic: the Gathering semi-pro player Eric Phillips.

Like me, Eric had an escape hatch in his program that reversed his decisions (rather than retreating to Nash) if he was losing by enough. Unlike me, his actually got implemented – the professor decided that given how well I was going to do anyway, I’d hit the complexity limit, so my escape hatch was left out.

Rather than get into implementation details, or proving the Nash equilibrium, I’ll discuss two things: How few levels people play on, and the motivating point: How things are already more distinct and random than you think they are, and how to take advantage of that.

Next Level

In the comments to the first two posts, most people focused on finding the Nash equilibrium. A few people tried to do something that would better exploit obviously stupid players, but none that tried to discover the opponents’ strategy.

The only reason not to play an exploitable strategy is if you’re worried someone will exploit it!

Consider thinking as having levels. Level N+1 attempts to optimize against Levels N and below, or just Level N.

Level 0 isn’t thinking or optimizing, so higher levels all crush it, mostly.

Level 1 thinking picking actions that are generically powerful, likely to lead to good outcomes, without considering what opponents might do. Do ‘natural’ things.

Level 2 thinking considers what to do against opponents using Level 1 thinking. You try to counter the ‘natural’ actions, and exploit standard behaviors.

Level 3 counters Level 2. You assume your opponents are trying to exploit basic behaviors, and attempt to exploit those trying to do this.

Level 4 counters Level 3. You assume your opponents are trying to exploit exploitative behavior, and acting accordingly. So you do what’s best again...
Fundamentals of Formalisation Level 3: Set Theoretic Relations and Enumerability
by philip_b — 2018-06-09T19:57:20.878Z

Followup to Fundamentals of Formalisation level 2: Basic Set Theory.

The big ideas:

Ordered Pairs
Relations
Functions
Enumerability
Diagonalization

To move to the next level you need to be able to:

Define functions in terms of relations, relations in terms of ordered pairs, and ordered pairs in terms of sets.
Define what a one-to-one (or injective) and onto (or surjective) function is. A function that is both is called a one-to-one correspondence (or bijective).
Prove a function is one-to-one and/or onto.
Explain the difference between an enumerable and a non-enumerable set.

Why this is important:

Establishing that a function is one-to-one and/or onto will be important in a myriad of circumstances, including proofs that two sets are of the same size, and is needed in establishing (most) isomorphisms.
Diagonalization is often used to prove non-enumerability of a set and also it sketches out the boundaries of what is logically possible.

You can find the lesson in our ihatestatistics course. Good luck!

P.S. From now on I will posting these announcements instead of Toon Alfrink.
Unraveling the Failure's Try
by LeoHolman — 2018-06-09T14:34:55.543Z

I'm sure this idea exists somewhere else, but I haven't found it yet. (Probably my fault, I've only been on here for a few weeks and I'm still working through the core sequences.)

The failure's try is a seemingly earnest attempt to overcome a challenge, but the challenger lacks the fundamental belief that they can surmount the challenge, and accept the challenge inevitably as evidence to their failure. That is to say, you put in an honest effort without believing you can do it so you can turn around and say it was impossible all along.

I think this is a readily identifiable symptom of cognitive bias, of unevenness of evidence because one starts with the conclusion and works backwards. The belief "I'm a failure" or at least "this task is impossible (for me)" exists somewhere in System 1, and subtly gravitates behavior toward manifesting itself.

I am well aware that I do this, and I'm not sure how to correct it. Somewhere in my belief network rests this idea that I am insufficient, a node of impostor syndrome, and biased evaluation. I got a 96 on my Chinese final, something I certainly would not have been able to do when I first arrived in China, but my reaction is not "I've progressed so much, I've learned all this material and tested well." It's... Well. Empty. Like I hadn't done anything at all. Like I didn't believe it happened. I've been working hard at learning this language for years, it's very important to me, and yet when faced with seemingly undeniable evidence that I've made substantial progress, I don't interpret success.

I think this node manifests in another area as well. I feel like I /must/ become useful in as many aspects as possible. After reflecting for a long while on this, I think it's because I believe otherwise people won't want anything to do with me. I fundamentally doubt the idea that my presence alone is something that can be enjoyed, I feel that I must constantly be raising my ability to help others before they could accept me.

So my question is, now that I am aware of the node, how do I unravel it? My understanding of counter-conditioning relies on specific, actionable behaviors. "Every time I want to eat ice cream I will think about my fitness goal, and instead work out, with enough time and careful planni...