trentmkelly/LessWrong-43k huggingface dataset:
Attainable Utility Landscape: How The World Is Changed
by TurnTrout — 2020-02-10T00:58:01.453Z

(This is one interpretation of the prompt, in which you haven't chosen to go to the moon. If you imagined yourself as more prepared, that's also fine.)

If you were plopped onto the moon, you'd die pretty fast. Maybe the "die as quickly as possible" AU is high, but not much else - not even the "live on the moon" AU! We haven't yet reshaped the AU landscape on the moon to be hospitable to a wide range of goals. Earth is special like that.

AU landscape as a unifying frame

Attainable utilities are calculated by winding your way through possibility-space, considering and discarding possibility after possibility to find the best plan you can. This frame is unifying.

Sometimes you advantage one AU at the cost of another, moving through the state space towards the best possibilities for one goal and away from the best possibilities for another goal. This is opportunity cost.

Sometimes you gain more control over the future: most of the best possibilities make use of a windfall of cash. Sometimes you act to preserve control over the future: most Tic-Tac-Toe goals involve not ending the game right away. This is power.

Other people usually objectively impact you by decreasing or increasing a bunch of your AUs (generally, by changing your power). This happens for an extremely wide range of goals because of the structure of the environment.

Sometimes, the best possibilities are made unavailable or worsened only for goals very much like yours. This is value impact.

Sometimes a bunch of the best possibilities go through the same part of the future: fast travel to random places on Earth usually involves the airport. This is instrumental convergence.

Exercise: Track what’s happening to your various AUs during the following story: you win the lottery. Being an effective spender, you use most of your cash to buy a majority stake in a major logging company. Two months later, the company goes under.

Technical appendix: AU landscape and world state contain equal information

In the context of finite deterministic Markov decision processes, there's a wonderful handful of theorems which basically say that the AU landscape and the environmental dynamics encode each other. That is, they contain the same information, just with different emphasis. This supports thinking of the AU landscape as a "dual" of the world state.

Let ⟨S,A,T,γ⟩ be a rewardless deterministic MDP with finite state and

...
A Simple Introduction to Neural Networks
by Rafael Harth — 2020-02-09T22:02:38.940Z

This post will discuss what neural networks are (chapter I), why they work (chapter II), and how they are trained (chapter III). It serves as the tenth entry in a sequence on Machine Learning, but it's written to be accessible without having read any previous part. Most of chapters I and II also doesn't require any advanced math.

Note that bold and italics means "I am introducing a new Machine Learning term."

I

Meet the neural network:

The term "neural network" describes a class of Machine Learning predictors which is inspired by the architecture of the human brain. It is an extremely important class: AlphaGo, AlphaGo Zero, AlphaStar, and GPT-2 are all based on neural networks.

The network above will be our running example throughout this post. Let's begin by defining its components. The blobs are called neurons. The arrows are called edges. Together, they define the graph underlying the neural network. (This usage of the term "graph" has nothing to do with a function graph – instead, a graph is simply a bunch of nodes with arrows between them.) If this graph has no cycles (= edges going around in a circle), the neural network is also called feed-forward. For this post, you can assume networks are always feed-forward. The green neurons are called input neurons, the red neurons are called output neurons, and the blue neurons are called bias neurons.

If it is possible to divide the entire set of neurons into an ordered list of subsets, such that each neuron only points towards neuron in the subset next in line, the network is called layered. For this post, you can assume networks are always layered. However, note that non-layered networks do exist: a network with edges A→B and A→C and B→C cannot be divided into layers.

As you can see, each neuron only points towards neurons in the next higher layer. Note that the first blue neuron, despite being in the input layer, isn't an input neuron.

So far, our view has been simplified. In reality, there's more going on:

The w∙,∙ written on the edges are numbers called weights. They will determine how important the value of the incoming edge is. The σ∙ written inside of the neurons are functions (in the classical sense: they take a value as input and output something else). From here on, if I talk about "the fifth neuron," I mean the neuron with σ5 written in it.

This picture is a good guide to explain how a neural network works.

F...
Did AI pioneers not worry much about AI risks?
by lisperati — 2020-02-09T19:58:53.395Z

It's seems noteworthy just how little the AI pioneers from the 40s-80s seemed to care about AI risk. There is no obvious reason why a book like "Superintelligence" wasn't written in the 1950s, but for some reason that didn't happen... any thoughts on why this was the case?

I can think of three possible reasons for this:

1. They actually DID care and published extensively about AI risk, but I'm simply not well enough schooled on the history of AI research.

2. Deep down, people involved in early AI research knew that they were still a long long way from achieving significantly powerful AI, despite the optimistic public proclamations that were made at that time.

3. AI risks are highly counter-intuitive and it simply required another 60 years of thinking to understand.

Anyone have any thoughts on this question?
Source of Karma
by jmh — 2020-02-09T14:13:30.650Z

I was thinking the other day it would be interesting to know something about the karma I was getting. Particularly the quality, to the extent that is possible to assess.

I suspect everyone would prefer to have those they respect up voting posts and comments but that type of transparency might not be desirable. I'm not sure if some type of weighting system would work or not -- how may karma points do the account giving an up vote have so you can get some average gauge of quality of the feedback.

I would not think that should be on the main page but something each person can look at under their own account.

Wonder what others think or if this has been suggested before?
State Space of X-Risk Trajectories
by David_Kristoffersson — 2020-02-09T13:56:41.402Z

Justin Shovelain developed the core ideas in this article and assisted in writing, David Kristoffersson was the lead writer and editor.

Cross-posted on the EA Forum

Abstract

Currently, people tend to use many key concepts informally when reasoning and forming strategies and policies for existential risks (x-risks). A well-defined formalization and graphical language for paths and choices would help us pin down more exactly what we think and let us see relations and contrasts more easily. We construct a common state space for futures, trajectories, and interventions, and show how these interact. The space gives us a possible beginning of a more precise language for reasoning and communicating about the trajectory of humanity and how different decisions may affect it.

Introduction

Understanding the possible trajectories of human civilization and the futures they imply is key to steering development towards safe and beneficial outcomes. The trajectory of human civilization will be highly impacted by the development of advanced technology, such as synthetic biology, nanotechnology, or artificial general intelligence (AGI). Reducing existential risks means intervening on the trajectory civilization takes. To identify effective interventions, we need to be able to answer questions like: how close are we to good and bad outcomes? How probable are the good and bad outcomes? What levers in the system could allow us to shape its trajectory?

Previous work has modeled some aspects of existential risk trajectories. For example, various surveys and extrapolations have been made to forecast AGI timelines [1; 2; 3; 4; 5], and scenario and risk modeling have provided frameworks for some aspects of risks and interventions [6; 7; 8]. The paper Long Term Trajectories of Human Civilization [9] offers one form of visualization of civilizational trajectories. However, so far none of these works have defined a unified graphical framework for futures, trajectories, and interventions. Without developing more and better intellectual tools to examine the possible trajectories of the world and how to shape them, we are likely to remain confused about many of the requirements for reaching a flourishing future, causing us to take less effective actions and leaving us ill-prepared for the events ahead of us.

We construct a state space model of existential risk, where our closeness to stably g

...
Does there exist an AGI-level parameter setting for modern DRL architectures?
by TurnTrout — 2020-02-09T05:09:55.012Z

Suppose the architecture includes memory (in the form of a recurrent state) and will act as the policy network for an observation-based RL agent. Evaluating the agent from a reasonable initial state, would you guess that there exists a model with robustly human+ capabilities for current architectures?

How many parameters would it take before you estimate there's a fifty-fifty chance of such a parameter setting existing? 1 billion? 1 trillion? More?
Who... (or what) designed this site and where did they come from?
by thedayismine — 2020-02-09T04:04:21.321Z

Hi there,

I'm brand new here and I have a question for this fine community.

Who designed this site?

The reason I ask is because the design of this site is superb. Every element—everywhere I look—is visually gorgeous to look at. Not only that, but all that I find "visually gorgeous" is really well thought out.

The way the links preview when you hover over them (and how sometimes, they display different things when you hover over them.
The way the comments... well, display the comments. Amazing.

Holy cannoli, that comment section...

Have you seen the comment section at the bottom of the pages here? Have you seen how it works?

It's gorgeous!

I've never seen anything like that before. Someone spent some serious time figuring out that comment system.

It's a work of art!

Any information on how this site came to look so darn good (and work so darn well) is much appreciated.

P.S. Although it wasn't the original reason I joined, the "things" discussed here are my kind of "things" and, the people that discuss it, sound like they're my kind of people.

Planets seemed to have aligned.

Thanks for making this place.
How to Frame Negative Feedback as Forward-Facing Guidance
by Liron — 2020-02-09T02:47:37.230Z

Your employee Fred always talks too much during your weekly staff meetings. It's been an ongoing issue. Everyone on your team is annoyed, and so are you. At this point, you have no choice but to give Fred some... negative feedback.

You sit down at your computer and start drafting what you're going to say to Fred:

Listen Fred, we think you're talking too much in our staff meetings and it's lowering the quality of the discussion. Can you try to talk a little less and let other people talk more?

But wait... let's be tactful here. Your goal is to optimize how you criticize Fred to maximize expected positive behavioral change. In that sense, your rough draft isn't super tactful yet.

I challenge you to try this now as a 5-minute exercise: What communication technique would you apply here? What exact words would you say to Fred?

...

...

...

The "shit sandwich" technique comes to mind, i.e. sandwiching your negative-feedback turd inside two slices of positive feedback. But let's use a much better technique: framing negative feedback as guidance. Here's how I'd do that:

Fred,

I think you have some room for improvement in the way you present ideas at our staff meetings. Sometimes I notice that you're making a valid point, which is a great contribution, but it doesn't get fully appreciated by the rest of the team. I want to share some techniques with you that I've seen our senior staff use to be perceived well in meetings.

For example, today when you brought up how our flux capacitors don't last long enough, no one really engaged with that topic. And that was probably frustrating for you, right? If you could tweak your communication style to easily get everyone to appreciate your ideas, that would be great for you and the team.

My advice on how to do this is basically to limit the number of points you bring into each staff meeting. And whenever you're going to say your point, try to first make other people feel like you've heard their point. Right now it feels like you're making a lot of different points and you're not acknowledging other people's points enough, so your points aren't getting fully appreciated. Do you agree with my perception?

A lot of what I've done here is explained in other books and articles: I follow the CORE structure (context, observation, result, exp...
Relationship Outcomes Are Not Particularly Sensitive to Small Variations in Verbal Ability
by Zack_M_Davis — 2020-02-09T00:34:39.680Z
Link: http://zackmdavis.net/blog/2020/02/relationship-outcomes-are-not-particularly-sensitive-to-small-variations-in-verbal-ability/

After a friendship-ending fight, you feel an impulse to push through the pain to do an exhaustive postmortem of everything you did wrong in that last, fatal argument—you could have phrased that more eloquently, could have anticipated that objection, could have not left so much "surface area" open to that class of rhetorical counterattack, could have been more empathetic on that one point, could have chosen a more-fitting epigraph, could have taken more time to compose your reply and squeeze in another pass's worth of optimizations—as if searching for some combination of variables that would have changed the outcome, some nearby possible world where the two of you are still together.

No solution exists. (Or is findable in polynomial time.) The causal forces that brought you to this juncture are multitudinous and complex. A small change in the initial conditions only corresponds to a small change in the outcome; you can't lift a two-ton weight with ten pounds of force.

Not all friendship problems are like this. Happy endings do exist—to someone else's story in someone else's not-particularly-nearby possible world. Not for you, not here, not now.
What can the principal-agent literature tell us about AI risk?
by apc — 2020-02-08T21:28:09.800Z

This work was done collaboratively with Tom Davidson.

Thanks to Paul Christiano, Ben Garfinkel, Daniel Garrett, Robin Hanson, Philip Trammell and Takuro Yamashita for helpful comments and discussion. Errors our own.

Introduction

The AI alignment problem has similarities with the principal-agent problem studied by economists. In both cases, the problem is: how do we get agents to try to do what we want them to do? Economists have developed a sophisticated understanding of the agency problem and a measure of the cost of failure for the principal, “agency rents”.

If principal-agent models capture relevant aspects of AI risk scenarios, they can be used to assess their plausibility. Robin Hanson has argued that Paul Christiano’s AI risk scenario is essentially an agency problem, and therefore that it implies extremely high agency rents. Hanson believes that the principal-agent literature (PAL) provides strong evidence against rents being this high.

In this post, we consider whether PAL provides evidence against Christiano’s scenario and the original Bostrom/Yudkowsky scenario. We also examine whether the extensions to the agency framework could be used to gain insight into AI risk, and consider some general difficulties in applying PAL to AI risk.

Summary

PAL isn’t in tension with Christiano’s scenario because his scenario doesn’t imply massive agency rents; the big losses occur outside of the principal-agent problem, and the agency literature can’t assess the plausibility of these losses. Extensions to PAL could potentially shed light on the size of agency rents in this scenario, which are an important determinant of the future influentialness of AI systems.

Mapped onto a PAL model, the Bostrom/Yudkowsky scenario is largely about the principal’s unawareness of the agent’s catastrophic actions. Unawareness models are rare in PAL probably because they usually aren’t very insightful. This lack of insightfulness also seems to prevent existing PAL models or possible extensions from teaching us much about this scenario.

There are also a number of more general difficulties with using PAL to assess AI risk, some more problematic than others.

PAL models rarely consider weak principals and more capable agents

PAL models are brittle

Agency rents are too narrow a measure

PAL models typically assume contract enforceability

PAL models typically assume AIs work for humans becau

...
A Cautionary Note on Unlocking the Emotional Brain
by  — 2020-02-08T17:21:03.112Z

[Cross-posted from Grand Unified Crazy. Relevant to Kaj's summary of the book.]

In children’s stories, the good guys always win, the hero vanquishes the villain, and everyone lives happily ever after. Real life tends to be somewhat messier than this.

The world of therapy presented by Unlocking the Emotional Brain reads somewhat like a children’s story. Loosely, it presents a model of the brain where your problems are mostly caused by incorrect emotional beliefs (bad guys). The solution to your problems is to develop or discover a correct emotional belief (good guy) that contradicts your incorrect beliefs, then force your brain to recognize the contradiction at an emotional level. This causes your brain to automatically resolve the conflict and destroy the incorrect belief, so you can live happily ever after.

Real life tends to be somewhat messier than this.

After about a month of miscellaneous experimentation on myself based on this book, my experiences match the basic model presented, where many psychological problems are caused by incorrect emotional beliefs (I don’t think this part is particularly controversial in psychological circles). It also seems to be true that if I force my brain to recognize a contradiction between two emotionally relevant beliefs, it will resolve the conflict and destroy one of them. Of course, as in real life where the good guy doesn’t always win, it seems that when I do this my brain doesn’t always destroy the right belief.

I have had several experiences now where I have identified an emotional belief which analytically I believe to be false or harmful. Per UtEB I have identified or created a different experience or belief that contradicts it, and smashed them together in my mind. A reasonable percentage of the time, the false belief emerges stronger than before, and I find myself twisting the previous “good” belief into some horrific experience to conform with the existing false belief.

In hindsight this shouldn’t be particularly surprising. Whatever part of your brain is used to resolve conflicting emotional beliefs and experiences, it doesn’t have special access to reality. All it has to work with are the two conflicting pieces and any other related beliefs you might have. It’s going to pick the wrong one with some regularity. As such, my recommendation for...
On the falsifiability of hypercomputation
by jessicata — 2020-02-07T08:16:07.268Z
Link: https://unstableontology.com/2020/02/07/on-the-falsifiability-of-hypercomputation/

[ED NOTE: see Vanessa Kosoy's comment here; this post assumes a setting in which the oracle may be assumed to return a standard natural.]

It is not immediately clear whether hypercomputers (i.e. objects that execute computations that Turing machines cannot) are even conceivable, hypothesizable, meaningful, clearly definable, and so on. They may be defined in the notation of Peano arithmetic or ZFC, however this does not imply conceivability/hypothesizability/etc. For example, a formalist mathematician may believe that the Continuum hypothesis does not have a meaningful truth value (as it is independent of ZFC), and likewise for some higher statements in the arithmetic hierarchy that are independent of Peano Arithmetic and/or ZFC.

A famous and useful criterion of scientific hypotheses, proposed by Karl Popper, is that they are falsifiable. Universal laws (of the form "∀x. p(x)") are falsifiable for testable p, as they can be proven false by exhibiting some x such that p(X) is false. In an oracle-free computational setting, the falsifiable hypotheses are exactly those in ∏₁ (i.e. of the form "∀n. p(n)" for natural n and primitive recursive p).

However, ∏₁ hypotheses do not hypothesize hypercomputation; they hypothesize (computably checkable) universal laws of the naturals. To specify the falsifiability criterion for hypercomputers, we must introduce oracles.

Let a halting oracle be defined as a function O which maps Turing machine-specifications to Booleans, which outputs "true" on exactly those Turing machines which eventually halt. Then, we can ask: is the hypothesis that "O is a halting oracle" falsifiable?

We can immediately see that, if O ever outputs "false" for a Turing machine which does eventually halt, it is possible to exhibit a proof of this, by exhibiting both the Turing machine and the number of steps it takes to halts. On the other hand, if O ever outputs "true" for a Turing machine which never halts, it is not in general possible to prove this; to check such a proof in general would require solving the halting problem, which is uncomputable.

Therefore, the hypothesis that "O is a halting oracle" is not falsifiable in a computational setting with O as an oracle.

However, there is a different notion of halting oracle whose definition is falsifiable. Let a constructive halting oracle be defined as a function O whic

...
More writeups!
by jefftk — 2020-02-07T03:10:02.313Z

A kind of post I would like to read more of is "I did X, and here's
how it went." You tend to see this most with research, but I've
enjoyed reading them on all sorts of things. This is one of the

main

kinds

of

post

I

write, and I would encourage others to
give it a try!

Here's an arbitrary selection of writeups I've enjoyed:

DB-19:
Resurrecting an Obsolete Connector

Your room can be as
bright as the outdoors

After one year of applying for EA jobs: It is really, really hard to get hired by an EA organisation

Wood joint
strength testing

Giving
Tuesday retrospective

Zeo sleep self-experiments

Hexing
the technical interview

EAGxBerkeley
2016 Retrospective

My
Business Card Runs Linux

Major reasons I like writeups include:

Exposure to problems that I didn't know were important to people.

Detailed looks at how people handled problems.

They're generally very concrete and written after the fact,
which makes it harder to write something that's neat but wrong.

When I'm thinking of doing something and find a writeup by
someone who did something similar it's fantastically useful.

When you voluntarily make something public, you do risk that people
will irresponsibly beat
you up over failings. I think it's generally worth it to go ahead
anyway, tell the whole story, and help build a norm of sharing things
so others can learn.