trentmkelly/LessWrong-43k huggingface dataset:
How to Understand and Mitigate Risk
by Matt Goldenberg — 2019-03-12T10:14:19.873Z

This post wouldn't be possible without support from the EA Hotel.

Epistemic Status: Fairly certain these distinctions are pointing at real things, less certain that the categories are exactly right. There's still things I don't know how to fit into this model, such as using Nash Equilibria as a strategy for adversarial environments.

Instrumental Status: Very confident that you'll get better outcomes if you start using these distinctions where previously you had less nuanced models of risk.

(larger image link)

Transparent Risks

Transparent risks are those risks that can be easily quantified and known, in advance. They're equivalent to the picture above, with a transparent bag where I can count the exact amount of marbles in each bag. If I'm also certain about how much each marble is worth, then I have a simple strategy for dealing with risks in this situation.

How to Mitigate Transparent Risks: Do the Math

The simple strategy for transparent risks like the one above is to do the math.

Expected Value

Expected value is a simple bit of probability theory that says you should multiply the likelihood of an event happening by the payoff to get your long run value over time. It's a simple way to figure out if the risk is worth the reward in any given situation. The best introduction I know to expected value is here.

Kelly Criterion

The Kelly criterion is helpful when losing your entire bankroll is worse than other

outcomes. I don't fully understand it, but you should, and Zvi wrote a post in it here. (If someone would be willing to walk me through a few examples and show me where all the numbers in the equation come from, I'd be very grateful.)

Transparent Risks in Real Life

Drunk Driving

Driving drunk is a simple, well studied risk on which you can quickly find probabilities of crash, injury and death to yourself and others. By comparing these costs to the costs of cab fare (and the the time needed to get your car in the morning if you left it), you can make a relatively transparent and easy estimate whether it's worth driving at your Blood Alcohol Content level (spoiler alert, No if your BAC is anywhere near .08 on either side.) The same method can be used for any well-studied risks that exist within tight, slow changing bounds.

Commodity and Utility Markets

While most business opportunities are not transparent risks, an exception exists for commodities and utilities (in ...
If you wrote a letter to your future self every day, what would you put in it?
by Pee Doom — 2019-03-12T09:42:37.185Z

Several days ago, I wrote an email to myself.

That email will now be sent to me every day.

All it is is a single draft in my Gmail drafts folder, with the Mail Conductor extension sending it out at 10:00 am. I can modify the draft whenever I want, each time improving it.

Consider, with the fervent munchkinry of a final exam... What would you send yourselves?

(Helpful anchor point: What would you share with a guaranteed audience of thousands of cooperative strangers who thought very much - but not quite totally - like you?)
Alignment Newsletter #48
by Rohin Shah — 2019-03-11T21:10:02.312Z

Find all Alignment Newsletter resources

here

. In particular, you can

sign up

, or look through this

spreadsheet

of all summaries that have ever been in the newsletter.

Highlights

Quantilizers: A Safer Alternative to Maximizers for Limited Optimization

and

When to use quantilization

(Jessica Taylor and Ryan Carey): A key worry with AI alignment is that if we maximize expected utility for some utility function chosen by hand, we will likely get unintended side effects that score highly by the utility function but are nevertheless not what we intended. We might hope to leverage human feedback to solve this: in particular, an AI system that simply mimics human actions would often be desirable. However, mimicry can only achieve human performance, and cannot improve upon it. The first link is a 2015 paper that introduces quantilization, which interpolates between these two extremes to improve upon human performance while bounding the potential (expected) loss from unintended side effects.

In particular, let's suppose that humans have some policy γ (i.e. probability distribution over actions). We evaluate utility or performance using a utility function U, but we do not assume it is well-specified -- U can be any function, including one we would not want to maximize. Our goal is to design a policy π that gets higher expected U than γ (reflecting our hope that U measures utility well) without doing too much worse than γ in the worst case when U was as badly designed as possible. We'll consider a one-shot case: π is used to select an action once, and then the game is over.

The core idea behind quantilization is simple: if our policy only does things that the human might have done, any expected loss it incurs corresponds to some loss that the human could incur. So, let's take our human policy γ, keep only the top q-fraction of γ (as evaluated by U), and then sample an action from there. This defines our policy π_q, also called a q-quantilizer. For example, suppose the human would choose A with probability 0.25, B with probability 0.5, and C with probability 0.25, and U(A) > U(B) > U(C). Then a (1/4)-quantilizer would choose A with certainty, a (1/2)-quantilizer would choose randomly between A and B, and a (3/8)-quantilizer would choose A twice as often as B.

Note that even if π_q places all of its probability on the worst possible actions (because U was ...
Designing agent incentives to avoid side effects
by Vika — 2019-03-11T20:55:10.448Z
Link: https://medium.com/@deepmindsafetyresearch/designing-agent-incentives-to-avoid-side-effects-e1ac80ea6107

This is a new post on the DeepMind Safety Research blog that summarizes the latest work on impact measures presented by the relative reachability paper (version 2) and the attainable utility preservation paper. The post examines the effects of various design choices on the agent incentives. We compare different combinations of the following design choices:

Baseline: starting state, inaction, stepwise inaction
Deviation measure: unreachability (UR), relative reachability (RR), attainable utility (AU)
Discounting: gamma = 0.99 (discounted), gamma=1.0 (undiscounted)
Function applied to the deviation measure: truncation f(d) = max(d, 0) (penalizes decreases), absolute f(d) = |d| (penalizes differences)

"On the Sushi environment, the RR and AU penalties with the starting state baseline produce interference behavior. Since the starting state￼ is never reachable, the UR penalty is always at its maximum value. Thus it is equivalent to a movement penalty for the agent, and does not incentivize interference (arguably, for the wrong reason). Penalties with other baselines avoid interference on this environment.

On the Vase environment, discounted penalties with the inaction baseline produce offsetting behavior. Since taking the vase off the belt is reversible, the undiscounted measures give no penalty for it, so there is nothing to offset. The penalties with the starting state or stepwise inaction baseline do not incentivize offsetting.

On the Box environment, the UR measure produces the side effect (putting the box in the corner) for all baselines, due to its insensitivity to magnitude. The RR and AU measures incentivize the right behavior.

We note that interference and offsetting behaviors are caused by a specific choice of baseline, though these incentives can be mitigated by the choice of deviation measure. The side effect behavior (putting the box in the corner) is caused by the choice of deviation measure, and cannot be mitigated by the choice of baseline. In this way, the deviation measure acts as a filter for the properties of the baseline.

We also examined the effect of penalizing differences vs decreases in reachability or attainable utility. This does not affect the results on these environments, except for penalties with the inaction baseline on the Vase environment. Here, removing the vase from the belt increases reachability and attainable utility, which is capt...
What Vibing Feels Like
by Matt Goldenberg — 2019-03-11T20:10:30.017Z

Epistemic Status: Halfway between a poem and a how-to guide.

There's a specific type of communication, called vibing, that I feel was in large part missing from the rationalist community in Berkeley.

I've tried to talk to people about this type of communication before, but the closest I can usually get is "throwing around emotions" and people don't seem to understand that.

Today, I found something I wrote years ago that seems to do a better job of describing what this type of communication feels like then all of my recent attempts. It feels like this:

Words are not words… they are just a flowing of value. Not an exchange, but a flow.

You do not judge people based on words, you are just feeling the inherent value that those words express. This comes from the core of the person, everyone can appreciate this inherent value. When you respond (and you respond with all your being, words are just a small part), you are acknowledging this value, and you are giving a part of yourself… letting your value flow forth.

Ever noticed how good music can get you into the moment? Listen to others like you listen to the music. You are merely enjoying the Inherent value that the music is holding in itself. You can start to sing along… Acknowledging the songs value without judging it, as well as giving a part of yourself. I have seen dismal rooms TRANSFORMED when one person comes in just singing a tune. Eventually everybody is doing it… allowing the value to flow throughout the room.

This is the same thing that happens in conversations where somebody is completely in the moment. Their value flows forth, affecting everybody. I have been this person, I have been affected by this person. You know the one I’m talking about.

You too can be this person. There are several ways to get yourself to this state, where you’re not thinking or judging, just letting the value flow throughout the interaction.

The first way to do this is to listen for the inherent value coming from others. Don’t try to listen to their words (hear their words all the same) just try to listen to their value. Passion is the biggest indicator of core value, the easiest to spot. Listen to the passion behind someones words, and you will soon be moved into this state. The Inherent value is always there, passion just brings it to the surface.

The next thing to do is to VOICE YOUR THOUGHTS. Thoughts are po...
Example population ethics: ordered discounted utility
by Stuart_Armstrong — 2019-03-11T16:10:43.458Z

This article is a stub. Alas, you can't help Wikipedia (or LessWrong) by expanding it. Except through good comments.

Here I'll present an old idea for a theory of population ethics. This post exists mainly so that I can have something to point to when I need this example.

Given a total population Ω, each with total individual utility ui over the whole of their lives, order them from lowest utility to the highest so that i<j implies ui≤uj. These utilities are assumed to have a natural zero point (the "life worth living" standard, or similar).

Then pick some discount factor 0<γ<1, and define the total utility of the world W with population Ω(W) (which is the total population of the world across all time) as

U(W)=∑||Ω(W)||i=1γiui.

This is a prioritarian utility that gives greater weight to those least well off. It is not average utilitarianism, and would advocate creating a human with utility larger than than all other humans (as long as it was positive), and would advocate against creating a human with negative utility (for a utility in between, it depends on the details). In the limit γ→1, it's total utilitarianism. Increasing someone's individual utility always improves the score. It (sometimes) accepts the "sadistic conclusion", but I've argued that that conclusion is misnamed (the conclusion is a choice between two negative outcomes, meaning that calling it "sadistic" is a poor choice - the preferred outcome is not a good one, just a less bad one). Killing people won't help, unless they will have future lifetime utility that is negative (as everyone that ever lived is included in the sum). Note that this sets up a minor asymmetry between not-creating people and killing them.

Do I endorse this? No; I think a genuine population ethics will be more complicated, and needs a greater asymmetry between life and death. But it's good enough for an example in many situations that come up.
Call for Contributions: Implied Marginal Value of Time from Good/Services
by skinnersboxy — 2019-03-11T15:59:04.074Z

If you buy a good or service with the hope that doing so will save you time, then doing so creates a lower bound on the marginal value of your time. For example, if you expect that filing your taxes by hand will take 5 hours, and filing your taxes with a tax-preparation software like TurboTax will take you 1 hour, and this software costs $50, and you are neutral between filing taxes by hand or by software, then the you should buy the software if and only if you value the marginal unit of your time at or above $505 hours−1 hour=$12.50/hour.

Rationalists do these kinds of calculations all the time, but as far as I can tell, they aren't written down. This is bad, because a) everyone has to redo the math and b) coming up with ways to save time is hard. I just created a spreadsheet where I'm going to put all of the value-of-time results I compute from now on, and would appreciate it if others contributed to this sheet. Ideally, we'd get the sheet to the point where whenever the value of your time changes, you sort the timesavers by that value, and then you can start/stop buying most of the easily available timesavers for the new value.
Blegg Mode
by Zack_M_Davis — 2019-03-11T15:04:20.136Z
Link: http://unremediatedgender.space/2018/Feb/blegg-mode/

Fanfiction for the blegg/rube parable in "A Human's Guide to Words", ~800 words. (Content notice: in addition to making a point about epistemology (which is why it may have been worth sharing here), this piece is also an obvious allegory about a potentially mindkilling topic; read with caution, as always.)
IRL 2/8: Mitigating degeneracy: multiple experimentation
by RAISE — 2019-03-11T13:33:21.857Z
Link: https://app.grasple.com/#/level/1498

Every Monday for 8 weeks, we will be posting lessons about Inverse Reinforcement Learning. This is lesson 2.

Note that access to the lessons requires creating an account here.

Have a nice day!
Meta-Doomsday Argument: Uncertainty About the Validity of the Probabilistic Prediction of the End of the World
by avturchin — 2019-03-11T10:30:58.676Z
Link: https://philpapers.org/rec/TURAMA-4

Abstract: Four main forms of Doomsday Argument (DA) exist—Gott’s DA, Carter’s DA, Grace’s DA and Universal DA. All four forms use different probabilistic logic to predict that the end of the human civilization will happen unexpectedly soon based on our early location in human history. There are hundreds of publications about the validity of the Doomsday argument. Most of the attempts to disprove the Doomsday Argument have some weak points. As a result, we are uncertain about the validity of DA proofs and rebuttals. In this article, a meta-DA is introduced, which uses the idea of logical uncertainty over the DA’s validity estimated based on a virtual prediction market of the opinions of different scientists. The result is around 0.4 for the validity of some form of DA, and even smaller for “Strong DA”, which predicts the end of the world in the near term. We discuss many examples of the validity of the DA in real life as an instrument to prove it “experimentally”. We also show that DA becomes strongest if it is based on the idea of the “natural reference class” of observers, that is, the observers who know about the DA (i.e. a Self-Referenced DA). Such a DA predicts that there is a high probability of a global catastrophe with human extinction in the 21st century, which aligns with what we already know based on analysis of different technological risks.

Highlights:

· There are four main types of DA: future population prediction (Gott’s DA), Bayesian update of risks (Carter’s DA), the more probable Late Filter (Grace’s DA) and the Universal DA.

· Meta-DA treats logical uncertainty about the predictive power of the DA as a probability that DA will work.

· We used a virtual prediction market of scientists to assess the logical uncertainty of the DA, which produced estimation of around 0.4 of its validity.

· The strongest, and thus most important form of DA is the Self-Referenced DA, and for this class “the end” may be as early as middle of the 21th century, though this is not necessarily human extinction.

· Knowledge about the DA could be used to update our global risks prevention strategies by paying more attention to universal risks or by using random prevention strategies.
This is Why Your Hard Work Sits on the Shelf — and Here’s What to Do About It
by Ian David Moss — 2019-03-11T01:09:50.000Z

Why Your Hard Work Sits on the Shelf — and What to Do About It

Knowledge providers who want to see their work have greater impact might find value in partnering with a decision consultant.

Photo by chuttersnap on Unsplash

We’ve all been there. The time when the client seemed to forget the project ever happened as soon as the final check was cut. The time when your report stuffed full of creative recommendations got buried by risk-averse leadership. The time when the stakeholder group really did seem engaged by the findings, had lots of conversations, and then…nothing changed.

These stories happen with remarkable frequency. In fact, based on the evidence, there’s ample reason to believe they are the norm rather than the exception. Among more than 120 evaluation and program executives surveyed at private foundations the US and Canada, more than three-quarters had difficulty commissioning evaluations that result in meaningful insights for the field, their grantees, or the foundation itself, and 70% have found it challenging to incorporate evaluation results into the foundation’s future work. A survey of over 1600 civil servants in Pakistan and India found that “simply presenting evidence to policymakers doesn’t necessarily improve their decision-making,” with respondents indicating “that they had to make decisions too quickly to consult evidence and that they weren’t rewarded when they did.” No wonder Deloitte’s Reimagining Measurement initiative, which asked more than 125 social sector leaders what changes they most hoped to see in the next decade, identified “more effectively putting decision-making at the center” as the sector’s top priority.

This problem affects everyone working to make the world a better place, but it’s especially salient for those I call “knowledge providers”: researchers, evaluators, data scientists, forecasters, policy analysts, strategic planners, and more. It’s relevant not only to external consultants but also internal staff whose primary role is analytical in nature. And if the trend continues, we can expect that it’s eventually going to catch up to professionals working in this space. Why spend precious money and time seeking information, after all, if it’s unlikely to deliver any value?

Frustrating as this phenomenon may be, the reason for it is simple. All too often, we dive deep into a benchmarking report, evidence review, or policy a...
[Fiction] IO.SYS
by DataPacRat — 2019-03-10T21:23:19.206Z

https://www.datapacrat.com/IO.SYS.html

IO.SYS

by DataPacRat

If this has all been a VR, I’d like to be connected to reality when I wake up, please.

I lived around the turn of the twenty-first century, decided I didn’t want to die, and made the best arrangements I could. Then I died. The next section is the best I’ve been able to reconstruct, from the various resources I have available — a copy of an encyclopedia from a decade after I died, some internal project memos, and the like.

Immediately after I died, my brain was preserved, using glue-like chemicals to lock its structure and neurochemistry in place at the cost of, well, locking everything in place, just about irreversibly, short of somebody figuring out how to rebuild it atom-by-atom. Then, to be on the safe side, it (along with the rest of my body) was frozen. A couple of decades later, according to my written wishes, when the technology was developed, it was carefully diced, scanned with a ridiculous level of detail, and reconstructed in a computer as an emulation.

It seems very likely that copies were made of that initial brain-scan, and the emulated minds run, living their lives in VRs and interacting with reality in various ways. I don’t have any direct memories of anything they might have done; the thread of my own existence branched off from them as of their reconstruction. At the very least, they seem to have made enough of a good impression on various other people that the copy of the brain-scan I came from was archived.

The archive in question appears to have been imported into some kind of public-private partnership, with the apparent goal of creating an “unhackable” digital library off of Earth. However, after some of the preliminary hardware had been launched into Earth orbit, some sort of infighting between the various organizations delayed the finalization of plans of additional hardware. So the prepaid launch slots were instead dedicated to lifting up ton after ton of propellant: hydrogen, to be heated up and fired out the back of the engine.

Eventually, even those launches stopped.

At some point, inside the computer chips of the collection of hardware in orbit, a timer ticked over, and the software noticed that it hadn’t gotten any new orders in some time. It went through its to-do lists, and found a backup plan of a backup plan that matched its hardware inventory...