trentmkelly/LessWrong-43k huggingface dataset:
The Costs of Reliability
by sarahconstantin — 2019-07-20T01:20:00.895Z

A question that used to puzzle me is “Why can people be so much better at doing a thing for fun, or to help their friends and family, than they are at doing the exact same thing as a job?”

I’ve seen it in myself and I’ve seen it in others. People can be hugely more productive, creative, intelligent, and efficient on just-for-fun stuff than they are at work.

Maybe it’s something around coercion? But it happens to people even when they choose their work and have no direct supervisor, as when a prolific hobbyist writer suddenly gets writer’s block as soon as he goes pro.

I think it has a very mundane explanation; it’s always more expensive to have to meet a specific commitment than merely to do something valuable.

If I feel like writing sometimes and not other times, then if writing is my hobby I’ll write when I feel like it, and my output per hour of writing will be fairly high. Even within “writing”, if my interests vary, and I write about whatever I feel like, I can take full advantage of every writing hour. By contrast, if I’ve committed to write a specific piece by a specific deadline, I have to do it whether or not I’m in the mood for it, and that means I’ll probably be less efficient, spend more time dithering, and I’ll demand more external compensation in exchange for my inconvenience.

The stuff I write for fun may be valuable! And if you simply divide the value I produce by my hours of labor or the amount I need to be paid, I’m hugely more efficient in my free time than in my paid time! But I can’t just trivially “up my efficiency” in my paid time; reliability itself has a cost.

The costs of reliability are often invisible, but they can be very important. The cost (in time and in office supplies and software tools) of tracking and documenting your work so that you can deliver it on time. The cost (in labor and equipment) of quality assurance testing. The opportunity cost of creating simpler and less ambitious things so that you can deliver them on time and free of defects.

Reliability becomes more important with scale. Large organizations have more rules and procedures than small ones, and this is rational. Accordingly, they pay more costs in reliability.

One reason is that the attack surface for errors grows with the number of individuals involved. For instance, large organizations often have rules against downloading software onto company computers without ...
Appeal to Consequence, Value Tensions, And Robust Organizations
by Matt Goldenberg — 2019-07-19T22:09:43.583Z

Epistemic Status: Strong opinions weakly held. Mostly trying to bring some things into the discourse that I think are too often ignored.

Some updates I've made based on the discussion in this post are here .

Introduction

Jessicata's Dialogue on Appeals to Consequences is an expansion of a response that she wrote to me a few months ago, arguing a particular point that I agree with: Namely, if you have an object level thing you want in the world, it's almost never worth lying or withholding information about that thing, because it breaks meta level norms about truthseeking that are much more important to accomplishing object level goals in general. However, there's a slightly more interesting case that I think is quite murkier, that the original comment was pointing to. That is, what if your truthseeking norms are in tension with OTHER meta level norms that are important? In general, how do you deal with instances where tensions between two important values cause you to not know what to do?

Dialogue

Let's imagine John and Jill are discussing John's behavior in a private space. Jill is a leader of the space, and John is someone who frequently attends the space and has lively discussions trying to get to the truth.

Jill: John, I've had several complaints about your tendency to steer conversations towards the divisive topic that everyone should be a Vegan, and I'm going to ask you to tone it down a bit when you're in our main space.

John: Are people saying that I'm making arguments that are false?

Jill: No, no one is saying that you're making false arguments. John: Are people saying that I'm derailing the conversation? I think you'll find that every instance I brought up veganism was highly relevant to the conversation.

Jill: Yes, some people have said that, but I happen to believe you when you say that you've only brought it up in relevant contexts for you.

John: Then what's the problem? I'm stating relevant true beliefs that add to the totality of the conversation and steer it in conversationally relevant directions.

Jill: The problem is twofold. Firstly, people find it annoying to retread the same conversation over and over. More importantly, this topic usually leads to demon conversations, and I fear that continued discussion of the topic at the rate its' currently discussed co...
Watch Elon Musk’s Neuralink presentation
by Alexei — 2019-07-19T21:48:11.162Z
Link: https://www.youtube.com/watch?v=lA77zsJ31nA

"Electric vehicles, rockets... and now brain-computer interfaces. Elon Musk's newest venture, Neuralink, aims to bridge the gap between humans and artificial intelligence by implanting tiny chips that can link up to the brain. At a press conference on July 16, Neuralink's ambitious plans were detailed for the first time, showcasing a future (a very distant future!) technology that could help people deal with brain or spinal cord injuries or controlling 3D digital avatars."
When does adding food make an organism reliably better?
by Slider — 2019-07-19T21:22:38.904Z

Food is important for an organism to grow and adapt. Yet most organism can overfeed and become obese. Many organism also artificially limit themselfs to a small size. This seems wasteful if food is readily available.

Are there any organisms that can actually take benefit from all the food available? Can those traits be transferred to more traditional organisms?

This question is formed to be comparatively silly when compared to a previous question. However should this question have answers those answers should be analogous to answers to the parent question.
Scope Insensitivity Judo
by Gordon Seidoh Worley — 2019-07-19T17:33:27.716Z

It's easy to bemoan scope insensitivity, a special case of that phenomenon where we mere humans end up caring more about the death of one person than one hundred, better remember the last bite of a meal than the first dozen, and think less is more and more is less. After all, if we didn't neglect scope we would be more rational, and so maybe happier and healthier, living in a world were everyone got more of what they wanted, since without scope insensitivity it wouldn't be so hard to convince people to help those far away who need more than those nearby who need less. But scope insensitivity is what we've got, so we have to learn to live with it.

Luckily there's plenty of reason to think we can take advantage of scope insensitive because people have already discovered ways to make the best of other forms of extension neglect. For example, adapting to duration neglect is something most people learn early on by adopting heuristics like "save the best for last" and "do the hardest part first". Salespersons and motivational speakers alike learn to exploit base rate neglect, sample size neglect, and the conjunction fallacy to convince people to do what they otherwise might not. And designers of all kinds of systems can mold incentives to work with rather than against human nature. Thus it stands to reason we can use our natural scope insensitivity to do more than fail to multiply.

I'll consider one such use case here, namely a practice of using scope insensitivity to prepare ourselves for high-stakes situations in low-stakes ones. This is a kind of scope insensitivity judo, or "gentle way", and just like in the martial art, we'll redirect the strength of our "opponent" to transform it into an unintended ally.

Dwell in the Dojo

Life is full of high-stakes situations: job interviews, first dates, nuclear missile crises. These generally feel like one-shot scenarios—there's one chance to get it right and if we fail all is lost. To wit, if we don't say the right things we'll lose our shot at that job forever, if we don't put out the right vibes that person will never fall in love with us, and if we push the big red button there'll be no second chances for anything.

We can make these multi-shot scenarios pretty easily with training, and there's some value in practicing i...
From Laplace to BIC
by johnswentworth — 2019-07-19T16:52:58.087Z

The previous post outlined Laplace approximation, one of the most common tools used to approximate hairy probability integrals. In this post, we'll use Laplace approximation to derive the Bayesian Information Criterion (BIC), a popular complexity penalty method for comparing models with more free parameters to models with fewer free parameters.

The BIC is pretty simple:

Start with the maximum log likelihood lnP[data|θmax]
Subtract k2lnN to penalize for complexity, where N is the number of data points and k is the number of free parameters (i.e. dimension of θ).

Thus: lnP[data|θmax]−k2lnN. Using this magic number, we compare any two models we like.

Let's derive that.

BIC Derivation

As usual, we'll start from P[data|model]. (Caution: don't forget that what we really care about is P[model|data]; we can jump to P[data|model] only as long as our priors are close enough to be swamped by the evidence.) This time, we'll assume that we have N independent data points xi, all with the same unobserved parameters - e.g. N die rolls with the same unobserved biases. In that case, we have

P[data|model]=∫θP[data|θ]dP[θ]=∫θ∏Ni=1P[xi|θ]p[θ]dθ

Next, apply Laplace approximation and take the log.

lnP[data|model]≈∑ilnP[xi|θmax]+lnp[θmax]+k2ln(2π)−12lndet(H)

where the Hessian matrix H is given by

H=d2dθ2lnP[data|θ]|θmax=∑id2dθ2lnP[xi|θ]|θmax

Now for the main trick: how does each term scale as the number of data points N increases?

The max log likelihood ∑iP[xi|θmax] is a sum over data points, so it should scale roughly proportionally to N.
The prior density and the k2ln(2π) are constant with respect to N.
H is another sum over data points, so it should also scale roughly proportionally to N.

Let's go ahead and write H as N∗(1NH), to pull out the N-dependence. Then, if we can remember how determinants scale:

lndet(N∗(1NH))=lndet(1NH)+k∗lnN

so we can re-write our Laplace approximation as lnP[data|model]≈∑ilnP[xi|θmax]+p[θmax]+k2ln(2π)−12lndet(1NH)−k2lnN=lnP[data|θmax]−k2lnN+O(1)

where O(1) contains all the terms which are roughly constant with respect to N. The first two terms are the BIC.

In other words, the BIC is just the Laplace approximation, but ignoring all the terms which don't scale up as the number of data points increases.

When Does BIC Work?

What conditions need to hold for BIC to work? Let's go back through the derivation and list out the key a...
Against Excessive Apologising
by Chris_Leong — 2019-07-19T15:00:34.272Z

Many people would say that if you realise that you are in the wrong, then you should always apologise. Perhaps, they'd exclude sociopathic situations where this would be used to manipulate you, but that'd be it.

However, it's easy to forget that apologising creates a cost for the person who is apologised to. They have to read your message and perhaps write a reply. This later component is tricky if they aren't convinced that you've made up for it. It reminds them of an experience they might want to forget. Further, it requires them to deal with a topic they may be completely sick and tired of.

If you apologise, it should be because it helps prevent or mend a rift with the other person. You should be extremely cautious about apologising when it's because that's what you think a nice person would do, as opposed to something more specific, since those are precisely the situations where you are likely to end up apologising with no benefit to anyone.

Now most people don't apologise enough and so this is probably the wrong advice for them. But Less Wrong samples a particular segment of the population and I suspect it includes a disproportionate number of those people who over-apologise.
[Link] Intro to causal inference by Michael Nielsen (2012)
by yagudin — 2019-07-19T12:19:50.817Z

This is a link post for Michael Nielsen's "If correlation doesn’t imply causation, then what does?" (2012).

I want to highlight the post for a few reasons:

(1) it is a well-written introduction by an experienced science communicator — Michael is an author of the most famous book on quantum computing;

(2) causal inference is an essential tool for understanding the world;

(3) two recent AI safety papers use causal influence diagrams to (a) understand agent incentives [arXiv, Medium] and (b) to provide a new perspective on some problems in AGI safety [arXiv].
When does adding more people reliably make a system better?
by Bird Concept — 2019-07-19T04:21:06.287Z

Prediction markets have a remarkable property. They reward correct contrarianism. They incentivise people to disagree with the majority consensus, and be right. If you add more traders to a market, in expectation they price will be more accurate.

More traders means both more fish and more sharks.

(The movie "The Big Short" might be a very sad portrait of the global financial system. But it's still the case that a system in a bad equilibrium with deeply immoral consequences rewarded the outcasts who pointed out those consequences with billions of dollars. Even though socially, no one bothered listening to them, including the US Government who ignored requests by one of the fund managers to share his expertise about the events after the crash.)

Lots of things we care about don't have this property.

Many social communities decline as more members join, and have to spend huge amounts of effort building institutions and rituals to prevent this.
Many companies have their culture decline as they hire more, and have to spend an incredible amount of resources simply to prevent this (which is far from getting better as more people join). (E.g. big tech companies can probably have >=5 candidates spend >=10 hours in interviews for a a single position. And that's not counting the probably >=50 candidates for that position spending >=1h.)
Online forums usually decline with growing user numbers (this happened to Reddit, HackerNews, as well as LessWrong 1.0).

In prediction markets the vetting process is really cheap. You might have to do some KYC, but mostly new people is great. This seems like a really imporant property for a system to have, and something we could learn from to build other such systems.

What other systems have this property?
Robust Agency for People and Organizations
by Raemon — 2019-07-19T01:18:53.416Z

Epistemic Status – some mixture of:

“My best guess, based on some theory, practice and observations. But very much _not_ battle-tested”
but also, “poetry that’s designed to get an idea across that isn’t necessarily precisely accurate", intended to get across the generators for my current worldview.
Was waiting to post this until I resolved some disagreements that seemed upstream, but I think that'll be awhile and maybe was a bad reason to delay anyhow. idk. YOLO.

tl;dr:

People are not automatically robust agents, and neither are organizations.

An organization can become an agent (probably?) but only if it’s built right. Your default assumption should probably be that a given organization is not an agent (and therefore may not be able to credibly make certain kinds of commitments).

Your default assumption, if you’re building an organization, should probably be that it will not be an agent (and will have some pathologies common to organizations).

If you try on purpose to make it an agent, have good principles, etc…

...well, your organization probably still won’t be an agent, and some of those principles might get co-opted by adversarial processes. But I think it’s possible for an organization to at least be better at robust agency (and, also better at being “good”, or “human value aligned”, or at least “aligned with the values of the person who founded it.”)

Becoming a robustly agentic person

For a few years I’ve been crystallizing what it means to be a robust agent, by which I mean: “Reliably performing well, even if the environment around you changes. Have good policies. Have good meta policies. Be able to interface well with people who might have a wide variety of strategies, some of who might be malicious or confused.”

People are not born automatically strategic, nor are they born an “agent.”

If you want robust agency, you have to cultivate it on purpose.

I have a friend who solves a lot of problems using the multi-agent paradigm. He spends a lot of effort integrating and empowering his sub-agents. He treats them like adults, makes sure they understand each other and trust each other. He makes sure each of them have accurate beliefs, and he tries to empower each of them as much as possible so they have no need to compete.

This… doesn’t actually work for me.

I’ve tried things like internal double crux or internal family systems, and so far, it’s just produc...
Thoughts on the 5-10 Problem
by Tofly — 2019-07-18T18:56:14.339Z

5 dollars is better than 10 dollars

The 5-10 Problem is a strange issue in which an agent reasoning about itself makes an obviously wrong choice.

Our agent faces a truly harrowing choice: it must decide between taking $5 (utility 5) or $10 (utility 10).

How will our agent solve this dilemna? First, it will spend some time looking for a proof that taking $5 is better than taking $10. If it can find one, it will take the $5. Otherwise, it will take the $10.

Fair enough, you think. Surely the agent will concede that it can't prove taking $5 is better than taking $10. Then, it will do the sensible thing and take the $10, right?

Wrong.

Our agent finds the following the following proof that taking $5 is better:

F→(U=5)F→(¬F→(U=0))□((F→(U=5))∧(¬F→(U=0)))→F□((F→(U=5))∧(¬F→(U=0)))→((F→(U=5))∧(¬F→(U=0)))(F→(U=5))∧(¬F→(U=0))F

Let's go over the proof.

Line 1: Taking $5 gives you $5.

Line 2: If F is true, then ~F->x is true for any x.

Line 3: If you find a proof that taking $5 gives you $5 and take $10 gives you $0, you'd take the $5.

Line 4: Combine the three previous lines

Line 5: Löb's Theorem

Line 6: Knowing that taking $5 gives you $5 and taking $10 gives you $0, you happily take the $5.

Simplified Example

To understand what went wrong, we'll consider a simpler example. Suppose you have a choice between drinking coffee (utility 1) and killing yourself (utility -100).

You decide to use the following algorithm: "if I can prove that I will kill myself, then I'll kill myself. Otherwise, I'll drink coffee".

And because a proof that you'll kill yourself, implies that you'll kill yourself, by Lob's Theorem, you will kill yourself.

Here, it is easier to see what went wrong-proving that you'll kill yourself is not a good reason to kill yourself.

This is hidden in the original 5-10 problem. The first conditional is equivalent to "if I can prove I will take $5, then I'll take $5".

Hopefully, it's now more clear what went wrong. How can we fix it?

Solution?

I once saw a comment suggesting that the agent instead reason about how a similar agent would act (I can't find it anymore, sorry). However, this notion was not formalized. I propose the following formalization:

We construct an agent A. Each time A makes a decision, it increments an internal counter n, giving each decision a unique identity. A ... ,
Prereq: Question Substitution
by Hazard — 2019-07-18T17:35:56.411Z

A quote from a 2002 paper by Kahneman and Frederick referencing some of Kahneman's earlier work with Tversky:

Early research on the representativeness and availability heuristics was guided by a simple and general hypothesis: when confronted with a difficult question people often answer an easier one instead, usually without being aware of the substitution.

It's an interesting hypothesis, and even without looking at studies it seems plausible. I can think of conversations where I've been frustrated by the fact that the other person can't seem to actually answer the question I've asked them and keeps wiggling around. But it makes me wonder.

How would you tell the difference between A:

Brain receives question
Question is substituted for an easier one.
Brain replies with answer to easier question

and B:

Brain receives question
Brain replies using a heuristic that it applies to this sort of question.

(thanks to this comment for helping me notice the discrepancy)

Hmmm, it seems like they could easily produce the same answer. Does question substitution explain any more than "the brain uses heuristics"?

Yes and no. I read the first few pages of the paper, and pretty quickly noticed that Kahneman doesn't seem to think that a literal question substitution is happening. Most of the paper explores the question of "what governs what heuristic gets applied when?" and in other phrasings Kahneman states "[people reply to questions] as if she had been asked [other question]."

Despite the fact that literal question substitution might not happen that much, I can still get useful information by using the substitution frame. If I'm investigating whether or not I used a heuristic I wasn't aware of, I can ask myself: "Was I trying to answer an easier question? What are similar but simpler questions related to the one I was trying to answer? Are there any questions I know I'm good at answering that I might have substituted for the original one?"

You might even say that question substitution is a great... heuristic for figuring out if you were using a heuristic, in lieu of a more detailed model of how and when your brain picks heuristics. Yes, by all means read the rest of the paper and launch a quest into understanding in detail how the mind works. And while you're working on that, feel free to use the i...