trentmkelly/LessWrong-43k huggingface dataset:
More on the Linear Utility Hypothesis and the Leverage Prior
by AlexMennen — 2018-02-26T23:53:35.605Z

This is the followup to “Against the Linear Utility Hypothesis and the Leverage Prior” that I had promised in the comments on that post. Apologies that it took me long enough that the previous post is no longer fresh in your mind even if you read it.

Leverage prior and convergence of expected utility

In my previous post, I offered the following counterargument to the leverage prior:

Now consider the following hypothesis: “There are 3↑↑↑3 copies of you, and a Matrix Lord will approach one of them while disguised as an ordinary human, inform that copy about his powers and intentions without offering any solid evidence to support his claims, and then kill the rest of the copies iff this copy declines to pay him $5. None of the other copies will experience or hallucinate anything like this.” Of course, this hypothesis is extremely unlikely, but there is no assumption that some randomly selected copy coincidentally happens to be the one that the Matrix Lord approaches, and thus no way for a leverage penalty to force the probability of the hypothesis below 1/3↑↑↑3.

People objected that this argument doesn't work, since the hypothesis I suggested should be considered alongside other hypotheses like “there are 3↑↑↑3 copies of you, and no Matrix Lords, but copies of you get approached by random people pretending to be Matrix Lords at whatever rate you would expect by default”. This hypothesis is quite a lot more likely than the previous one, and if you pay Pascal's Mugger, then in this world, something like (3↑↑↑3)*10^-10 copies of you lose $5, whereas in the other, vastly less likely world, 3↑↑↑3 copies of you would die. An expected utility calculation should show that paying $5 is not worth it. In general, any hypothesis in which you alone have astronomically large influence can be compared to a hypothesis in which there is a similarly-sized universe filled with copies of you without astronomical influence, some of which encounter evidence that they do have such influence. This suggests using a leverage prior.

That's a good point, but on the other hand, hypotheses like “there are 3↑↑↑3 copies of you, and no Matrix Lords, but copies of you get approached by random people pretending to be Matrix Lords at whatever rate you would expect by default” have to be compared with other hypotheses like “there are (3↑↑↑3)^2 copies of you, 3↑↑↑3 of which will be approa...
Goal Factoring
by alkjash — 2018-02-26T23:30:01.074Z

This is part 18 of 30 of Hammertime. Click here for the intro.

Up until today, Hammertime focused on improving one’s ability to achieve one’s goals. The next two techniques, Goal Factoring and Internal Double Crux, are designed to figure out what goals to pursue. For the largest goals in life, you should be able to make a detached decision about whether they’re worth pursuing before you throw your all into them.

Day 18: Goal Factoring

Previously: Day 7, Aversion Factoring.

Goal Factoring is a CFAR technique for systematically figuring out all the subgoals and aversions you have around an action, and what to do about them. The basic algorithm:

Pick an action. It can be something you already do.
Factor the action into goals and aversions. Write down all the costs and aversions to pursuing the action, and continue to factor sub-goals until they feel like irreducible components.
Brainstorm possible replacement actions. Try to design another action that achieves the goals better and reduces the costs and aversions. This action can be an upgrade of your current action, something else altogether, or even a combination of two or more actions. Make a new plan.
Reality check. Imagine instituting your new plan. Decide if you’re satisfied. Also, decide if it’s feasible by Murphyjitsu.

This is already quite a complicated and useful beast. Three things to keep in mind:

Use a focusing check to find all the subgoals and aversions. If I say out loud, “The only reason I want to go to the gym is physical health,” I feel a curtain of discontent that reminds me physical attractiveness is also important. Remember that honesty and attention to detail are essential to finding aversions, and this applies to goals as well!

Goal factoring might solve the problem at any step. Writing down your true motivations can be enough to figure out the right course of action. About three months ago, I noticed that the main motivation for my video game addiction is “Prove to my parents that it’s possible to be successful without giving up video games.” Writing this down made it impossible to endorse this action any longer.

Prepare to accept all possible worlds. Keep an open mind going into Goal Factoring: you’re allowed to consider all the alternatives. You’re also allowed to keep doing what you’re currently doing afterwards. Try to release any attachment to the action itself beyond its instrumental value. Ge...
Inconvenience Is Qualitatively Bad
by Alicorn — 2018-02-26T23:27:05.474Z

My most complicated cookie recipe has four layers. Two of these require stovetop cooking, and the other two require the use of the oven separately before the nearly-complete cookies are baked in yet a third oven use, for a total of three different oven temperatures. I have to separate eggs. I have to remember to put butter out hours in advance so it'll be softened when I get underway. Spreading the fruit neatly and then the almond goop on top of that without muddling the layers is finicky and almost none of the steps parallelize well.

They're delicious, but at what cost?

People who don't cook as a hobby would never, ever make these cookies. And this is reasonable. They shouldn't. On most days I shouldn't either. They are staggeringly inconvenient.

But they're made up of individual steps that you could mostly figure out if you really wanted to. Lots and lots of little steps. This is why I want to scream whenever I hear someone try to add steps to someone else's life. Especially if they say "just".

"Just" Google it. "Just" rinse out your recyclables. "Just" add another thing to remember and another transition to your to-do list and another obligation to feel guilt about neglecting and another source of friction between you and your real priorities. It "just" takes a minute. Don't you care?

Anyone who didn't have any immune defense against things that just take a minute would spend fifteen hours a day on one-minute tasks, if expending the energy required to switch between the tasks didn't kill them before it got that bad. But "it would be inconvenient" doesn't tend to feel like a solid rebuttal - to either party; the one attempting to impose can just reiterate "but it'll only take a minute".

Everyone needs algorithms to cut down on inconveniences.

Some I am aware of:

Chunking. Things feel less inconvenient (and accordingly are) if they are understood in batches, as one thing and not thirty. (This is related, I think, to a lot of manifestations of executive dysfunction - chunking doesn't work as well.) People naturally do more of this with things they're good at - I think a lot of being good at things just is being able to take them in larger chunks and finding larger amounts of the thing trivial.
Delegating. For

...
The Hamming Problem of Group Rationality
by PDV — 2018-02-26T18:59:33.745Z

The upcoming moderation changes are long-term catastrophic. They’re deliberately making it easy for high-status people to silence anyone and everyone they don’t want to hear from, without penalty or stigma. I trust literally no one with this power, even if they have the best of intentions; the social-monkey incentives to abuse it are strong, and of all instincts that one’s probably the best at getting around conscious intentions; according to one of the stronger prevailing theories, it's literally what our intelligence developed to do.

This is especially bad because as I see it, protecting group epistemics from corrupting social incentives is the Hamming problem of group rationality. The main difference between pursuing the art of rationality alone vs in a group is that in a group you have to manage politics, and politics is an especially strong mind-killer at the small-group level, even more so than at the national tribal level.

So as far as I’m concerned, the dev team has forfeited the war to win one temporary battle (getting Eliezer back on the site). And as significant a battle as that is, it’s not worth the war.
Focusing
by alkjash — 2018-02-26T06:10:00.614Z

This is part 17 of 30 of Hammertime. Click here for the intro.

You know how they say we only use 10 percent of our brains? I think we only use 10 percent of our hearts.

~ Owen Wilson

It is with some trepidation that I venture into the “fuzzy System 1” side of instrumental rationality. I worry that these introspective techniques optimize too much for cathartic eureka moments, and that the resulting feelings far overstate their true value.

Nevertheless, there is a definite power to these methods. You have subconscious beliefs, values, and strategies that you’re unaware of, or at least can’t articulate. Gendlin’s Focusing is a starting point for plumbing these hidden depths.

Day 17: Focusing

Background: “Focusing” for skeptics.

tl;dr: your brain hallucinates sensory experiences that have no correspondence to reality. Noticing and articulating these “felt senses” gives you access to the deep wisdom of your soul.

I’ll start by explaining my most gears-like model for why focusing works, and then describe some exercises towards strengthening the Focusing muscle.

One of the predictions of my model is that felt senses are only one piece of the nonverbal puzzle – the patterns in our dreams and our tastes for fiction and mythology, for instance, serve the same function. This will be the content of a future post.

Left and Right Brain

This model is derived from Jordan Peterson’s lectures on psychology, and in particular this conversation. I reserve the right to call everything fake if you try to falsify it.

Human beings are both predator and prey. This duality is so central to human evolution that the brain is divided left and right to serve the two different purposes separately.

The left brain is the predator brain, the center for “approach” mechanisms. It’s built for tracking a particular prey animal, articulating rules about behavior, and solving concrete problems. To fix your attention on a target is to activate your left brain and get ready to hunt it down. In the direction you look, there is clarity and legibility. Over that direction, you gain power and mastery.

“Sin” derives from the Greek word for missing the mark: human beings are aiming creatures.

The right brain is the prey brain, the center for “flight” mechanisms. It’s built for hypothesizing a venomous fog of worst-case scenarios: snakes in every tree, traps under every bramble. The right brain is constantly on edge, sea...
Mapping the Archipelago
by alkjash — 2018-02-26T05:09:49.833Z

I got excited reading Meta-tations on Moderation: Towards Public Archipelago for two reasons: there's a clear island of the archipelago I've been mostly avoiding on LessWrong, and the whole place has been growing at a high enough rate to demand fracturing.

Since we have the chance to direct the growth of the brand new archipelago, let's start a discussion down one level of meta: what specific islands do you want to see? Second, how should discussion and moderation norms differ between them?

Three islands of current LW, according to me:

AI Risk: Serious discussion for serious folk. No smiles allowed.
Instrumental Rationality: The means to get to the ends. Whatever they might be.
Fluff and Fiction: Blood for the Art God! Fun over fact.

(Guess which island I avoid.)
Experimental Open Threads
by Chris_Leong — 2018-02-26T03:13:16.999Z

Meta-tations on Moderation: Towards Public Archipelago suggested that we should be running more experiments with how discussion is run and listed several examples in the post. What if we ran monthly experimental open threads? I can imagine several formats that would work better as open threads than as a thread belonging to a specific commentator.

Here's a few thoughts of possible experiments:

Double crux - Top level comment would be claims. People who disagree can comment underneath to double crux with the poster. Third parties are discouraged from adding comments which inject their opinion into the double crux
Change my view - Top level comments would be claims that the author must believe and be willing to defend. Second level comments must disagree with at least one point of the claim or ask for clarification
Socratic method - Top level comments would be claims. Second level comments would be discouraged from directly saying that someone is wrong and instead encouraged to ask them questions instead to get them to think
Explain your perspective - Very similar to Socratic method, but the Socratic method is still focused on persuasion. This would also involve asking questions, but the commentators would be encouraged to try to understand people's views instead of trying to share them
Dependency sharing - Top level comments list a view and give examples of what might convince you to
Ask me anything - Top level comments lists an experience someone has had or specific expertise. Lower level comments ask them questions

A key part of rationality that has been somewhat underemphasised on Less Wrong is group rationality. Achieving big goals requires groups of people and these people need to be able to co-ordinate and efficiently share their knowledge. We could sit around and try to theorise about how to make this work or we could simply start running experiments.

A few questions:

Would these be suitable for frontpage posts? I don't see this experiment working with blog posts, as they would be unlikely to hit critical mass. (Sidenote: I don't think posting these threads should net me karma, if they take off it may be worthwhile to have a way to mark threads as open at some point)
Any suggestions of useful experiments to run?
Lastly, if I ran some of these experiments, would anyone commit to participating significantly in at least the first N (default N=3) t

...
Walkthrough of 'Formalizing Convergent Instrumental Goals'
by TurnTrout — 2018-02-26T02:20:09.294Z

Introduction

I found Formalizing Convergent Instrumental Goals (Benson-Tilsen and Soares) to be quite readable. I was surprised that the instrumental convergence hypothesis had been formulated and proven (within the confines of a reasonable toy model); this caused me to update slightly upwards on existential risk from unfriendly AI.

This paper involves the mathematical formulation and proof of instrumental convergence, within the aforementioned toy model. Instrumental convergence says that an agent A with utility function U will pursue instrumentally-relevant subgoals, even though this pursuit may not bear directly on U. Imagine that U involves the proof of the Riemann hypothesis. A will probably want to gain access to lots of computronium. What if A turns us into computronium? Well, there's plenty of matter in the universe; A could just let us be, right? Wrong. Let's see how to prove it.

My Background

I'm a second-year CS PhD student. I've recently started working through the MIRI research guide; I'm nearly finished with Naïve Set Theory, which I intend to review soon. To expose my understanding to criticism, I'm going to summarize this paper and its technical sections in a somewhat informal fashion. I'm aware that the paper isn't particularly difficult for those with a mathematical background. However, I think this result is important, and I couldn't find much discussion of it.

Intuitions

It is important to distinguish between the relative unpredictability of the exact actions selected by a superintelligent agent, and the relative predictability of the general kinds-of-things such an agent will pursue. As Eliezer wrote,

Suppose Kasparov plays against some mere chess grandmaster Mr. G, who's not in the running for world champion. My own ability is far too low to distinguish between these levels of chess skill. When I try to guess Kasparov's move, or Mr. G's next move, all I can do is try to guess "the best chess move" using my own meager knowledge of chess. Then I would produce exactly the same prediction for Kasparov's move or Mr. G's move in any particular chess position. So what is the empirical content of my belief that "Kasparov is a better chess player than Mr. G"?

...

The outcome of Kasparov's game is predictable because I know, and understand, Kasparov's go

... ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Will AI See Sudden Progress?
by KatjaGrace — 2018-02-26T00:41:14.514Z

Will advanced AI let some small group of people or AI systems take over the world?

AI X-risk folks and others have accrued lots of arguments about this over the years, but I think this debate has been disappointing in terms of anyone changing anyone else’s mind, or much being resolved. I still have hopes for sorting this out though, and I thought a written summary of the evidence we have so far (which often seems to live in personal conversations) would be a good start, for me at least.

To that end, I started a collection of reasons to expect discontinuous progress near the development of AGI.

I do think the world could be taken over without a step change in anything, but it seems less likely, and we can talk about the arguments around that another time.

Paul Christiano had basically the same idea at the same time, so for a slightly different take, here is his account of reasons to expect slow or fast take-off.

Please tell us in the comments or feedback box if your favorite argument for AI Foom is missing, or isn’t represented well. Or if you want to represent it well yourself in the form of a short essay, and send it to me here, and we will gladly consider posting it as a guest blog post.

I’m also pretty curious to hear which arguments people actually find compelling, even if they are already listed. I don’t actually find any of the ones I have that compelling yet, and I think a lot of people who have thought about it do expect ‘local takeoff’ with at least substantial probability, so I am probably missing things.

Crossposted from AI Impacts.
Self-regulation of safety in AI research
by Gordon Seidoh Worley — 2018-02-25T23:17:44.720Z

In many industries, but especially those with a potentially adversarial relationship to society like advertising and arms, self regulatory organizations (SROs) exist to provide voluntary regulation of actors in those industries to assure society of their good intentions. For example:

TrustArc (formerly TRUSTe) has long provided voluntary certification services to web companies to help them assure the public that companies are following basic practices that allow consumers to protect their privacy. They have been successful enough to, outside the EU, keep governments from much regulating online businesses.
The US Green Building Council offers multiple levels of LEED certification to provide both targets and proof to the public that real estate developers are protecting environmental commons.
The American Medical Association, The American Bar Association, and the National Association of Realtors are SROs that function as de facto official regulators of their industries despite being non-governmental organizations (NGOs).
Financial regulation was formerly and sometimes still is done via SROs, although governments have progressively taken a stronger hand in the industry over the last 100 years.

AI, especially AGI, is an area where there are many incentives to violate societal preferences and damage the commons and it is currently unregulated except where it comes into contact with existing regulations in its areas of application. Consequently, there may be reason to form an AGI SRO. Some reasons in favor:

An SRO could offer certification of safety and alignment efforts being taken by AGI researchers.
An SRO may be well positioned to reduce the risk of an AI race by coordinating efforts that would otherwise result in competition.
An SRO could encourage AI safety in industry and academia while being politically neutral (not tied to a single university or company).
An SRO may allow AI safety experts to manage the industry rather than letting it fall to other actors who may be less qualified or have different concerns that do not as strongly include prevention of existential risks.
An SRO could act as a "clearinghouse" for AI safety research funding.
An SRO could give greater legitimacy to prioritizing AI safety efforts among capabilities researchers.

Some reasons against:

An SRO might form a de facto "guild" and keep out qualified researchers.
An SRO could cr

...
The abruptness of nuclear weapons
by paulfchristiano — 2018-02-25T17:40:35.656Z

Nuclear weapons seem like the marquee example of rapid technological change after crossing a critical threshold.

Looking at the numbers, it seems to me like:

During WWII, and probably for several years after the war, the cost / TNT equivalent for manufacturing nuclear weapons was comparable to the cost of conventional explosives, (AI impacts estimates a manufacturing cost of $25M/each)
Amortizing out the cost of the Manhattan project, dropping all nuclear weapons produced in WWII would be cost-competitive with traditional firebombing (which this thesis estimates at 5k GBP (=$10k?) / death, vs. ~100k deaths per nuclear weapon) and by 1950, when stockpiles had gown to >100 weapons, was an order of magnitude cheaper. (Nuclear weapons are much easier to deliver, and at that point the development cost was comparable to manufacturing cost).

Separately, it seems like a 4 year lead in nuclear weapons would represent a decisive strategic advantage, which is much shorter than any other technology. My best guess is that a 2 year lead wouldn't do it, but I'd love to hear an assessment of the situation from someone who understands the relevant history/technology better than I do.

So my understanding is: it takes about 4 years to make nuclear weapons and another 4 years for them to substantially overtake conventional explosives (against a 20 year doubling time for the broader economy). Having a 4 year lead corresponds to a decisive strategic advantage.

Does that understanding seem roughly right? What's most wrong or suspect? I don't expect want to do a detailed investigation since this is pretty tangential to my interests, but the example is in the back of my mind slightly influencing my views about AI, and so I'd like it to be roughly accurate or tagged as inaccurate. Likely errors: (a) you can get a decisive strategic advantage with a smaller lead, (b) cost-effectiveness improved more rapidly after the war than I'm imagining, or (c) those numbers are totally wrong for one reason or another.

I think the arguments for a nuclear discontinuity are really strong, much stronger than any other technology. Physics fundamentally has a discrete list of kinds of potential energy, which have different characteristic densities, with a huge gap between chemical and nuclear energy densities. And the dynamics of war are quite sensitive to energy density (nucle...