trentmkelly/LessWrong-43k huggingface dataset:
In the presence of disinformation, collective epistemology requires local modeling
by jessicata — 2017-12-15T09:54:09.543Z

In

Inadequacy and Modesty

, Eliezer describes modest epistemology:

How likely is it that an entire country—one of the world’s most advanced countries—would forego trillions of dollars of real economic growth because their monetary controllers—not politicians, but appointees from the professional elite—were doing something so wrong that even a non-professional could tell? How likely is it that a non-professional could not just suspect that the Bank of Japan was doing something badly wrong, but be confident in that assessment?

Surely it would be more realistic to search for possible reasons why the Bank of Japan might not be as stupid as it seemed, as stupid as some econbloggers were claiming. Possibly Japan’s aging population made growth impossible. Possibly Japan’s massive outstanding government debt made even the slightest inflation too dangerous. Possibly we just aren’t thinking of the complicated reasoning going into the Bank of Japan’s decision.

Surely some humility is appropriate when criticizing the elite decision-makers governing the Bank of Japan. What if it’s you, and not the professional economists making these decisions, who have failed to grasp the relevant economic considerations?

I’ll refer to this genre of arguments as “modest epistemology.”

I see modest epistemology as attempting to defer to a canonical perspective: a way of making judgments that is a Schelling point for coordination. In this case, the Bank of Japan has more claim to canonicity than Eliezer does regarding claims about Japan's economy. I think deferring to a canonical perspective is key to how modest epistemology functions and why people find it appealing.

In social groups such as effective altruism, canonicity is useful when it allows for better coordination. If everyone can agree that charity X is the best charity, then it is possible to punish those who do not donate to charity X. This is similar to law: if a legal court makes a judgment that is not overturned, that judgment must be obeyed by anyone who does not want to be punished. Similarly, in discourse, it is often useful to punish crackpots by requiring deference to a canonical scientific judgment.

It is natural that deferring to a canonical perspective would be psychologically appealing, since it offers a low likelihood of being punished for deviating while allowing deviants to be punished, creating a sense of unity a...
Meaning wars
by Elo — 2017-12-14T23:17:01.874Z

Originally posted: http://bearlamp.com.au/meaning-wars/

Everyone thinks the attention game is about attention. It is (of course) but it isn't. It's about meaning. We give attention to the things that we find meaningful. Attention being a rough proxy our brain provides for meaning. That means we spend time on, thinking about, sharing the ideas, information and experiences we find meaningful. The mind-changing ideas, the discoveries, the strong emotions. The important stuff right?

Well no. That's not always what we spend time on. But let's look concretely at some examples. Think back to when TED talks first started. Enough of us had the experience when we first watched the videos and got attached to the feeling that we'd just participated in insightful and valuable information. But then we clicked next video and did it all again. And again. And again. Until eventually 8 hours later we felt stale about the whole idea of receiving an epiphany in a video in bite-sized form.

I don't watch TED talks any more and I have to ask myself why, and how that fits in the world of me wanting insights and epiphany. Are these videos interesting but not actionable? That's relevant but it's off the mark.

So why did we click, and why did we pay attention? And why did it all go wrong? How did things get unstuck? These videos call our attention, but don't matter to us. Even though they are fascinating and attractive. An info-hazard: be careful or you will be sucked into "many-ted-talks", where-did-my-afternoon-go?

Why?

Well - meaning. We seek meaning. We seek to matter, and we seek to do things we care about. (this is not insightful, it's obvious).

When you watch your first video, it's pretty new, it's unique and insightful. The second delivers the same. And the fourteenth? It doesn't matter how interesting this one is, it's probably not the same wonderful feeling as the first video. It's getting to be the same delivery of information. Despite being exciting, it's also getting old. It's losing its meaning...

We start out wanting meaning, we start out getting meaning, and after a while we don't really get the same thing any more. We are not designed to notice meaning wearing off - we expect it to keep being there. Until it's well and truly worn out so bad that ...
Against the Linear Utility Hypothesis and the Leverage Penalty
by AlexMennen — 2017-12-14T18:38:53.103Z

[Roughly the second half of this is a reply to: Pascal's Muggle]

There's an assumption that people often make when thinking about decision theory, which is that utility should be linear with respect to amount of stuff going on. To be clear, I don't mean linear with respect to amount of money/cookies/etc that you own; most people know better than that. The assumption I'm talking about is that the state of the rest of the universe (or multiverse) does not affect the marginal utility of there also being someone having certain experiences at some location in the uni-/multi-verse. For instance, if 1 util is the difference in utility between nothing existing, and there being a planet that has some humans and other animals living on it for a while before going extinct, then the difference in utility between nothing existing and there being n copies of that planet should be n utils. I'll call this the Linear Utility Hypothesis. It seems to me that, despite its popularity, the Linear Utility Hypothesis is poorly motivated, and a very poor fit to actual human preferences.

The Linear Utility Hypothesis gets implicitly assumed a lot in discussions of Pascal's mugging. For instance, in Pascal's Muggle, Eliezer Yudkowsky says he “[doesn't] see any way around” the conclusion that he must be assigning a probably at most on the order of 1/3↑↑↑3 to the proposition that Pascal's mugger is telling the truth, given that the mugger claims to be influencing 3↑↑↑3 lives and that he would refuse the mugger's demands. This implies that he doesn't see any way that influencing 3↑↑↑3 lives could not have on the order of 3↑↑↑3 times as much utility as influencing one life, which sounds like an invocation of the Linear Utility Hypothesis.

One argument for something kind of like the Linear Utility Hypothesis is that there may be a vast multiverse that you can influence only a small part of, and unless your utility function is weirdly nondifferentiable and you have very precise information about the state of the rest of the multiverse (or if your utility function depends primarily on things you personally control), then your utility function should be locally very close to linear. That is, if your utility function is a smooth function of how many people are experiencing what conditions, then the utility from influencing 1 life should be 1/...
The Perils of the Security Mindset taken too far
by  — 2017-12-13T22:34:08.173Z

Epistemic status: A few initial thoughts.

For your project to be secure no one should know of your project's existence. Or better than that your project should show all the outwards signs of one project but be another.

To be secure, your inner workings should be opaque, so you become less predictable. Therefore people trust you less. In Newcomb's problem one of the common strategies people come up with to trick omega is to use quantum sources of information to become less predictable. The common counter to this is that Newcomb only fills both boxes in the case it can predict you.

If you are opaque, it is not known who you associate with. Even if people trust you, they might not trust that you would not associate with people they do not trust.

If you are opaque, your security model is not known. Even if people trust you not to leak things on purpose, they might not trust you to do so accidentally.

You trust people less than is optimal. There are false negatives to your decisions to trust people.

You degrade the rationality of other people. There are at least two things that mean people aren't as rational as they could be, a lack of brain power and a lack of information. Hiding information means people can be a lot less rational or effective. This cost is borne by everyone else but you, so you might not be accounting for it.

Hiding your strategic views, hides your flaws. No one can know if you are being too paranoid, because you hide your threat model.

Brain power is hoovered up trying to model other people modelling you, to make sure that you don't tip your hand.

If you can possibly avoid taking the security mindset this far, do so. Do all you can in the freedom of openness. Secrecy can also be a mindkiller.
Guarding Slack vs Substance
by Raemon — 2017-12-13T20:58:10.775Z

Builds on concepts from:

Slack
Goodheart's Imperius
Nobody Does the Thing They are Supposedly Doing

Summary: If you're trying to preserve your sanity (or your employees') by scaling back on the number of things you're trying to do... make sure not to accidentally scale back on things that were important-but-harder-to-see, in favor of things that aren't as important but more easily evaluated.

Epistemic Effort: Had a conversation, had some immediate instinctive reactions to it, did not especially reflect on it. Hope to flesh out how to manage these tradeoffs in the comments.

Zvi introduced the term "Slack" to the rationaljargonsphere a few months ago, and I think it's the most clearly useful new piece of jargon we've seen in a while.

Normally, when someone coins a new term, I immediately find people shoehorning it into conversations and concepts where it doesn't quite fit. (I do this myself an embarrassing amount, and the underlying motivation is clearly "I want to sound smart" which bodes ill).

By contrast, I experienced an explosion of people jargon-dropping Slack into their conversations and every single instance was valid. Lack-of-slack was a problem loads of people had been dealing with, and having a handle for it was a perfect instance of a new name enabling higher level discussion.

This hints at something that should be alarming: "slack" is a useful term because nobody has enough of it.

In particular, it looks like many organizations I'm familiar with run at something like -10% slack, instead of the 40% slack that apparently is optimal across many domains.

Gworley noted in the comments of Zvi's post:

If you work with distributed systems, by which I mean any system that must pass information between multiple, tightly integrated subsystems, there is a well understood concept of maximum sustainable load and we know that number to be roughly 60% of maximum possible load for all systems.

The probability that one subsystem will have to wait on another increases exponentially with the total load on the system and the load level that maximizes throughput (total amount of work done by the system over some period of time) comes in just above 60%. If you do less work you are wasting capacity (in terms of throughput); if you do more work you will gum up the works and waste time waiting even if all

...
Calling Bullshit - Lectures and Readings on Evaluating  Scientific Research
by Kyle Hipke — 2017-12-13T20:15:27.987Z
Link: http://callingbullshit.org/index.html

This is a relatively recent course from UW which has all of the lecture videos and reading materials now available online. It's very useful for learning how to evaluate scientific research without having enough knowledge to evaluate the actual content of the research papers.
Oracle paper
by Stuart_Armstrong — 2017-12-13T15:04:27.000Z

Available on the arXiv, my paper on two types of Oracles (AIs constrained to answering questions only), and how to use them more safely.

An Oracle is a design for potentially high power artificial intelligences (AIs), where the AI is made safe by restricting it to only answer questions. Unfortunately most designs cause the Oracle to be motivated to manipulate humans with the contents of their answers, and Oracles of potentially high intelligence might be very successful at this. Solving the problem, without compromising the accuracy of the answer, is tricky. This paper reduces the issue to a cryptographic-style problem of Alice ensuring that her Oracle answers her questions while not providing key information to an eavesdropping Eve. Two Oracle designs solve this problem, one counterfactual (the Oracle answers as if it expected its answer to never be read) and one on-policy (limited by the quantity of information it can transmit).
Oracle paper
by Stuart_Armstrong — 2017-12-13T14:59:08.052Z

Available on the arXiv, my paper on two types of Oracles (AIs constrained to answering questions only), and how to use them more safely.

An Oracle is a design for potentially high power artificial intelligences (AIs), where the AI is made safe by restricting it to only answer questions. Unfortunately most designs cause the Oracle to be motivated to manipulate humans with the contents of their answers, and Oracles of potentially high intelligence might be very successful at this. Solving the problem, without compromising the accuracy of the answer, is tricky. This paper reduces the issue to a cryptographic-style problem of Alice ensuring that her Oracle answers her questions while not providing key information to an eavesdropping Eve. Two Oracle designs solve this problem, one counterfactual (the Oracle answers as if it expected its answer to never be read) and one on-policy (limited by the quantity of information it can transmit).
Bayes and Paradigm Shifts - or being wrong af
by abbeybee — 2017-12-13T04:48:17.844Z

So I've been thinking about Bayesian probability and paradigm shifts. One of the early examples that Price published after discovering Bayes' theorem (after Bayes died) was of someone who, upon awakening for the first time with no other information on cosmology, if they knew Bayes theorem, could then update their probability that the sun would rise again the next day, each day they saw it rise again. So with time, as they see the sun rise more and more times, they become more and more 'certain' that it will rise again the next day (ie their priors become higher).

However, not having any knowledge of the universe or physics, they are unaware that there is a near certainty that this sun will someday supernova and no longer rise again. If they made thousands of generations of sun tracking bayesians, every day they would see the sun rise and update their probability, and become more certain that it would rise again. By the time it didn't rise, they would be wildly certain that it would rise again. So the more certain they became, actually the more WRONG they became. That sun was always almost certainly doomed at the same 99.999....% level the whole time (maybe not to each given new day, but eventually) and they just didn't have access to good enough priors to recognize this.

So as a result of bad priors, they are maybe increasing their accuracy relative to any given day (the sun only dies on 1 in a billion days) but decreasing their accuracy of it's eventual transformation into a black hole or some such phenomena which will likely kill the shit out of them.

I think this kind of misinformed search for accuracy is very symbolic of a bayesian look at paradigm shifts (even as it could also b used as a limited critique of bayesian statistics). Once they get access to just the knowledge that other stars exist, it opens up a huge range of other variables they didn't know about in the calculation of their priors. So while we're chugging along in our search for accuracy, we may be building relative accuracy, while building absolute an error until our paradigm catches up with a new and deeper layer of information.
TSR #5 The Nature of Operations
by Hazard — 2017-12-12T23:37:06.066Z

**This is part of a series of posts where I call out some ideas from the latest edition of The Strategic Review (written by Sebastian Marshall), and give some prompts and questions that I think people might find useful to answer. I include a summary of the most recent edition, but it's not a replacement for reading the actual article. Sebastian is an excellent writer, and your life will be full of sadness if you don't read his piece. The link is below.

Background Ops # 5: The Nature of Operations

SUMMARY

Religio → Strategy → Tactics → Operations

Zhukov was pretty cool. You should def read to article to know why.
Ops can be “hard to get” because they often seem too simple to be considered.
Guidance

“Though operations cannot be understood by the individual pieces alone, the best practices of implementing and running the individual pieces are critical to learn.”
“Study and learn how different pieces of operations fit together over time, practice analyzing where one’s operations are the weakest, and how to improve operations across the spectrum.”

I really like using this hierarchy to think about things. Just remembering that these levels of planning exist helps me notice when I’m not acting on them. How does my religio inform my strategy? How does my strategy inform my tactics? How do my tactics inform my operations? When I’m falling apart, these stay as rhetorical questions. When I’m on top of things, they get answered.

Sebastian’s definition of Operations seems pretty accurate. The coordination of tactics over time. However, I prefer to think about operations via a question. What do you do to ensure that the things you intend to do actually get done? That seems to be at the core of operations, which seem to be at the core of getting good at things and living a quality life. Lots of low hanging fruit for improving one’s life is stuff you’ve heard before. Understanding operations lets you go, “Ohhhh, that’s why nothing worked. I was only trying to try, not actually doing.”

It seems like there are two big categories of operations. The first are operations that ensure you get a particular thing done. Examples would be a weightlifting regimen that a coach makes for you, a morning routine that you use to feel ready to tackle the day, or going grocery shopping every other friday to ensure you’ve got enough food. The second are operations that allow you to get a part...
12/12/2017 Update: Creating Sequences
by Raemon — 2017-12-12T22:34:05.684Z

Over the past few weeks, we've been pushing updates that improve LW 2.0's Sequence functionality, as well as cleaning up some of the less well known sequences.

Those updates are now complete and ready for general use. Highlights include:

The Library

If you click on the "Recommended Reading" title on the front page, or on "The Library" in the main menu, you'll be taken to our new Library page. This includes sections for our Core Reading, Curated Sequences, and Community Sequences.

Core Reading is essential content that has stood the tests of time, that users are generally expected to have read.

Curated Sequences are additional high quality essays that the admins have decided to feature.

Community Sequences are where most new sequences will appear. (Currently sorted by "newest first", although we'll probably update the sorting mechanisms soon)

Creating New Sequences

Under the "Community Sequences" header, you'll see a "Create new sequence" button. This is deliberately somewhat hard-to-find, since we don't want new users to immediately start creating sequences willy nilly.

To create a sequence, you'll start by giving it a name, description, and images for the Banner and Thumbnail. It's recommend that the Banner and Thumbnail be the same picture (although perhaps cropped differently so that they look good at large and small scales).

Banner images must be at least 1600 pixels wide. Eventually we'll add some tools to help users find appropriate images. Meanwhile, if you need help finding an appropriate banner, you can ping us and we'll help you out. (In general, a good tip is to go to Google Image search, click "tools" tab, and restrict results to images whose size is at least 2 MP, and whose usage rights are set to "noncommercial re-use)

After clicking the "create" button, you'll be able to add and change the ordering of posts.

User Profile Sequences

Once you've created at least one sequence, your User Profile page will gain a Sequences section. This will list your sequences, and includes another "create sequence" button.

You can see an example of this at Luke Muehlhauser's user page.

Happy Sequencing!

Creating better sequence tools is a core element of the LW 2.0 strategy. Some of our next goals are to improve your ability to ke...
Quick thoughts on empathic metaethics
by lukeprog — 2017-12-12T21:46:08.834Z

Years ago, I wrote an unfinished sequence of posts called "No-Nonsense Metaethics." My last post, Pluralistic Moral Reductionism, said I would next explore "empathic metaethics," but I never got around to writing those posts. Recently, I wrote a high-level summary of some initial thoughts on "empathic metaethics" in section 6.1.2 of a report prepared for my employer, the Open Philanthropy Project. With my employer's permission, I've adapted that section for publication here, so that it can serve as the long-overdue concluding post in my sequence on metaethics.

In my previous post, I distinguished "austere metaethics" and "empathic metaethics," where austere metaethics confronts moral questions roughly like this:

Tell me what you mean by 'right', and I will tell you what is the right thing to do. If by 'right' you mean X, then Y is the right thing to do. If by 'right' you mean P, then Z is the right thing to do. But if you can't tell me what you mean by 'right', then you have failed to ask a coherent question, and no one can answer an incoherent question.

Meanwhile, empathic metaethics says instead:

You may not know what you mean by 'right.' But let's not stop there. Here, let me come alongside you and help decode the cognitive algorithms that generated your question in the first place, and then we'll be able to answer your question. Then we can tell you what the right thing to do is.

Below, I provide a high-level summary of some of my initial thoughts on what one approach to "empathic metaethics" could look like.

Given my metaethical approach, when I make a “moral judgment” about something (e.g. about which kinds of beings are moral patients), I don’t conceive of myself as perceiving an objective moral truth, or coming to know an objective moral truth via a series of arguments. Nor do I conceive of myself as merely expressing my moral feelings as they stand today. Rather, I conceive of myself as making a conditional forecast about what my values would be if I underwent a certain “idealization” or “extrapolation” procedure (coming to know more true facts, having more time to consider moral arguments, etc.).[1]

Thus, in a (hypothetical) "extreme effort" attempt to engage in empathic metaethics (for thinking about my own mora...