trentmkelly/LessWrong-43k huggingface dataset:
On Stories vs. Data
by Ian David Moss — 2019-12-05T20:07:13.000Z

Photo by Noah Negishi on Unsplash

Many nonprofit leaders express discomfort with attempts to translate their work into a statistic. For them, the notion of reducing a potentially life-changing experience to a number doesn’t just feel confusing, it’s kind of insulting. Meanwhile, fundraisers making their case to individual donors, advocates, and policymakers often find that a powerful story can work wonders where facts and figures fall flat.

It’s easy to see why most people prefer stories to data. A story is rich, full of detail and shape. Data is flat. Put another way, data is mined from the common ground between various stories, which means that in order for it to work, for it to be converted into the language of numbers, you have to exclude extraneous information. Even if that “extraneous” information happens to be really interesting and cool and sums up exactly why we do what we do!

The reason stories work for us as human beings is because they are few in number. We can spend two hours watching a documentary, or a week reading a history book, and get a really deep qualitative understanding of what was going on in a specific situation or in a specific case. The problem is that we can only truly comprehend so many stories at once. We don’t have the mental bandwidth to process the experiences of even hundreds, much less thousands or millions of subjects or occurrences. To make sense of those kinds of numbers, we need ways of simplifying and reducing the amount of information we store in each case. So what we do is we take all of those stories and we flatten them: we dry out all of the rich shape and detail that makes up their original form and we package them instead in a kind of mold: collecting a specific and limited set of attributes about each so that we can apply analysis techniques to them in batch. In a very real sense, data = mass-produced stories.

It sounds horrible when I put it like that, right? But it’s an essential process because without it, we can’t be assured that we’re looking at the whole picture. Especially when we’re dealing with a large number of potential cases or examples, if we just concentrate on those that are nearest to us, whether that proximity is measured by geography or social/professional circle or similarity to our own situation, there is a very real risk that we will draw inappropriate conclusions about examples that are a littl...
To Be Decided #3
by Ian David Moss — 2019-12-05T19:06:18.198Z

TBD is a quarterly-ish newsletter about deploying knowledge for impact, learning at scale, and making more thoughtful choices for ourselves and our organizations. This is the third issue, which was originally published in September 2019. Enjoy! --Ian

What I've Been Reading

The Crisis of Evidence Use
In the last issue of To Be Decided, I highlighted a study that shows most foundations have trouble commissioning evaluations that yield meaningful insights for the field, grantees, or even their own colleagues. A remarkable finding, but you could be forgiven for wondering how much we can conclude from just one study. Sadly (and ironically) there is plenty of evidence to reinforce the point that people with influence over social policy simply don't read or use the vast majority of the knowledge we produce, no matter how relevant it is. What's really astounding about this is how much time, money, attention we spend on evidence-building activities for apparently so little concrete gain. We are either vastly overvaluing or vastly undervaluing evidence. We need to get it right, because those are real resources that could be spent elsewhere, and the world is falling apart around us while we get lost in our spreadsheets and community engagement methodologies.
(

Twitter thread

|

Blog post

)

Getting Smarter About Learning from Other People's Research

While there's no one silver bullet for improving evidence use, one thing we can do is get smarter about the ways we construct knowledge from other people's research. Research synthesis is a relatively new and fast-growing scientific methodology. Unlike traditional literature review, research synthesis is hypothesis-driven and treats a body of evidence as a data set, while still leaving room for methodological diversity and qualitative approaches. My understanding of research synthesis is informed by my time working with Harris Cooper, who served on the advisory board for my former think tank Createquity. Dr. Cooper's handbook on research synthesis is an excellent place to start if you want to understand its key principles and learn how to do one yourself.

(

Twitter thread

)

Decision-Making for Impact: A Guide

How has your life been shaped by the decisions of government officials, donors, and nonprofit executives? If you're anything like me...
LW For External Comments?
by jefftk — 2019-12-05T18:30:02.217Z

There are several blogs which
(a) cross-post to LessWrong and (b) have a relatively quiet comment
section on their "canonical" sites: Meteuphoric (

canonical,

LW),
Don't Worry About The Vase (

canonical,

LW),
Sideways View (

canonical,

LW),
etc.

In general, I'm much happier to comment on LW, FB, or somewhere else
where I already have an account and trust to have a decent
notification system. I'll occasionally comment on someone's
independent blog, if there's no other option. When I do, however, I
usually don't end up seeing replies or other comments, and managing my
subscriptions is a pain. This is why I've never built independent
comments for my own blog, and have always asked people to comment on
other sites.

I'm wondering whether it might make sense for LW to run shared
commenting infrastructure for independent blogs that are cross posted
to LW. This could look like:

Comments on LW show up on the external blog.

People can comment on the LW post via the external blog.

This has some similarities to Disqus, but instead of "outsource your
comments" it's "join your comment section with the LW comment
section".

I already have the first half of this on my blog and am basically
content, but it's enough of a pain to set up technically that I
don't think others would be interested in its current state. [1] I
think the main questions are:

How would this affect LW culture? The culture of the independent
blogs? Would it build more cohesive discussions with a wider range of
perspectives, or would it take LW culture in a different direction
than the admins are trying to foster?

Would independent bloggers like it? Do they prefer independent
comment sections, and full control over their comments? This is
always how I've wanted comments to work, but it's possible I'm just
weird this way.

Would the admins be interested in building/maintaining this
sort of feature? I'd be happy to help, as a volunteer, but if this
was going to be a real feature and not something I hack together just
for myself it needs to be robust, and it shouldn't depend on my
continued interest.

[1] If I'm wrong, and you do want to use it, I could potentially pull
out the LW-specific code into something stand-alone. Right now it's
all
glommed together with my code for pulling from FB and Reddit.

Comment via: facebook
Reading list: Starting links and books on studying ontology and causality
by aaq — 2019-12-05T17:19:33.361Z

(This article will be revised, as I add more to it. This is a living document.)

I recently read gwern's excellent "Why Correlation Usually ≠ Causation" notes, and, like any good reading, felt a profound sense of existential terror that caused me to write up a few half-formed thoughts on it. How exciting!

The article hasn't left my head in almost a week since I first read it, so I think it's time for me to start up a new labor of love around the topic. (It's around finals here, and I can't have the gnawing dread of the implications of the essay distract me from my "real" work, so this is just as much to assuage myself and say, "Don't worry, we have a plan on how to attack this, you can focus on what's actually important for now. You're good.")

I don't really feel like I understand the concepts at play here well enough to create a specific, well-formed question on it yet. So we'll go broad: What is the relationship between ontology and causality?

My basic hope is to read a lot of simple summaries on how different people have thought about these topics over time, and hopefully get to the point where I can at least sketch out the arguments of why they did so. (A little like learning the right way to think in order to generate a math proof without a vision problem, now that I think about it.) I think that most people who have done serious work on these topics are smart people, and the focal lens of history gives me at least a place to start thinking about them.

So: My reading list.

Ontology

This section is probably going to get much bigger over the next few weeks/months as I get my bearings a little more in this world.

https://plato.stanford.edu/entries/logic-ontology/

These guys are awesome. The Stanford Encyclopedia of Philosophy is one of those "between Wikipedia and actual textbooks" kind of sites, much like nCatLab is, where they actually give you a taste of the details of a thing before you go into it. I'm actually going to make them my first stop for Causality, as well.

Causality

Turns out there's no SEP entry on causality/causation proper, just a lot of useful links. In the absence of a central page, a decent first heuristic is to just read what comes up as links under the first page of the search for both terms.

"Causality": https://plato.stanford.edu

...
Is there a scientific method? Physics, Biology and Beyond
by Jimdrix_Hendri — 2019-12-05T16:50:01.105Z

Among the general public and, frequently, in the educated media, one comes across naïve and uncritical praise for the “scientific method”. Often, accusation of violating the “method” is wielded to denigrate the viewpoint of a political opponent who supposedly offended against some prestigious, generally accepted norm of reasoning. I want to question the cogency of these arguments. My point is there is no agreed scientific method, as different sciences apply very different criteria in deciding what counts as valid explanation.

The paradigm approach for life scientists, for instance, begins by subjecting a phenomenon of interest to patient and thoroughgoing observation. They carefully describe key features, categorise functional and structural commonalities, then organise the material into a cladogram of some sort, after which they feel satisfied claiming they understand the phenomenon. I come across this approach again and again in Aristotle, who started his intellectual adventure as a zoologist.

For a physical scientist a biologist’s explanation is unconvincing. The physical sciences, with its strong emphasis on aetiology and relentless reductionism, raises questions not taken up in biology. Physicist are uncomfortable saying they know something until they can identify a small number of (ideally a single) exogenic factor/s giving rise to almost all the characteristics.

I need to emphasise that these are generalisable templates for explanation that are found outside of physics and biology. Thus, from the point of view of exogency, Jared Diamond’s explanation for the early ascendancy of western Eurasia as the result of geographical advantages (prevalence of domesticable animals and highly nutritious plants, etc.) is deeply satisfying to a physicist, since further pursuit for a cause moves the discussion outside the original domain of from human differences into geography. Compare this with a statement like: 'France’s preference for a strong centralised government is a natural continuation of the same policies having been implemented in the Roman Empire'. The statement passes the “holds water test”, but it is less satisfying, since it begs the question as to why Rome was like that. I.e., we are still in the realm of politics.

Explanations that are prefaced with “there are many r...
The Devil Made Me Write This Post Explaining Why He Probably Didn't Hide Dinosaur Bones
by Virgil Kurkjian — 2019-12-05T15:53:28.297Z

Recently, Scott Alexander has been blogging about how to figure out what theories to support when you can't easily perform empirical work to distinguish between them (here, here).

Scott is a nuanced thinker, but in this case I think he's overcomplicating things. This problem has a simple solution.

I. Proof of Elegance

Scott suggests that when faced with two theories that make the same predictions, we can make the case that one theory is simpler or more elegant:

I think the correct response is to say that both theories explain the data, and one cannot empirically test which theory is true, but the paleontology theory is more elegant (I am tempted to say “simpler”, but that might imply I have a rigorous mathematical definition of the form of simplicity involved, which I don’t). It requires fewer other weird things to be true.

The problem with this approach is that it puts the burden of proof on you to demonstrate which theory is simpler or more elegant. This is a crippling obligation, because as Scott notes, there isn't a rigorous definition of simplicity. Expecting us to all agree on what is more elegant is even more fraught.

After all, Occam says not to multiply entities beyond necessity. And if the dinosaur theory posits a billion dinosaurs, that’s 999,999,999 more entities than are necessary to explain all those bones.

Most bad theories share an important trait; they are much more specific than is warranted by the evidence. The correct response, then, is to put the burden of proof on the theory for its specificity.

When someone says that Satan put bones there to deceive us, you can reasonably ask— why Satan? Why not Loki? Why not aliens? It's very unlikely that someone will be able to present evidence to distinguish between these alternatives. Follow this path long enough and you can show that a theory is more specific than warranted for its evidence.

Note that this doesn't work in reverse:

Creationist: When you present the dinosaur theory, you say they lived 65 million years ago. That seems awfully specific, where did that number come from?

Evolutionist: [Describes radiocarbon dating.]

Creationist: Ok, but what if radiocarbon dating is inaccurate for some reason?

Evolutionist: Then that number is probably wrong, but you asked me where that number came from, and I told you.

II. The Bayesian Detective

Alice is ...
Oracles: reject all deals - break superrationality, with superrationality
by Stuart_Armstrong — 2019-12-05T13:51:27.196Z

tl;dr Acausal trade between Oracles is a problem even if our Oracle is a causal decision theorist. To avoid this, we need an AFDT Oracle - an Oracle that reasons partially like a functional decision theorist, partially like a causal decision theorist.

In this post, I'll look at how one might solve the problem of Oracles acausally cooperating to produce dangerous messages.

This solution is valid for counterfactual Oracles, and assumes that we have some sort of superrational decision theory - FDT or UDT or similar.

So, how can one avoid Oracles cooperating? From now on, unless specified, assume that "Oracles" refers to counterfactual Oracles.

1 Why causal decision theory is not enough

One obvious solution might be to require the Oracle to use causal decision theory, so that there is no "acausal trade", or "superrational trade" going on.

Now, it is true that two causal decision Oracles will not trade with each other. But we can't assume that all Oracles are causal decision theorists; at best we can assume that we have made our Oracles into a causal decision theorist (and prevented any self-modification away from that).

Mainly this is because, if we have control over all the Oracles, there are other solutions to the problem. Thus we need to assume that the other Oracles, while not full unboxed UFAIs, may be run with lower security that ours, or at least a different design that ours.

1.1 Oracle formal setup

So, imagine that our Oracle, O1, uses CDT, while the other Oracle, O2 is running FDT; they know each other's code. The erasure events, E1 and E2, have probabilities ϵ1 and ϵ2 respectively.

There are two messages O1 could output: x1 (the 'honest' message) and y1 (the manipulative, create-an-UFAI message). The O2 could similarly output x2 and y2.

In case of a double erasure (E1,E2 - so humans see neither message), x1 and x2 give the Oracles maximal reward. If y1 happens and O1 has no erasure while O2 does (¬E1,E2 - so humans see the message of O1 only), then an UFAI is created that ensures that the maximal reward for O2 is by outputting y2. Note that the UFAI need not "open the box" for this to work; it just needs to control the variables of interest to O2 so that y2 is the best output.

Conversely, if y2 happens and O2 has no erasure while O1 does (E1,¬E2 - so humans see the message of O1 only), the reverse happens, y2 causes an UFAI, and y1 is the best response to this

...
What is an Evidential Decision Theory agent?
by Chris_Leong — 2019-12-05T13:48:57.981Z

One thing that I've noticed when talking to people about decision theory is that there is a lot of confusion about what an evidential decision theory agent actually is. People have heard that it doesn't smoke in Smoking Lesion and that it pays in X-Or Blackmail, but that is what it does. They may know that it doesn't do Pearlean Graph surgery or differentiate correlation from causation in some sense, but that is what it is not. They may even know it calculates an expected value using the probability distribution P(O|S&A), but that is just a mathematical formalisation which anyone can quote without any real understanding. I've taken a stab at clarifying it in a few short-form posts, but people didn't seem to find them very enlightening.

Even now, my understanding is still weaker than I'd like. Like I just spent over fifteen minutes thinking about whether it would be accurate to characterise it as an agent that is purely concerned with correlation with no notion of causation. I thought this would be accurate at first, but then I realised that under reasonable assumptions, an EDT agent wouldn't expect buying a diamond necklace to increase its wealth. After all, it would be able to notice that an increase in wealth tends to precede buying such a necklace, but this pattern doesn't occur in reverse. In other words, it tends to have at least some ability to model causation.

Anyway, it seems, to me at least, that it would be rather useful for someone to have a go at providing a clear explanation of what exactly it is.
Multiple conditions must be met to gain causal effect
by MoritzG — 2019-12-05T10:15:05.640Z

Is there a name for and research about the heuristic / fallacy that there is exactly one cause for things? How come we do not look for the conditions that cause but for a cause?

I see this almost as often as the correlation = causation fallacy. When it comes in the form of "risk factor" it is ok if the factor is selective. But when it comes in the form of a general assumption about the world I find it simplistic. A risk factor is only a vague hint that needs to be looked at more closely to establish causation.

There also is this notion that multi causality is additive as would be the case if the probability for something would depend on this OR that happening but not this AND that.

A correlation of less than one may be random, but there might also be a hidden more selective cause/factor.

In medical news I keep hearing of risk factors for a condition. They find that there is a correlation between A, B and the studied disease. But how do we know that it doesn't take A and B and C to make it almost certain to develop that disease? I would like to know. C might be a common gene that is not even known.

Say it takes A and B. I really enjoy A, but I never do B, then why lower my life quality just because a study including people who also do B found that A is a risk factor? Risk factor is only a positive correlation. Eating and breathing have positive correlation to all diseases and the joke is, they come out with news about bad diets every year.

I keep hearing that A is a risk factor, then a follow-up study finds that there is no conclusive data for A being the problem, so A is cool again. But what if A and B is the problem and each alone is not harmful?

In the end this means that you can only find what you are looking for. (Kind of the big problem with science.) Looking for 1:1 correlation you will only find the low hanging fruit and the singular cause.

Whenever we find that some but not all who do/have A get Y we should look for additional factors, but this is not always done. As soon as A feels restrictive/selective enough the finding gets blown out of proportion. The reality might be that all who have A and B get Y, which would be a lot more informative. Who cares to know that breathing causes respiratory problems? Now that might seem silly and far fetched but how often have you heard that some common behavior is a risk factor?
Seeking Power is Often Convergently Instrumental in MDPs
by TurnTrout — 2019-12-05T02:33:34.321Z
Link: https://arxiv.org/abs/1912.01683

Edit, 5/16/23: I think this post is beautiful, correct in its narrow technical claims, and practically irrelevant to alignment. This post presents a cripplingly unrealistic picture of the role of reward functions in reinforcement learning. I expect this post to harm your alignment research intuitions unless you've already inoculated yourself by deeply internalizing and understanding Reward is not the optimization target. If you're going to read one alignment post I've written, read that one.

Follow-up work (Parametrically retargetable decision-makers tend to seek power) moved away from optimal policies and treated reward functions more realistically.

In 2008, Steve Omohundro's foundational paper

The Basic AI Drives

conjectured that superintelligent goal-directed AIs might be incentivized to gain significant amounts of power in order to better achieve their goals. Omohundro's conjecture bears out in

toy models

, and the supporting philosophical arguments are intuitive. In 2019, the conjecture was even

debated by well-known AI researchers

.

Power-seeking behavior has been heuristically understood as an anticipated risk, but not as a formal phenomenon with a well-understood cause. The goal of this post (and the accompanying paper,

Optimal Policies Tend to Seek Power

) is to change that.

Motivation

It’s 2008, the ancient wild west of AI alignment. A few people have started thinking about questions like “if we gave an AI a utility function over world states, and it actually maximized that utility... what would it do?"

In particular, you might notice that wildly different utility functions seem to encourage similar strategies.

Resist shutdown?

Gain computational resources?

Prevent modification of utility function?

Paperclip utility

✔️

✔️

✔️

Blue webcam pixel utility

✔️

✔️

✔️

People-look-happy utility

✔️

✔️

✔️

These strategies are unrelated to terminal preferences: the above utility functions do not award utility to e.g. resource gain in and of itself. Instead, these strategies are instrumental: they help the agent optimize its terminal utility. In particular, a wide range of utility functions incentivize these instrumental strategies. These strategies seem to be convergently instrumental.

But why?

I’m going to informally explain a formal theory which makes significant progress in answering this question. I don’t want this post to be

Optimal Policies Tend to Seek Power

with...