trentmkelly/LessWrong-43k huggingface dataset:
Goal-directedness: exploring explanations
by Morgan_Rogers — 2022-02-14T16:20:31.570Z

This is the first post in my Effective-Altruism-funded project aiming to deconfuse goal-directedness. Comments are welcomed. All opinions expressed are my own, and do not reflect the attitudes of any member of the body sponsoring me.

In my preliminary post, I described my basic intuitions about goal-directedness, and focussed on explainability. Concisely, my initial, informal working definition of goal-directedness is that an agent's behaviour is goal-directed to the extent that it is better explained by the hypothesis that the agent is working towards a goal than by other types of explanation.

In this post I'm going to pick away at the most visible idea in this formulation: the concept of an explanation (or at least the aspect of it which is amenable to formalization with minimal effort), and especially the criteria by which an explanation is judged. More precisely, this post is a winding path through some mathematical ideas that could be applied to quantitatively judge explanations; this collection of ideas shall be examined more closely in subsequent posts once some desirata are established. By the end I'll have a naïve first picture of how goal-directedness might be measured in terms of goal-based explanations, while having picked out some tools for measuring explanations which goal-directedness can be contrasted with.

What constitutes a good explanation?

In science (and rational discourse more broadly), explanations are judged empirically: they are used to generate predictions, and these are compared with the results of existing or subsequent observations. The transition from explanations to predictions will be covered in the next section. Here we'll break down the criteria for comparison.

The most obvious way an explanation can fail is if it predicts phenomena which are not observed, or conversely if it fails to predict phenomena which are observed. We can present this criterion as:

Accuracy. The more accurately the observations match the predictions, the better the explanation is considered to be.

That is not the end of the story. For instance, more value is attributed to explanations which produce a greater variety of predictions for future observations. An exhaustive account of existing observations with no extra consequences barely qualifies as an explanation, whereas an explanation that appears to reveal the fundamental nature of the object(s) of scienti...
Paper is True
by Zvi — 2022-02-14T14:10:03.302Z

Postscript to: Rock is Strong

Always choosing Rock, broadly construed, is freerolling on the efforts of others and/or you are lying. You are in at some important sense defecting. The problem of an information cascade is real. You are not contributing new true information, instead reinforcing the information and heuristics already out there.

Often this is the most accurate option available to you, or at least the most efficient. Doing the work yourself is hard. Many can’t do better that way and even for those who can it is a lot of work. On average those who disagree are more wrong and end up doing worse. Other times, accuracy is not what is desired. None of that makes it easy to pick the correct rock, but often it is straightforward enough.

The defection that comes from not doing your part for the long term epistemic commons often is made up for not only by your lack of a practical option to do otherwise, but also in some cases by the gains from accuracy.

Rewarding those who worship the rock, like rewarding any form of defection, is also defecting. The Queen causes everyone to die in a volcanic eruption because she set up bad incentives and is encouraging everyone to spend down the commons.

This can also be thought of partially as explore versus exploit. The majority of actions should be exploit, and most people should mostly be exploiting. There is more variance in ability to explore than exploit, so most people should explore even less than average and let others explore and report back. Exploration is a public good, so it needs to be rewarded beyond its private payoff or there won’t be enough of it. When competition is sufficiently intense or other factors reward exploitation too much, exploration can die out entirely.

This is the context for a Marginal Revolution post that nagged at me enough to give me enough initial motivation to write this whole thing. Quoted in full, and this is not about the examples it is about the method:

No, you don’t always have to agree with the majority of the educated people, but I would say this. For whatever set of views you think is justified, try to stick to the versions of those views held by well-educated, reasonable, analytically-inclined people. You will end up smarter over time, and in better places. Peer effects are strong, including across your ideological partners.

When I hear that a particular group defend

...
Rock is Strong
by Zvi — 2022-02-14T14:00:02.399Z

Everybody wants a rock. It’s easy to see why. If all you want is an almost always right answer, there are places where they almost always work.

The Security Guard

He works in a very boring building. It basically never gets robbed. He sits in his security guard booth doing the crossword. Every so often, there’s a noise, and he checks to see if it’s robbers, or just the wind.

It’s the wind. It is always the wind. It’s never robbers. Nobody wants to rob the Pillow Mart in Topeka, Ohio. If a building on average gets robbed once every decade or two, he might go his entire career without ever encountering a real robber.

At some point, he develops a useful heuristic: it he hears a noise, he might as well ignore it and keep on crossing words: it’s just the wind, bro.

This heuristic is right 99.9% of the time, which is pretty good as heuristics go. It saves him a lot of trouble.

The only problem is: he now provides literally no value. He’s excluded by fiat the possibility of ever being useful in any way. He could be losslessly replaced by a rock with the words “THERE ARE NO ROBBERS” on it.

That last line is making a bunch of hidden assumptions, in addition to the non-hidden assumption that there are (almost) never any robbers. All of them boil down to ‘the purpose of the security guard is to know whether there is a robbery and then act appropriately depending on the answer’ where the act if a robbery is detected could be ‘call 911’ and/or something more actively.

That is one good reason to hire a security guard. Yet something tells me that wasn’t the best description of why this man is doing crossword puzzles in the Pillow Mart in Topeka, Ohio.

My guess is that the security guard’s primary purpose is to make sure everyone knows that there exists a security guard. This has many nice properties, such as these Five Good Reasons.

Potential robbers know there is a security guard.
Employees know there is a security guard.
Insurance company knows there is a security guard.
In case of robbery you can say you had a security guard.
Anyone who tries to walk in faces mild social awkwardness.

When you have a security guard you get to tell the story that you have a security guard. Without one, you can’t tell that story. Dead rocks tell no tales, and when they try (such as writing “THIS IS A SECURITY GUARD” with or without adding “ALSO THERE ARE NO ROBBERS”) it is not very eff...
Some limitations of reductionism about epistemology
by Richard_Ngo — 2022-02-14T08:50:02.419Z

This post is largely based on a lightning talk I gave at a Genesis event on metacognition, with some editing to clarify and expand on the arguments.

Reductionism is the strategy of breaking things down into smaller pieces, then trying to understand those smaller pieces and how they fit together into larger pieces. It’s been an excellent strategy for physics, for most of science, for most of human knowledge. But my claim is that, when the thing we're trying to understand is how to think, being overly reductionist has often led people astray, particularly in academic epistemology.

I’ll give three examples of what goes wrong when you try to be reductionist about epistemology. Firstly, we often think of knowledge in terms of sentences or propositions with definite truth-values - for example, “my car is parked on the street outside”. Philosophers have debated extensively what it means to know that such a claim is true; I think the best answer is the bayesian one, where we assign credences to propositions based on our evidence. Let’s say I have 90% credence that my car is parked on the street outside, based on leaving it there earlier - and let’s assume it is in fact still there. Then whether we count this as “knowledge” or not is mainly a question about what threshold we should use for the definition of “knows” (one which will probably change significantly depending on the context).

But although bayesianism makes the notion of knowledge less binary, it still relies too much on a binary notion of truth and falsehood. To elaborate, let’s focus on philosophy of science for a bit. Could someone give me a probability estimate that Darwin’s theory of evolution is true? [Audience answer: 97%] Okay, but what if I told you that Darwin didn’t know anything about genetics, or the actual mechanisms by which traits are passed down? So I think that 97% points in the right direction, but I think it’s less that the theory has a 97% chance of being totally true, and more like a 97% chance of being something like 97% true. If you break down all the things Darwin said into a list of propositions: animals inherit from their parents, and 100 different things - almost certainly at least one of these is false. That doesn’t change the fact that overall, the theory is very close to true (even though we really have no idea how to measure or quantify that closeness).

I don’t think this i...
Strevens on scientific explanation
by Richard_Ngo — 2022-02-14T08:10:01.421Z

This post discusses two books by Michael Strevens: The Knowledge Machine: How Irrationality Created Modern Science and Thinking Off Your Feet: How Empirical Psychology Vindicates Armchair Philosophy. I loved the former, which tries to answer the core question in philosophy of science: why does science work so well? It’s a masterful synthesis of previous ideas and important new arguments, and it’ll be my go-to recommendation from now on for people interested in the field. The latter was… slightly less persuasive. But let’s start with the good stuff.

The Knowledge Machine begins with a review of two of the key figures in philosophy of science: Popper and Quine. Historically, philosophers of science focused on identifying a “scientific method”: a specific way of generating theories, designing experiments, and evaluating evidence which, when followed, led scientists to the truth. Popper’s influential account of the scientific method focused on scientists trying to refute their hypotheses. He claimed that only severe tests which attempt to falsify a hypothesis can give us reason to provisionally accept it. Along with other early philosophers of science, Popper’s work promoted (according to Strevens) “an ideal of the scientist as a paragon of intellectual honesty, standing up for truth in the face of stifling opposition from the prevailing politics, culture, and ideology”.

However, over the last half-century or so a range of criticisms of this flattering view of science have emerged. Most prominent is Kuhn, who in his book The Structure of Scientific Revolutions characterises scientists as constrained within a specific paradigm of thinking, unable to rationally decide between different paradigms. Soon afterwards, in his book The Sleepwalkers, Koestler described the emergence of early science not as the triumph of a superior method, but rather as “a history of collective obsessions and controlled schizophrenias”. More recently, Feyerabend’s book Against Method espoused a position he called “epistemological anarchism”, arguing that “anything goes” in the pursuit of truth.

Strevens focuses on Kuhn, but his arguments are in line with the positions of the others. He summarises a range of case studies of scientists ignoring inconvenient data, deciding questions via political maneuvering, and generally behaving in ways no “scientific method” would endorse. These case studie

...
Memory decay in human brains from options pricing theory
by Jalex S — 2022-02-14T07:52:46.343Z

ACT-R theory predicts that the "strength" of declarative memory in human brains decays as ∑i1√ti, where ti is the amount of time since the ith exposure to the fact. The power law here seems strange – surely things should either decay discretely or exponentially. My first reaction when I heard this was to think very confusedly about reaction rates between neurotransmitters, and my second reaction was to look for the empirical evidence.[1] As it turns out, the power law relationship was empirically established in the late 90s. [2] Ready for another shock? This matches closely with an empirical observation about information decay in financial markets! Our goal in this post is to give a theoretical explanation of this coincidence.

The motivating idea behind our argument is that impact decay is caused by the market literally forgetting what happened. This is cute but naive[3] – we shouldn't hope to use neuroscience to explain something essentially mathematical. Instead, we'll leverage this analogy to find a semi-rigorous derivation in the appropriate direction.

Epistemic status:

the ideas are probably not new, but I haven't seen them collected in one place before. There's some math, and it's a bit sketchy.

This post grew out of conversations with LW user Kirby Sikes, which in turn were inspired by this comment of Terry Stewart. We'll take a somewhat meandering path to the main result, walking the same path we used to discover it.

Thanks again to Kirby Sikes for helpful comments on a draft.

Conventions

We emphasize phrases in bold and define phrases in italics.

Refer back here if you lose track of notation: T is the "final" time, t is an intermediate time, and t0 is the "initial" time. N is a number of shares or contracts. stdev is short for standard deviation. K is a "strike price" and e is an "edge", both in units of dollars. X is a random variable. u is a unit vector.

We're forced to lean on some knowledge of probability theory. There's a tiny bit of calculus – it can be safely black-boxed if you don't understand it.

Background: options pricing

An options contract gives the owner the right (but not the obligation) to buy[4] a unit of stock at a specific strike price on a specific expiration date. It's called an "option" beca...
Book review: The Age of Surveillance Capitalism
by Richard_Ngo — 2022-02-14T07:50:01.273Z

I recently finished Shoshana Zuboff’s book The Age of Surveillance Capitalism. It’s received glowing reviews, but left me disappointed. Zuboff spends much of the book outraged at the behaviour of big tech corporations, but often neglects to explain what’s actually bad about either the behaviour itself or the outcomes she warns it’ll lead to. The result is far more polemical than persuasive. I do believe that there are significant problems with the technology industry - but mostly different problems from the ones she focuses on. And she neglects to account for the benefits of technology, or explain how we should weigh them against the harms.

Her argument proceeds in three stages, which I’ll address in turn:

Companies like Google and Facebook have an “extraction imperative” to continually “expropriate” more personal data about their users.
They use this for “the instrumentation and instrumentalisation of behaviour for the purposes of modification, prediction, monetisation, and control.”
Ultimately, this will lead to “a form of tyranny” comparable to (but quite different from) totalitarianism, which Zuboff calls instrumentarianism.

On data: I agree that big companies collect a lot of data about their users. That’s a well-known fact. In return, those users get access to a wide variety of high-quality software for free. I, for one, would pay thousands of dollars if necessary to continue using the digital products that are currently free because they’re funded by advertising. So what makes the collection of my data “extraction”, or “appropriation”, as opposed to a fair exchange? Why does it “abandon long-standing organic reciprocities with people”? It’s hard to say. Here’s Zuboff’s explanation:

Industrial capitalism transformed nature’s raw materials into commodities, and surveillance capitalism lays its claims to the stuff of human nature for a new commodity invention. Now it is human nature that is scraped, torn, and taken for another century’s market project. It is obscene to suppose that this harm can be reduced to the obvious fact that users receive no fee for the raw material they supply. That critique is a feat of misdirection that would use a pricing mechanism to institutionalise and therefore legitimate the extraction of human behaviour for manufacturing and sale. It ignores the key point that the essence of the exploitation here is the rendering of our lives a

...
Are there any preventive steps someone can take after being exposed to strep throat?
by Maxwell Peterson — 2022-02-14T03:01:53.625Z

Googling this question tells me that the best way to avoid strep is by washing your hands often. But say, hypothetically, that someone’s girlfriend has told him that she has strep the day before Valentine’s Day. Knowing he’ll be exposed, are there any steps he should take to reduce the chance and severity of infection? (His hypothetical tonsils were removed long ago). Since this person is hypothetical, advice to avoid his girlfriend on Valentine’s Day will not reach him.

For example, for COVID, there are reasons to think that taking Zinc lozenges (https://www.lesswrong.com/posts/5DKqK3hEzzBoGF47C/consider-taking-zinc-every-time-you-travel) and vitamin D (https://astralcodexten.substack.com/p/covidvitamin-d-much-more-than-you ) might help. Strep throat is not a respiratory virus, but rather a bacterial infection, so does the same advice apply? Or are there other things that seem promising to try? Antibiotics are a natural thought, but is taking antibiotics before any sign of infection reasonable? And if so, would a doctor even prescribe preventatively?
Some thoughts on vegetarianism and veganism
by Richard_Ngo — 2022-02-14T02:40:01.615Z

I feel pretty confused about whether I, as an effective altruist, should be vegetarian/vegan (henceforth abbreviated veg*n). I don’t think I’ve seen anyone explicitly talk about the arguments which feel most compelling to me, so I thought I’d do that here, in a low-effort way.

I think that factory farming is one of the worst ongoing moral atrocities. But most of the arguments I’ve heard for veg*nism, which I found compelling a few years ago, hinge on the effects that my personal consumption would have on decreasing factory farming (and sometimes on climate change). I now don’t find this line of thinking persuasive - my personal consumption decisions just have such a tiny effect compared to my career/donation decisions that it feels like I shouldn’t pay much attention to their direct consequences (beyond possibly donating to offset them).

But there are three other arguments which seem more compelling. First is a deontological argument: if you think something is a moral atrocity, you shouldn’t participate in it, even if you offset the effects of your contribution. In general, my utilitarian intuitions are much stronger than my deontological ones, but I do think that following deontological principles is often a very good heuristic for behaving morally. The underlying reason is that humans by default think more naturally in terms of black-and-white categories than shades of grey. As Yudkowsky writes:

Any rule that's not labeled "absolute, no exceptions" lacks weight in people's minds. So you have to perform that the "Don't kill" commandment is absolute and exceptionless (even though it totally isn't), because that's what it takes to get people to even hesitate. To stay their hands at least until the weight of duty is crushing them down. A rule that isn't even absolute? People just disregard that whenever.

Without strong rules in place it’s easy to reason our way into all sorts of behaviour. In particular, it’s easy to understimate the actual level of harm that certain actions cause - e.g. thinking of the direct effects of eating meat but ignoring the effects of normalising eating meat, or normalising “not making personal sacrifices on the basis of moral arguments”, or things like that. And so implementing rules like “never participate in moral atrocities” sends a much more compelling signal than “only participate in moral atrocities when you think that’s net-positi...
Some ideas for interacting with reporters
by mike_hawke — 2022-02-14T00:38:42.740Z

[Edit: This written mostly with recent examples of bad journalism in mind (including the NYT/SSC affair). Not intended for less fraught situations.]

Refuse to grant them an interview unless they agree to prominently include a link to your own side of the story.

Consider contacting competing reporters and offering them the same deal. Tell them that you are shopping around.
Get them to agree to it in writing, and require them to display the language of the agreement somewhere on the page of the final article.

Find a polite way to say, "nothing personal, but your profession carries a reputation for untrustworthiness." Nah, the commenters changed my mind about this. It would be better to just say, "no comment."
Screenshot everything they say.
Consider getting into the habit of giving glomar responses.
Build a reputation (or advertise an official policy) for waiting at least X days before responding to requests for comment.
Build a reputation for being a respectable, sober expert. Then, when a journalist wants to collapse your forecast into a point estimate, go ahead and give them such a point estimate...but make it absurd enough that the smart & conscientious readers might feel compelled to do a fact check.

"Top Forecasting Team Says World Population in 2050 Will be Only Six Thousand!"
"Human-Level AI in 3100-3200 Years, No Sooner, No Later!"
"Comfortable Martian Colony on June 3rd, 2079!"
"The Next Pandemic Will Occur on My Mom’s 88th birthday, According to Leading Biosecurity Researcher"