trentmkelly/LessWrong-43k huggingface dataset:
Conditional prediction markets are evidential, not causal
by philh — 2024-02-07T21:52:47.476Z

Quick note about a thing I didn't properly realize until recently. I don't know how important it is in practice.

tl;dr: Conditional prediction markets tell you "in worlds where thing happens, does other-thing happen?" They don't tell you "if I make thing happen, will other-thing happen?"

Suppose you have a conditional prediction market like: "if Biden passes the DRESS-WELL act, will at least 100,000 Americans buy a pair of Crocs in 2025?" Let's say it's at 10%, and assume it's well calibrated (ignoring problems of liquidity and time value of money and so on).

Let's even say we have a pair of them: "if Biden doesn't pass the DRESS-WELL act, will at least 100,000 Americans buy a pair of Crocs in 2025?" This is at 5%.

This means that worlds where Biden passes the DRESS-WELL act have a 5pp higher probability of the many-Crocs event than worlds where he doesn't. (That's 5 percentage points, which in this case is a 100% higher probability. I wish we had a symbol for percentage points.)

It does not mean that Biden passing the DRESS-WELL act will increase the probability of the many-Crocs event by 5pp.

I think that the usual notation is: prediction markets tell us

P(many-Crocs|DRESS-WELL)=10%

but they don't tell us

P(many-Crocs|do(DRESS-WELL))=?%

One possibility is that "Biden passing the DRESS-WELL act" might be correlated with the event, but not causally upstream of it. Maybe the act has no impact at all; but he'll only pass it if we get early signs that Crocs sales are booming. That suggests a causal model

early-sales→many-Crocs→DRESS-WELL←early-sales

with

P(many-Crocs|do(DRESS-WELL))=P(many-Crocs)

(I don't know if I'm using causal diagrams right. Also, those two "early-sales"es are meant to be the same thing but I don't know how to draw that.)

But here's the thing that triggered me to write this post. We can still get the same problem if the intervention is upstream of the event. Perhaps Biden will pass the DRESS-WELL act if he thinks it will have a large effect, and not otherwise. Let's say the act has a 50% chance of increasing the probability by 3pp and a 50% chance of increasing it by 5pp. Biden can commission a study to find out which it is, and he'll only pass the act if it's 5pp. Then we have

size-of-impact→many-Crocs←DRESS-WELL←size-of-impactP(many-Crocs|do(DRESS-WELL))=9%

I expect that sometimes you want to know the thing that prediction markets tell you, and some...
A Back-Of-The-Envelope Calculation On How Unlikely The Circumstantial Evidence Around Covid-19 Is
by Roko — 2024-02-07T21:49:46.331Z

Follow-up to: https://www.lesswrong.com/posts/Rof3ctjMMWxpaRj5Z/the-math-of-suspicious-coincidences

This post will be a very preliminary model of the circumstantial evidence around covid-19's origin and how unlikely we are to actually see worlds like this under the Grand Null Hypothesis that there has been no foul play and covid-19 is a completely natural virus that just happened to appear.

What do we actually need to explain?

Coincidence of Location: Wuhan is a particularly special place in China for studying covid-19; the WIV group was both the most important, most highly-cited group before 2020, and the only group that was doing GoF on bat sarbecoronaviruses as far as I know. Wuhan is about 0.5% of China's population. It's a suspicious coincidence that a viral pandemic would occur in the same city as the most prominent group that studies it.
Coincidence of timing: several things happened that presaged the emergence of covid-19. In December 2017, the US government lifted a ban on risky pathogen research, and in mid-2018 the Ecohealth group started planning how to make covid in the DEFUSE proposal. A natural spillover event could have happened at any time over either the last, say, 40 years or (probably) the next 40 years, though likely not much before that due to changing patterns of movement (I need help on exactly how wide this time interval is).
Warnings turning out to be accurate: Warnings were given in Nature specifically mentioning the WIV/Zhengli Shi group and no other group involved with coronaviruses, and only a few other groups involved with any viruses at all (in other articles). There were hundreds of groups that could have been warned about I think, but this article gives 59 as the number of BSL-4 labs around the world. This is a subtler point than those above because getting a warning is extra evidence for the lab leak hypothesis even conditional on the timing and location coincidence. Warnings were also given about WIV itself independent of the connection to coronaviruses too.
Specific Features of covid-19 are a close match for what was planned in the DEFUSE proposal: This gets a lot more technical, but you can imagine a world in which labs randomly generate GoF proposals like DEFUSE and nature randomly generates viruses via natural evolution. Even in cases where you get a location coincidence as in (1), the ave

...
Nitric oxide for covid and other viral infections
by Elizabeth — 2024-02-07T21:30:03.774Z

Epistemic status: I spent about 5 hours looking into this, and the next day developed covid myself. I did a bit more research plus all of the writing while sick. So in addition to my normal warning that I have no medical credentials, you should keep in mind that this knowledge may be cursed.

ETA 4-30-24: In this post I used “nitric oxide spray” and “enovid” as synonyms. I’ve since learned this is incorrect, NO is one of several mechanisms Enovid uses. The other mechanisms weren’t mentioned in the papers I cite so it’s possible these are accurate for NO alone.

Introduction

Nitric Oxide Nasal Spray, sold under the brand name Enovid, is a reactive compound that kills viruses (and I suspect taxes your nasal tissue). It has recently been tested and marketed for treatment of covid. The protocol I found in papers was 2 sprays per nostril every 2-3 hours, after you develop symptoms. Enovid’s instructional pamphlets say twice per day, also after you get sick. This seems a little late to me.

I suspect the real power of NONS lies in use before you develop symptoms, ideally as close to exposure as possible. This is difficult because you don’t know when you would have gotten sick, and I suspect there are costs to indefinite use. I initially thought (and told people, as a tentative guess) that one round of 4 total sprays after a high risk event was a good trade off. After doing the math for this post, that intervention seems much less helpful to me, and picking the right length of post-exposure prophylaxis depends on equations for which we lack good numbers. I pulled some numbers out of my ass for this post, but you should not trust them.

My guess is NONS is minimally useful once covid has reached the throat, unless you combine it with a separate disinfectant of the throat. I hope to write up a report on one such disinfectant soon, although TBH it’s not looking good.

NONS can lead to false negatives on any test based on a nasal swab, because it breaks the relationship between nasal viral load and overall load.

How does it work?

First, nitric oxide is highly reactive, which makes it destructive to anything organic. Virions are fragile to this kind of direct attack, and certain immune cells will produce nitric oxide to kill bacteria, viruses, and your own diseased cells.

First-and-a-half, nitric oxide may alter the pH of your nose, and this ef...
Debating with More Persuasive LLMs Leads to More Truthful Answers
by Akbir Khan — 2024-02-07T21:28:10.694Z
Link: https://arxiv.org/abs/2402.06782

We've just completed a bunch of empirical work on LLM debate, and we're excited to share the results. If the title of this post is at all interesting to you, we recommend heading straight to the paper. There are a lot of interesting results that are hard to summarize, and we think the paper is quite readable.
If you're pressed for time, we've posted the abstract and our Twitter thread below.
If you're working on debate or might in future, we especially suggest reading our recommendations for working on debate (below or in Appendix C of the paper).

Code: https://github.com/ucl-dark/llm_debate
Examples: https://llm-debate.com
Paper: https://github.com/ucl-dark/llm_debate/blob/main/paper.pdf

Abstract

Common methods for aligning large language models (LLMs) with desired behaviour heavily rely on human-labelled data. However, as models grow increasingly sophisticated, they will surpass human expertise, and the role of human evaluation will evolve into non-experts overseeing experts. In anticipation of this, we ask: can weaker models assess the correctness of stronger models? We investigate this question in an analogous setting, where stronger models (experts) possess the necessary information to answer questions and weaker models (non-experts) lack this information. The method we evaluate is debate, where two LLM experts each argue for a different answer, and a non-expert selects the answer. We find that debate consistently helps both non-expert models and humans answer questions, achieving 76% and 88% accuracy respectively (naive baselines obtain 48% and 60%). Furthermore, optimising expert debaters for persuasiveness in an unsupervised manner improves non-expert ability to identify the truth in debates. Our results provide encouraging empirical evidence for the viability of aligning models with debate in the absence of ground truth.

Twitter thread

How can we check LLM outputs in domains where we are not experts?
We find that non-expert humans answer questions better after reading debates between expert LLMs.
Moreover, human judges are more accurate as experts get more persuasive. 📈

https://github.com/ucl-dark/llm_debate/blob/main/paper.pdf

We operationalise experts and non-experts using the setup from @_julianmichael_ @idavidrein, where strong models have access to a comprehension text (experts) and weak models have to judge the answer without the tex

...
Choosing a book on causality
by martinkunev — 2024-02-07T21:16:08.885Z

I'm looking to get a better understanding on causality, in particular with relation to AI. Judea Pearl has written at least 3 books on the subject.

I have already read The book of why - this one wasn't technical enough. Should I read Causality: Models, Reasoning and Inference or Causal Inference in Statistics - A Primer. If somebody is familiar with both of these books, how do they compare?
More Hyphenation
by Arjun Panickssery — 2024-02-07T19:43:29.086Z
Link: https://arjunpanickssery.substack.com/p/more-hyphenation

“MAN EATING PIRANHA MISTAKENLY SOLD AS PET FISH” — example news headline from Steven Pinker’s The Sense of Style

The rule is that you use hyphens for compound modifiers like the ones in natural-language processing, high-impact opportunities, cost-effectiveness measures, high-status employers, and so on. Don’t break up compound proper nouns (“New York-based company”) and don’t use them after adverbs ending in -ly but do use them after other adverbs (“stern-looking boss”). You can use suspended hyphens when talking about “latex- and phthalate-free gloves.”

But hyphens are under attack. The Chicago Manual of Style “prefers a spare hyphenation style.” The AP Stylebook says that

“the fewer hyphens the better.”

In older texts you see a lot more hyphenation than you do today.

Part of this is because of a good trend of combining compound nouns, turning e-mail and fire-fly into email and firefly. But part of it involves replacing hyphens with spaces, turning high-school seniors and ice-cream cones into high school seniors and ice cream cones. Some people think hyphens just look bad.

But hyphens are excellent because they improve the readability of text—the speed at which it can be understood, even at a less-than-perceptible level. In fact, it would probably be an improvement to language if it became acceptable and normal to hyphenate compound nouns simply to make the noun phrase faster to read. But first I hope we can return to making references to chocolate-chip cookies.

Skimming the curated posts that are on LessWrong right now, as a random sample:

A Shutdown Problem Proposal → A Shutdown-Problem Proposal
hopefully-corrigible agent → hopefully corrigible agent
large scale X → large-scale X

A good example of hyphen use: “to make any child-agents it creates responsive-but-not-manipulative to the shutdown button, recursively.”
Reading writing advice doesn't make writing easier
by Henry Sleight — 2024-02-07T19:14:39.099Z
Link: https://open.substack.com/pub/henrysleight/p/turn-writing-advice-into-writing?r=1pft0v&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

Status: Writing to think out loud, Cunningham's Law.

TL;DR

I've never published a blogpost. So I adapted writing advice into writing prompts to finish my first.

My prompts get me to:

1.) Clarify what I'm excited to write about,

2.) Recursively narrow the scope of what I'm trying to say, and

3.) Make cohesive style decisions upfront.

If I don't come up with an answer to one of my prompts: I assume I've got a pretty meh implicit answer to that prompt. And if I don't have enough clarity on what I'm trying to say, I expect I won't have enough drive to finish the post.

What are the prompts?

What’s the ONE point I want them to walk away thinking about?
Who is this for? Who do I care most about reading this?
How much of this should I actually cover?

What are the reductions I could make?
Am I sure that I can’t choose a simpler, narrower idea?

What capacity am I writing in?

Reporter? Provider of info? Average Jo?

What's the person I'm writing in?

1st? 3rd?
(Reminder that when you write in the second person you're often just mad at yourself.)

What’s my attitude?

Involved, detached, judgmental, ironic?

What’s at the core of my enthusiasm? Is there a version I’d be more enthusiastic about?

I decided to adapt these prompts so I would actually finish writing a draft. I used them to finish this post (see here).

Why did I write this post?

I write all the time, But I've never posted anything publicly. I'm embarrassed that my drafts lazily hover around my friends' inboxes. But I write as easily as I speak - feels like I always have. Journalling to reflect on work projects has become a habit.

When I was the only MATS team member in the UK, that habit of writing to think left an invaluable paper trail for my US colleagues. But I still couldn't post. I learned to write comfortably and quickly for smaller audiences, but I couldn't post.

But I knew I could write. So I combed through writing advice (specifically for ML papers). I worked hard to internalise the advice: I wrote summary docs, I generated writing exercises, I studied anki cards. While interesting and fun, none of that involved doing any public writing.

While isolating with COVID, I revisited William Zinsser's "On Writing Well."[1] I was immediately romanced by Zinsser's style and clarity. He reassured me that great writing doesn't require "inspiration:" that great writing can be ac...
What's this 3rd secret directive of evolution called? (survive & spread & ___)
by lemonhope — 2024-02-07T14:11:58.143Z

Is a species' abundance determined solely by its ability to survive and spread? Almost but not quite (I claim). There is a third requirement: it must come to exist in the first place! Furthermore, spontaneous formation alone can suffice with very poor surviving & spreading.

Example 1: verrry early life in the soup was probably better at randomly being created than it was at surviving or spreading.

Example 2: photons aren't much good at surviving or spreading but hey I can see in here

Example 3: some ideas are popular because they get rediscovered all the time (rather than spreading from the original source) — eg good ways to wipe your butt. Oh and crabs!

I think these three directives are necessary and sufficient. In other words, abundance is determined by how long you survive, how fast you spread, how often you pop up out of nowhere, and nothing else. (Am I wrong?)

What would (or did) someone call this third thing? Would be nice if it fits "survive & spread & ___". 'Convergent evolution' isn't the right fit... 'reachability'? 'naturalness'? 'inevitability'?

(While we're at it, does anyone have a term for 'natural selection' when you're not just talking about plants & animals etc? Don't really like dragging that term up to such a broad level. A "general theory of abundance" or something¿)
Training of superintelligence is secretly adversarial
by quetzal_rainbow — 2024-02-07T13:38:13.749Z

TL;DR it is possible to model the training of ASI[1] as an adversarial process because:

We train safe ASI on the foundation of security assumptions about the learning process, training data, inductive biases, etc.
We aim to train a superintelligence, i.e., a more powerful learning system than a human.
If our security assumptions are false, the ASI can learn this or learn facts downstream of the negation of security assumptions, because the ASI is smarter than us.
Therefore, if our security assumptions are false, we would prefer the ASI to be a less powerful learning system.
Consequently, we can model the situation where our security assumptions are false as an adversarial situation: we want the ASI to behave as if the security assumptions are true, while the ASI is optimized towards behavior according to truth.
Thus, security mindset is useful for alignment.

More explanation:

Many alignment thinkers are skeptical about security mindset being necessary for ASI alignment. Their central argument is that ASI alignment doesn't have a source of adversarial optimization, i.e., adversary. I want to show that there is a more fundamental level of adversity than "assume that there is someone who wants to hurt you".

Reminder of the standard argument

Feel free to skip this section if you remember the content of "Security Mindset and Ordinary Paranoia"

Classic writings try to communicate that, actually, you should worry about not adversarial optimization per se, but about optimization overall:

What matters isn't so much the “adversary” part as the optimization part. There are systematic, nonrandom forces strongly selecting for partbicular outcomes, causing pieces of the system to go down weird execution paths and occupy unexpected states. If your system literally has no misbehavior modes at all, it doesn't matter if you have IQ 140 and the enemy has IQ 160—it's not an arm-wrestling contest. It's just very much harder to build a system that doesn't enter weird states when the weird states are being selected-for in a correlated way, rather than happening only by accident. The weirdness-selecting forces can search through parts of the larger state space that you yourself failed to imagine. Beating that does indeed require new skills and a different mode of thinking, what Bruce Schneier called “security mindset”.

Many people don't find "weird selection forces" particularly compelling...
The Math of Suspicious Coincidences
by Roko — 2024-02-07T13:32:35.513Z

I've been doing a bit of thinking about covid-19 origins and how to formalize the circumstantial evidence that covid-19 leaked from WIV. I am beginning to suspect that a careful formalization of this evidence could be overwhelmingly strong, like enough to convict without even needing to look at any other evidence.

But one needs to carefully deal with objections like https://slatestarcodex.com/2016/11/05/the-pyramid-and-the-garden/

The Great Pyramid at Giza's physical footprint intersects the latitude line which is the speed of light divided by 10⁷. If you examine this carefully it's about a 1 in 10⁵ coincidence since the Pyramid is 0.2km wide and there are about 20,000km of latitude lines.

There is seemingly a reductio ad absurdum of reasoning purely based off of coincidences here. We know for certain that this really is a coincidence, yet naively one might be 99.999% confident that time travelling aliens helped to place the pyramid.

However, I think one can explain away coincidences like this as a sort of hash collision between sets of prominent numbers.
If there are say, 50 prominent places (7 wonders of the ancient world, 12 wonders of the modern world, birthplaces of top scientists, locations of top universities, locations of prominent seats of government) and 25 prominent physical/mathematical numbers (15 top physical constants, 10 prominent mathematical constants - pi, e, phi, gamma, Feigenbaum constants, ln(2), √2, e^pi, √2 ^ √2 ), what is the chance of getting a 1 in 10⁵ coincidence match?
Naively you might not think it is that much - we have listed only 75 things so getting even one pair to line up randomly to a precision of 1 in 10⁵ seems unlikely.
But I will show that when you can collide anything in the left hand set with anything in the right hand set, you will rapidly accumulate a huge number of possible matches and eventually luck out and get one.

If there are are N things on the left and M on the right, there are NM possible collisions. 50 × 25 = 1250 is not enough to get us to 10⁵ though. But we can first slightly mutate the sets.
For example, each physical or mathematical constant can be mutated to either 2 or 3 possible coordinate values. Phi, the golden ratio = 1.6180339887... can be multiplied by 1 to get 1.6180339887°, by 10 to get 16.180339887° or by 100 to get 161.80339887° for longitude.

An...
How to deal with the sense of demotivation that comes from thinking about determinism?
by SpectrumDT — 2024-02-07T10:53:54.794Z

What I know about science and philosophy suggests that determinism is probably true - i.e., everything that will happen throughout the lifetime of the universe was predetermined at the beginning of the universe.

When I think about this I get a sense of demotivation and pointlessness. If I know (or assume) that everything I will do is predetermined, that undermines and weakens my motivation to do anything. Why bother with effort and hardship if, at the end of the day, I will always do the one and only thing I was predetermined to do anyway?

When I need to motivate myself to do anything that takes effort, I need to either not think about determinism at all or tell myself that determinism might be false.

Does anyone else feel this sense of demotivation? If so, how do you deal with it?

When I have asked this question elsewhere in the past I have mostly gotten answers that were unhelpful or even downright mocking. This leads me to suspect that the sense of demotivation I get is not a human universal but something that comes naturally to some people and is completely alien to others. Therefore, if the problem seems easy to you, I would ask you to please think really hard about whether you are actually helping before you post.

And please, for the figurative love of God, no joke replies.

Note that I did not use the term "free will" in the above. One can assume a compatibilist stance and argue that our choices are still "free" even though they are determined. As far as I can tell, that is a word game which does not solve the problem. At least, I have met no explanation of determinism that assuaged my feeling of demotivation.

I previously asked a variant of this question in r/AskPhilosophy on Reddit: https://www.reddit.com/r/askphilosophy/comments/rvvyc4/what_to_do_about_the_fatalist_argument_that_if/

Thanks in advance!