trentmkelly/LessWrong-43k huggingface dataset:
Counterfactuals from ensembles of peers
by David Johnston — 2022-01-04T07:01:06.196Z

Bruno De Finetti once argued that probabilities are subjective representations of forecasts of real events. In particular, he rejected the idea that statements like "there is a random probability p of this coin landing on heads when it is tossed, distributed according to Q". For De Finetti, probabilities were subjective and had to refer to real events. He famously proved a representation theorem: given a subjective probability that is symmetric to permutation, we can prove the existence a random variable that represents to the "probability of each event". This theorem justified a common but metaphysically mysterious construction - random variables that represent "the probability of something" - by showing that they can be derived from random variables that represent actual events, if those random variables are exchangeable. This concern is similar (but not identical) to Jaynes' concern with the mind projection fallacy - roughly, if probabilities are subjective things, we should be careful not to start treating them as objective things.

Counterfactuals, I think, have some similarities to "probabilities of coins". In particular, I have about as much difficulty understanding what "the probability this coin lands on heads" actually means as I do understanding "the counterfactual outcome of xxx is yyy" means.

Chris says counterfactuals "only make sense from within the counterfactual perspective". One interpretation of this is: we want a theory of what makes a counterfactual statement true or useful or predictive, and such a theory can only be constructed starting with the assumption that some other counterfactual statements are true.

I think one disanalogy between probability models and counterfactuals is that there are many reasonable attempts to propose abstract and widely applicable principles for constructing good probability models. Some examples of this are principles of coherence imply that subjective uncertainty should be represented by probability models and Solomonoff or max entropy priors.

On the other hand, there are few abstract and widely applicable theories of what makes a counterfactual hypothesis appropriate. Commonly, we rely on specific counterfactual intuitions: for example, randomization guarantees that potential outcomes are independent of treatment assignment. As I understand, David Lewis' theory is one exception to this rule - i...
Some thoughts about an estimator by Taleb
by paladim — 2022-01-04T05:37:07.658Z

I recently read Maximum ignorance probability, with applications to surgery's error rates by N.N. Taleb where he proposes a new estimator for the parameter p of a Bernoulli random variable. In this article, I review the main points of it and also share my own thoughts about it.

The estimator in question (which I will call maximum ignorance estimator) takes the following form

^p=1−I−10.5(n−m,m+1)

where I is the regularized beta function, n is the number of independent trials and m is the number of successes.

This estimator is derived by solving the following equation

Fp(m)=q

where Fp is the cumulative distribution function of a binomial with n trials and probability p of success. In words, this estimator sets p to a value such that the probability of observing m successes or less is exactly q. How do we pick q? The author sets q to 0.5 as it maximizes the entropy (more on this later).

Finally, the estimator is applied to a real world problem. A surgeon works in area with a mortality rate of 5% and he has performed 60 procedures with no fatalities. What can we say about his error probability? By applying the estimator described earlier ^p=0.01148.

Taleb argues that the empirical approach (^p=m/n) does not provide a lot of information because the sample is small, i.e. the estimate is ^p=0, however, we "know" that this is value is not 0, it is just that we have not observed enough samples to see a failure.

On the other hand, the Bayesian would pick a Beta prior for p. A Beta distribution has two parameters and we have only one constraint (mortality rate of 5% on the area) which leaves us with one degree of freedom. The choice of this remaining degree of freedom is arbitrary and it is shown that it has a significant impact on the final estimate obtained.

Having gone through the main points of the article, here follow my own thoughts:

While the choice of q=0.5 is justified by entropy maximization, it is just a built in assumption. In the surgeon example, it means that we are assuming that if the surgeon did 60 procedures more, we believe that there is a probability of 50% of having one or more fatalities. Is this a reasonable assumption? I don't know, but I would argue it is an assumption about a question which we did not ask, i.e.

...
Flinches
by ship_shlap — 2022-01-03T23:51:27.067Z

Flinches are an essential part of our thought process, and also an important part of rationality. They appear when we are presented with an idea, and they are difficult to first detect. Even more difficult is to have the will to not follow through with them and stay objective and curious.

"Chocolate bars dipped in honey are super yummy!" Alice said. "Sounds gross," my mind says (flinch!).

Here, I experienced non-evidential influence - hostility to Alice's idea - because it's something I've never tried before. It's also an idea that would take effort to confirm, or the motive is something along those lines. We all know about the classical "Ew I don't want to try X food!" and then we end up liking it. "It doesn't hurt to be a 'lil' irrational here". Keep in mind that choosing to be rational or irrational is in itself irrational. It leaves you prone to rationalization, and other such blunders. The flinch happens when we are first presented an idea and we quickly form an opinion on it/have a reaction to it. The opinion is often a cached thought, an emotional and 'pre-programmed' reaction which we have to notice and ignore in order for us to think rationally. Therefore flinches are often subliminal, and people often make the mistake of rationalizing it, believing that noticing the flinch is the only necessary part. Flinches are a great enemy of curiosity. Becoming aware of the flinch is the first step, and the second step is to 'act upon it' - to ignore it and test the idea with curiosity. A major part of being curious is to fight against our flinches and to obsess with the idea presented - to find its ups and downs.

When Galileo provided the Copernican theory (held that the Earth revolved around the sun), everyone disagreed (flinch) with his idea because the idea of the earth being the center of the universe felt easier, anything else must've been absurd. Galileo faced charges for his theory.
"Death is an inevitable and tragic part of life." Bob says. "What!? Death is what provides meaning to life! How could you call it tragic or bad?" (flinch!! I'm not saying this view is right or wrong) We often flinch when an idea opposes another which is widely accepted, and appears obvious.
Robin Warren and Barry Marshall received a lot backlash for their discovery of H. Pylori which were the real cause of the stomach ulcers which many fell victim to. "I

...
An Observation of Vavilov Day
by Elizabeth — 2022-01-03T21:10:02.107Z

Content note: this post contains discussion of starvation.

I aspire to be a person who does good things, and who is capable of doing hard things in service of that. This is a plan to test that capacity.

I haven’t been in a battle, but if you gave me the choice between dying in battle and slowly starving to death, I would immediately choose battle. Battles are scary but they are short and then they are over.

If you gave me a chance to starve to death to generate some sufficiently good outcome, like saving millions of people from starvation, I think I would do it, and I would be glad to have the opportunity. It would hurt, but only for a few weeks, and in that time I could comfort myself with the warm glow of how good this was for other people.

If you gave me a chance to save millions of people by starving, and then put food in front of me, I don’t think I could do it. I would do okay for a few days, maybe a week, but I worry that eventually hunger would incapacitate the part of my brain that allows me to make moral trade-offs at my own expense, and I would wake up to find I’d eaten half the food. I want to think I’d manage it, but if the thought experiment gods didn’t let me skip the hard part with more proactive measures, I’m not confident I could.

During the siege of Leningrad, scientists and other staff of the Institute of Plant Study faced the above choice, and to the best of our knowledge, all of them chose hunger. 12 of them died for it, the rest merely got close (English language sources list 9 deaths, which is the number of scientists who died in service of the seed bank but not the total number of people). They couldn’t kill themselves because they were needed to protect the food from rats and starving citizens. Those survival odds are better than the certain death of my hypothetical, but they didn’t have the same certainty of impact either, so I think it balances out.

That’s heroism enough, but a fraction of what’s present in this story. Those scientists worked at an institute founded by Nikolai Vavilov, a Soviet botanist who has the misfortune to be right on issues inconvenient to Joseph Stalin. Vavilov’s (correct) insistence that his theories could feed Russians and those of Stalin’s favored scientist couldn’t got him arrested, tortured, and sent to a gulag, where he eventually starved to death.

The seeds Vavilov and his staff protected now cover 80% o...
Apply for research internships at ARC!
by paulfchristiano — 2022-01-03T20:26:18.269Z

(Update: we are no longer accepting applications for interns.)

In addition to

hiring full-time researchers

, ARC is also hiring for 1-3 month internships. We’re open to interns for summer 2022 in particular, but if a different time works for you that’s fine too.

Internships are appropriate for undergraduates and graduate students, or for anyone considering a career change.

To apply for an internship, you can use

this application

and check the “internship” box. If you submitted an application previously there’s no need to resubmit; we’ll clarify if there is ambiguity.

Salary for interns is $15k/month. We encourage interns to work from our Berkeley office especially at the start of their internship.

(The rest of this post is copied from our

previous hiring post

.)

What is ARC?

ARC is a non-profit organization focused on theoretical research to align future machine learning systems with human interests. We are aiming to develop alignment strategies that would continue to work regardless of how far we scaled up ML or how ML models end up working internally.

Probably the best way to understand our work is to read Eliciting Latent Knowledge, a report describing some recent and upcoming research, which illustrates our general methodology.

We currently have 2 research staff (Paul Christiano and Mark Xu). We’re aiming to hire another 1-2 researchers in early 2022. ARC is a new organization and is hoping to grow significantly over the next few years, so early hires will play a key role in helping define and scale up our research.

Who should apply?

Most of all, you should send in an application if you feel excited about proposing the kinds of algorithms and counterexamples described in our report on ELK.

We’re open to anyone who is excited about working on alignment even if you don't yet have any research background (or your research is in another field). You may be an especially good fit if you:

Are creative and generative (e.g. you may already have some ideas for potential strategies or counterexamples for ELK, even if they don't work).
Have experience designing algorithms, proving theorems, or formalizing concepts.
Have a broad base of knowledge in mathematics and computer science (we often draw test cases and counterexamples from these fields).
Have thought a lot about the AI alignment problem, especially in the limit of very powerful AI systems.

Hiring will be a priority for us in e...
How to End a Pandemic
by MikkW — 2022-01-03T20:26:07.009Z

Disclaimer: I am not an epidemiologist, and my knowledge of the field is limited to what I have learned from the following experiences: 1) being a person who has lived through 2 years of a pandemic 2) having played a decent amount of Plague, Inc. (for whatever that's worth) 3) having taken a microbiology class when I was in college. So what I say should be taken in the sprit of proposing potential solutions, not as an assertion of what we know will work. This is babble, not prune.

So, we're in the middle of a pandemic. COVID-19 has been causing problems for 2 years, and the US (like many countries) is now seeing the largest number of daily cases that it's ever experienced (!). The question I want to explore is: What can we do to make this pandemic end? A related question asks how we can eradicate other viruses that are causing problems, including flu and the common cold.

Some might hope (as I've heard some suggest) that with Omicron swiftly sweeping through swathes of people, a very large fraction (90%+) will be infected with the virus, and some of those people will die, but the remainder will become immune, thereby leaving COVID-19 no more room to propagate. One might hope this ends the pandemic globally over a period no longer than 90 days.

Indeed, we should expect that this very dynamic will cause Omicron to spell out its own end, as well as the end of the Delta variant and the original variant that started this mess. But I would also remind you that there are many, many particles of Coronavirus around the world, and many of the particles carry a wide variety of mutations that are seen in neither Omicron, Delta, nor the original variant. In my estimation, it is almost inevitable that one of those variations will provide enough resistance to immunity to allow it to cause problems long after the Omicron wave is over- and I don't expect that that variation would be the last variant of Covid we face, either.

But who knows, it could be that I'm not right on that. If Omicron really does spell out the end of the Covid-19 pandemic, then consider the rest of this post to be a discussion of what we might do if another pandemic happens down the road (which I hear experts assign too high of a probability to for my comfort).

Our toolbox

We have various tools in our toolbox: Masks and other protective equipment; lockdowns; quarantines; border closures; But the most powerful ...
Prizes for ELK proposals
by paulfchristiano — 2022-01-03T20:23:25.867Z

We are no longer accepting submissions. We'll get in touch with winners and make a post about winning proposals sometime in the next month.

ARC

recently released a technical report on

eliciting latent knowledge

(ELK), the focus of our current research. Roughly speaking, the goal of ELK is to incentivize ML models to honestly answer “straightforward” questions where the right answer is unambiguous and known by the model.

ELK is currently unsolved in the worst case—for every training strategy we’ve thought of so far, we can describe a case where an ML model trained with that strategy would give unambiguously bad answers to straightforward questions despite knowing better. Situations like this may or may not come up in practice, but nonetheless we are interested in finding a strategy for ELK for which we can’t think of any counterexample.

We think many people could potentially contribute to solving ELK—there’s a large space of possible training strategies and we’ve only explored a small fraction of them so far. Moreover, we think that trying to solve ELK in the worst case is a good way to “get into ARC’s headspace” and more deeply understand the research we do.

We are offering prizes of $5,000 to $50,000 for proposed strategies for ELK. We’re planning to evaluate submissions received before February 15.

For full details of the ELK problem and several examples of possible strategies, see

the writeup

. The rest of this post will focus on how the contest works.

Contest details

To win a prize, you need to specify a training strategy for ELK that handles all of the counterexamples that we’ve described so far, summarized in the section below—i.e. where the breaker would need to specify something new about the test case to cause the strategy to break down. You don’t need to fully solve the problem in the worst case to win a prize, you just need to come up with a strategy that requires a new counterexample.

We’ll give a $5,000 prize to any proposal that we think clears this bar. We’ll give a $50,000 prize to a proposal which we haven’t considered and seems sufficiently promising to us or requires a new idea to break. We’ll give intermediate prizes for ideas that we think are promising but we’ve already considered, as well as for proposals that come with novel counterexamples, clarify some other aspect of the problem, or are interesting in other ways. A major purpose of th...
Causality and determinism in social science - An investigation using Pearl's causal ladder
by tailcalled — 2022-01-03T17:51:54.021Z

This is a link post for a Substack post I made, which itself is a remaster of set of messages I sent in another forum, because I thought it was important enough to deserve being publicized properly.

Let’s start with some philosophy of causality, specifically a quick review of Judea Pearl’s Ladder of Causality. Imagine that you want to build a suspension bridge, and you need to get a cable that doesn’t break for the bridge. How could you go about doing this?

One possibility would be to look at historic cases where suspension bridges broke, and try to figure out how they differ from cases where bridges didn’t break. Maybe breaking bridges tend to use cheap rather than expensive materials, so you get an extra-expensive cable made of a gold alloy to be on the safe side. Maybe they tended to be built in places with many records of large storms, and break shortly after meteorologists warn of a new hurricane, so you destroy the historical records and get rid of the meteorologists in the area.

These solutions are obviously terrible. The issue is that the associations above is not causation; while you can probably find statistical tendencies with material costs or storm warnings correlating with bridge crashes, you can’t intervene on these variables to prevent the bridges from crashing; they’re only correlated with bridge stability due to them being side-effects of other factors. They are mere correlations, which in Judea Pearl’s causal ladder would be called rung 1.

We need a causal understanding of the world, both for decision-making and for many forms of theory and research. Causality is ultimately about counterfactuals or potential outcomes; the ability to consider not just what happens, but also what would happen differently if some variable had been different. But how can we know this? The answer is that it can be computed if we have an appropriate causal model of the world.

For instance, if you know the strength of a cable, you know how much load it can bear until it deforms or breaks. If you know how much load the bridge will put on the cable, you can know whether any given cable will break, and can select a cable that causes the bridge to keep standing. Further, if the bridge collapses, you might be able to figure out the load that was on the bridge at the time, and see whether the bridge broke because of an insufficiently strong cable. A theory that allows these...
Shadows Of The Coming Race (1879)
by Capybasilisk — 2022-01-03T15:55:02.070Z

I recently came across the short essay below by George Eliot, the male pen name of Victorian author Mary Ann Evans. It's quite remarkable in its prescience of current AI debates, particularly how long the "servant-master" relationship can persist between humans and their machines.

What's striking for me is that I'd read several of her novels such as Middlemarch and Adam Bede, but nothing about these fairly typical Victorian works ever hinted that she was thinking about the kinds of questions raised in this essay.

Shadows Of The Coming Race

My friend Trost, who is no optimist as to the state of the universe
hitherto, but is confident that at some future period within the
duration of the solar system, ours will be the best of all possible
worlds--a hope which I always honour as a sign of beneficent
qualities--my friend Trost always tries to keep up my spirits under the
sight of the extremely unpleasant and disfiguring work by which many of
our fellow-creatures have to get their bread, with the assurance that
"all this will soon be done by machinery." But he sometimes neutralises
the consolation by extending it over so large an area of human labour,
and insisting so impressively on the quantity of energy which will thus
be set free for loftier purposes, that I am tempted to desire an
occasional famine of invention in the coming ages, lest the humbler
kinds of work should be entirely nullified while there are still left
some men and women who are not fit for the highest.

Especially, when one considers the perfunctory way in which some of the
most exalted tasks are already executed by those who are understood to
be educated for them, there rises a fearful vision of the human race
evolving machinery which will by-and-by throw itself fatally out of
work. When, in the Bank of England, I see a wondrously delicate machine
for testing sovereigns, a shrewd implacable little steel Rhadamanthus
that, once the coins are delivered up to it, lifts and balances each in
turn for the fraction of an instant, finds it wanting or sufficient, and
dismisses it to right or left with rigorous justice; when I am told of
micrometers and thermopiles and tasimeters which deal physically with
the invisible, the impalpable, and the unimaginable; of cunning wires
and wheels and pointing needles which will register your and my
quickness so as to exclude flattering opinion; of a machine for dra

...