trentmkelly/LessWrong-43k huggingface dataset:
Base Rates and Reference Classes
by jsteinhardt — 2021-11-24T22:30:18.741Z
Link: https://bounded-regret.ghost.io/base-rates-and-reference-classes/

Part of lecture notes for the upcoming Stat157 class on Forecasting.

Let's start by considering the following question:

What is the probability that Joe Biden is President of the United States on Nov. 1st, 2024?

[Note: This was written on Nov. 3rd, 2021.]

To answer this question, one strategy I would use is to look at base rates: the rate of occurrence of similar events. For instance:

What fraction of presidential terms are fully completed (last all 4 years)? The answer to this is 49 out of the 58 total terms, or around 84%.
On the other hand, we know that Biden has already made it through 288 days of his term. If we remove the 5 presidents who left office before that, there are 49 out of 53 or around 92%.
But alternately, Joe Biden is pretty old (78 to be exact). If we look up death rate per year in actuarial tables, it's around 5.1% per year, so this leaves him with a ~15% chance of death or a 85% chance of surviving his term.

These are all examples of using base rates.

Base rates can be powerful as they allow us to draw analogies with related cases even when we don't have directly relevant data or a trend line to extrapolate. In that sense they are like zeroth-order forecasting, but can work even when you don't have a clear time series or other trend to base a forecast on.

Decomposing the Problem

We can take base rates one step further. If Biden doesn't complete his term, the main reasons I can think of are:

Death by natural causes
Death by assassination
Impeachment / resignation
The presidency no longer exists (due to a coup, invasion, etc.)

So, we could estimate all of these using base rates, then add them up.

The last two are easiest: only one president has left office due to resignation (Nixon), so 1 out of 58 on impeachment / resignation. I'd put the presidency no longer existing at <1%, so together these add up to maybe 2%.

[At the time of writing, PredictIt gave Biden a 22% chance of resigning before the end of his term. I feel confused about this.]

Death by assassination: 4 out of 58 presidents, or around 7%. But only one of these was more than 288 days into the term, which would instead give 1.7%. I would subjectively put the probability a bit between these, at 3.5%.

Death by natural causes: we previously gave this a 15% chance. But that's neglecting that life expectancy increases with income, and Biden is pretty rich. Eyeballing the mortality curves here, I'm go...
Why do you need the story?
by George3d6 — 2021-11-24T20:26:01.569Z
Link: https://cerebralab.com/Why_do_you_need_the_story?

Boy: Why are you washing your hands?
Shaman: Because this root is poisonous.
Boy: Then why pull it out of the ground?
Shaman: Because the pulp within it can cure the mosquito disease.
Boy: But, I thought you said it was poisonous.
Shaman: It is, but the outside is more so than the inside.
Boy: Still, if it's poisonous, how can it cure things.
Shaman: It's the dose that makes the poison.
Boy: So can I eat the pulp?
Shaman: No, because you are not sick, it's the disease that makes the medicine.
Boy: So it's good for a sick man, but bad for me?
Shaman: Yes, because, the sick man must suffer in order to be cured.
Boy: Why?
Shaman: Because he made the spirits angry, that's why he got the disease.
Boy: So why will suffering make it better?
Shaman: Suffering is part of admitting guilt and seeking forgiveness, the root helps the sick man atone for his transgression. Afterwards, the spirits lift the disease.
Boy: The spirits seem evil.
Shaman: The spirits are neither evil, nor are they, good, as we understand it.
Boy: Ohhh
Shaman: Here, sit, let me tell you how our world was made...

Ok, that sounds like a half believable wise mystic, right? That would make for a decent intro to a mediocre young-adult fantasy coming of age powertrip novel

Now, imagine if the shaman just left it as "The root is poisonous if you are not sick, but it helps if you are sick and eat a bit of its pulp, most of the time". No rules of thumb about dose making the poison, no speculations as to how it works, no postulation of metaphysical rules, no analogies to morality, no grand story for the creation of the universe.

That'd be one boring shaman, right? Outright unbelievable. Nobody would trust such a witcherman to take them in the middle of the jungle and give them a psychedelic brew, he's daft. The other guy though... he might be made of the right stuff to hold ayahuasca ceremonies.

This is not limited to shamans though, but it's more obvious to see in the case of shamans, because you don't use their services.

If you are willing to stop adverting your eyes from uncomfortable uncertainty, you'll see it in most modern specialists, from electricians, to doctors to nuclear physicists. We are all just 2 or 3 reality-validated claims away from babbling nonsense in order to give answers to a a "why".

Ok, let me try again:

Boy: Why did you give my dad ibuprofen?
Doctor: Because it helps cure diseases, like colds.
Boy: So if I ta

...
[AN #169]: Collaborating with humans without human data
by Rohin Shah — 2021-11-24T18:30:03.795Z

Listen to this newsletter on The Alignment Newsletter Podcast.

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Please note that while I work at DeepMind, this newsletter represents my personal views and not those of my employer.

SECTIONS

HIGHLIGHTS

TECHNICAL AI ALIGNMENT

LEARNING HUMAN INTENT

FORECASTING

MISCELLANEOUS (ALIGNMENT)

NEAR-TERM CONCERNS

RECOMMENDER SYSTEMS

NEWS

HIGHLIGHTS

Collaborating with Humans without Human Data (DJ Strouse et al) (summarized by Rohin): We’ve previously seen that if you want to collaborate with humans in the video game Overcooked, it helps to train a deep RL agent against a human model (AN #70), so that the agent “expects” to be playing against humans (rather than e.g. copies of itself, as in self-play). We might call this a “human-aware” model. However, since a human-aware model must be trained against a model that imitates human gameplay, we need to collect human gameplay data for training. Could we instead train an agent that is robust enough to play with lots of different agents, including humans as a special case?

This paper shows that this can be done with Fictitious Co-Play (FCP), in which we train our final agent against a population of self-play agents and their past checkpoints taken throughout training. Such agents get significantly higher rewards when collaborating with humans in Overcooked (relative to the human-aware approach in the previously linked paper).

In their ablations, the authors find that it is particularly important to include past checkpoints in the population against which you train. They also test whether it helps to have the self-play agents have a variety of architectures, and find that it mostly does not make a difference (as long as you are using past checkpoints as well).

Read more: Related paper: Maximum Entropy Population Based Training for Zero-Shot Human-AI Coordination

Rohin's opinion: You could imagine two different philosophies on how to build AI systems -- the first option is to train them on the actual task of interest (for Overcooked, training agents to play against humans or human models), while the second option is to train a more robust agent on some more general task that hope...
Paxlovid Remains Illegal: 11/24 Update
by Zvi — 2021-11-24T13:40:01.199Z

What Alex Tabarrok called The Paxlovid Paradox is getting noticed by the people who notice such things, and completely ignored by everyone else. I’ve split off this week’s Paxlovid update to help make the situation easier to notice, and easier to remember and reference later, lest we forget.

Supply and Demand

The good news is that we are confident Paxlovid is safe and effective and the purchasing department is acting accordingly. Once we decide it is legal, we’ve secured our supply. The Biden administration has agreed to

pay $5 billion for 10 million treatments

. I previously thought this was 10 million pills as did the source, but I’ve been informed it was 10 million treatments, which means I was about to be happy to pay ten times what we’re actually paying. Kind of neat.

They’re going to make a profit from saving people’s lives. Outrageous!

The bad news, of course, is that Paxlovid Remains Illegal.

The Paxlovid Paradox

Scott Alexander is the latest to point out that this is murderous madness1. As I observed, and was later quoted at Marginal Revolution:

The trial was stopped due to ‘ethical considerations’ for being too effective. You see, we live in a world in which:

It is illegal to give this drug to any patients, because it hasn’t been proven safe and effective.

It is illegal to continue a trial to study the drug, because it has been proven so safe and effective that it isn’t ethical to not give the drug to half the patients.

Who, if they weren’t in the study, couldn’t get the drug at all, because it is illegal due to not being proven safe and effective yet.

So now no one gets added to the trial so those who would have been definitely don’t get Paxlovid, and are several times more likely to die.

But our treatment of them is now ‘ethical.’

For the rest of time we will now hear about how it was only seven deaths and we can’t be sure Paxlovid works or how well it works, and I expect to spend hours arguing over exactly how much it works.

For the rest of time people will argue the study wasn’t big enough so we don’t know the Paxlovid is safe.

Those arguments will then be used both by people arguing to not take Paxlovid, and people who want to require other interventions because of these concerns.

FDA Delenda Est.

Scott Alexander notes that the current prediction market median approval date is January 1, 2022, with a 92% chance of approval by March. I’ve asked Polymarket ...
HIRING: Inform and shape a new project on AI safety at Partnership on AI 
by Madhulika Srikumar — 2021-11-24T08:27:05.955Z

Hello! I am a program lead at Partnership on AI (PAI), a non-profit bringing together academic, civil society, industry, & media organizations to advance responsible AI. Work with me and my colleagues to inform and shape a new project on AI safety at PAI!

Apply: Safety-Critical AI (SCAI) Senior Consultant

Applications are due on

December 10, 2021

. This is a remote, contract, consulting position with a part-time commitment beginning in January 2022.

We are seeking a senior consultant for a forthcoming workstream on norms for responsible deployment of AI products to

1. Inform the research and program agenda for the project,

2. Create a roadmap outlining research directions and practical outputs

What the new project is about

The goal of the new workstream on deployment norms is to develop recommendations and resources to improve industry coordination on AI safety ie. reducing risks posed by deployment of powerful AI. This workstream will consider questions around how to enable access to models, encourage incident sharing, and facilitate better understanding of risks, to enhance safety efforts. You will have the *unique* opportunity to shape a multistakeholder workstream and to develop best practices alongside individuals in civil society, academia and industry from

PAI’s Partner network

, and to inform key decision makers in government and industry.

Who we are looking for

The position is designed for well established researchers or practitioners who have an extensive track record of leading research, policy, or strategy work in AI safety; and have a strong interest in developing governance strategies to prevent a

“race to the bottom”

on safety. We are NOT being prescriptive about years of experience or educational background. We welcome applications from candidates from different disciplinary backgrounds including engineering, statistics, political science or law.

What have we been upto on AI safety

PAI develops tools, recommendations, and other resources by inviting voices from across the AI community and beyond to share insights that can be synthesized into actionable guidance. Some of our other ongoing projects in our Safety-Critical AI Program include:

Publication Norms for Responsible AI

- project examining how novel research can be published in a way that maximizes benefits while mitigating potential harms

AI Incident

...
How much Bayesian evidence from rapid antigen and PCR tests?
by mingyuan — 2021-11-24T06:54:21.265Z

Suppose I am running a several-hundred-person indoor event in the US where all attendees are required to be fully vaccinated. Suppose also that no one will come to the event if they have COVID-like symptoms. Given full vaccination and lack of symptoms, how much additional risk reduction is there from requiring people to get a negative rapid test result at the door? Also, same question, but for requiring a negative result on a PCR test <48 hours before the event.

(I did not make an effort to look into this myself because I figured someone here would already know the answer, which would be way faster than me trying to figure it out from scratch. Please someone just tell me, thank you.)
French long COVID study: Belief vs Infection
by Bucky — 2021-11-23T23:14:57.252Z

Thanks to JustisMills and Ruby for reviewing this post. Any errors are my own.

TLDR: The French long COVID study which suggests that belief in having had COVID is correlated with long COVID symptoms but that actually having had COVID is not correlated with long COVID symptoms used the wrong statistical tool to obtain this result.

In reality, the study data show that long COVID symptoms are correlated with having had COVID and agree with Scott’s conclusions in Long COVID: Much more than you wanted to know.

Study

The authors suggest that nearly all long COVID symptoms might not be caused by SARS-CoV-2 (except for those associated with anosmia). I believe that this is true in some cases but not remotely close to the extent suggested by the paper.

Study Design

Roughly speaking the experimental setup was:

Send out a whole load of serology tests to ~36,000 people in May-Nov 2020 (Serology tests are antibody tests which are intended to show if you have ever had COVID).
Perform the tests (~27,000 received back), then give participants their results.
Send out a questionnaire about whether participants think they’ve had COVID and what persistent symptoms they’ve had (Dec 2020-Jan 2021).
Exclude some people for reasons.

e.g. Participants who thought that they had had COVID after they did the serology test were excluded.

Run some logistic regressions on different symptoms vs belief in having had COVID and/or serology results.

You may have spotted the first problem. We’re trying to test whether people’s belief in whether they’ve had COVID or their actually having had COVID is a better predictor of long COVID symptoms but we’ve given participants their serology results before we ask them if they think they’ve had COVID.

You’d think that this would ruin the results – belief in having had COVID should be extremely well correlated with having a positive serology result.

Fortunately (?!) this doesn’t seem to be the case. Of everyone who had a positive serology results, only 41.5% replied that they thought they’d had COVID. Of everyone who thought they’d had COVID, 50.4% had had a negative serology result.

I’m super confused by this but I’ll take this at face value for the moment and move on to the analysis.

Combined effects logistic regression with correlated predictors

The main reported result comes from model 3 of the study's analysis. This is the combined effects logistic regression m...
AI Tracker: monitoring current and near-future risks from superscale models
by Edouard Harris — 2021-11-23T19:16:05.554Z
Link: https://aitracker.org/

TLDR: We've put together a website to track recent releases of superscale models, and comment on the immediate and near-term safety risks they may pose. The website is little more than a view of an Airtable spreadsheet at the moment, but we'd greatly appreciate any feedback you might have on the content. Check it out at

aitracker.org

.

Longer version:

In the past few months,

several

successful

replications

of GPT-3 have been publicly announced. We've also seen the

first serious attempts

at scaling significantly beyond it, along with indications that large investments are being made in commercial infrastructure that's intended to

simplify training

the next generation of such models.

Today's race to scale is qualitatively different from previous AI eras in a couple of major ways. First, it's driven by an unprecedentedly tight feedback loop between incremental investment in AI infrastructure, and expected profitability [1]. Second, it's inflected by nationalism: there have been

public statements

to the effect that a given model will help the developer's home nation maintain its "AI sovereignty" — a concept that would have been alien just a few short years ago.

The replication and proliferation of these models likely poses major risks. These risks are uniquely hard to forecast, not only because many capabilities of current models are novel and might be used to do damage in imaginative ways, but also because the capabilities of future models can't be reliably predicted [2].

AI Tracker

The first step to assessing and addressing these risks is to get visibility into the trends they arise from. In an effort to do that, we've created AI Tracker: a website to catalog recent releases of superscale AI models, and other models that may have implications for public safety around the world.

You can visit AI Tracker at

aitracker.org

.

Each model in AI Tracker is labeled with several key features: its input and output modalities; its parameter count and total compute cost (where available); its training dataset; its known current and extrapolated future capabilities; and a brief description and industry context, among others. The idea behind the tracker is to highlight these models in the context of the plausible public safety risks they pose, and place them in their proper context as instances of a scaling trend.

(There's also a FAQ at the bottom of the

page

, if you'd like to know a b...
Laplace's rule of succession
by Ege Erdil — 2021-11-23T15:48:47.719Z

Laplace's rule of succession is a commonly used heuristic in forecasting the occurence of events which have little to no historical precedents. The basic problem setup is this: suppose you flip a coin with an unknown bias N=H+T times and get H heads and T tails. We want to infer something about the bias p of the coin from this information, where we define p to be the probability of the coin coming up heads.

One obvious idea is to estimate p by ^p=H/(H+T). This works well if we have some decent number of heads and tails, but it fails badly in the edge case where the tosses are very imbalanced. In the extreme case, if we get all heads and no tails, we would be estimating ^p=1 which is obviously a bad estimate: it implies that we're sure the coin has a probability 1 of coming up heads, which seems like an extreme inference to make from a sequence of N heads. After all, if we merely had p<1/(2N), we would not expect to see any heads to begin with. An inference that goes beyond this bound seems intuitively unjustified.

The simple version of Laplace's rule is a heuristic which says we should pretend we got one heads and one tails in excess of the results we have actually obtained and compute the naive estimator under that assumption. In other words, we would estimate ^p=(H+1)/(H+T+2). This avoids the pitfall mentioned in the previous paragraph even though it is a biased estimator: for N tosses we have

E[^p]=pN+1N+2=p(N+2)+(1−2p)N+2=p+1−2pN+2

which is not equal to the true value p unless p=1/2. This is unavoidable, because if we want to avoid overconfidence in the case of N heads, we have to make the estimator biased in the case p=1; and by continuity it will also at least be biased in some neighborhood of 1. It is, however, asymptotically unbiased: this means that as N→∞, we have E[^p]→p.

It's easy to see why this is an improvement over the naive estimator for avoiding overconfidence, but it looks more like an ad-hoc fix than an approach with theoretical justification. To give the justification behind Laplace's rule, I first have to talk about conjugate priors.

Conjugate priors

In Bayesian inference, we often have the following problem: there is a family of models we're considering that are parametrized by a vector θ∈Rn of parameters and we have a prior P(θ) over these parameters. Given the prior, we observe some event X, and we want to do Bayesian updating to get a...
AI Safety Needs Great Engineers
by Andy Jones — 2021-11-23T15:40:18.358Z

Top line: If you think you could write a substantial pull request for a major machine learning library, then major AI safety labs want to interview you today.

I work for

Anthropic

, an industrial AI research lab focussed on safety. We are bottlenecked on aligned engineering talent. Specifically engineering talent. While we'd always like more ops folk and more researchers, our safety work is limited by a shortage of great engineers.

I've spoken to several other AI safety research organisations who feel the same.

Why engineers?

May last year, OpenAI released GPT-3, a system that did surprisingly well at a surprisingly broad range of tasks. While limited in many important ways, a lot of AI safety folk sat up and noticed. Systems like GPT-3 might not themselves be the existential threat that many of us are worried about, but it's plausible that some of the issues that will be found in such future systems might already be present in GPT-3, and it's plausible to think solving those issues in GPT-3 will help us solve equivalent issues in those future systems that we are worried about.

As such, AI safety has suddenly developed an empirical subfield. While before we could only make predictions about what might go wrong and how we might fix those things, now we can actually run experiments! Experiments are not and should never be the entirety of the field, but it's a new and promising direction that leverages a different skill set to more 'classic' AI safety.

In particular, the different skill set it leverages is engineering. Running experiments on a real - if weak - AI system requires a substantial stack of custom software, with projects running from hundreds of thousands to millions of lines of code. Dealing with these projects is not a skillset that many folks in AI safety had invested in prior to the last 18 months, and it shows in our recruitment.

What kind of engineers?

Looking at the engineers at Anthropic right now, every one of them was a great software engineer prior to joining AI safety. Every one of them is also easy to get on with. Beyond that, common traits are

experience with distributed systems
experience with numerical systems
caring about, and thinking a lot about, about AI safety
comfortable reading contemporary ML research papers
expertise in security, infrastructure, data, numerics, social science, or one of a dozen other hard-to-find specialities.

This is not a...