trentmkelly/LessWrong-43k huggingface dataset:
Five views of Bayes' Theorem
by Adam Scherlis — 2022-07-02T02:25:30.962Z

I think rearranging Bayes' Theorem sheds light on it in interesting ways.

P(A|B) = P(B|A) P(A)/P(B)

This is the most common form: you want to know the conditional probability of A given B, but what you actually know is the probability of B given A (and your priors on A and B). Bayes lets you swap the two events.

One way to think about this one: to swap around the conditional, multiply by a correction factor of P(A)/P(B) to change the "units" from "some sort of probability of B" to "some sort of probability of A". (This is literally a unit conversion if P(A) and P(B) are measures over two different spaces!)

P(A|B) P(B) = P(B|A) P(A)

These are two ways of writing the joint probability P(A, B), corresponding to two ways of sampling (A, B) sequentially: B and then A, or A and then B. The first thing you sample comes from your prior, and the second thing comes from a conditional distribution that depends on the first thing.

P(A|B)/P(B|A) = P(A)/P(B)

A lot of the time, people say things like "A is pretty likely given B, but B isn't quite as likely given A." Or: "many Bs are As, but not that many As are Bs." These are equivalent to saying "A is more likely than B, a priori" or "there are more As than Bs".

(I actually wrote this post because people kept saying the first thing around me and I had to think for a moment each time to realize it was the same as the second thing.)

P(A|B)/P(A) = P(B|A)/P(B)

Let's say both sides are equal to three. That means:

A is three times more likely than priors if B.

But also:

B is three times more likely than priors if A.

So if people in San Francisco are three times as likely to own crypto as the general population, then people who own crypto are three times as likely to live in SF.

This almost implies that "the relative risk of A given B is the relative risk of B given A", but "relative risk" is apparently defined as P(A|B)/P(A|¬B) rather than P(A|B)/P(A). Well, at least it's approximately true when A and B are both rare.

P(B) = P(A) P(B|A)/P(A|B)

This one's weird. I guess it's useful if you sampled B after A, but then realized you want to know the prior probability of B? And for some reason P(A|B) is what you have lying around?

Leave a comment if you know a cute intuition for this one!
[Linkpost] Existential Risk Analysis in Empirical Research Papers
by Dan H — 2022-07-02T00:09:49.399Z
Link: https://arxiv.org/pdf/2206.05862.pdf

I’ve noted before that it can be difficult to separate progress in safety from progress in capabilities. However, doing so is important, as we want to ensure that we are making differential progress on safety, rather than just advancing safety as a consequence of advancing capabilities. In particular, I think that research should rigorously evaluate trade-offs between improving safety and advancing capabilities.

This paper introduces the concept of “capabilities externalities,” and provides concrete steps for how to analyze the risk of different research papers. It also introduces “x-risk sheets,” which can be added to papers and help researchers discuss their impact on AI x-risk. Since discussion of AGI is becoming increasingly normalized in mainstream ML, I think careful discussions of x-risk are increasingly feasible within mainstream ML. The paper is largely aimed at mainstream ML researchers, and it is one step towards normalizing the discussion of existential risks.
Agenty AGI – How Tempting?
by PeterMcCluskey — 2022-07-01T23:40:16.610Z
Link: http://www.bayesianinvestor.com/blog/index.php/2022/07/01/agenty-agi-how-tempting/

[Epistemic status: mostly writing to clarify my intuitions, with just a
few weak attempts to convince others. It's no substitute for reading
Drexler's writings.]

I've been struggling to write more posts relating to Drexler's vision
for AI (hopefully to be published soon), and in the process got
increasingly bothered by the issue of whether AI researchers will see
incentives to give AI's broad goals that turn them into agents.

Drexler's CAIS paper convinced
me that our current trajectory is somewhat close to a scenario where
human-level AI's that are tool-like services are available well before
AGI's with broader goals.

Yet when I read LessWrong, I sympathize with beliefs that developers
will want quite agenty AGI's around the same time that CAIS-like
services reach human levels.

I'm fed up with this epistemic learned
helplessness,
and this post is my attempt to reconcile those competing intuitions.

Please recall that Drexler's distinction here focuses on a system's
goals, not its knowledge. Software is more agenty when its goals cover a
wide range of domains, and long time horizons. Services are designed to
produce specific outputs using a system's current procedures.

The Easy Part?

An ESRogs comment on
LW
nudged me a bit more toward Drexler's position.

"The smart part is not the agent-y part": intelligence and agency are
independent, with intelligence being the hard part, and agency being
more like prompt engineering.

This seems mostly correct, at least for Drexler's meaning of agency.

There are still significant problems with getting agency correct, but
they're mostly associated with figuring out what we want, and likely
not helped much by math or computer science expertise.

Orthogonality squared

ESRogs also suggests a potentially valuable concept that he calls
"Orthogonality squared". I.e. the domain of an agent's goals is
independent of its intelligence.

We can, in principle, create AI's that don't care about atoms. At
least for some simple tasks, it's fairly natural to have the AI care
only about abstractions such as math. You can't alter the laws of
arithmetic by rearranging atoms - laws of arithmetic are in a different
realm from atoms. An AI that cares only about a distant part of Tegmark
IV
won't care whether we shut down an arrangement of atoms in our realm.

Many people concerned about AI safety seem reluctant to accept this.
E.g. Rob Bensinger's initial rep...
AXRP Episode 16 - Preparing for Debate AI with Geoffrey Irving
by DanielFilan — 2022-07-01T22:20:18.456Z

YouTube link

Many people in the AI alignment space have heard of AI safety via debate - check out episode 6 of AXRP if you need a primer. But how do we get language models to the stage where they can usefully implement debate? In this episode, I talk to Geoffrey Irving about the role of language models in AI safety, as well as three projects he’s done that get us closer to making debate happen: using language models to find flaws in themselves, getting language models to back up claims they make with citations, and figuring out how uncertain language models should be about the quality of various answers.

Topics we discuss:

Status update on AI safety via debate

Language models and AI safety

Red teaming language models with language models

GopherCite

Uncertainty estimation for language reward models

Following Geoffrey’s work, and working with him

Daniel Filan:
Hello, everybody. Today I’ll be speaking with Geoffrey Irving. Geoffrey is a safety researcher at DeepMind who leads the scalable alignment team. We’ll be speaking about three papers he’s co-authored: Red Teaming Language Models with Language Models, whose first author is Ethan Perez; Teaching Language Models to Support Answers with Verified Quotes, aka the GopherCite paper, whose first authors are Jacob Menick, Maja Trebacz and Vladimir Mikulik; and Uncertainty Estimation for Language Reward Models, whose first author is Adam Gleave. For links to what we’re discussing, you can check the description of this episode and you can read the transcript at axrp.net. Welcome to the show, Geoffrey.

Geoffrey Irving:
Thank you.

Status update on AI safety via debate

Daniel Filan:
So I guess my first question is, what happened to AI Safety via Debate?

Geoffrey Irving:
It is the most important question. So I think the thing that happened is I’m still doing that stuff. I’m building up towards it. And the overall research agenda here is figure out and implement the protocol for humans and machines discussing problems that gives us good answers that we endorse after reflection. So that’s debate, but then broadened: there’s a lot of different versions of debate. There’s different things you can add onto it. You could do debate plus evidence or debate plus more aggressive red teaming. There’s a large space of protocols there.

Geoffrey Irving:
And these papers are kind of pieces of that story, but we were also ...
Examples of practical implications of Judea Pearl's Causality work
by ChristianKl — 2022-07-01T20:58:58.066Z

I was recently talking with a friend about the practical usefulness of modern academic epistemology. Intuitively, I would have wanted to say that Judea Pearl's work causality has a lot of practical implications, however, I couldn't think of any examples.

Do you have examples of conclusions you have drawn because you learned about Pearl's causality that you wouldn't have drawn otherwise, or can point to other people making practical use of the concepts?
Minerva
by Algon — 2022-07-01T20:06:55.948Z
Link: https://ai.googleblog.com/2022/06/minerva-solving-quantitative-reasoning.html

Google Research's new AI tackles natural language math problems and handily outperforms the SOTA[1]. It is a pre-trained PaLM [2]finetuned on some maths datasets (which use LaTeX) composed of maths webpages and Arxiv papers (38.5B tokens). The three models trained were as follows.

When generating answers, Minerva is given the same prompt of four questions with correct a chain of reasoning and a consistent format for the final, correct answer. Then the actual question is given. Minerva then outputs a chain of reasoning and a corresponding answer a number of times, with the most common answer chosen. Minerva is graded only on the final answer.

This voting algorithm is called maj@1k and saturates faster than pass@k (generates k answers, if one is right then the answer is graded correctly) but doesn't perform as well for large k. This is quite reasonable, as majority voting will continue to choose the most common answer, with the estimate's error decreasing with larger k. Whereas pass@k allows the model more tries for large k.

Datasets

The datasets used are:

MATH: High school math competition level problems

MATH dataset. Note that a PhD CS student who wasn't fond of maths achieved 40% accuracy on this dataset, and a three time IMO gold medalist achieved 90%.

MMLU-STEM: A subset of the Massive Multitask Language Understanding benchmark focused on STEM, covering topics such as engineering, chemistry, math, and physics at high school and college level.

MMLU example questions

GSM8k: Grade school level math problems involving basic arithmetic operations that should all be solvable by a talented middle school student.

The datasets have questions which vary in difficulty. Predictably, the model performed worse on harder questions, with false positives linearly with question difficulty on

Results

Now time for a suprise quiz! For the purposes of this quiz, assume we're talking about the most accurate minerva model (540B parameters using maj1@k sampling. k=64 for MATH and k=16 for MMLU). And we'll be averaging over results on subtopics[3]. Note the SOTA is OpenAI's davinci-002, which obtained absolute (averaged) scores of about 20% and 49%.

Elicit Prediction (forecast.elicit.org/binary/questions/eMIWVdS3B)

Elicit Prediction (forecast.elicit.org/binary/questions/kWY5jIKFh)

Elicit Prediction (forecast.elicit.o

...
Disarming status
by sano — 2022-07-01T20:00:20.801Z

“...with a utility function that has both a (standard) absolute and a relative consumption components and is — as is standardly assumed — concave in absolute consumption, the marginal utility from additional consumption through the absolute term approaches zero as income rises. The relative component hence becomes increasingly important as income rises. Status seeking, on this view, becomes increasingly important with economic growth... ”[1]

Prologue

Status concerns have long been recognized by economists as a major factor driving economic growth. For example, Adam Smith in The Theory of Moral Sentiments[2] emphasized that pursuit of wealth is a response more to want of status, than to pressures of existential needs, or desire to improve one's living conditions. The behavior is also generally seen as beneficial, as it leads to economic growth, which is instrumental for raising the standard of living. On the other hand, economists increasingly[3] point out that consumption-based status seeking leads to positional arms races that waste resources, have suboptimal effects on individual well-being and contribute to environmental degradation.

The claim

The point I want to focus on in this post is superiority of the psychological, over "material" motives, for innovation and willingness to work. The pattern is discernible not only in the behavior of the mainstream economy workers who manifest the preference indirectly through status-oriented consumption, but also in the more direct psychological needs-oriented type of motivation, typical for the almost billion volunteers globally. I believe this idea has implications that so far have not been adequately considered in economic theory or practice; in particular, that an opportunity exists for efficiency improvement, by exploiting salience of the psychological motivation in a novel type of work reward.

The distinction between psychological and material concerns implies that a reward for any work we do, has in general two functions: it can be a source of exchange value, and it can also provide a psychological needs-supporting signal. The monetary reward in the capitalist market economy (CME) for example, has both of these functions; it can be exchanged for goods or services, and its amount is also a strong signal of competence[5]. In other economic arrangements, either of the two functions might be weak, or m...
Paper: Forecasting world events with neural nets
by Owain_Evans — 2022-07-01T19:40:12.788Z

Paper authors: Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, Dan Hendrycks

This post contains the

paper’s

abstract and excerpts from the paper (with slight modifications).

Paper Abstract

Forecasts of climate, geopolitical conflict, pandemics, and economic indicators help shape policy and decision making. In these domains, the judgment of expert humans contributes to the best forecasts. Can these forecasts be automated?

To this end, we introduce Autocast, a dataset containing thousands of forecasting questions and an accompanying 200GB news corpus. Questions are taken from forecasting tournaments (including Metaculus), ensuring high quality, real-world importance, and diversity. The news corpus is organized by date, allowing us to precisely simulate the conditions under which humans made past forecasts (avoiding leakage from the future).

Motivated by the difficulty of forecasting numbers across orders of magnitude (e.g. global cases of COVID-19 in 2022), we also curate IntervalQA, a dataset of numerical questions and metrics for calibration.

We test language models on our forecasting task and find that performance is far below a human expert baseline. However, performance improves with increased model size and incorporation of relevant information from the news corpus.

An example question from the Autocast dataset (taken from Good Judgment Open)
The dataset simulates the task faced by human forecasters. The model must generate a forecast each day for a series of days (here 01-14-22 to 03-23-22) given access to the news on that day (but not after that day). So the model makes retrodictions without "cheating" (i.e. having access to future information)

Paper Introduction

Forecasting plays a crucial role in the modern world. Climate forecasts shape the policies of governments and companies. Economic forecasts influence investment and employment. In 2020, forecasts about the spread of COVID-19 led to national lockdowns and border closures, slowing the spread of the virus. Consequently, machine learning (ML) models that make accurate forecasts across a broad range of topics could enable more informed decision making at scale and improve

ML safety

.

Two main approaches to forecasting are described in the

forecasting

literature

: statistical and judgmental forecas...
Reframing the AI Risk
by Thane Ruthenis — 2022-07-01T18:44:32.478Z

Follow-up to: Reshaping the AI Industry: Straightforward Appeals to Insiders

Introduction

The central issue with convincing people of the AI Risk is that the arguments for it are not respectable. In the public consciousness, the well's been poisoned by media, which relegated AGI to the domain of science fiction. In the technical circles, the AI Winter is to blame — there's a stigma against expecting AGI in the short term, because the field's been burned in the past.

As such, being seen taking the AI Risk seriously is bad for your status. It wouldn't advance your career, it wouldn't receive popular support or peer support, it wouldn't get you funding or an in with powerful entities. It would waste your time, if not mark you as a weirdo.

The problem, I would argue, lies only partly in the meat of the argument. Certainly, the very act of curtailing the AI capabilities research would step on some organizations' toes, and mess with people's careers. Some of the resistance is undoubtedly motivated by these considerations.

It's not, however, the whole story. If it were, we could've expected widespread public support, and political support from institutions which would be hurt by AI proliferation.

A large part of the problem lies in the framing of the arguments. The specific concept of AGI and risks thereof is politically poisonous, parsed as fictional nonsense or a social faux pas. And yet this is exactly what we reach for when arguing our cause. We talk about superintelligent entities worming their way out of boxes, make analogies to human superiority over animals and our escape from evolutionary pressures, extrapolate to a new digital species waging war on humanity.

That sort of talk is not popular with anyone. The very shape it takes, the social signals it sends, dooms it to failure.

Can we talk about something else instead? Can we reframe our arguments?

The Power of Framing

Humanity has developed a rich suite of conceptual frameworks to talk about the natural world. We can view it through the lens of economy, of physics, of morality, of art. We can empathize certain aspects of it while abstracting others away. We can take a single set of facts, and spin innumerable different stories out of them, without even omitting or embellishing any of them — simply by playing with emphases.

The same ground-truth reality can be comprehensively described in many different ways, simply ...
Who is this MSRayne person anyway?
by MSRayne — 2022-07-01T17:32:59.525Z

I've been on this site as a lurker for a few years now, and started interacting more heavily the past month or so. I've been reluctant to engage much, because, to be honest, I'm a very unusual person - even by the standards of LessWrong - and I don't feel like I fit in or have any idea how to contribute effectively. So I'm writing this both to introduce myself as a member of the community (so you can get a sense of who this MSRayne weirdo is who keeps commenting on your posts), and to ask: how can someone like me contribute?

(Warning: I'm very long winded and if I had stopped and pruned this to be easier to read I would probably not have written it at all. I'm going to trust you not to make fun of me for things I say here. I think LessWrongers can be trusted not to do that.)

A hero is born

I was raised by reclusive narcissists who think children are essentially extensions of their parents who ought not have any agency or opinions of their own. I am still unraveling ways in which things I passively assume to be objective truths are actually just opinions my parents burned into me by consistently punishing me for contradicting them. I was totally socially isolated most of my life, homeschooled until 16 with only a few short-term periods of interaction with others of my own age (all of them entirely superficial with no friendships resulting), never had a job (I didn't want to have yet another person telling me what to do, and I didn't and don't care about anything money can buy), never went to college (my older brother went twice and has never had a good job so it seemed like a total waste of money, and anyway I would have to do two terrifying things - leave home, and be around other people), etc. I still live with my parents and rarely leave the house. In fact I rarely leave my bedroom.

I hated my "schooling", which was essentially just my parents giving me textbooks, telling me to work through them, and (if mom) ignoring me while I mostly daydreamed and doodled, and (if dad) punishing me severely for every mistake or even for having handwriting that didn't look like it came out of a typewriter.

A normal child would have learned to hate study in general as a result of this, but I'm hyper curious and always loved reading, so instead it made me learn to hate work and discipline and all externally imposed responsibilities and avoid anything that reminds me of any of t...
Limerence Messes Up Your Rationality Real Bad, Yo
by Raemon — 2022-07-01T16:53:10.914Z

There’s a pretty basic rationality fact that I don’t see talked about much on LW despite it’s obvious relevance. So I am here to write The Canonical Rationality Post on the topic:

Limerence (aka "falling in love"[1]) wrecks havoc on your rationality.

Evolution created you to breed and raise families and stuff. It gave you complex abstract reasoning because that was a useful problem solving tool. Evolution didn’t do that great a job of aligning humans with its goals (see:

masturbation is an inner alignment failure, birth control is an outer alignment failure

). But, it looks like in some sense evolution was aware it had an alignment problem to solve. It gave us capacity for reason, and also it built in a massive hardcoded override for situations where no fuck you your brain is not for building rocketships and new abstract theories, your brain is for producing children and entangling yourself with a partner long enough to raise them. Now focus all your attention on the new prospective mate you are infatuated with.

To be clear, I like limerence. I put decent odds on CEV considering it a core human value. I think attempts to dissociate from your basic human drives are likely to have bad second-order consequences.

But, still, it’s been boggling how wonky my judgment gets when I was under the influence. And it’s boggling how arbitrary it felt afterwards (one day I’d be “wow this person is amazing”, and then a week later after I got some distance I looked back and thought “wow, Raemon’s thinking was so silly there"). It felt, in retrospect, like I'd been drugged.

I'm not sure how the rationality-warping of limerence compares to other major "evolution-hard-coded-some-overrides" areas like "access to power/money/status". But the effects of limerence feel sharpest/most-pronounced to me, and the most "what the hell just happened?" after the warp has cleared.

Mutual Infatuation and the Limerence Event Horizon

The place where this gets extra intense is when you and the object of your affects are both into each other. When only one of you is quietly pining for the other, your feelings might be warped but... they don't really have anywhere to go.

But if two people both like each other they tend to ascend this graph:

The y-axis is labeled "irrationality" for succinctness, but a better label might be "your bottom-line-is-written-ness." It's not tha...