trentmkelly/LessWrong-43k huggingface dataset:
Shortcuts With Chained Probabilities
by jefftk — 2021-02-18T02:00:05.283Z

Let's say you're considering an activity with a risk of death of one
in a million. If you do it twice, is your risk two in a million?

Technically, it's just under:

1 - (1 - 1/1,000,000)^2 = ~2/1,000,001

This is quite close! Approximating

1 - (1-p)^2 as

p*2 was only off by 0.00005%.

On the other hand, say you roll a die twice looking for a 1:

1 - (1 - 1/6)^2 = ~31%

The approximation would have given:

1/6 * 2 = ~33%

Which is off by 8%. And if we flip a coin looking for a tails:

1/2 * 2 = 100%

Which is clearly wrong since you could get heads twice in a row.

It seems like this shortcut is better for small probabilities; why?

If something has probability p, then the chance of it
happening at least once in two independent tries is:

1 - (1-p)^2
= 1 - (1 - 2p + p^2)
= 1 - 1 + 2p - p^2
= 2p - p^2

If

p is very small, then

p^2 is negligible,
and

2p is only a very slight overestimate. As it gets
larger, however, skipping it becomes more of a problem.

This is the calculation that people do when adding micromorts: you
can't die from the same thing multiple times, but your chance of death
stays low enough that the inaccuracy of naively combining these
probabilities is much smaller than the margin of error on our
estimates.

Comment via: facebook
AXRP Episode 4 - Risks from Learned Optimization with Evan Hubinger
by DanielFilan — 2021-02-18T00:03:17.572Z

YouTube link

This podcast is called AXRP, pronounced axe-urp and short for the AI X-risk Research Podcast. Here, I (Daniel Filan) have conversations with researchers about their papers. We discuss the paper and hopefully get a sense of why it's been written and how it might reduce the risk of artificial intelligence causing an existential catastrophe: that is, permanently and drastically curtailing humanity's future potential.

In machine learning, typically optimization is done to produce a model that performs well according to some metric. Today's episode features Evan Hubinger talking about what happens when the learned model itself is doing optimization in order to perform well, how the goals of the learned model could differ from the goals we used to select the learned model, and what would happen if they did differ.

Daniel Filan:
Today, I'll be talking to Evan Hubinger about risks from learned optimization. Evan is a research fellow at the Machine Intelligence Research Institute, was previously an intern at OpenAI working on theoretical AI safety research with Paul Christiano. He's done software engineering work at Google, Yelp and Ripple, and also designed the functional programming language, Coconut. We're going to be talking about the paper "Risks from Learned Optimization in Advanced Machine Learning Systems". The authors are Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse and Scott Garrabrant. So Evan, welcome to AXRP.

Evan Hubinger:
Thank you for having me Daniel.

Daniel Filan:
So the first thing I wanted to say about this paper is it has a nice glossary at the end with what various terms mean. And I think this is unusual in machine learning papers. So just straight off the bat, I wanted to firstly thank you for that and let potential readers of the paper know that it's maybe easier to read than they might think.

Evan Hubinger:
Thank you. We worked hard on that. I think one of the things that we set out to do in the paper was to define a lot of things and create a lot of terminology. And so in creating a lot of terminology, it's useful to have a reference for that terminology, and so we figured that would be helpful for people trying to go back and look at all of the terms that we're using.

Evan Hubinger:
And terms evolve over time. It's not necessarily the case that the same terms that we use in our paper are going to be used in the futu...
The Problem with Giving Advice
by Rossin — 2021-02-17T21:52:48.430Z

Epistemic Status: I'm highly confident this is a phenomenon that occurs with a lot of advice people give, but I'm quite uncertain about the best way to deal with it when trying to give advice to more than one person.

The main thing people fail to consider when giving advice is that someone with ostensibly the same problem may require a vastly different solution than themselves. The underlying cause of the problem in many cases may be the exact opposite of what it was for you.

Many issues in life are caused by being too extreme. Either extreme is problematic-both too much of something or too little. Often people only think about one extreme (or one possible failure mode, we’re not limited to only two!) when they give advice because that is the problem they themselves had to overcome. It fails to occur that not only might this advice not be helpful, it might be actively detrimental to a person struggling with the opposite problem. To use a concrete example, imagine a runner is trying to improve their 5K time and asks a more accomplished runner for advice. The faster runner suggests that if the questioner wants to break through their plateau they will probably need to do more high intensity interval training. After all, the faster runner can remember that was what they needed to do to progress past their own plateau. Unfortunately, the slower runner’s problem is that they are overtraining, their muscles do not have sufficient time to recover and become stronger, limiting improvement. This advice will thus be completely counterproductive, and will probably lead to injury if the runner tries too hard to follow it.

The issue is that it is instinctive to ask “how would this advice have affected me?” when evaluating possible advice to give, rather than “is this the sort of scenario in which that sort of advice would be useful, accounting for the individual receiving the advice?” From this one might be tempted to derive the following lesson: never give anyone important advice unless you have thoroughly questioned what their problem is and are very confident you understand both the problem and the underlying cause.

If you have the resources and time to give people individual advice I think this is a reasonable principle to abide by. But we often do not have this luxury, sometimes we want to give advice to multiple people at once. Sometimes we just don’t ...
Covid: CDC Issues New Guidance on Opening Schools
by Zvi — 2021-02-17T20:00:01.602Z

Author’s Note: This was originally the title topic of this week’s Covid post (as ‘School Daze’), but the post was getting long and this is its own issue, so I’m moving the guidelines discussion to its own post.

As usual, all discussions of school are confusing for me, because I consider 21st-century American schools as they currently exist to mostly be a dystopian nightmare, obedience factory and prison system. That makes it hard to root for the resumption of in-person education.

Still, I do root for it, for two reasons. First, I recognize that this is what almost all parents want for their kids, and second, that the alternative that is being implemented in practice is not home unschooling or kids getting to be kids again. It is ‘remote learning’ and it is a toxic cesspool that drives large percentages of kids into depression, makes it impossible for many parents to work or relax, and generally makes standard schooling look like paradise while also neither teaching the few things school successfully teaches nor offering contact with fellow human beings. It’s the absolute worst in every way other than not catching Covid, and it is saddening to me that more children are not withdrawn from school even under these conditions.

Thus, I am increasingly comfortable treating ‘get kids back in school’ the same way I would if I thought of school the way (for example) my parents think of schools.

The controversy over how to deal with schools continues. Few on any sides are showing much sanity. It’s understandable, as children are not a sanity-friendly topic in modern America.

One school is doing this with five million dollars of plexiglass (link has video, this is a still frame):

Then between classes they plan to require disinfecting, while they do nothing about airflow, and I don’t have any more idea how the hell to teach students under these conditions than the teacher who made the video does. It doesn’t seem possible.

To be fair, that’s in no way recommended by anyone. It’s certainly not what the CDC’s guidelines from last week say.

Those guidelines also aren’t intentionally suggesting this insanity, which is also happening:

Isn’t it great that we can get all the disadvantages of school without the pesky advantage of perhaps sometimes having someone to teach a student something, or the safety of not putting a bunch of people into the same room for...
Preparing to land on Jezero Crater, Mars: Notes from NASEM livestream
by Yitz — 2021-02-17T19:35:01.198Z

I just attended a livestream of NASA scientists talking about the upcoming Perseverance landing, and the goals they hope to achieve during the mission. A recording of the full meeting can be found here: https://livestream.com/accounts/7036396/events/9513656. I was taking notes during the talk, and thought I'd share them here, in case anybody finds them informative. I have not edited my notes, and am simply pasting them in full for now, so there's a whole bunch of grammatical errors and unoptimized wording. Nonetheless, I hope you find some of this interesting!

This program is trying to understand history of geology and climate on Mars, exploring possibilities of life, comparing and contrasting with Earth.

Some Mars processes are really different than earth, which provides new perspective on us. Our mission is about the ancient past, it’s tough to survive rn where we’re going. We are explorers, we want to plan for future human missions.

Mars missions: follow the water—>explore habitatablity—>seek signs of life—>prepare for future human exploration

Our current questions we are asking can only be asked as a result of past missions.

We want to get precious samples and bring them back to Earth. We are always thinking about the future, thinking about hope. This is about technology as well.

Helicopter is audacious (in a good way) tech demonstration.

Objective A: Geology, B: astrobiology, C: sample caching (+D: prepare for humans)

We need to understand geological context of 3 and 1/2 billion year old crater, was about same time life began on Earth.

This will be most advanced sampling system ever sent to another planet, will help us to determine *which* samples to bring back

Why Jezero crater? Spent about five years assessing sites, this was selected.

Deposits suggest crater was likely friendly to life in the past, there’s an ancient delta showing water flow with sediments, which could have preserved organic molecules and other potential life signs. We also see water-bearing minerals

Cameras: we have 23 cameras on this rover, plus two on helicopter. Primary camera is mast camera, we also have ground-penetrating radar, plus spectrometer which are designed to study chemical composition of rocks and soil, to identify potential best samples with organic molecules.

All missions are built on past missions, couldn’t do it without our robotic friends :3

Ma...
[AN #138]: Why AI governance should find problems rather than just solving them
by Rohin Shah — 2021-02-17T18:50:02.962Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

Please note that while I work at DeepMind, this newsletter represents my personal views and not those of my employer.

HIGHLIGHTS

‘Solving for X?’ Towards a problem-finding framework to ground long-term governance strategies for artificial intelligence (Hin-Yan Liu et al) (summarized by Rohin): The typical workflow in governance research might go something like this: first, choose an existing problem to work on; second, list out possible governance mechanisms that could be applied to the problem; third, figure out which of these is best. We might call this the problem-solving approach. However, such an approach has several downsides:

1. Such an approach will tend to use existing analogies and metaphors used for that problem, even when they are no longer appropriate.

2. If there are problems which aren’t obvious given current frameworks for governance, this approach won’t address them.

3. Usually, solutions under this approach build on earlier, allegedly similar problems and their solutions, leading to path-dependencies in what kind of solutions are being sought. This makes it harder to identify and/or pursue new classes of solutions

4. It is hard to differentiate between problems that are symptoms vs. problems that are root causes in such a framework, since not much thought is put into comparisons across problems

5. Framing our job as solving an existing set of problems lulls us into a false sense of security, as it makes us think we understand the situation better than we actually do (“if only we solved these problems, we’d be done; nothing else would come up”).

The core claim of this paper is that we should also invest in a problem-finding approach, in which we do not assume that we even know what the problem is, and are trying to figure it out in advance before it arises. This distinction between problem-solving and problem-finding is analogous to the distinction between normal science and paradigm-changing science, between exploitation and exploration, and between “addressing problems ...
2021-03-01 National Library of Medicine Presentation: “Atlas of AI: Mapping the social and economic forces behind AI”
by IrenicTruth — 2021-02-17T18:23:47.320Z

Monday, March 1, 2021 at 4 pm: https://videocast.nih.gov/watch=41566 (it will be archived at the same link)

I thought this would be of interest to people here, especially to those interested in AI alignment.

The full announcement follows:

National Library of Medicine Lecture on Science, Technology & Society – “Atlas of AI: Mapping the social and economic forces behind AI”
Please join us on Monday, March 1 at 4 pm, when Dr. Kate Crawford will deliver the inaugural annual National Library of Medicine (NLM) Lecture on Science, Technology, and Society, entitled, “Atlas of AI: Mapping the social and economic forces behind AI.” The lecture will be videocast and is open to all https://videocast.nih.gov/watch=41566.
Dr. Crawford is the inaugural Visiting Chair of AI and Justice at the École Normale Supérieure, as well as a Senior Principal Researcher at Microsoft Research, and the cofounder of the AI Now Institute at New York University.
NLM is pleased to have her launch this annual series, which aims to raise awareness and spark conversations about the intersection of societal and ethical considerations with the conduct of biomedical research and the use of advanced technologies.
Machine learning systems are already playing a significant role in many of our social institutions, including healthcare, education, hiring and criminal justice. Despite the patina of objectivity and neutrality, many scholars have shown how these systems can reproduce and intensify forms of structural bias and discrimination. In this talk, Dr. Crawford shares insights from her new book, Atlas of AI, to show the historical origins, labor practices, infrastructures, and epistemological assumptions that underlie the production of artificial intelligence. The classificatory logics and predictive approaches raise challenges that extend well beyond the current bias debate. Dr. Crawford offers new paths for thinking through the research ethics and policy implications of the turn to machine learning, which are increasingly urgent in a time of a pandemic and growing inequality in the United States.
Dr. Crawford is a leading scholar of science, technology, and society. Over a 20-year career, her work has focused on understanding large scale data systems and AI in the wider contexts of history, politics, labor, and the environment. Dr. Crawford has held academic ...
Are we prepared for Solar Storms?
by Alexander Gietelink Oldenziel — 2021-02-17T15:38:03.338Z

The COVID19- Pandemic has brought one awful fact to global attention:

The Powers that Be are woefully unprepared for once-in-century global risks.

Are we prepared for Solar Storms (solar flares, coronal mass ejection events, etc)?

A basic summary:

A more advanced explanation (that concludes fears are mostly overblown):
Safely controlling the AGI agent reward function
by Koen.Holtman — 2021-02-17T14:47:00.293Z

In this fifth post in the sequence, I show
the construction a counterfactual planning agent with an
input terminal that can be used to iteratively improve the agent's
reward function while it runs.

The goal is to construct an agent which has has no direct incentive
to manipulate this improvement process, leaving the humans in control.

The reward function input terminal

I will define an agent with an input terminal can be used to improve
the reward function of an agent. The terminal contains the current
version of the reward function, and continuously sends it to the
agent's compute core::

This setup is motivated by the observation that it is unlikely that
fallible humans will get a non-trivial AGI agent reward function right
on the first try, when they first start it up. By using the input
terminal, they can fix mistakes, while the agent keeps on running, if
and when such mistakes are discovered by observing the agent's
behavior.

As a simplified example, say that the owners of the agent want it to
maximize human happiness, but they can find no way of directly
encoding the somewhat nebulous concept of human happiness into a
reward function. Instead, they start up the agent with a first reward
function that just counts the number of smiling humans in the world.
When the agent discovers and exploits a first obvious loophole in this
definition of happiness, the owners use the input terminal to update
the reward function, so that it only counts smiling humans who are not
on smile-inducing drugs.

Unless special measures are taken, the addition of an input terminal
also creates new dangers. I will illustrate this point by showing the
construction of a dangerous agent ITF further below.

Design and interpretation of the learning world

As a first step in defining any agent with an input terminal, I have
to define a model of a learning world which has both the agent and
its the input terminal inside it. I call this world the learning
world, because the agent in it is set up to learn the dynamics of its
learning world environment.

See this earlier post in the
sequence
for a general introduction to the graphical language I am using to
define world models and agents.

As a first step to constructing the learning world diagram, I take the
basic diagram of an agent interacting with its environment:

To model the input terminal, I then split each environment state node
St into t...
Graphical World Models, Counterfactuals, and Machine Learning Agents
by Koen.Holtman — 2021-02-17T11:07:47.249Z

This is the second post in a sequence. For the introduction post, see
here.

Graphical World Models

A world model is a mathematical model of a particular world.
This can be our real world, or an imaginary world. To make a
mathematical model into a model of a particular world, we need to
specify how some of the variables in the model relate to observable
phenomena in that world.

We introduce our graphical notation for building world models by
creating an example graphical model of a game world. In the game
world, a simple game of dice is being played. The player throws a
green die and a red die, and then computes their score by adding the
two numbers thrown.

We create the graphical game world model in thee steps:

We introduce three random variables
and relate them to observations we can make when the game is played
once in the game world. The variable X represents the observed
number of the green die, Y is the red die, and S is the score.

We draw a diagram:

We define the two functions that appear in the annotations
above the nodes in the diagram:
D(d)=(if d∈{1,2,3,4,5,6} then 1/6 else 0) ,sum(a,b)=a+b .

Informal interpretation of the graphical model

We can read the above graphical model as a description of how we might
build a game world simulator, a computer program that generates random
examples of game play. To compute one run of the game, the simulator
would traverse the diagram, writing an appropriate observed value into
each node, as determined by the function written above the
node. Here are three possible simulator runs:

We can interpret the mathematical expression P(S=12), the
probability that S equals 12, as being the exact probability that
the next simulator run puts the number 12 into node S.

We can interpret the expression E(S), the expected value of S,
as the average of the values that the simulator will put into S,
averaged over an infinite number of runs.

The similarity between what happens in the above drawings and what
happens in a spreadsheet calculation is not entirely
coincidental. Spreadsheets can be used to create models and
simulations without having to write a full computer program from
scratch.

Formal interpretation of the graphical model

In section 2.4 of the paper, I
define the exact formal semantics of graphical world models.
These formal definitions allow one to calculate the exact valu...
Overconfidence is Deceit
by Duncan Sabien (Inactive) — 2021-02-17T10:45:04.276Z

Author's note: This essay was written as part of an effort to say more of the simple and straightforward things loudly and clearly, and to actually lay out arguments even for concepts which feel quite intuitive to a lot of people, for the sake of those who don't "get it" at first glance. If your response to the title of this piece is "Sure, yeah, makes sense," then be warned that the below may contain no further insight for you.

Premise 1: Deltas between one’s beliefs and the actual truth are costly in expectation

(because the universe is complicated and all truths interconnect; because people make plans based on their understanding of how the world works and if your predictions are off you will distribute your time/money/effort/attention less effectively than you otherwise would have, according to your values; because even if we posit that there are some wrong beliefs that somehow turn out to be totally innocuous and have literally zero side effects, we are unlikely to correctly guess in advance which ones are which)

Premise 2: Humans are meaningfully influenced by confidence/emphasis alone, separate from truth

(probably not literally all humans all of the time, but at least in expectation, in the aggregate, for a given individual across repeated exposures or for groups of individuals; humans are social creatures who are susceptible to e.g. halo effects when not actively taking steps to defend against them, and who delegate and defer and adopt others’ beliefs as their tentative answer, pending investigation (especially if those others seem competent and confident and intelligent, and there is in practice frequently a disconnect between the perception of competence and its reality); if you expose 1000 randomly-selected humans to a debate between a quiet, reserved person outlining an objectively correct position and a confident, emphatic person insisting on an unfounded position, many in that audience will be net persuaded by the latter and others will feel substantially more uncertainty and internal conflict than the plain facts of the matter would have left them feeling)

Therefore: Overconfidence will, in general and in expectation, tend to impose costs on other people, above and beyond the costs to one’s own efficacy, via its predictable negative impact on the accuracy of those other people’s beliefs, including further downstream effects of those people’s

...