trentmkelly/LessWrong-43k huggingface dataset:
Building and using the subagent
by Stuart_Armstrong — 2020-02-12T19:28:52.320Z

Overall summary post here.

When designing a low-impact agent A, one typically wants them to maximise R0−ρ, where R0 is some positive reward function we want A to increase (ie the actual goal) and ρ is some impact measure or impact penalty. Keeping ρ low is supposed to ensure that A doesn’t make a big difference to the world.

I’ve shown that some impact measures may have issues with subagents. We can decompose the issue of subagents into two questions:

Can A build SA, an “R0−ρ”-maximising subagent, without triggering a high ρ?

Does SA allow A to cheat the impact measure and get higher reward?

I’ll consider three impact measures: the twenty billion questions measure from my old low-impacts paper, the relative reachability measure from Krakovna et al's paper, and the attainable utility measure from Turner et al's paper.

Then the answers to those two questions for those three impact measures are generally[1]:

ρPossible to build SA?Incentive to build SA?Twenty billion QYesNoR-ReachabilityYesDepends on baselineAttainable utilityProbablyYes

Building the subagent

The twenty billion questions impact measure does not prevent the agent from building a subagent within its box. That’s because that measure is keyed to changes in the world, not to the power level of the agent. Until there are changes in the world, it will not trigger.

The relative reachability impact measure similarly won’t penalise the construction of a subagent (as long as the “states” it uses to define reachability don’t include specific information about the agent and any subagents). The reasoning is similar: the penalty triggers when an “irreversible” state is reached, not when the agent becomes powerful.

Building the subagent for the attainable utility impact measure is harder, since this typically increases the power of the agent. But the question is not “will any subagent trigger the impact measure?”, but “can the agent build a subagent in some way that doesn’t trigger the impact measure?” The example here shows that this is possible in some cases. Given that we don't have any impossibility results here, we can expect that a superintelligent agent will likely find a way of doing it.

Using the subagent

Does having the subagent help to evade the impact measure (and hence increase R0 more)? For twenty billion questions, the answer is generally no[2]. Having SA as an independent agent is no diffe

...
[AN #86]: Improving debate and factored cognition through human experiments
by Rohin Shah — 2020-02-12T18:10:02.213Z

Find all Alignment Newsletter resources

here

. In particular, you can

sign up

, or look through this

spreadsheet

of all summaries that have ever been in the newsletter. I'm always happy to hear feedback; you can send it to me by replying to this email.

Audio version

here

(may not be up yet).

Highlights

Writeup: Progress on AI Safety via Debate

(Beth Barnes et al) (summarized by Rohin): This post reports on work done on creating a

debate

(

AN #5

) setup that works well with human players. In the game, one player is honest (i.e. arguing for the correct answer) and one is malicious (i.e. arguing for some worse answer), and they play a debate in some format, after which a judge must decide which player won the debate. They are using Thinking Physics questions for these debates, because they involve questions with clear answers that are confusing to most people (the judges) but easy for some experts (the players).

Early freeform text debates did not work very well, even with smart, motivated judges. The malicious player could deflect on questions they didn't want to answer, e.g. by claiming that the question was ambiguous and redirecting attention by asking new questions. In addition, when the malicious player got to go first and give an incorrect "framework" for finding the answer, and then made individually true claims to "fill in" the framework, it was hard for the honest player to rebut it. So, they moved to a framework without such asymmetries: both players gave a claim (simultaneously), both gave constructive arguments, and both rebutted the other's arguments. In addition, part of the appeal of debate is that the agents can "zoom in" on the particular disagreement they have, and the judge need only analyze the smallest disagreement in order to declare an overall winner.

This suggests the following setup: players simultaneously provide an answer supported with subclaims. Then, after looking at the other player's answer and subclaims, they can provide objections (perhaps on particular subclaims, or the overall structure of the argument). Players can concede or challenge objections left by the other player; the debate then recurses on one of the challenged objections (currently, the malicious player chooses which claim to recurse on, in order to be conservative). There is a depth limit to the recursion, once it is reach...
Suspiciously balanced evidence
by gjm — 2020-02-12T17:04:20.516Z

What probability do you assign to the following propositions?

"Human activity has caused substantial increases in global mean surface temperature over the last 50 years and barring major policy changes will continue to do so over at least the next 50 years -- say, at least half a kelvin in each case."
"On average, Christians in Western Europe donated a larger fraction of their income last year to non-religious charitable causes than atheists."
"Over the next 10 years, a typical index-fund-like basket of US stocks will increase by more than 5% per annum above inflation."
"In 2040, most global electricity generation will be from renewable sources."

These are controversial and/or difficult questions. There are surely a lot of people who think they know the answers and will confidently proclaim that these propositions are just true or, as the case may be, false. But you are a sophisticated reasoner, able to think in probabilistic terms, and I expect your answers to questions like these mostly lie between p=0.1 and p=0.9 or thereabouts, just like mine. No foolish black-and-white thinking for the likes of us!

(I confess that my estimate for the first of those propositions is above p=0.9. But it's not above p=0.99.)

... But isn't it odd that the evidence should be so evenly balanced? No more than 3 bits or so either way from perfect equality? Shouldn't we expect that the totality of available evidence, if we could evaluate it properly, would make for a much larger imbalance? If we encounter only a small fraction of it (which we'd need to, to explain such evenly balanced results), shouldn't we expect that randomness in the subset we happen to encounter will in some cases make us see a large imbalance even if any given single piece of evidence is about as likely to go one way as the other? What's going on here?

Let me add a bit of further fuel to the fire. I could have tweaked all those propositions somewhat -- more than 3%, or more than 7%, above inflation; more than 40%, or more than 60%, or 2050 instead of 2040. Surely that ought to change the probabilities quite a bit. But the answers I'd have given to the questions would still have probabilities between 0.1 and 0.9, and I bet others' answers would have too. Can things really be so finely enough balanced to justify this?

I can think of two &qu... ,
Demons in Imperfect Search
by johnswentworth — 2020-02-11T20:25:19.655Z

One day, a gradient descent algorithm ball was happily rolling down a high-dimensional surface hill. All it wanted was to roll as far down as possible. Unbeknownst to the ball, just off to the side was a steep drop-off - but there was a small bump between the ball and the drop-off. No matter; there was enough random noise on the ball that it would jump the bump sooner or later.

But the ball was headed into unfriendly territory.

As the ball rolled along, the bump became taller. The farther it rolled, the taller the bump grew, until no hope remained of finding the big drop anytime before the stars burned out. Then the road began to narrow, and to twist and turn, and to become flatter. Soon the ball rolled down only the slightest slope, with tall walls on both sides constraining its path. The ball had entered the territory of a demon, and now that demon was steering the ball according to its own nefarious ends.

This wasn’t the first time the ball had entered the territory of a demon. In early times, the demons had just been bumps which happened to grow alongside the ball’s path, for a time - chance events, nothing more. But every now and then, two bumps in close proximity would push the ball in different directions. The ball would roll on, oblivious, and end up going in one direction or the other. Whichever bump had "won" would continue to steer the ball's trajectory - and so a selection process occurred. The ball tended to roll alongside bumps which more effectively controlled its trajectory - bumps which were taller, bumps which steered it away from competing bumps. And so, over time, bumps gave way to barriers, and barriers gave way to demons - twisty paths with high walls to keep the ball contained and avoid competing walls, slowing the ball's descent to a crawl, conserving its potential energy in case a sharp drop were needed to avoid a competitor's wall.

The ball’s downhill progress slowed and slowed. Even though the rich, high-dimensional space was filled with lower points to explore, the highly effective demons had built tall walls to carefully contain the ball within their own territory, drawing out its travels indefinitely.

The Pattern

This tale visualizes a pattern:

There is some optimization process - in this case, some variant of gradient descent.
The optimizing search is imperfect: gradient descent only looks

...
Will COVID-19 survivors suffer lasting disability at a high rate?
by jimrandomh — 2020-02-11T20:23:50.664Z

The case fatality rate of 2019-nCoV (aka Coronavirus, COVID-19) is still uncertain, with estimates floating around ranging from

0.16%-5.7%

, higher among the elderly and people with preexisting conditions, and lower among everyone else. However, the death rate doesn't capture all of the harms from infection. There is also time lost during the infection and recovery, there is the possibility of accelerated aging, and there is the possibility of long-term nonfatal disability, such as chronic fatigue.

Since 2019-nCoV has only existed for about two months, there is no data on the long-term outcomes of its survivors. However, the rate of lasting disability among survivors is important for deciding what responses are appropriate. I'm particularly interested in estimating the risk of chronic fatigue from nCoV infection. If that risk is high, this would greatly increase the importance of avoiding it personally and of suppressing it in communities of people doing important work, and would also greatly increase the expected economic impact.

As a starting point, I chose a similar but more severe virus, SARS, which was successfully contained in 2003. Out of 208 Canadian survivors of SARS, 22 (10%) appear in

this study

of subjects "who remained unable to return to their former occupation" with "clinical similarities to patients with fibromyalgia syndrome". This implies a high lower bound on the rate of disability among SARS survivors. However, this is only one virus, and may not be representative of severe respiratory illnesses.

Good answers to this question would be:

Papers estimating the rates of postviral fatigue from other viruses, especially respiratory viruses, viruses with severity comparable to 2019-nCoV, and among non-elderly patients
Models of postviral fatigue and how they relate to 2019-nCoV
Data on whether and how much lung damage from non-viral sources causes chronic fatigue
Early data on 2019-nCoV which bears on this question

Any research help on this question is greatly appreciated, even if it provides only a bit of information about a small corner of the problem, or reports that a strategy for answering the question failed to pan out.
The Relational Stance
by Raemon — 2020-02-11T05:16:06.900Z

There's a concept that's pretty core to my understanding of relationships, which I'm sure must have been written up somewhere but I don't know of a good explanation. Here is a first pass.

The concept is, well, 'relating' – the meaning that you ascribe to a relationship, and the stance from which you are oriented towards it.

I think relating is particularly important when you choose that meaning deliberately.

The Coworker

If you have a coworker you who make jokes with, and get reasonable amounts of work done with, that... could just be some random acquaintance who you pass the time with...

Or it could be someone you choose to make a larger part of your life (a friend, or trusted colleague, that you would help out in time of need, or go to bat for).

Or it could be someone that you don't expect to go to bat for or help in crisis... but nonetheless, when you reflect upon them, you decide "this person is an important part of my life. If I left this job I wouldn't stay in touch with them, our relationship is ephemeral. But I might still ascribe importance to them being part of my life, right now. Their jokes are silly but something about them makes life better in a way I deeply appreciate. The little kindnesses, or competences, that they demonstrate at work are meaningful to me."

There are many ways you could choose to relate to a person, independent of what kind of activities you do together and commitments you've made.

The Romantic Partner

The original context I got this idea from was some relationship advice book I stumbled upon as a teenager, which said "Love is not an emotion. Love it a choice."

Sometime later, I fell in love for the first time. Over the course of a few months, I noticed a few different things going on in my head:

I had a crush on them, i.e. feelings of limerence and infatuation
I had some kind of strong and more significant-seeming feelings, that I thought might be love.
Several months later, I found that although the limerence feelings came and went without warning, and the stronger #2 feelings came and went on slower timescales (weeks or months), there some other aspect of my orientation towards them that didn't go away at all. And it wasn't a feeling, it was something different. There was a particular way I cared about them, felt they were a good person, wanted them to be part of my life, wanted them to succeed and be happy even if I wasn'

...
Intelligence without causality
by Donald Hobson — 2020-02-11T00:34:28.740Z

TL;DR
In this work, I explain how a mind can function in a universe devoid of cause and effect. I show that such a design can accommodate arbitrary restrictions that the laws of physics might happen to place on its outputs.
I originally decided to write this post when I decided that I wanted to refer to this idea within something else that I intend to write, and then realized that I couldn't think of anywhere that the whole idea was clearly expressed.
Imagine a acausal universe, as described in https://www.lesswrong.com/posts/o5F2p3krzT4JgzqQc/causal-universes (The rest of this post probably won't make sense if you haven't read and understood this first)
What kind of mind design would work in such a universe. An acausal mind doesn't have separate inputs and outputs, it just has acausal connections, which for the purposes of this post, I call aputs.
An aput is a correlation between some part of the external world and the AI algorithm that is enforced by the laws of physics. Lets assume that all the aputs are binary, true or false. Let B={0,1} The AI is then a black box that is consistent with some but not necessarily all possible aputs. In terms of the exponentially big computer simulating the universe, the AI has a set of aputs a1...an and the AI has an action set S. S⊂Bn. The computer simulating the universe searches through all possible bitstrings. If the subset of bits that consist of the AI's aputs is not in S, then the bitstring is discarded. Of course, the AI might not be a black box to the physics of the universe. The AI might contain some internal variables, with multiple consistency checks relating them to the surroundings. The AI might not be guaranteed existence at all. If the state of the world where the AI is never turned on is self consistent, there isn't anything the AI can do about it.
Of course, sometimes the laws of physics put constraints on S. These physical constraints can be modeled by forcing the AI to pick S∈C for some set C determined by the laws of physics.
Some physically relevant constraints on C include causality.

C={S⊂Bn:∀(a1,...,an)∈S ∀x∈Bm∃1y∈Bn−mst(a1,...am)=x⟹(am+1,...,an)=y}

for fixed m<n. (∃1 means there exists a unique value)
If we divide the aputs into blocks (i1,o1,i2,o2,...im,om) for easier notation, then...
Welcome to Effective Altruism Melbourne [Edit With Your Details]
by kierenwatkins — 2020-02-10T21:53:10.569Z

(The following are our suggestions for what kind of information is best to include in the welcome post of your group, feel free to replace them with whatever you think is best)

What kind of events does your group usually run? What does it usually do?

How frequently does your group organize events or meet?

Who would be a good fit for you group?

Should they have any particular skills or have done some specific background reading?
Simulation of technological progress (work in progress)
by Daniel Kokotajlo — 2020-02-10T20:39:34.620Z

I've made a model/simulation of technological progress, that you can download and run on your laptop.

My goal is to learn something about intelligence explosions, takeoff speeds, discontinuities, human-level milestones, AGI vs. tools, bottlenecks, or something else. I'll be happy if I can learn something about even one of these things, even if it's just a minor update and not anything close to conclusive.

So far I've just got a very basic version of the model built. It works, but it's currently unclear what--if anything--we can learn from it. I need to think more about whether the assumptions it uses are realistic, and I need to explore the space of parameter settings more systematically.

I'm posting it here to get feedback on the basic idea, and maybe also on the model so far if people want to download it and play around. I'm particularly interested in evidence/arguments about whether or not this is a productive use of my time, and arguments that some hidden assumption my model makes is problematically determining the results.

If you want to try out the model yourself, download NetLogo here and then open the file in this folder.

How the model works:

The main part of the model consists of research projects, which are lists of various types of task. Civilization completes tasks to complete research projects, and when projects get finished, civilization gets a "bonus" which allows it to do new types of task, and to do some old types faster.

The projects, the lists of tasks needed to complete them, the speeds at which civilization can do the tasks, and the bonuses granted by completing projects are all randomly generated, typically using exponential distributions and often with parameters you can change in the UI. Other important parameters can be changed in the UI also, such as how many task types are "off limits" for technological improvement, and how many task types are "temporarily off limits" until some specified level of technology is reached.

As explained so far, the model represents better technology leading to more research directions (more types of task become available) and faster progress (civilization can do tasks in less time).

Projects are displayed as dots/stars which flicker as work is done on them. When they complete, they turn into big green circles. Their location in the display repre...
Why do we refuse to take action claiming our impact would be too small?
by hookdump — 2020-02-10T19:33:02.632Z

I've seen this often in problems like climate change or animal exploitation:

"The solution is up to others. The powerful. The governments. The policy makers."

In this way people frequently delegate their share of responsibility to more powerful or visible entities.

To illustrate with an hypothetical example: If we suddenly found out that mobile phone frequencies destroy the planet, instead of stopping using them, many people would say:

"My actions won't make any difference. Instead it's up to the government to ban cell phones. Why should I be the fool that starts sacrificing, while everybody else keeps enjoying cell phones?"

But the only reason the government needs to ban cell phones is that the world is full of irresponsible people who need to be coerced into doing the right thing!

Does this phenomenon have a name? Does anybody here know the underlying psychological mechanism? Is it a genuine blindness about the sea being made up of millions of small droplets? An excuse to avoid responsibility? Something else?
Gricean communication and meta-preferences
by Charlie Steiner — 2020-02-10T05:05:30.079Z

There's a bit of an analogy between communication and value learning that I've been thinking about recently.

Communication:

By "Gricean communication," I mean Paul Grice's model of communication as actions based on recursive modeling.

A classic example is me flashing my headlights to communicate that you need to check yours.

Why do I take this weird action of flashing my headlights? Well, because I expect you to check yours. But I don't expect this because I think you have some evolved, unconscious reflex to check your headlights when you see a flashing light; I expect you to model my behavior and deduce that I expect you to check your headlights.

So this simple act invokes several layers of recursive agent-shaped models:

I flash my headlights because I model you (layer 4)...
as modeling me as making a reasoned decision to flash my lights (layer 3)...
in which I model you (layer 2)...
as interpreting my actions in terms of a model (layer 1)...
where I flash my lights when yours are in an unusual state.

The layer 1 model is "Charlie thinks your lights should be checked, so he flashes his."

Layer 2 is "You know that I'm flashing my lights because I think you should check yours."

Layer 3 is "Charlie knows that you'll interpret the flash as indicating checking your lights."

Layer 4 is "You know that I'm flashing my lights because I expect you to interpret that as a signal about what I'm thinking."

If we could only reason at layer 1, we would only be capable of reasoning about other people as stimulus-response machines - things in the environment with complicated behaviors that are worth learning about, but are more or less black boxes. As we increase the depth of these recursive models of our partner in communication, we're committing ourselves to more informative priors about how we expect people to reason and behave.

(See also previous similar discussion on LW)

Now, if we wanted, we could keep adding layers in an infinite tower. But more generally, we don't need to use lots of layers in the real world, and so we don't. As Dennett puts it in an excellent paper about Gricean communication among monkeys:

A fourth-order system might want you to think it understood you to be requesting that it leave. How high can we human beings go? "In principle," forever, no doubt, b

...