trentmkelly/LessWrong-43k huggingface dataset:
Will AI undergo discontinuous progress?
by Sammy Martin — 2020-02-21T22:16:59.424Z

This post grew out of conversations with several people, including

Daniel Kokotajlo

,

grue_slinky

and

Linda Lisefors

, and is based in large part on a collection of scattered comments and blog-posts across lesswrong, along with some podcast interviews - e.g.

here

. The in-text links near quotes will take you to my sources.

I am attempting to distinguish two possibilities which are often run together - that progress in AI towards AGI (‘takeoff’) will be discontinuous and that it will be fast, but continuous. Resolving this distinction also addresses the claim that there has been a significant shift in arguments for AI presenting an existential risk: from older arguments discussing an ultra-fast intelligence explosion occurring in a single ‘seed AI’ to more moderate scenarios.

I argue that the ‘shift in arguments on AI safety’ is not a total change in basic assumptions (which some observers have claimed) but just a reduction in confidence about a specifically discontinuous takeoff. Finally, I try to explicitly operationalize the practical differences between discontinuous takeoff and fast, continuous takeoff.

Further Reading

Summary: Why AI risk might be solved without additional intervention from Longtermists

Paul Christiano’s original post

MIRIs Thoughts on Discontinuous takeoff

Misconceptions about continuous takeoff

AI Impacts original post

Soft Takeoff can still lead to Decisive Strategic Advantage

Defining Discontinuous Progress

What do I mean by ‘discontinuous’? If we were to graph world GDP over the last 10,000 years, it fits onto a

hyperbolic growth pattern

. We could call this ‘continuous’ since it is following a single trend, or we could call it ‘discontinuous’ because, on the scale of millennia, the industrial revolution exploded out of nowhere. I will call these sorts of hyperbolic trends ‘continuous, but fast’, in line with Paul Christiano, who argued for continuous takeoff, defining it this way:

AI is just another, faster step in the

hyperbolic growth we are currently experiencing

, which corresponds to a further increase in rate but not a discontinuity (or even a discontinuity in rate).

I’ll be using Paul’s understanding of ‘discontinuous’ and ‘fast’ here. For progress in AI to be discontinuous, we need a switch to a ...
Theory and Data as Constraints
by johnswentworth — 2020-02-21T22:00:00.783Z

There’s a widely-known legend about statistician Abraham Wald’s work on planes during WWII (the veracity of the legend is examined

here

). As the story goes, the military collected data on planes coming back from missions, marking the location of any bullet holes. They soon had statistics showing how many had been hit on the engine, the fuel system, etc. Based on this data, commanders wanted to add extra armor to reinforce the areas which were most often hit.

Wald, however, suggested adding extra armor to the places which were least often hit. His reasoning: planes hit in those areas were the planes which didn’t come back.

The moral of the tale: useful insights do not come from data alone. Background knowledge, interpretation and model-building - all of which we’ll bundle into the word “theory” - are necessary elements as well. In

the framework

of

this sequence

: theory and data are both constraints on the production of useful insights. Immediate question: how taut are each of those constraints? What’s the limiting factor?

Let’s think about the tautness of each constraint in the Wald legend. How expensive was the data, and how expensive was the theory? In this case, the data was presumably far more expensive: it required people on the ground examining hundreds of returning airplanes per mission, collecting all of that hand-written data, adding it all up by hand, and relaying it via telephone or radio or mail to the Statistical Research Group in New York City (where Wald worked). The theory, on the other hand, probably took Wald a day at most. Of course there would also be some overhead on both sides - Wald needed to write up the theory in a manner convincing to military commanders, and military commanders needed to set up the whole data-collection project - but there again, the data would have been far more expensive than the theory.

Conclusion: the data constraint was much more taut than the theory constraint. Theory was abundant relative to the scarcity of data.

… at least during WWII.

The Internet

More recently, you may have heard about a fancy new technology called “the internet” which makes it really, really cheap for people to share data. Given such a technology shift, we’d expect the data constraint to become slack much more often, leaving the theory constraint taut.

What does t...
Training Regime Day 7: Goal Factoring
by Mark Xu — 2020-02-21T17:55:29.848Z

Introduction

Humans have these things that we call goals. They also take these things called actions. Ideally, a human takes actions that move the world closer to one of the human's goals. You want your actions to be connected to your goals in this sense.

Usually, when an action is first introduced into your action repertoire, it has a very clear connection to your goal. However, once an action is introduced, it often tends to stick around. Sometimes, the action-goal connection doesn't change, and so continuing to take that action is a good idea. Other times, the action-goal connection shifts, but you still continue to take that action.

The purpose of goal factoring is two fold. The first purpose is to evaluate the strength of the action goal connection for any given action. The second purpose is to find actions that have a stronger connection to your goals.

Cleaning your closet

Closets can have many things inside them. The closet I will discuss today contains all of your goals. However, you can't just put stuff in a closet willy nilly, it must be organized in boxes. The boxes that you put your goals in are your actions.

We proceed as if we are cleaning a regular closet. First, we pick a box. Then we open the box. Sometimes, we realize that the box is way too big for the thing inside it, so we find a smaller box. Sometimes, we don't even want the stuff, so we throw it away. Other times, we find that the stuff fits pretty well inside the box, so we leave it as it is. Goal factoring is the process of opening the action-box to find the goals inside.

In terms of action-goal connection, the first case corresponds to a case where the action is an ineffective way to achieve the goal. For example, suppose my goal is to stay up to date with current events and my action is browsing Reddit for hours a day. It seems evident that there are ways that will keep me both better informed about current events and take less time than browsing Reddit.

The second case corresponds to goals shifting as time passes. For example, I used to value having up-to-date information about Magic: the Gathering, resulting in me checking various MtG websites quite frequently. However, recently I discovered that I no longer really cared what was happening in the MtG world, yet I continued to check the websites.

The third case is when the action-goal connection remains strong. For example, I have a goa

...
You are an optimizer. Act like it!
by agentydragon — 2020-02-20T23:46:30.240Z

In the long run, optimizers win. So act like an optimizer.
Optimizers use all available resources to take optimal decisions.
Optimizers are motivated to have beliefs that correspond to reality, because they are needed as inputs for the function that determines the action.
If you feel something is true, it's not the same thing as believing it's true.
Don't do something just because you feel it's the right thing.
Do it if you believe it to be the correct thing to do.
Not if you feel it. If you believe it.
Don't make the decision based on what your S1 alone is telling you.
(Sure, S1 is also good for some stuff but you would not use it to correctly solve x^2 - 64 = 0.)
You are always in control of your actions.
When you, the optimizer, don't move the body (e.g., binging etc.), you have taken an action that caused the connection from your beliefs to your actions to be cut.
That does not mean you don't always have control of your actions.
You are a subprogram running on a smartass monkey.
Sometimes the CPU executes you, sometimes it doesn't.
Some conditions cause you to get executed more, and move the monkey.
Some conditions cause another program to execute.
These conditions can be affected by the monkey's actions.
And when you are able to exert influence over the monkey's body, you can attempt to choose such monkey actions that optimize the probability you will be able to reach your goals.
And if your goals require you (and not some other process) taking actions over the monkey, you attempt to get yourself scheduled.
(Of course some processes might be best left to some other program, although that's said with a lot of uncertainty and remains to be proven.)
(At least, execution of some other subagent might be good for monkey happiness, which might be needed as prevention of interruptions from high-priority "hunger, sadness, ..." monkey processes.)
S1 can be used as an interface for talking with other monkey processes. Yep, feels good. I have at least some monkey subagents agreeing on this being a good idea.
Okay, just lost control for a while. Let's make this a post and cross [interrupt]
It's some social process interrupting.
Interrupting...
Interruptions can be stopped. I should do that at some point, like disabling notifications.

... cross my fingers it will cause more schedulings. I will need to think about what...
Goal-directed = Model-based RL?
by adamShimi — 2020-02-20T19:13:51.342Z

Epistemic Status: quick write-up, in reaction to a serendipitous encounter with an idea. I see the main value of this post as decently presenting a potentially interesting take on a concept in AI safety to the community.

While skimming my copy of Reinforcement Learning: an introduction for the part on AlphaGo Zero, I found a section called Habitual and Goal-directed behavior. That caught my attention, because one idea I keep going back to is goal-directed behavior from the Value Learning Sequence; when studying the sequence, I was intrigued by the idea. But the lack of formalization made me uncertain about my position on the argument, that not all useful (and possibly superintelligent) agents have to be goal-directed.

Going back to my serendipitous discovery, the section is part of the Psychology chapter: it compares model-based and model-free RL to goal-directed and habitual behavior in psychology. To clarify the RL terms used here:

Model-based RL is the version of RL where the agents learns both from direct experience with the environment and from simulated experience with a model, which entails that it builds and updates a model of the environment.
Model-free RL is the version of RL where the agents only learns from direct experience with the environment. Usually, the only thing available to the agent is its value function or its policy; there is no model of the environment.

As for the different behaviors, let's quote the book itself:

Goal-directed behavior, according to how psychologists use the phrase, is purposeful in the sense that it is controlled by knowledge of the value of goals and the relationship between actions and their consequences.

and

Habits are behavior patterns triggered by appropriate stimuli and then performed more-or-less automatically.

There is also a summary:

Habits are sometimes said to be controlled by antecedent stimuli, whereas goal-directed behavior is said to be controlled by its consequences

Now, is this version of goal-directed behavior linked to the version that Rohin Shah wrote about? I think so. They are probably not the same, but examining the similarities and differences might clarify the part relevant to AI safety.

Comparison between the two versions of goal-directed behavior

Similarities

In intuitions about goal-directed behavior, the first example of goal-directed behavior vs non-goal directed behavior concerns policies for ...
Training Regime Day 6: Seeking Sense
by Mark Xu — 2020-02-20T17:33:29.011Z

Introduction

Seeking sense is very easy to describe and very hard to actually do in practice. The basis of this technique is a claim is that the things that go on in other people's heads make sense from their perspective and that it's valuable to try to figure out what's happening in their brains. (Other people know things too!)

We can divide this technique into two parts: the skill and the attitude.

A seeking sense attitude is approaching other brains in a way that makes you curious about how they work. When you observe someone doing something that ranks very low according to your values, instead of dismissing them as bad, you find yourself very curious as to what process is making them take that action. This notion is partially captured by words like "empathy" or "compassion". To me, it feels like the seeking sense attitude means I have to treat other people like human beings instead of just action/reaction APIs.

The seeking sense skill is the ability to actually resolve the curiosity generated by the seeking sense attitude. In some sense, this is the rationality version of having social skills. Having this ability allows you to observe people's actions and infer things about their values, how they think, what they know about the world, etc.

Both of these things are related to boggling at other people's brains.

Seeking Sense

The first thing that you must accept in order to seek sense properly is the claim that minds actually make sense. This sounds extremely reasonable, but often, I find that it's difficult to accept that that a given mind might make sense. For example, people who seem to be taking actions that aren't optimizing for anything confuse me a lot; it takes a strong perspective shift for me to get into the mode where I think what they're doing might make sense.

Something that might help is realizing that your actions might not make sense from other people's perspectives. When I imagine these people, I find myself compelled to say "Well, actually, my actions make perfect sense because of X, Y and Z." I imagine myself doing this because I think I have a decent understanding of why I do the things I do. In many cases, however, people don't quite understand their own minds well enough to offer clear explanations. This makes seeking sense difficult because you can’t just ask them to explain. Nevertheless, you can imagine what sorts of explanations that t

...
Curiosity Killed the Cat and the Asymptotically Optimal Agent
by michaelcohen — 2020-02-20T17:28:41.955Z

Here's a new(ish) paper that I worked on with Marcus Hutter.

If an agent is exploring enough to be guaranteed strong performance in the limit, that much exploration is enough to kill it (if the environment is minimally difficult and dangerous). It's nothing too surprising, but if you're making a claim along these lines about exploration being dangerous, and you need something to cite, this might work.

My attitude towards safe exploration is: exploration isn't safe. Don't do it. Have a person or some trusted entity do it for you. The paper can also be read as a justification of that view.

Obviously, there are many more details in the paper.
How do you survive in the humanities?
by polymathwannabe — 2020-02-20T15:19:10.598Z

Two years ago, I fulfilled a huge part of my dream of doing what I love and started studying creative writing full time. This program is run by a humanities department, so I knew I'd have to stomach epistemic nonsense from time to time, which most of the time I can do while reminding myself that in the end it's the diploma and the connections that I care about.

But some days are harder than others, and every time a teacher says something plainly wrong, I can't seem to decide between the cost of derailing the class and the cost of letting my classmates walk home with a wrong idea. For example, in first semester I had a long discussion with a teacher about questions in general; he didn't believe that questions are supposed to get answered, that debates are supposed to be settled. I tried my best to make the case that a long-standing debate managed in such a way that it never ends is a very poorly managed debate, but he thought that was the whole point of a debate. This was the same teacher who later said that an essay could make strong points with impeccable arguments and solid evidence and still he had the essential freedom to not believe it. Last year, a small disagreement over the interpretation of a poem got seriously derailed to the definition of life, and the teacher started saying one badly constructed modus ponens after another, and I didn't know whether to address each argument or to simply point out he didn't know how to make an argument and should never have been given a teaching position. Last week, a teacher recited the standard clichés against logical positivism and claimed that science was oblivious to the suffering of the poor; upon further probing, it turned out he was confusing positivism with optimism. I wasted half an hour trying to explain his error, to no avail. At one point he uttered the phrase "I prefer to believe..." and it was impossible to make him see how irresponsible such a statement is. Yesterday, a teacher was explaining the composition of a literary essay, and she claimed that an essay writer isn't required to provide justification for their claims. I asked, "Then why should I believe anything the essay says?" and she replied, "You're free to decide whether you believe it or not," and I was just too exhausted from last week to explain that's not how...
Making Sense of Coronavirus Stats
by jmh — 2020-02-20T15:12:51.292Z

[Update: Some have pointed out the definition of mortality rate should be that of deaths to some defined population, typically a median estimate over a period of time (week, month, year...) and not limited only to those infected. (New update here. The ratios I have been considering are called Case Fatality Rate.)

It was also correctly pointed out that my second calculation was simply wrong. The ration should not be deaths/recovered but deaths/(deaths+recovered) -- that is deaths to the total population considered. ]

From what I've seen WHO and other health organizations are saying the mortality rate for the new coronavirus outbreak is between 2 and 3 percent. That seems to be based on the ratio of the reported deaths to the reported cases of infection.

That doesn't seem right to me.

The last statistics I looked at (news report) was:

Total - 75,768

Recovered - 16,329 (21.6%)

Deaths - 2129 (2.8%)

However, that leave us with a bit over 75% with an unknown end state.

If I try to infer the outcome for all the reported infections I can think of two ways to estimate the end results. One, is to assuming the remaining cases will produce a similar outcome as has been observed so far. Using that assumption I can then iterate through the unknown cases using the 21.6% percent will recover, 2.8% will die and ~75% will move to the next round.

The other way would be to assume the ratio of the current deaths to current recoveries is a good measure of the mortality rate.

Using the first approach the total deaths approaches 8,740 people from the current population of 75,768. The second approach results in a higher number, 9879. Both of these numbers would suggest this version of the coronavirus is pretty bad, in terms of mortality rates. In the context of the other two coronavirus outbreaks, it seem closer to SARS, though a bit worse, than to MERS.

Should I think this approach to estimating mortality rates for new diseases without out a long history (like the flu) might be more accurate than the standard approach that seems to be deaths/total infections. This standard approach would seem to systematically under estimate the mortality as initially one would expect infections to be rising more rapidly than deaths.

The implication is that health organizations/bureaucracies might tend to be slow to react. The early numbers are nothing to worry about, and might even show a decreasin...
Mapping downside risks and information hazards
by MichaelA — 2020-02-20T14:46:30.259Z

This post was written for Convergence Analysis.

Many altruistic actions have downside risks; they might turn out to have negative effects, or to even be negative overall. Perhaps, for example, the biosecurity research you might do could pose information hazards, the article you might write could pose memetic downside risks, or that project you might start could divert resources (such as money or attention) from more valuable things.

This means that one way to be more effective in one's altruism is to prevent or mitigate downside risks and information hazards. This post is intended to aid with that goal by clarifying the concepts of downside risks and information hazards, and how they relate to each other and to other “types” of effects (i.e., impacts). This will be done through definitions, visuals, and examples.

I believe that the ideas covered in this post are fairly well-known, and should seem fairly intuitive even for those who hadn’t come across them before. But I still think there’s value in showing visually how the concepts covered relate to each other (i.e., “mapping” them onto the space of possible types of effects), as this post does.

Positive vs negative

First I’ll present and explain the space in which I’ll later map downside risks and information hazards. This space is shown in the following figure (along with a handful of examples to illustrate the ideas[1]):

This space divides all possible effects into either positive and intended (top left), negative and intended (top right), negative and unintended (bottom right), or positive and unintended (bottom left).

In this model, an action is judged as normatively or morally “positive” and “negative” from the perspective of the person using this model in analysing, observing, or considering one or more effects. We could call this person the “analyst”. This analyst may also be the “actor”; the person who is performing, or might perform, the actions that will lead to the effects in question. But the analyst will not necessarily be the actor.

For example, if I am using this model in deciding whether I should take up a diplomatic career, then I am both the analyst and the actor. From my perspective, making great power wars less likely is a positive effect, and so I will be more likely to take up a diplomatic career if I think it will lead to that effect.

However, if I am using this model in thinking about the po...