trentmkelly/LessWrong-43k huggingface dataset:
Tessellating Hills: a toy model for demons in imperfect search
by DaemonicSigil — 2020-02-20T00:12:50.125Z

If you haven't already, take a look at this post by johnswentworth to understand what this is all about: https://www.lesswrong.com/posts/KnPN7ett8RszE79PH/demons-in-imperfect-search

The short version is that while systems that use perfect search, such as AIXI, have many safety problems, a whole new set of problems arises when we start creating systems that are not perfect searchers. Patterns can form that exploit the imperfect nature of the search function to perpetuate themselves. johnswentworth refers to such patterns as "demons".

After reading that post I decided to see if I could observe demon formation in a simple model: gradient descent on a not-too-complicated mathematical function. It turns out that even in this very simplistic case, demon formation can happen. Hopefully this post will give people an example of demon formation where the mechanism is simple and easy to visualize.

Model

The function we try to minimize using gradient descent is called the loss function. Here it is:

L(→x)=−x0+ϵn∑j=1xj⋅splotchj(→x)

Let me explain what some of the parts of this loss mean. Each function splotchj(→x) is periodic with period 2π in every component of →x. I decided in this case to make my splotch functions out of a few randomly chosen sine waves added together.

ϵ is chosen to be a small number so in any local region, ϵ∑nj=1xj⋅splotchj(→x) will look approximately periodic: A bunch of hills repeating over and over again with period 2π across the landscape. But over large enough distances, the relative weightings of various splotches do change. Travel a distance of 20π in the x7 direction, and splotch7 will be a larger component of the repeating pattern than it was before. This allows for selection effects.

The −x0 term means that the vector →x mainly wants to increase its x0 component. But the splotch functions can also direct its motion. A splotch function might have a kind of ridge that directs some of the x0 motion into other components. If splotch7 tends to direct motion in such a way that x7, increases, then it will be selected for, becoming stronger and stronger as time goes on.

Results

I used ordinary gradient descent, with a constant step size, and with a bit of random noise added in. Figure 1 shows the value of x0 as a function of time, while figure 2 shows the values of x1,x2,…x16 as a function of time.

Fig 1:

Fig 2:

There are three phases to the evolution: In t

...
EA Kansas City planning meetup, discussion & open questions
by samstowers — 2020-02-19T23:12:21.968Z

Hi all!

We had a meeting planning the future of the Effective Altruism Kansas City group. We discussed:

Group values and goals
What we as a (currently small) local group could do to maximize our positive impact
Communication structure for internal and external communications, individual responsibilities
How we should direct members' desire for action
Whether it makes sense to spend any time/resources on local charities

Some discussion on each below.

Group values and a vision

Up to this point, we've only really done discussion groups and a few workshops - nothing in particular we did really involved action on behalf of the group (with the exception of we had at least one person sign up for the GWWC pledge). People have been asking where the group is going and when they can do something about the problems we've been talking about.

Actions we could take to maximize positive impact

A problem facing Effective Altruists is that the good we do can feel very distant - donating some portion of your salary isn't sexy or salient. So in terms of action itself, doing something like volunteering at a food pantry may not be as impactful as donating the equivalent in salary to an effective charity. However, such action could be argued to:

Keep interest in altruism alive and well by staying connected to the community
Signal boost Effective Altruism ideas both by getting the word out and signalling virtue in a way people understand

Additionally, applying EA tools in a local context (impact evaluations, searching for the biggest problems and most effective solutions) can be a good introduction for people who haven't asked these classic questions before.

Along these lines, these are some projects we're considering taking on as a group:

Helping a group of us get practiced presenting EA ideas well, reach out and schedule presentations/workshops with other organizations to spread EA ideas and get giving pledges
Doing basic impact estimations on local charities and publishing our research, to help money that's earmarked to be spent locally be as effective as it can be (while encouraging donors to look internationally and prominently mentioning the different in impact). Think of this like a local GiveWell. We hypothesize that the impact differences between these local (US) charities will be smaller than the gap between international options, but large enough to be

...
On unfixably unsafe AGI architectures
by Steven Byrnes — 2020-02-19T21:16:19.544Z

There's loads of discussion on ways that things can go wrong as we enter the post-AGI world. I think an especially important one for guiding current research is:

Maybe we'll know how to build unfixably unsafe AGI, but can't coordinate not to do so.

As a special case, I will suggest that we might have a x-risk-level accident as the culmination of a series of larger and larger accidents.

(This is an extreme case of what John Maxwell (following Nate Soares) calls an alignment roadblock.)

I'm sure this has been discussed before, but it sometimes seems to slip through the cracks in recent discussions, where instead I sometimes an implicit assumption that x-risk-level catastrophic accidents will not happen if we have ample warning in the form of minor accidents—and thus (this theory goes) we should think only about (1) fast takeoff, (2) deceptive systems (such as Paul Christiano's "influence-seekers") that pretend to be beneficial until it's too late to stop them, (3) researchers being reckless due to race dynamics, and (4) other problems that are not "accidents" per se. But even if we avoid all those problems, and thus get ample experience in the form of minor accidents, I don't think that's necessarily enough.

1. Is there such a thing as an "unfixably unsafe AGI"?

By "unfixable", I mean that to solve the problem, we need to massively backtrack and take a different path to AGI (see Appendix) ... or that a safer AGI architecture simply doesn't exist.

By "unsafe", I mean ... well, I'm not really sure what this term should mean. Is it "less unsafe than the non-AGI status quo humanity on fast-forward" (a low bar!), or "the most safe that's technologically possible" (an almost impossibly high bar!), or some absolute metric like "<X% chance of extinction" for some X? It's your choice, readers! As your safety standards get lower, the existence of "unfixably unsafe AGI" becomes less likely, but a bigger problem if it does happen.

To keep things concrete, let's have in mind an Example failure mode: Goal instability under learning and reflection: The AGI will have an internal concept of (for example) "doing what the human overseer would want", and this concept will develop and churn as the agent develops better understanding of people and the world. (See "Ontological crisis".) At some point—according to this failure mode—the internal goals / constraints / etc. may fall out o

...
[AN #87]: What might happen as deep learning scales even further?
by Rohin Shah — 2020-02-19T18:20:01.664Z

Find all Alignment Newsletter resources

here

. In particular, you can

sign up

, or look through this

spreadsheet

of all summaries that have ever been in the newsletter. I'm always happy to hear feedback; you can send it to me by replying to this email.

Audio version

here

(may not be up yet).

Highlights

Scaling Laws for Neural Language Models

(Jared Kaplan, Sam McCandlish et al) (summarized by Nicholas): This paper empirically measures the effect of scaling model complexity, data, and computation on the cross entropy loss for neural language models. A few results that I would highlight are:

Performance depends strongly on scale, weakly on model shape: Loss depends more strongly on the number of parameters, the size of the dataset, and the amount of compute used for training than on architecture hyperparameters.

Smooth power laws: All three of these show power-law relationships that don’t flatten out even at the highest performance they reached.

Sample efficiency: Larger models are more efficient than small models in both compute and data. For maximum computation efficiency, it is better to train large models and stop before convergence.

There are lots of other interesting conclusions in the paper not included here; section 1.1 provides a very nice one page summary of these conclusions, which I'd recommend you read for more information.

Nicholas's opinion: This paper makes me very optimistic about improvements in language modelling; the consistency of the power law implies that language models can continue to improve just by increasing data, compute, and model size. However, I would be wary of generalizing these findings to make any claims about AGI, or even other narrow fields of AI. As they note in the paper, it would be interesting to see if similar results hold in other domains such as vision, audio processing, or RL.

A Constructive Prediction of the Generalization Error Across Scales

(Jonathan S. Rosenfeld et al) (summarized by Rohin): This earlier paper also explicitly studies the relationship of test error to various inputs, on language models and image classification (the previous paper studied only language models). The conclusions agree with the previous paper quite well: it finds that smooth power laws are very good predictors for the influence of dataset size and model capacity. (It fixed the amount of compute, and so did not investigate...
Training Regime Day 5: TAPs
by Mark Xu — 2020-02-19T18:11:05.649Z

Introduction

TAP stands for Trigger Action Pattern/Planning. "Pattern" is a descriptive term, i.e. it's describing what's actually happening. "Planning" is what I hope to teach you how to do, i.e. how to create certain Trigger Action Patterns in yourself.

In the literature, these are sometimes called "implementation intentions", but that is a dumb name so we're not going to call it that. These have been studied pretty thoroughly and are probably one of the techniques that CFAR teaches that has the most rigorous evidence that demonstrates effectiveness.

In my view, the reason that TAPs work is because your brain doesn't really like making decisions. Making decisions is extremely costly, so, in a given situation, your brain will usually resort to doing the thing that it usually does in this situation. The idea of "installing" a TAP in yourself is to convince your brain that the action that you want it to do is also the action that you usually do in this situation.

You can think of TAPs as basically building habits. If you've read the power of habit, this general process will be very familiar to you. Today, I'll be covering how to give yourself habits that you do want; and maybe will cover how to get rid of habits that you don't want in TAPs 2 (maybe).

Steps

The steps of installing a TAP in yourself are pretty simple:

Pick an action

Pick a trigger

Practice

Picking an action

It is crucial that the action is specific. It should abundantly clear to you if the action has happened yet. If the answer could ever reasonably be "maybe," then you should pick a more specific action.

"Calm down" is not a good action because it is hard to tell if you are calm. "Take 5 deep breaths" is a much better action because it is easy to count breaths. However, it might be difficult to figure out if a breath is deep. "Breath in for 7 seconds and breathe out for 7 seconds 5 times" is a very good action because it is very clear when you have done it.

Importantly, the action should be something that actually advances your goals. The point is not to have a bunch of TAPs, the point is to get the things that you actually want. Sometimes, the process of making your action specific enough to satisfy the specificity requirement, the action no longer advances your goals. In this case, you should either choose a new action or conclude that a TAP might not be a good way to solve this problem

The act

...
Does donating to EA make sense in light of the mere addition paradox ?
by George3d6 — 2020-02-19T14:14:51.569Z

TL;DR Assuming that utilitarianism as a moral philosophy has flaws along the way, such that one can follow it but only to some unkown extent, what would be the moral imperative for donating to EA ?

I've not really found a discussion around this on the internet so I wonder if someone around here as thought about it.

It seems to me that EA makes perfect sense in light of a utilitarian view of morals. But it seems that a utilitarian view of morals is pretty shaky in that, if you follow it to it's conclusion you get to the merge addition paradox or the "utility monster".

So in light of that does donating to an EA organisation (as in: one that tries to save or improve as many lives as possible for as little money as possible) really make any sense ?

I can see it intuitively making sense, but barring a comprehensive moral system that can argue for the value of all human life, it sems intuition is not enough. As in, it also intuitively make sense to put 10% of your income into low-yield bonds, so in case one of your family members or friends has a horrible (deadly or severely life-quality diminishing) problem you can help them.

From an intuitive perspective "helping my mother/father/best-friend/pet-dog" seem to topple "helping 10 random strangers" for most people, thus it would seem that it doesn't make sense barring you are someone that's very rich and can thus safely help anyone close to him and still have some wealth leftover.

I can also see EA making sense from the perspective of other codes of ethics, but it seems like most people donating to EA don't really follow the other prescripts of those codes that the codes hold to be more valuable.

E.g.

You can argue that helping people not die is good under Christian ethics, but it's better to help them convert before they die so that they can avoid eternal punishment.
You can argue that helping people not die under a pragmatic moral system (basically a more system that's simply a descriptive version of what we've seen "works"), but at the same time, most pragmatic moral systems would probably yield the result that helping your community rather than helping strangers halfway across the globe (simply because that would have been viewed as better by most people in past and current generation, so it's probably correct).
I seems donating is in

...
Stuck Exploration
by Chris_Leong — 2020-02-19T12:31:55.276Z

Recently, we've been considering agents that instead of just being given the problem setup have to actually learn it from experience. For example, instead of being told that an agent is a perfect predictor, they just have to play until they realise this. When an agent encounters a problem like iterated Parfit's Hitchhiker, there is a weird effect that can occur during exploration if the predictor is perfect.

We will turn Parfit's Hitchiker into an iterated problem by changing being left in the desert to a large utility penalty, rather than just dying. Suppose an agent has decided that next time it is in town that it won't pay. As long as it is committed to this, it'll never actually end up in town and so it'll never actually fulfil this commitment. It'll always be predicted to defect and so will never get the chance. We will call this stuck exploration because the exploration never resolves.

One trivial way around this is to never commit based on the next time, but instead control exploration by a pseudo-random variable based on the time. However, avoiding Stuck Exploration doesn't necessarily mean that the agent will necessarily end up understanding the situation. The agent will notice that it always pays in town; ie. that it never takes the explore option when there. An EDT agent would be able to figure out that it is likely being predicted, while a dualistic agent like AIXI wouldn't be able to directly make this connection. Of course, AIXI would notice that something weird was happening and this would distort the model of the algorithm it chooses in some way. Perhaps it would a spurious link, but we won't try to delve into this at this stage.

This post was a result of discussions with Davide Zagami and supported by the EA Hotel and AI Safety Research Program. Thanks to Pablo Moreno and Luke Miles for feedback.
Does anyone have a recommended resource about the research on behavioral conditioning, reinforcement, and shaping?
by Eli Tyre — 2020-02-19T03:58:05.484Z

In animals or in humans. I was reading this post, and the linked paper, earlier today, and would like to know more of the science in this area.

A textbook would be great.

Individual papers, or anthologies, or youtube videos, or blog posts, or whatever, are also welcome.

Thanks,

Eli
Assembling Sets for Contra
by jefftk — 2020-02-19T03:40:02.164Z

Contra dances usually run about eight minutes, which is really kind of
long. [1] To keep things interesting, it's common for bands to play
sets: a series of tunes that go together. But what does "go together"
mean, and how do you make your own sets?

The first consideration, and it's a big one, is that the tunes in your
set all need to work for the same dances. If you make a set from one
tune that works well for repeated A-part balances and another tune
that wants repeated B-part balances, the tune is not going to match the dance at least half
the time. I recommend Andrew VanNorstrand's excellent Musician's
Guide to Contra Choreography for a lot of ideas on how to think
about playing for dancers, including how to think about what sort of
music supports what sort of figures.

When putting together sets you want to have a range of options so that
when the caller asks for "a really smooth pretty one, but with a
balance at the top of the B" you have something you can pull out.
Over time you'll get a sense of what callers tend to ask for, but a
good starting place for a band about to play a standard 3hr 11-12
dance evening might be ~14 sets:

One opening set to start the night with, probably reels, with very
clear good phrasing

Two smooth marchy sets

Two or three driving reel sets

Two or three chunky reel sets

One reel set that's chunky in the A part and smooth in the B

One reel set that's chunky in the B part and smooth in the A

One or two energetic sets that feel different: rags, quebecois,
bouncy jigs

One or two smooth pretty jig sets

One or two groovy jig sets

This is not at all a rule, just my guess about what I would want to
have prepared.

The second consideration is how the tunes relate to each other. This
includes energy level, apparent tempo, key, mode, etc. A very common
(and effective!) story to tell with tunes is one of increasing energy:
the tune change brings lift and excitement. This usually means
moving in the direction of more notes, higher keys (G to A, not A to
G), greater intensity.

Telling the opposite story well is harder, though possible. You can
build excitement, and then drop down at the tune change. When I've
heard bands do this well, sometimes it has felt like the hall is
taking a sigh, stretching and relaxing, as the music opens up and
takes a breath. Other good versions have involved replacing
exuberan...
Is there an intuitive way to explain how much better superforecasters are than regular forecasters?
by William_S — 2020-02-19T01:07:52.394Z

Is there an intuitive way to explain how much better superforecasters are than regular forecasters? (I can look at the tables in https://www.researchgate.net/publication/277087515_Identifying_and_Cultivating_Superforecasters_as_a_Method_of_Improving_Probabilistic_Predictions but I don't have an intuitive understanding of what brier scores mean, so I'm not sure what to think about it).
We Want MoR (HPMOR Discussion Podcast) Completes Book One
by moridinamael — 2020-02-19T00:34:15.864Z

The We Want MoR podcast, a chapter by chapter read-through and discussion of Harry Potter and the Methods of Rationality by a Rationalist and a newbie, has just finished up Book One (Ch. 1-21) of HPMOR. (Apple Podcasts link.)

Less Wrong used to be a major locus of discussion for HPMOR, so I thought it made sense to share this here, especially now that the show has passed this milestone. If folks don't necessarily want to keep HPMOR content on Less Wrong, there are ongoing lively discussions of the weekly episodes on the r/hpmor subreddit.

Disclosure, I was the guest on this episode.
Is there software for goal factoring?
by philip_b — 2020-02-18T19:55:37.764Z

Is there software for goal factoring (the CFAR technique)? I want to use it to create a directed graph, which is not necessarily a tree and may have cycles (this requirement disqualifies mind mapping software). Nodes are goals, they have attached text. There's an edge from goal x to goal y iff fulfillment of x directly contributes to fulfillment of y. There should be an easy way to see the graph and modify it, preferrably in a visual way.
What are information hazards?
by MichaelA — 2020-02-18T19:34:01.706Z

This post was written for Convergence Analysis. Cross-posted to the EA Forum.

The concept of information hazards is highly relevant to many efforts to do good in the world (particularly, but not only, from the perspective of reducing existential risks). I’m thus glad that many effective altruists and rationalists seem to know of, and refer to, this concept. However, it also seems that:

People referring to the concept often don’t clearly define/explain it
Many people (quite understandably) haven’t read Nick Bostrom’s original (34 page) paper on the subject
Some people misunderstand or misuse the term “information hazards” in certain ways

Thus, this post seeks to summarise, clarify, and dispel misconceptions about the concept of information hazards. It doesn’t present any new ideas of my own.

In his paper, Bostrom defines an information hazard as:

A risk that arises from the dissemination or the potential dissemination of (true) information that may cause harm or enable some agent to cause harm.

He emphasises that this concept is just about how true information can cause harm, not how false information can cause harm (which is typically a more obvious possibility).

Bostrom’s paper outlines many different types of information hazards, and gives examples of each. The first two types listed are the following:

Data hazard: Specific data, such as the genetic sequence of a lethal pathogen or a blueprint for making a thermonuclear weapon, if disseminated, create risk.

[...] Idea hazard: A general idea, if disseminated, creates a risk, even without a data-rich detailed specification.

For example, the idea of using a fission reaction to create a bomb, or the idea of culturing bacteria in a growth medium with an antibiotic gradient to evolve antibiotic resistance, may be all the guidance a suitably prepared developer requires; the details can be figured out. Sometimes the mere demonstration that something (such as a nuclear bomb) is possible provides valuable information which can increase the likelihood that some agent will successfully set out to replicate the achievement.

He further writes:

Even if the relevant ideas and data are already “known”, and published in the open literature, an increased risk may nonetheless be created by drawing attention to a particularly potent possibility.

This leads him to...