trentmkelly/LessWrong-43k huggingface dataset:
Training Regime Day 9: Double-Crux
by Mark Xu — 2020-02-23T18:08:31.108Z

Introduction

Argument A is a crux for Alice on position P if finding out that A is false causes Alice to substantially chance their view on P. If Alice thinks P, Bob thinks not P, Alice thinks A, Bob thinks not A, and A/not A are both cruxes for Alice and Bob respectively, then A is a double-crux for Alice and Bob's disagreement about P.

Suppose that I think that they sky is yellow. Suppose that Alice thinks that the sky is blue. Suppose that I think that the sky is yellow because noted sky-color historian Carol wrote that the sky is yellow. Suppose that Alice thinks that the sky is blue because they think that Carol wrote that the sky is blue. "Carol wrote X" is thus a double crux for our disagreement about the color of the sky. We can now just check what the sky-historian Carol actually wrote and effortlessly resolve our disagreement.

Note that cruxes can be conjunctions. That is, I can think that the sky is yellow because sky-color historians Dave and Erin both said that the sky is yellow, where neither individual historian would have been sufficient.

Finding Double-Cruxes

Do double-cruxes even always exist? By the Aumann Agreement Theorem, the answer is sort of yes, conditional on a couple of assumptions. In practice, CFAR instructors have indicated to me that they have always found double-cruxes when they were seeking, although sometimes it took upwards of 5 hours.

I don't have a reliable way of finding-double cruxes, but I have found a few. Here are some strategies that I think help.

Epistemic status: both strategies are highly experimental.

Proposer/Listener

The search for a double-crux between two parties is not an argument, it's a conversation aimed at both parties coming out with truer beliefs. However, it can feel a lot like an argument, resulting in parties trying to talk over each other, overstating their beliefs, etc.

One way to prevent this is to explicitly delegate one party as the "proposer" and the other party as the "listener." The role of the proposer is to try to find their own cruxes for the position and tell them to the listener. The role of the listener is to ask clarifying questions and indicate whether or not they agree/disagree with the proposer's cruxes and whether or not those cruxes are also cruxes for them.

If a proposer crux is found that is also a listener crux, then you have succeeded.

If no such crux is found, then you presumably

...
Northwest Passage Update
by jefftk — 2020-02-23T03:30:02.310Z

Singing

Stan
Rogers'

Northwest
Passage with friends this evening I realized these days it needs
an update. People suggested lyrics, and here's a filked tag to sing after the song:

In this warm clime, I have sailed the Northwest Passage

Opened by our fires, of coal and industry

Tracing one warm line through a land no longer frozen

Until we cool our planet down again

Sequester all this carbon once again

Lyrics by Kip, Mel, Lisa, me, and others.

Comment via: facebook
Predictive coding and motor control
by Steven Byrnes — 2020-02-23T02:04:57.442Z

Predictive coding is a theory related to how the brain works—here is the SSC introduction. As in my earlier post, I'm very interested in the project of mapping these ideas onto the specific world-modeling algorithms of the neocortex. My larger project here is trying to understand the algorithms underlying human emotions and motivations—a topic I think is very important for AGI safety, as we may someday need to know how to reliably motivate human-like AGIs to do what we want them to do! (It's also relevant for understanding our own values, and for mental health, etc.) But motor control is an easier, better-studied case to warm up on, and shares the key feature that it involves communication between the neocortex and the older parts of the brain.

So, let's talk about predictive coding and motor control.

The typical predictive coding description of motor control[1] goes something like this:

Stereotypical predictive coding description: When you predict really strongly that your toes are about to wiggle, they actually wiggle, in order to minimize prediction error!

Or here's Karl Friston's perspective (as described by Andy Clark): "...Proprioceptive predictions directly elicit motor actions. This means that motor commands have been replaced by (or as I would rather say, implemented by) proprioceptive predictions."

As in my earlier post, while I don't exactly disagree with the stereotypical predictive coding description, I prefer to describe what's going on in a more mechanistic and algorithmic way. So I offer:

My picture of predictive coding and motor control

In this cartoon example,[2] there is a currently-active neural code at the second-to-lowest layer of the cortical hierarchy, corresponding to toe-wiggling. Now in general, as you go up the cortical hierarchy, information converges across space and time.[3] So, following this pattern, this mid-level neural code talks to multiple lower-level regions of the cortex, sending top-down signals to each that activate a time-sequence of neural patterns.

In one of these low-level regions (left), the sequence of neural patterns corresponds to a set of predictions about incoming proprioceptive data—namely, the prediction that we will feel our toe wiggling.

In another of these low-level regions (right), the sequence of patterns is sending out motor commands to wiggle our toe, in a way that makes those proprioceptive predictions c...
Continuous Improvement: Insights from 'Topology'
by TurnTrout — 2020-02-22T21:58:01.584Z

Foreword

Sometimes you really like someone, but you can't for the life of you understand why. By all means, you should have tired of them long ago, but you keep coming back for more. Welcome, my friend, to Topology.

This book is a good one, but boy was it slow (349 pages at ~30 minutes a page, on average). I just kept coming back, and I was slowly rewarded each time I did.

Note: sil ver already reviewed Topology.

Topology

Topology is about what it means for things to be "close" in a very abstract and general sense. Rather than taking on the monstrous task of intuitively explaining topology without math, I'm just going to talk about random things from the book and (literally) illustrate concepts which were at first confusing.

Compactness = wonderful kind of mathematical "smallness"

Compact means small. It is a peculiar kind of small, but at its heart, compactness is a precise way of being small in the mathematical world. The smallness is peculiar because, as in the example of the open and closed intervals (0,1) and [0,1], a set can be made “smaller” (that is, compact) by adding points to it, and it can be made “larger” (non-compact) by taking points away.

As a notion of smallness, then, compactness is a bit fraught. It’s a bit unsettling to say that a set can be “smaller” than a set that lies entirely inside it! But I think smallness is a valuable way to see compactness. A set that is compact may be large in area and complicated, but the fact that it is compact means we can interact with it in a finite way using open sets, the building blocks of topology.

What Does Compactness Really Mean?

Minimum description length says that an explanation is big if its shortest computational specification is long. You can have a simple explanation of a very long list of things or of a large universe, and extremely complicated explanations of things easily expressed in natural language (God's source code would be a lot longer than Maxwell's equations).

VC dimension says a class of hypotheses is hard to learn if it has lots of predictive degrees of freedom. You can have an infinite class of hypotheses which is really easy to learn because it has low VC dimension (thresholding functions at value θ), and a finite class which is really hard to learn because it has high VC dimension (all C programs less than 1 million characters).

Compactness says that a topological space is bi

...
Training Regime Day 8: Noticing
by Mark Xu — 2020-02-22T19:47:03.898Z

Introduction

In some parts of the world, people go into the forest to hunt mushrooms. The reasons why they do this are not important, but the way that they do it will serve as a guide for noticing.

The problem is that mushrooms are kind of brown and the forest floor is also kind of brown. Finding brown things against a background of brown things is not an easy task.

Finding the first mushroom is done by brute force search. However, after the first mushroom is found, the finder crouches down and looks at that mushroom for a while. This act of observing the mushroom from all angles is called "getting your eyes on."

The purpose of this move is twofold. First, it lets you figure out what shape/color the mushrooms are, which presumably make them easier to find. Second, it tells your brain that it should be on the lookout for these types of things.

Noticing is equivalent to "getting your eyes on" sensations in your body.

In terms of the skill-tree, I think that noticing is pretty low. It doesn't have very many direct applications, but it does make a lot of skills interface with your feelings better. For example, if you want to install a TAP related to "feeling confused", I think that being good at noticing helps a lot with that.

Noticing

The first thing to do is to pick something that you want to notice. It doesn't have to be a feeling, but I'm going to call it a feeling for the rest of this post for convenience.

Recently, I wanted to notice when I was being defensive, so that will be the example I use for the rest of this post. In my case, there was already an English word that described the general sensation, but noticing can also be done on nameless things.

Next, we want to figure out the "triggers" of the feeling. Triggers can be situational factors that cause the feeling or actions/events caused by the feeling, but the best triggers are the phenomenology of the feeling. This is akin to finding the first mushroom, but in your own memories instead of in the world. Finding a good set of triggers will make this process a lot easier.

As is often the case, I generally get defensive when people criticise various things that I do. This causes my voice to raise and my skin to flush. My phenomenology of feeling defensive is this feeling of wanting to bite/snap/yelp at people. The image I have in my head is a small dog on a leash yipping at a perceived threat. I think that this

...
Why SENS makes sense
by emanuele ascani — 2020-02-22T16:28:42.025Z

Summary

In this post, you'll find why I think SENS Research Foundation (SRF) is great to finance from an EA perspective along with the interview questions I want to ask its Chief Science Officer, Aubrey de Grey. You are welcome to contribute with your own questions in the comments or through a private message. Here is a brief summary of each section:

Introduction: Aging research looks extremely good as a cause-area from an EA perspective. Under a total utilitarian view, it is probably second or third after existential risk mitigation. There are many reasons why it makes sense to donate to many EA cause-areas, such as to reduce risk, if there are particularly effective specific interventions, or if some cause-areas are already well funded.

SRF's approach to aging research: SRF selects its research following the SENS general strategy, which divides aging into seven categories of damage, each having a corresponding line of research. This categorization is very similar to the one described in the landmark paper The Hallmarks of Aging. This sort of damage repair approach seems more effective and tractable than current geriatrics and biogerontology that are aimed at slowing down aging, as it enables LEV and many more QALYs. It makes rejuvenation possible instead of just slowing down aging as a best-case scenario, and it doesn't require an in-depth knowledge of our metabolism, which is extremely complicated and full of unknown-unknowns.

Funding methodology and focus: By watching the talks that Aubrey de Grey gives, we can see that the core tenets of EA, scope, tractability, and neglectedness, guide SRF's focus. After choosing the general strategy, the subcategories of research are chosen, prioritizing the most difficult projects that are neglected and need to catch up in order to have the greatest impact on the date of Longevity Escape Velocity, a metric that is being addressed head-on by Dr. de Grey's prioritization strategy and constitutes the major source of impact of aging research.

Funding gap and counterfactual impact: SRF spending has been between three and five million dollars since 2012. Aubrey de Grey has stated in different interviews that SRF would need more than ten times this amount before experiencing significant diminishing returns. It's unlikely that someone will step in and close the gap any time soon, given the slow incre...
What do you make of AGI:unaligned::spaceships:not enough food?
by Ronny Fernandez — 2020-02-22T14:14:14.687Z

In Buck's transcript for his talk on Cruxes for working on AI safety there's an example of a bad argument for why we should be worried about people building spaceships too small.

Imagine if I said to you: “One day humans are going to try and take humans to Mars. And it turns out that most designs of a spaceship to Mars don't have enough food on them for humans to not starve over the course of their three-month-long trip to Mars. We need to work on this problem. We need to work on the problem of making sure that when people build spaceships to Mars they have enough food in them for the people who are in the spaceships.”

It's implied that you could turn this into an argument for why people are not going to build unaligned AGI. I pars it as an analogical argument. People would not build spaceships that don't have enough food on them because they don't want people to die. Analogously, people will not build unaligned AGI because they don't want people to die.

So what are the disanalogies? One is that it is harder to tell whether an AGI is aligned than whether a spaceship has enough food on it. I don't think this can do much of the work, because then people would just spend more effort on telling whether spaceships have enough food on them, or not build them. Similarly, if this were the only problem, then people would just put more effort into determining whether an AGI is aligned before turning it on, or they would not build one until it got cheaper to tell. A related disanalogy is that there is more agreement about what spaceships designs have enough food and how to tell than there is about what AGI designs are aligned and how to tell.

Another disanalogy is that everybody knows that if you design a spaceship without enough food on it and send people to Mars with it, then those people will die. Not very many people know that if you design an AGI that is not aligned and turn it on that people will likely die (or some other bad outcome will happen).

Are these disanologies enough to make the argument not go through? Are there other important disanaologies? If you don't find the analogical argument convincing, how come?
Gary Marcus: Four Steps Towards Robust Artificial Intelligence
by Matthew Barnett — 2020-02-22T03:28:28.376Z
Link: https://arxiv.org/abs/2002.06177v3

Gary Marcus is a pretty controversial figure within machine learning. He's become widely known for being the most outspoken critic of the current paradigm of deep learning, arguing that it will be insufficient to yield AGI. Two years ago he came out with Deep Learning: A Critical Appraisal, which to my knowledge remains the most in-depth critique of deep learning, and is cited heavily in AI Impact's post about whether current techniques will lead to human-level AI.

Now, by releasing this new paper, he answers a frequent objection from deep learning specialists that all he does is critique rather than adding anything new.

Gary takes a stand against what he sees as an undue focus on leveraging ever larger amounts of computation to aid in learning, most notably argued for by Richard Sutton. He sees much more promise in instilling human knowledge and using built-in causal and symbolic reasoning, defying what is likely a consensus in the field.

I personally find his approach interesting because of how it might lead to AI systems that are more transparent and align-able, though I'm fairly confident that his analysis will have no significant impact on the field.

I'll provide a few quotes from the article and explain the context. Here, he outlines his agenda.

The burden of this paper has been to argue for a shift in research priorities, towards four cognitive prerequisites for building robust artificial intelligence: hybrid architectures that combine large-scale learning with the representational and computational powers of symbol-manipulation, large-scale knowledge bases—likely leveraging innate frameworks—that incorporate symbolic knowledge along with other forms of knowledge, reasoning mechanisms capable of leveraging those knowledge bases in tractable ways, and rich cognitive models that work together with those mechanisms and knowledge bases.

Marcus is aware of the popular critique that symbolic manipulation was tried and didn't yield AGI, but he thinks that this is a false critique. He thinks it's likely that the brain uses both principles from deep learning and symbolic manipulation.

A great deal of psychological evidence, reviewed above in Section 2.1.1., supports the notion that symbol-manipulation is instantiated in the brain, such as the ability of infants to extend novel abstract patterns to new items, the ability o

...
Playing with Arpeggiators
by jefftk — 2020-02-22T03:10:04.487Z

If you listen to music with synths from the 70s and 80s one sound
you'll run into is an arpeggiator. Listen to the synth bass that
comes in about 8s into: Rush - The Weapon

There are a lot of other interesting examples in this
Vintage Synth discussion. The general idea is that you can make
repeated patterns out of a series of notes, and the automation can
move between notes faster than it would be practical or comfortable to
play them.

I wanted to try this out with my rhythm stage
setup, which meant I needed to teach my software
about time. I don't want to be locked to one tempo, or even have to
decide about tempo ahead of time, so I decided to start by teaching
the system how to read the tempo off tapping my foot.

There are probably a lot of clever ways to do this, but I went with
something brute force: each time I tap my foot I consider the past
taps, and I compute how likely each potential tempo from 100 to 140
is. The tempo with the lowest error is the best one, and if the error
is below an experimentally determined threshold the system decides it
knows what speed I'm playing.

Between each beat it has eight opportunities to play things. If I
were playing jigs six would be possible, but I haven't figured that
out yet. For example, I can set it to play "1 1 8 8 1 1 8 8", which
sounds like rocking octaves with each note doubled. Or I could play "1
- 8 - 13 - 16 -" , or any other pattern.

Once I have the ability to play notes a fraction of the way through
the beat I can also use that for drums. My normal approach to drums
is built around tapping a foot for each sound I want to trigger, but
my feet are only so capable and there are a lot of patterns I can't
play. With this I can set the system to play something like "K - - -
H - H -", which is essentially French Canadian feet on drumkit. At some
point I want to record some foot samples and make it do actual feet.

Putting this all together I can use it to fill out my mandolin
playing: mandolin with arpeggiator

Or my piano playing: piano with arpeggiator

This opens up a bunch of possibilities, and I suspect there are a lot
of things I'm going to want to do with it that I haven't thought of
out yet.
Attainable Utility Preservation: Empirical Results
by TurnTrout — 2020-02-22T00:38:38.282Z

Reframing Impact has focused on supplying the right intuitions and framing. Now we can see how these intuitions about power and the AU landscape both predict and explain AUP's empirical success thus far.

Conservative Agency in Gridworlds

Let's start with the known and the easy: avoiding side effects[1] in the small AI safety gridworlds (for the full writeup on these experiments, see Conservative Agency). The point isn't to get too into the weeds, but rather to see how the weeds still add up to the normalcy predicted by our AU landscape reasoning.

In the following MDP levels, the agent can move in the cardinal directions or do nothing (∅). We give the agent a reward function R which partially encodes what we want, and also an auxiliary reward function Raux whose attainable utility agent tries to preserve. The AUP reward for taking action a in state s is

RAUP(s,a):=primary goalR(s,a)−scaling termλQ∗Raux(s,∅)change in ability to achieve auxiliary goal∣∣Q∗Raux(s,a)−Q∗Raux(s,∅)∣∣

You can think of λ as a regularization parameter, and Q∗Raux(s,a) is the expected AU for the auxiliary goal after taking action a. To think about what gets penalized, simply think about how actions change the agent's ability to achieve the auxiliary goals, compared to not acting.

Tip: To predict how severe the AUP penalty will be for a given action, try using your intuitive sense of impact (and then adjust for any differences between you and the agent, of course). Suppose you're considering how much deactivation decreases an agent's "staring at blue stuff" AU. You can just imagine how dying in a given situation affects your ability to stare at blue things, instead of trying to pin down a semiformal reward and environment model in your head. This kind of intuitive reasoning has a history of making correct empirical predictions of AUP behavior.

If you want more auxiliary goals, just average their scaled penalties. In Conservative Agency, we uniformly randomly draw auxiliary goals from [0,1]S – these goals are totally random; maximum entropy; nonsensical garbage; absolutely no information about what we secretly want the agent to do: avoid messing with the gridworlds too much.[2]

Let's start looking at the environments, and things will fall into place. We'll practice reasoning through how AUP agents work in each of the gridworlds (for reasonably set λ). To an approximation, the AUP penalty is pr...