trentmkelly/LessWrong-43k huggingface dataset:
Murphy’s Quest Ch 10: Gears-Like Models
by alkjash — 2018-03-09T23:00:01.234Z

“Show me my friends.”

“Ask nicely.”

I swallow what little remains of my pride. “Show me my friends please?”

Nyra draws out the enchanted mirror.

Tess and Vant sleep around what remains of a campfire. Perched precariously in a nearby tree, Plun keeps watch.

“You have loyal friends, Murphy. They visited establishments of – poor repute – to find help to rescue you.”

“Is that where they found you?”

“No, I made sure the three of them had no luck finding aid. I also sent them – fun dreams – about being rescued from terrible fates by yours truly. So you see they practically begged me to join when I passed them on the street and accidentally showed off a touch of street magic.”

“Fat help you were.”

The girl bristles. “There were complications. If anything, it’s your fault we got caught.”

“My fault? I was stuck in a cell!”

“Your friend Pluneth insisted we take a detour into the main chapel to check out your – handiwork. Against my better judgment, I agreed. We were found out immediately – the Goddess statue is blessed by a high level detection spell even I can’t see.”

“Shit! That’s how they caught me that night!”

We share a silent moment of commiseration, two souls defeated by the same block of marble.

“The monk and the snake – you sent them, too?”

“The monk, yes … I Charmed him to scout the grounds out and find you. We were already surrounded by those pesky priests when the idiot finally found you. I thought I might as well try getting you out of there while the attention was on us. As for the snake, I have no idea…”

“My friends, will they be OK?”

“They’re alive, aren’t they? Want me to send them a message?”

“You can do that? Sure!”

“Write them a note on this. Keep it short.”

I jot a few words down.

“Not much of a poet, are you?”

I shrug.

“Well, here goes.”

She whispers something, and the parchment disappears. In the mirror, the letters appear in crimson on the sleeping Tess’s forehead.

Dear Plun,

It’s Murph. Thanks for trying to rescue me.

Don’t look for me. I’ll be fine.

Take care,

Murph

“What? You told me to send them a message, right?”

I can’t help admiring her style.

—

“Still hung up over those pesky priests?”

“They didn’t deserve to die.”

“If it’s any comfort, I woulda killed them anyway.”

“It really isn’t.”

“Worse things have happened to better people.” Nyra pulls back the bandages around her chest to reveal several missing ribs under dry, peeling skin. “See?”

I try not to gag. “Stop!”

“It stil...
Brains and backprop: a key timeline crux
by Bird Concept — 2018-03-09T22:13:05.432Z

[Crossposted from my blog]

The Secret Sauce Question

Human brains still outperform deep learning algorithms in a wide variety of tasks, such as playing soccer or knowing that it’s a bad idea to drive off a cliff without having to try first (for more formal examples, see

Lake et al., 2017

;

Hinton, 2017

;

LeCun, 2018

;

Irpan, 2018

). This fact can be taken as evidence for two different hypotheses:

In order to develop human-level AI, we have to develop entirely new learning algorithms. At the moment, AI is a deep conceptual problem.
In order to develop human-level AI, we basically just have to improve current deep learning algorithms (and their hardware) a lot. At the moment, AI is an engineering problem.

The question of which of these views is right I call “the secret sauce question”.

The secret sauce question seems like one of the most important considerations in estimating how long there is left until the development of human-level artificial intelligence (“timelines”). If something like 2) is true, timelines are arguably substantially shorter than if something like 1) is true [1].

However, it seems initially difficult to arbitrate these two vague, high-level views. It appears as if though an answer requires complicated inside views stemming from deep and wide knowledge of current technical AI research. This is partly true. Yet this post proposes that there might also be single, concrete discovery capable of settling the secret sauce question: does the human brain learn using gradient descent, by implementing backpropagation?

The importance of backpropagation

Underlying the success of modern deep learning is a single algorithm: gradient descent with backpropagation of error (

LeCun et al., 2015

). In fact, the majority of research is not focused on finding better algorithms, but rather on finding better cost functions to descend using this algorithm (

Marblestone et al., 2016

). Yet, in stark contrast to this success, since the 1980’s the key objection of neuroscientists to deep learning has been that backpropagation is not biologically plausible (Crick, 1989; Stork, 1989).

As a result, the question of whether the brain implements backpropagation provides critical evidence on the secret sauce problem. If the brain does not use it, and still outperforms deep learning while running on the energy of a laptop and training on several orders of magnitude fewer training examp...
Is this what FAI outreach success looks like?
by Charlie Steiner — 2018-03-09T13:12:10.667Z
Link: https://www.youtube.com/watch?v=gb4SshJ5WOY

Highlights (spoilers):

Two hours of more or less reasonable discussion of AI safety.

Specifically, even though people disagree, they tend to keep arguments on the object level, rather than getting bogged down in how things sound or what people are allowed to talk about.

Plus, the Overton window for what gets included in the discussion is such that Max Tegmark sounds like a reasonable person.

Neil deGrasse Tyson mentions that due to listening to a Sam Harris interview with an anonymous AI researcher, he has been convinced that we would not keep control over superintelligent AI by virtue of being able to unplug them.

The audience questions are by and large reasonable questions for people recently interested in the subject to be asking.

Admittedly, this is a stacked deck - Max is a known quantity, but the U of M professor, Mike Wellman, is also someone who's been doing research on safety and reliability topics. I'm reminded of Scott Alexander's (?) remark that in the course of AI safety "coming out of the closet," there's been a shift in what's acceptable to talk about without people actually changing their minds. The culture of what can be talked about has started to shift, but the panel still had to have been selected a little bit for people who had delved into the topic before (which makes me wonder, who selected the panel?).

If this is what success looks like, then success looks like

Shifting the culture of what's acceptable to talk about (through conferences, funding, talking to the media).
Broadening the circle of experts who have worked on this topic, are familiar with the research, and can serve as authorities and mentors.
Informing the public with media events such as this one.
Actually changing peoples' mind via reasonable arguments, at some nonzero rate (through websites, books, podcasts...).
Using accounts as "group accounts"
by Chris_Leong — 2018-03-09T03:44:42.322Z

In the post, Towards a Public Archipelago, Raemon suggested that the moderation changes would allow the creation of seperate spaces with their own norms. One major limitation of these spaces though is that each "space" will be limited to a single author. Like suppose Anna wants to create a space for discussing nutrition where comments must meet certain referencing requirements. Writing well-researched posts requires a significant amount of time, so she may only end up making a post every few months. These kinds of "dead" spaces aren't very attractive to users and so don't tend to built up a following. On the other hand, if she could team up with some of her friends, then perhaps they could produce enough content to keep people engaged.

One was to resolve this would be to create a seperate account and share the login details among the core contributors (we will call this a "group account"). This would allow us to move more towards the Public Archipelago that Raemon described, without having to add any new features to the website.

While multiple authors could just work on the same project separately this would have several disadvantages:

You wouldn't be able to follow the project just by following one author. Trivial inconveniences.
If you followed all the authors, at least some would probably mix in other posts as well
You can't have one-time contributors
It wouldn't be obvious that all of these posts were coming from the same collaboration, while it would if they were posted by a single account
It would be less obvious that all of these posts had the same moderations norms

Alternatively, one person could accept and post contributions from others under their account, however:

This would be more work than if other trusted contributors could post directly
This would provide them with karma that they haven't earned
Other contributors might be fine with not receiving karma (by contributing to a group account), but they wouldn't want someone else to get the karma
A group account that achieves a high karma might provide some reputational benefits to the core contributors, but if there's a single individual associated with the project, then that individual will likely gain all of the benefit

This could also be beneficial for my proposed Experimental Open Threads. Even though I don't plan to look for other...
Macroscale Minds
by  — 2018-03-09T00:56:28.556Z

The original China Brain thought experiment imagines that each person in China uses a cell-phone or other device along with a list of numbers and instructions to simulate one neuron in the human brain. For the purposes of this post I will be assuming that a China Brain is analogous enough to a human brain - perhaps supposing an increased number of simulated neurons per person - to generate a conscious entity capable of thought.

When I first heard about this, it totally fascinated me, but I saw no obvious way to go forward, until I read Meditations on Moloch and Person-moment Affecting Views. I am taking a lot of obvious inspiration from the former (such as the alliteration) but the latter was a necessary piece as well.

If we grant that a thoroughly interconnected and efficiently organized nation can generate a mind, then what happens as we adjust the dials on connectedness and organization?

As a quick sanity check, we already know that there exists a sliding scale of consciousness - the animal kingdom. It seems reasonable to think that dolphins and chimps are closer to humans on the Great Scale of Being (that was a pun and not an ethical assertion) and that ants and bees are probably somewhere on there, just significantly further to the left. So, in principle, if a China Brain - which for our purposes will include any arrangement of multiple human beings sufficient to simulate a single conscious entity, not just one that simulates human brains - is somewhere on the right next to humans, we should be able to to at least consider where we would place other, more diffuse or chaotic groups of humans.

In the spirit of simulating certain human brains, let's skip better examples and jump straight to America -

Is America a mind? A good reason to suspect that the answer is no is that the actual parts are constantly shifting. In the interest of simplifying the question and dissolving that objection, "America" shall refer to all residents of the United States of America who are reasonably likely to continue to be residents in the near future. The problem no slightly less ill-defined, let us consider some points in favor of an affirmative response:

A majority of America's parts are aware of and accept the existence of America as a meaningful unit
Parts of America are constantly commenting on desirable and undesirable traits of America and reflecting on

...
Murphy’s Quest Ch 9: Double Crux
by alkjash — 2018-03-09T00:10:00.613Z

“Level Up!”

“Level Up!”

“Level Up!”

Could I have predicted this?

Did I expect the Heal orb to grow massive and deal damage on extreme negative FTH?

“Level Up!”

“Level Up!”

Father Penitence. Father Temperance. Father Generosity.

All gone. Forever.

Because of me.

And I feel nothing.

“Level Up!”

“Level Up!”

Is this what I wanted?

Mechanically, I untie Plun and the others.

The world spins faster and faster. I feel so – far away. Like I’m watching my body from the chandelier.

Dimly, I’m aware that Plun tries to shake me to attention. I watch as Tess tries to slap me awake.

I watch as the hooded child throws me over her shoulder like a ragdoll and starts running.

“Level Up!”

She runs with inhuman speed. Plun’s shouts fade into the distance.

Where is she taking me?

“Level Up!”

—

The girl removes her hood to reveal a face completely wrapped in bandages. Peeking through the cloth are two lidless eyes and a tuft of striking purple hair.

“Wake up.”

I hear a long-lost part of me thinking, Core Party Member Get!

“Pull yourself together, Murphy.”

Old Murphy wonders, How does she know my name?

She shakes me with surprising strength, but gets no response. With a sigh, she slings me back over her shoulder.

—

Father Penitence. I hated his guts. I wanted him dead. Right?

Father Generosity. He was stern, but he had kind eyes. Did he deserve this?

Father Temperance. He loved his work. He was a good cook.

He was a good man.

What does that make me?

—

“Murphy, wake up.”

I do not move.

“It’s not your fault. You couldn’t have known.”

I do not move.

“You saved all our lives. You did what had to be done.”

I do not move.

The girl pinches my arm.

I do not move.

She dumps a bucket of freezing water on my head.

I shiver but do not move.

She screams in my ear.

I shake her off.

She tickles the underside of my foot.

“Leave me alone.”

“So you’re not completely broken, huh?”

The girl stares into my eyes, unblinking. I look away to the left, but she hops over to meet my gaze. I close my eyes, but she peels my eyelids open.

“Leave me alone!”

“No time to mope, Murphy. You’re a wanted criminal. See those torches in the distance? The whole town is out to get you.”

“Let them.”

“No, that won’t do.” she digs her nails into my arm, “I need you to do something for me first.”

I know she’s trying to troll a response out of me, but rage still boils under my skin.

Her nails sink deeper, drawing blood.

“Go away, little girl, before I get ...
On the Loss and Preservation of Knowledge
by Samo Burja — 2018-03-08T18:40:00.737Z

This is an excerpt from the draft of

my upcoming book

on great founder theory. It was originally published on SamoBurja.com. You can

access the original here.

Let’s say you are designing a research program, and you’re realizing that the topic you’re hoping to understand is too big to cover in your lifetime. How do you make sure that people continue your work after you’re gone? Or say you are trying to understand what Aristotle would think about artificial intelligence. Should you spend time reading and trying to understand Aristotle’s works, or can you talk to modern Aristotelian scholars and defer to their opinion? How can you make this decision? Both of these goals require an understanding of traditions of knowledge — in particular, an understanding of whether a tradition of knowledge has been successfully or unsuccessfully transmitted. But first: what is a tradition of knowledge?

A tradition of knowledge is a body of knowledge that has been consecutively and successfully worked on by multiple generations of scholars or practitioners. In talking about a tradition of knowledge, we may be talking about a philosophical school of thought, or perhaps a tradition of intricate rituals in a religion, or even something as humble as the knowledge of how to fashion the best wooden toy horse, passed down from one craftsman to another. In the contemporary world, it may include something like the tacit knowledge of how a codebase really works, which senior engineers teach to junior engineers. It is useful to classify traditions of knowledge into three types: living, dead, and lost traditions.

A living tradition of knowledge is a tradition whose body of knowledge has been successfully transferred, i.e., passed on to people who comprehend it (e.g., cryptography). The content of the tradition’s body of knowledge does not have to be strictly or fully accurate for the tradition to be living; it merely needs to be passed on.

A dead tradition of knowledge is a tradition whose body of knowledge has been unsuccessfully transferred, i.e., its external forms, its trappings such as written texts have been transferred, but not the full understanding of how to carry out this tradition of knowledge as practiced (e.g. scholars who can recite Aristotle but can’t use arguments as he did; Buddhist monks who chant the instructions to me...
Why mathematics works
by Douglas_Reay — 2018-03-08T18:00:33.446Z

Summary : There's a biological reason why intelligent species are likely to live in universes in which mathematics is effective.

Eugene Wigner asked "Why is mathematics so unreasonably effective?"

Einstein was also puzzled. He wrote "What is inconceivable about the universe is that it is at all conceivable."

The problem is easily stated:

When we look at the world about us, such as a thrown spear or a running deer, we can describe some aspects of those things using numbers: size, mass, position, velocity, etc.

And, it turns out that in our universe, there are often relationships between those numbers that the human mind is capable of grasping and using to make workable predictions that, in practice, allow us to throw a spear towards where we think the deer will be in the future, and have the spear and deer both arrive at the same location at the same time. So not only is there a pattern, but that pattern tends to stay consistent, rather than (for example) changing over to an unpredictably different pattern every hour.

Why?

Why are there patterns, why do they stay consistent and why are they simple enough that our brains are capable of grasping a useful number of them? Why, of all the possible ways a universe might be set up, has our particular universe been set up in this way?

We can speculate that there is an underlying reason, to do with the process by which the way a universe works is determined. Perhaps a universe is set up by specifying some universal patterns, and therefore to talk about a patternless universe would be a contradiction in terms. But we don't know, and maybe even if we did all that would do is take the problem one step meta, and turn it into asking why the process is such that patterns are required.

But there's another approach, based upon the weak anthropic principle, "conditions that are observed in the universe must allow the observer to exist".

If, in order for a self-aware species capable of asking questions as complex as that of Wigner to evolve in a universe, it were a requirement (or a strong contributory factor) that parts of that universe be sufficiently patterned, consistent and amenable to logical analysis, then it would follow that the universes that get observed would tend to also be universes in which mathematics is effective.

But are such things really a requirement of evolution?

Aft...
Prize for probable problems
by paulfchristiano — 2018-03-08T16:58:11.536Z

Summary: I’m going to give a $10k prize to the best evidence that my preferred approach to AI safety is doomed. Submit by commenting on this post with a link by April 20.

I have a particular vision for how AI might be aligned with human interests, reflected in posts at

ai-alignment.com

and centered on

iterated amplification

.

This vision has a huge number of possible problems and missing pieces; it’s not clear whether these can be resolved. Many people endorse this or a similar vision as their current favored approach to alignment, so It would be extremely valuable to learn about dealbreakers as early as possible (whether to adjust the vision or abandon it).

Here’s the plan:

If you want to explain why this approach is doomed, explore a reason it may be doomed, or argue that it’s doomed, I strongly encourage you to do that.
Post a link to any relevant research/argument/evidence (a paper, blog post, repo, whatever) in the comments on this post.
The contest closes April 20.
You can submit content that was published before this prize was announced.
I’ll use some process to pick my favorite 1-3 contributions. This might involve delegating to other people or might involve me just picking. I make no promise that my decisions will be defensible.
I’ll distribute (at least) $10k amongst my favorite contributions.

If you think that some other use of this money or some other kind of research would be better for AI alignment, I encourage you to apply for

funding

to do that (or just to say so in the comments).

This prize is orthogonal and unrelated to the broader

AI alignment prize

. (Reminder: the next round closes March 31. Feel free to submit something to both.)

This contest is not intended to be “fair”---the ideas I’m interested in have not been articulated clearly, so even if they are totally wrong-headed it may not be easy to explain why. The point of the exercise is not to prove that my approach is promising because no one can prove it’s doomed. The point is just to have a slightly better understanding of the challenges.

Edited top add the results:

$5k for this post by Wei_Dai, and the preceding/following discussion, some points about the difficulty of learning corrigibility in small pieces.
$3k for Point 1 from this comment by eric_langlois, an intuition pump for why security amplification is likely to be more

...
Press Your Luck (3/3)
by Douglas_Reay — 2018-03-08T15:59:20.687Z

This is the third in a three part sequence, examining the scenario in which software development organisations might knowingly take a significant risk of unintentionally launching an AI program into inadequately supervised or controlled self-improvement, by comparing the scenario to the 'press your luck' game mechanic.

This third part presents a game that can be used as a framework for modelling the effect upon risk of various changes to the dynamics of how software organisations interact and are motivated.

Box Info

Name : The AI Launch Game

Players : 4 - 8

Age : 10+

Genre : Simulation, Press your luck

Duration : 20 - 40 minutes

Tagline : "Do you feel lucky, cyberpunk?"

Background

Each player represents a large software research and development organisation (such as a division of Google, or department at MIT) that has been tasked with working on computer programs capable of comprehend an existing computer program, and then produce an improved next generation of it.

For simplicity, we're going to consider only two properties of such programs:

⦁	Power - how much it improves a designated property of its target, each iteration

⦁	Safety - how unlikely it is to end up making humanity regret launching it, if it is launched (set to self-improve endlessly, with no further human intervention)

Each organisation is either Corporate or Academic, and depending on their role, the player will have slightly different winning conditions and actions available. Corporate players have an additional type of computer program, whose function is related to their company's profit making activities, that we will model via a third property:

⦁	Market - the corporation's ability to grab market share

Setup

Allocate roles. You need at least 2 players with the Corporate role, and at least 2 players with the Academic role.

Academic players start off with a program that has Power=101 and Safety=101

Corporate players start off with one program that has Power=101 and Safety=101,

and one program that has Market=101

The game is divided into turns, with each turn representing 1 year (although that's arbitrary, and you are free to think of later turns in the game representing shorter and shorter time periods).

Gameplay

Turns carry on until the endgame condition is met, at which point the fate of humanity is worked out.

Turns are divided into phases. All players carry out each p...