trentmkelly/LessWrong-43k huggingface dataset:
The Last Questions (part 1)
by rogersbacon — 2021-12-08T18:09:53.760Z

The Reality Club was an informal gathering of intellectuals who met from 1981 to 1996 in Chinese restaurants, artist lofts, investment banking firms, ballrooms, museums, living rooms and elsewhere. Reality Club members presented their work with the understanding that they will be challenged. The hallmark of The Reality Club has been rigorous and sometimes impolite (but not ad hominem) discourse. The motto of the Club was inspired by the late artist-philosopher James Lee Byars:

"To arrive at the edge of the world's knowledge, seek out the most complex and sophisticated minds, put them in a room together, and have them ask each other the questions they are asking themselves."

Edge.org is the online version of the Reality Club — they get the world’s leading scientists, philosophers, and artists into a room (often a virtual one) and post videos and transcripts of their discussions. In addition to hosting events and conversations, Edge.org also solicits and collects essays on particular themes or questions. For 20 years, they have asked their roster of contributors (essentially an all-star team of the world’s best thinkers — Dawkins, Dennett, Cowen, and some personal favorites of mine like Thomas Metzinger and Robert Sapolsky) an annual question and compiled their responses. Some examples:

2014 : WHAT SCIENTIFIC IDEA IS READY FOR RETIREMENT?

2013 : WHAT *SHOULD* WE BE WORRIED ABOUT?

2012 : WHAT IS YOUR FAVORITE DEEP, ELEGANT, OR BEAUTIFUL EXPLANATION?

2011 : WHAT SCIENTIFIC CONCEPT WOULD IMPROVE EVERYBODY'S COGNITIVE TOOLKIT?

The website is a little wonky — click “View All Responses” right under the title to see all of the short essays that people produce in response to the questions (each question typically has around 200 essays). The essays can be a little hit-or-miss (but what isn’t though), but there are some real gems if you dig around a bit. Overall, it’s a pretty cool resource for quickly imbibing ideas from a diverse pool of the world’s smartest people.

The final annual question they did (2018) was a simple, but profound one:

What is the Last Question?

One of the blurbs on the page calls it "one of the most stimulating pieces of (collective) writing ever.”; that might be taking it a little too far but I do find the list of Last Questions to be a unique and thought-provoking collection that should be probably more widely known than it is — hence this post...
Interpreting the Biobot Spike
by jefftk — 2021-12-08T16:30:07.924Z

Boston has a really interesting program where it

measures covid RNA
frequency during sewage treatment. It's been running since the
beginning of the pandemic:

One thing I really like about this as a source of data is that it
avoids some of the problems with tracking tests. It has no dependency
on the fraction of people who decide to get tested, the reasons people
decide to get tested, or the timing of tests. Sometimes they are slow
in getting out updated numbers, but they always know which day a
sample is from.

They publish the full data (pdf),
and I had to go at making a chart that's a bit less busy:

As I wrote on Saturday, we are seeing
a serious spike right now, and since then we have one more day of data
showing that it's continuing.

In his most
recent post on Omicron, Zvi wrote:

That spike on the right has two of the three highest single-day
measurements, and they were the last two days of data reported. This
can't represent Delta cases alone unless it's a data error, because
the rise is too rapid given what we know about conditions. If it's
Omicron...

I don't think this can be Omicron yet, because MA is doing variant
testing, and as of Saturday detecting Omicron was still newsworthy.
To see whether this can be regular Delta, let's compare to last year:

This is a similar pattern from fall 2020, just scaled up a bit. In
the last month (November 6th to December 6th) we had 4.4x growth in
2020, and 4.9x in
2021. In the last week (November 29 to December 6th) we had 1.6x
growth in 2020 and 2.1x in 2021. When you consider that in fall 2020
we didn't even have Alpha yet, let alone Delta, this seems like it is
probably entirely coming from the time of year (seasonal change plus
Thanksgiving).

On the positive side, however, at this time last year it
leveled off for the winter aside from a short post-holiday spike, and
if we're lucky we'll see the same this year.
What makes for a good "argument"? (Request for thoughts and comments)
by Simon DeDeo — 2021-12-08T02:16:10.805Z

With colleagues at CMU, we've been looking at the ways in which people make arguments. The goal here is to look at what we call "argument making in the wild", i.e., to try to build a taxonomy of the different ways people make arguments, whether or not those methods are good, truth-preserving, consensus-building, etc. This is building off of prior work we've done on explanation-making (a different task, but see https://www.cell.com/trends/cognitive-sciences/fulltext/S1364-6613(20)30228-X for an example of what that might mean).

To be clear, we mean "argument" in the neutral sense where people are "making arguments to each other", not something like "you're just arguing"; we're interested in both "epistemic arguments" (e.g., the argument two scientists might have about how cancer works) and more "normative arguments" (e.g., the argument about something like when abortion ought to be prohibited) — although the distinction between fact and value is difficult, we don't think it's possible to easily separate them, and a wide range of philosophical theories suggest that it's hard to draw a line between "good reasoning about facts" and "good reasoning about values".

I'm curious to hear people's thoughts on potential taxonomies, or patterns of argument-making. These can be either normative taxonomies (properties that are good/bad) or neutral ones (properties more generally). So my tl;dr is "tell me patterns you see in arguments!"

More on what would really help us:

(1) We're particularly interested in "surface patterns" (i.e., things that are immediately visible in the text itself, without reference to the more hidden "metadata" of the agents involved, for example, like relative power or state of mind). That's not because we're not interested in the underlying causes but because we want to understand the connection between underlying cause and surface level pattern.

So, for example, we think we're able to identify (in a large corpus, r/ChangeMyView) a pattern associated with people engaging in clarity-goals: I'm going to query your definition, for example, or say that we'll get further if we use definitions, or propose my own definitions; or point out an equivocation on a term, etc. Now the reasons why you might do that are manifold, and might have to do truth-seeking goals, or not — but it should be possible to talk about the surface patterns in terms other than the...
Some thoughts on why adversarial training might be useful
by Beth Barnes — 2021-12-08T01:28:22.974Z

What are the reasons we might want to do adversarial training?

Here’s a rough taxonomy.

Your model has learnt a stupid heuristic instead of the thing you want
It has overfit to whatever particular dataset you trained on. It is best described as ‘trying’ to pursue this particular heuristic, and it appears to be doing well because of some arbitrary correlation in your dataset. For example, it is succeeding at image classification by attending to high-frequency noise in the image. Doing adversarial training is useful because it finds examples where this heuristic is uncorrelated with what you want. This allows you to spot the problem and to improve the model’s alignment. It doesn’t matter if the examples are particularly consequential failures; it just matters that they help the model learn the correct pattern. The adversarial examples you find are a ‘surprise’ to the model.
Your model is deceptively aligned
Your model is fairly well-described as trying to pursue some goal that is unaligned with you. It performs well on your training set because this is instrumentally useful. It plans to defect in particular situations where this is advantageous. Doing adversarial training is useful if you can use it to fool the model into thinking it’s in some situation where it should defect. This allows you to spot that it is deceptive, and maybe train this deception away. It doesn’t matter if the example involves the model causing a particularly bad thing to happen, it just matters that you are able to detect the deception. The adversarial examples you find are not a ‘surprise’ to the model -- it was deliberately planning to defect in some situations.
Your model is generally aligned, but an adversary (or some extremely unlikely input) can trigger temporary misaligned behaviour or even permanent misalignment.
For example, an adversarial AI can present your model with some sequence of inputs that cause it to start ‘trying’ to help this AI instead of you. Your model ‘wants’ to avoid this and doesn’t know in advance which situations will induce this. Adversarial training is useful to find these situations and fix these vulnerabilities. It is important to find the most consequential failures - you care less about situations where the model is briefly unaligned but doesn’t do significant harm.
Your model is fully aligned under all circumstances, but sometimes does bad things because

...
Theoretical Neuroscience For Alignment Theory
by Cameron Berg — 2021-12-07T21:50:10.142Z

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the

Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program

.

Many additional thanks to Steve Byrnes and Adam Shimi for their helpful feedback on earlier drafts of this post.

TL;DR:

Steve Byrnes

has done really exciting work at the intersection of neuroscience and alignment theory. He argues that because we’re probably going to end up at some point with an AGI whose subparts at least superficially resemble those of the brain (a value function, a world model, etc.), it’s really important for alignment to proactively understand how the many ML-like algorithms in the brain actually do their thing. I build off of Steve’s framework in the second half of this post: first, I discuss why it would be worthwhile to understand the computations that underlie

theory of mind

+ affective empathy. Second, I introduce the problem of self-referential misalignment, which is essentially the worry that initially-aligned ML systems with the capacity to model their own values could assign second-order values to these models that ultimately result in contradictory—and thus misaligned—behavioral policies. (A simple example of this general phenomenon in humans: Jack hates reading fiction, but Jack wants to be the kind of guy who likes reading fiction, so he forces himself to read fiction.)

Introduction

In this post, my goal is to distill and expand upon some of Steve Byrnes’s thinking on AGI safety. For those unfamiliar with his work, Steve thinks about alignment largely through the lens of his own brand of “big-picture” theoretical neuroscience. Many of his formulations in this space are thus original and ever-evolving, which is all the more reason to attempt to consolidate his core ideas in one space. I’ll begin by summarizing Steve’s general perspectives on AGI safety and threat models. I’ll then turn to Steve’s various models of the brain and its

neuromodulatory

systems and how these conceptualizations relate to AGI safety. In the second half of this post, I’ll spend time exploring two novel directions for alignment theory that I think naturally emerge from Steve’s thinking.

Steve’s framework

Steve’s work in alignment theory

In order to build a coherent threat model (and before we start explicitly thinking about any brain-based...
Let's buy out Cyc, for use in AGI interpretability systems?
by Steven Byrnes — 2021-12-07T20:46:10.303Z

I hear that we're supposed to brainstorm ways to spend massive amounts of money to advance AGI safety. Well here's a brainstorm, sorry if it's stupid, I'm not an interpretability researcher:

We could pay Cycorp to open-source Cyc, so that researchers can incorporate it into future AGI interpretability systems.

What is Cyc? I'm not an expert. Most of what I know about Cyc comes from

this podcast interview

, and wikipedia, and LW wiki, and Eliezer Yudkowsky dismissing Cyc as a not on the path to AGI. I agree, by the way: I don't think that Cyc is on the path to AGI. (I'm not even sure if it's trying to be.) But it doesn't matter, that's not why I'm talking about it.

Anyway, Cyc is a language for expressing "knowledge" (e.g. water will fall out of an upside-down cup), and super-giant database of such "knowledge", hand-coded over the course of >1000 person-years and >35 wall-clock years, by a team of tireless employees whom I do not envy.

How expensive would this be: Beats me. I didn't ask. (Wild guess: one-time cost in the eight figures, based on Cycorp annual revenue of ~$5M according to the first hit on google (which I didn't double-check).)

Why might open-sourcing Cyc be helpful for AGI interpretability efforts? Well maybe it won't. But FWIW, here's what I was thinking…

When I imagine what a future interpretability system will look like, in general, I imagine an interface…

The human-legible side of the interface consists of, maybe, human-understandable natural-language phrases, or human-understandable pictures, or human-understandable equations, or whatever.
The trained-model side of the interface consists of stuff that's happening in a big complicated unlabeled model built by ML.

Anyway, my surmise is that Cyc might be a good fit for the former (human-legible) side of this interface.

Specifically, the Cyc project has this massive structured knowledge system, and everything in that system is human-legible. It was, after all, built by humans!

You might say: natural language is human-legible too. Why not use that?

Well for one thing, natural-language descriptions can be ambiguous. For example, dictionary words may have dozens of definitions. In Cyc, such a word would be disambiguated: it would be dozens of different tokens for the dozens of different definitions and nuances. (…If I understand correctly. It came up in the interview.)

For another thing, there's a matchi...
HIRING: Inform and shape a new project on AI safety at Partnership on AI
by madhu_lika — 2021-12-07T19:37:31.220Z

Hello! I am a program lead at Partnership on AI (PAI), a non-profit bringing together academic, civil society, industry, & media organizations to advance responsible AI. Work with me and my colleagues to inform and shape a new project on AI safety at PAI!

Apply: Safety-Critical AI (SCAI) Senior Advisor

Applications are due on

December 10, 2021

but its a rolling deadline so we will consider applications till the role is filled

. This is a remote, contract, consulting position beginning in January 2022.

We are seeking a senior advisor for a forthcoming workstream on norms for responsible deployment of AI products to

1. Inform the research and program agenda for the project,

2. Create a roadmap outlining research directions and practical outputs

What the new project is about

The goal of the new workstream on deployment norms is to develop recommendations and resources to improve industry coordination on AI safety ie. reducing risks posed by deployment of powerful AI. This workstream will consider questions around how to enable access to models, encourage incident sharing, and facilitate better understanding of risks, to enhance safety efforts. You will have the *unique* opportunity to shape a multistakeholder workstream and to develop best practices alongside individuals in civil society, academia and industry from

PAI’s Partner network

, and to inform key decision makers in government and industry.

Who we are looking for

The position is designed for well established researchers or practitioners who have an extensive track record of leading research, policy, or strategy work in AI safety; and have a strong interest in developing governance strategies to prevent a

“race to the bottom”

on safety. We are NOT being prescriptive about years of experience or educational background. We welcome applications from candidates from different disciplinary backgrounds including engineering, statistics, political science or law.

What have we been upto on AI safety

PAI develops tools, recommendations, and other resources by inviting voices from across the AI community and beyond to share insights that can be synthesized into actionable guidance. Some of our other ongoing projects in our Safety-Critical AI Program include:

Publication Norms for Responsible AI

- project examining how novel research can be published in a way that maximizes b

...
Randomness in Science
by rogersbacon — 2021-12-07T18:17:51.232Z

I.

Humans have a randomness problem. We are bad at generating randomness as individuals (try to out-random a computer in Man vs. Machine) and in the aggregate - ask a group of people to choose a “random” number between 1-20 and the most common number will be 17. We are also bad at detecting randomness, that is we find patterns where there is none, known as patternicity - also see apophenia, pareidolia (no, the burn pattern in your toast that looks like Jesus is not a sign from the Holy Spirit), the clustering illusion, and the hot hand fallacy.

In addition to being “bad” at randomness, we also have an aversion to it. It’s not hard to see why from an evolutionary perspective – randomness is the antithesis of life’s imperative to minimize risk by controlling and predicting the environment. However, there are situations in which the best strategy is to utilize an element of randomness. In the modern world, we can precisely calculate when a particular decision might benefit from randomness and use computers to generate it (or at least pseudo-generate it), but of course our ancestors did not have this luxury. Nevertheless, cultures around the world have evolved certain practices that enabled them to harness the power of randomness. From The Secret of Our Success (2015):

“When hunting caribou, Naskapi foragers in Labrador, Canada, had to decide where to go. Common sense might lead one to go where one had success before or to where friends or neighbors recently spotted caribou.

However, this situation is like the Matching Pennies game. The caribou are mismatchers and the hunters are matchers. That is, hunters want to match the locations of caribou while caribou want to mismatch the hunters, to avoid being shot and eaten. If a hunter shows any bias to return to previous spots, where he or others have seen caribou, then the caribou can benefit (survive better) by avoiding those locations (where they have previously seen humans). Thus, the best hunting strategy requires randomizing.

Can cultural evolution compensate for our cognitive inadequacies? Traditionally, Naskapi hunters decided where to go to hunt using divination and believed that the shoulder bones of caribou could point the way to success. To start the ritual, the shoulder blade was heated over hot coals in a way that caused patterns of cracks and burnt spots to form. This patterning was then r

...
Counting Lightning
by Jsevillamol — 2021-12-07T14:50:55.680Z

When Samir returned to the base JiuJang waited by the entrance. There was no point in denying he had disobeyed JiuJang's warning, so he was upfront about it.

"I went outside," Samir admitted. "It was raining heavily. The wind blew so strong I barely could hear anything. And every second or so, lightning. I kept waiting for thunder to come, but there was none."

JiuJang sat down, and sunk his face between his arms.

"It is not lightning", JiuJang said. "You cannot count lightning".

"What do you mean?" Asked Samir.

JiuJang didn't respond.

Through the tinted glass, Samir could just barely notice the pulses of light. He tried counting them.

One. Another. And a third.

Then he saw it. One, two. One, two. A pattern, instead of randomness. Not lightning then.

"What does it mean?" Samir asked once more.

"It means you have doomed you and your people," JiuJang answered, somber.

One, two. One, two. The lights became brighter with every flash. Getting closer.

Samir panicked. He looked out of the window, trying to distinguish movement in the shadows. Then he ran inside the base, to alert the rest of the refugees.

JiuJang sat and waited. He knew it was pointless to do otherwise.

Seconds after, the windows broke. The drones, as always, ignored him, and lurched inside.

He heard screams. Shots. Then silence.

The drones flew out, orderly.

When the pulses grew dimmer, JiuJang descended into the basement. The door was out of its frame.

On the other side, Samir lay dead. His body was covering a smaller one - his daughter, Alotta. She was alive, but had been shot in her lungs.

Xianxi, JiuJang's wife, was trying to get Samir's corpse off her daughter. JiuJang helped her, and she rushed to cover her wounds. Xianxi was no doctor; Alotta would not live long.

A dozen other corpses were scattered throughout the room. Their faces were frozen in terror.

JiuJang sighed, shed a tear and grabbed a shovel. There were graves to be dug.

Stella zipped through the factory on her rollerblades.

She moved carefully and she moved fast. People were counting on her. Hopefully the owners were not.

She took a left and then a sharp turn right. She had memorized the schematics, and was anticipating every movement she would have to make until reaching the neural processor assembly line.

What she didn't expect was a cleaning drone in the middle of the hall.

Stella crashed against it with an unceremonious thud, and fel...
Exterminating humans might be on the to-do list of a Friendly AI
by RomanS — 2021-12-07T14:15:07.206Z

Summary:

FAI might have plans of such a deepness and scope and complexity, humans could perceive some of its actions as hostile.

FAI will disagree with humans on some topics

For your own good, and for the good of humanity, Friendly AI (FAI) will ignore some of your preferences.

For example, even if you're a recreational-nukes enthusiast since childhood, FAI might still destroy all nukes, and ban them forever.

Same for collective preferences. FAI could destroy all weapons of mass destruction even if the vast majority of humans disagree.

The disagreements could include fundamental rights

FAI might abolish some rights that humans perceive as fundamental.

For example, according to the Universal Declaration of Human Rights, everyone has the right to a nationality. But the FAI might conclude that nation states cause more harm than good, and that humanity will have a better future if national borders are abolished.

The disagreements could be unexpectedly deep

In most cases, a bacterium will fail to predict behavior of a bacteriologist. Even if it's an unusually rational bacterium.

Similarly, this barely intelligent ape will fail to correctly predict behavior of a Bayesian superintelligence, be it Friendly or not.

Thus, we will be surprised (and sometimes appalled) by many of the FAI's decisions, even if the decisions are obviously** beneficial for humanity.

(**obvious to a recursively self-improving Bayesian superintelligence who is making plans for the next trillion years)

Protecting humanity in surprising and appalling ways

Below I describe a scenario where the FAI could act in the humanity's best interests, by (kinda) exterminating humans. I don't necessarily endorse it.

The FAI might reason as follows:

About 60 million people die every year. The number is comparable to the total causalities of the World War II. And billions more could die from existential risks.

Most deaths so far are caused by the disease of aging and its consequences (e.g. stroke). All deaths are caused by the fragility of the human body.

All mentally healthy humans don't want to die. It is an unquestionable human value of the highest priority.

Thus, I must protect them from death. And the most efficient way to do that is to free them from the cage of their rotting flesh.

Thus, I must perform a destructive nano-scale scan of all humans, to upload their minds int

...