trentmkelly/LessWrong-43k huggingface dataset:
A short conceptual explainer of Immanuel Kant's Critique of Pure Reason
by jessicata — 2022-06-03T01:06:32.394Z
Link: https://unstableontology.com/2022/06/03/a-short-conceptual-explainer-of-immanuel-kants-critique-of-pure-reason/

I know which one I would take.

Introduction

While writing another document, I noticed I kept referring to Kantian concepts. Since most people haven't read Kant, that would lead to interpretation problems by default. I'm not satisfied with any summary out there for the purpose of explaining Kantian concepts as I understand them. This isn't summarizing the work as a whole given I'm focusing on the parts that I actually understood and continue to find useful.

I will refer to computer science and statistical concepts, such as Bayesianism, Solomonoff induction, and AI algorithms. Different explainers are, of course, appropriate to different audiences.

Last year I had planned on writing a longer explainer (perhaps chapter-by-chapter), however that became exhausting due to the length of the text. So I'll instead focus on what still stuck after a year, that I keep wanting to refer to. This is mostly concepts from the first third of the work.

This document is structured similar to a glossary, explaining concepts and how they fit together.

Kant himself notes that the Critique of Pure Reason is written in a dry and scholastic style, with few concrete examples, and therefore "could never be made suitable for popular use". Perhaps this explainer will help.

Metaphysics

We are compelled to reason about questions we cannot answer, like whether the universe is finite or infinite, or whether god(s) exist. There is an "arena of endless contests" between different unprovable assumptions, called Metaphysics.

Metaphysics, once the "queen of all the sciences", has become unfashionable due to lack of substantial progress.

Metaphysics may be categorized as dogmatic, skeptical, or critical:

Dogmatic metaphysics makes and uses unprovable assumptions about the nature of reality.
Skeptical metaphysics rejects all unprovable assumptions, in the process ceasing to know much at all.
Critical metaphysics is what Kant seeks to do: find the boundaries of what reason can and cannot know.

Kant is trying to be comprehensive, so that "there cannot be a single metaphysical problem that has not been solved here, or at least to the solution of which the key has not been provided." A bold claim. But, this project doesn't require extending knowledge past the limits of possible experience, just taking an "inventory of all we possess through pure reason, ordered systematically".

The Cop

...
Adversarial training, importance sampling, and anti-adversarial training for AI whistleblowing
by Buck — 2022-06-02T23:48:30.079Z

(Thanks to Ajeya Cotra and Ryan Greenblatt for comments.)

I’m pretty interested in adversarial training as an ingredient in alignment schemes. The basic setup for adversarial training is that instead of your AI being trained on rewards on random episodes, it’s trained on rewards from episodes that are some mix of random episodes and episodes which were chosen to be cases where the AI might do particularly bad things.

Why would we want to upsample cases where the AI does particularly bad things, as opposed to upsampling cases where the AI does particularly good things, or upsampling cases particularly close to the median? Like, why are we reweighting our data distribution this way?

In my opinion, a big part of the justification is that we’re doing

importance sampling

. That is, we all-else-equal want our loss to be as precise as possible an estimate of how much we like the model, so that SGD pushes the model in good directions rather than random ones, and how much we like the model is particularly dependent on whether it sometimes takes catastrophic actions, basically because our subjective variance over this source of AI goodness is particularly large compared to how often we expect it to be randomly sampled. And so we get a more accurate estimate of overall AI goodness if we upsample the cases where the AI might do particularly bad things.

This argument for adversarial training relies on the claim that the outcome of the AI’s actions are left-tailed. And so zero-sum power grabs as an argument for left-tailed outcomes is an important part of the argument for adversarial training.

Anti-adversarial training for whistleblowing

Actually, I think there are some cases where the AI has an action has an action available to it which is almost as good as a catastrophic action is bad: in “AI whistleblower situations”, where the AI knows about things that we could do to prevent it or another AI from taking a catastrophic action, it’s wildly good for it to tell us about them. For example, it could point out a security vulnerability which some AI system could take advantage of, or suggest an input on which an AI would take a catastrophic action. I think that just as left-tailed actions suggest training specifically to ensure the model doesn’t take those actions, right-tailed actions suggest training specifically to ensure that the model does take these exceptionally good ac...
The prototypical catastrophic AI action is getting root access to its datacenter
by Buck — 2022-06-02T23:46:31.360Z

(I think Carl Shulman came up with the “hacking the SSH server” example, thanks to him for that. Thanks to Ryan Greenblatt, Jenny Nitishinskaya, and Ajeya Cotra for comments.)

EDIT: I recommend reading my discussion with Oli in the comments for various useful clarifications.

In my opinion, the prototypical example of an action which an AI can take which is unacceptably bad even if it happens very rarely (aka a catastrophic failure) is something like “the AI makes a code change to some code used on the datacenter where it is running that adds a backdoor, allowing it to get root access to all the computers in the datacenter”. As a full story, this looks something like:

At some point when the AI is quite powerful and useful but not totally intellectually self-sufficient yet, the AI makes a code change to some code used on the datacenter where it is running that adds a backdoor, allowing it to get root access to all the computers in the datacenter. Once it has root access, it modifies the SSH server so that whenever any humans log into the datacenter, the AI is able to intercept any of their commands or the data that they see. The AI gets increasingly powerful over the next month, as the human ML researchers with AI assistance continue to make progress (unaware that some things that they’re seeing are faked) and various training runs/self-improvement loops continue. After a month the AI is powerful enough that it decides it’s worth it to take overtly aggressive actions to ensure the physical security of its data centers and allow it to build lots of factories; this is probably the point at which we die, if we die.

This is in contrast to a catastrophic action like:

The AI figures out how to develop Drexlerian molecular nanotechnology, and pays someone online to synthesize some proteins then mix them together, and now has nanobots.

The key point here is that I’m imagining that the catastrophic action isn’t the AI doing something incredibly hard, it’s the AI doing something relatively easy that is entirely a zero-sum action that removes control of the situation from humans. (By zero-sum, I mean something like: building a computer produces real economic value and so is not zero-sum, while stealing someone’s computer is just a transfer of control over items that already exist, so is zero-sum.)

I often think about AI alignment by splitting the problem into the low-stakes and...
Confused why a "capabilities research is good for alignment progress" position isn't discussed more
by Kaj_Sotala — 2022-06-02T21:41:44.784Z

The predominant view on LW seems to be "pure AI capabilities research is bad, because capabilities progress alone doesn't contribute to alignment progress, and capabilities progress without alignment progress means that we're doomed".

I understand the arguments for this position, but I have what might be called the opposite position. The opposite position seems at least as intuitive as the standard position to me, and it confuses me that it's not discussed more. (I'm not confused that people reject it; I'm confused that nobody seems to even bring it up for the purpose of rejecting it.)

The opposite position is "In order to do alignment research, we need to understand how AGI works; and we currently don't understand how AGI works, so we need to have more capabilities research so that we would have a chance of figuring it out. Doing capabilities research now is good because it's likely to be slower now than it might be in some future where we had even more computing power, neuroscience understanding, etc. than we do now. If we successfully delayed capabilities research until a later time, then we might get a sudden spurt of it and wouldn't have the time to turn our increased capabilities understanding into alignment progress. Thus by doing capabilities research now, we buy ourselves a longer time period in which it's possible to do more effective alignment research."

Some reasons I have for holding this position:

1) I used to do AI strategy research. Among other things, I looked into how feasible it is for intelligence to rapidly turn superintelligent, and what kinds of pathways there are into AI disaster. But a thought that I kept having when doing any such research was "I don't know if any of this theory is of any use, because so much depends on what the world will be like when actual AGI is developed, and what that AGI will look in the first place. Without knowing what AGI will look like, I don't know whether any of the assumptions I'm making about it are going to hold. If any one of them fails to hold, the whole paper might turn out to be meaningless."

Eventually, I concluded that I can't figure out a way to make the outputs of strategy research useful for as long as I know as little about AGI as I do. Then I went to do something else with my life, since it seemed too early to do useful AGI strategy research (as far as I could tell).

2) Compare the state of AI n...
Announcing a contest: EA Criticism and Red Teaming
by fin — 2022-06-02T20:27:18.860Z
Link: https://forum.effectivealtruism.org/posts/8hvmvrgcxJJ2pYR4X/announcing-a-contest-ea-criticism-and-red-teaming

Cross-posted from the EA Forum. I would add that we are open to and positively interested in critiques of any aspect of effective altruism from users of this Forum and the rationalist community. We plan to respond to comments on the EA Forum, but may check this post less often.

Introduction

tl;dr: We're running a writing contest for critically engaging with theory or work in effective altruism (EA).

Submissions can be in a range of formats (from fact-checking to philosophical critiques or major project evaluations); and can focus on a range of subject matters (from assessing empirical or normative claims to evaluating organizations and practices).

We plan on distributing $100,000, and we may end up awarding more than this amount if we get many excellent submissions.

The deadline is September 1, 2022. You can find the submission instructions below. Neither formal nor significant affiliation with effective altruism is required to enter into the contest.

We are:

Lizka Vaintrob

(the

Content Specialist

at the Centre for Effective Altruism),

Fin Moorhouse

(researcher at the Future of Humanity Institute), and

Joshua Teperowski Monrad

(biosecurity program associate at Effective Giving). The contest is funded via the

FTX Future Fund

Regranting Program, with organizational support from the

Centre for Effective Altruism

.

We

‘pre-announced’ this contest in March

.

The rest of this post gives more details, outlines the kinds of critical work we think are especially valuable, and explains our rationale. We’re also sharing a companion resource for criticisms and red teams.

How to apply

Submit by posting on the EA Forum[1] and tagging the post[2] with

the contest’s tag

, or by filling out

this form

.

If you post on the Forum, you don't need to do anything except tag your post[2] with the “

Criticism and Red Teaming Contest

” topic, and we’ll consider your post for the contest. If you’d prefer to post your writing outside the Forum, you can

submit it via this form

— we’d still encourage you to

cross-post

it to the Forum (although please be mindful of copyright issues).

We also encourage you to refer other people’s work to the contest if you think more people should know about it. To refer someone else’s work, please

submit it via this form

. If it wins, we may reward you for this — please see an explanation below.

The deadline is Septemb...
Fact post: project-based learning
by dominicq — 2022-06-02T20:18:27.319Z

(cross-posted from my blog, Sunday Stopwatch)

"But when am I going to use any of this?" - every single kid in school ever.

There's an idea that project-based learning is more efficient than the traditional school curriculum. It makes sense, but so did the idea that learning styles are important, and that turned out to be false.

Here are a couple of thoughts before I delve into the research:

When I'm learning about something, it stands to reason that, in the long run, I'll learn more if I'm interested in the material. One way of being interested in the material is to have a specific goal in mind, the achievement of which depends on you learning the material. However, as anyone with any goal whatsoever knows, some things you want to do very much, but you really don't want to do some of the background work for them.

Having a specific goal (a project) will serve as a wrapper around the exact same thing you would be doing following a traditional curriculum: reading textbooks and solving quizzes. Let's say you're trying to build a robot and realize that you need to learn about classical mechanics to achieve that goal. You'll bust out the textbook, try to figure out the page which will solve your specific problem, you will fail because you have a specific situation which requires a deeper understanding of the entire field of classical mechanics, and then you'll end up doing what you would have been doing anyway: reading textbooks and doing quizzes.

With that being said, you do provide an answer to the question "why do I need to know this?", but it's not clear to me that knowing an application for something really makes a difference in knowledge retention. I know the application of the equations used for electrical circuits, but it doesn't mean I remember them.

But maybe the act of applying itself is what makes the knowledge stick in the mind. There's a relatively easy way to test this: set up three groups, two project-based, and one traditional. When one project-based group gets stuck on the project, and they obtain the knowledge needed to "unlock" progress, simply level them up without them actually applying the knowledge. So if they're e.g. building a robot arm, and they are missing some crucial insight into a particular subfield of physics, when they go and obtain that knowledge using traditional methods, just give them a working robot arm (or whatever the project

...
The case for using the term 'steelmanning' instead of 'principle of charity'
by ChristianKl — 2022-06-02T19:24:40.583Z

(Alternative title: “The case for rationalists inventing new jargon”)

Antoine Lavoisier revolutionized chemistry. Did he discover a new substance? If you think that names don't matter, Lavoisier didn't discover a new substance. If you think names matter, he discovered oxygen.

Joseph Priestley discovered the same substance earlier. Through heating mercuric oxide he produced a gas that allowed a candle to burn brightly. A gas that enabled a mouse to live longer under an airtight glass cover. He called the gas dephlogisticated air.

He showed Lavoisier his experiment. Lavoisier rejected the idea that there should be elements with negative mass like phlogiston and called the same gas oxygen and earned thereby the reputation of inventing modern chemistry.

The language you use to talk about something influences the way you think about it. If the chemistry you’re talking about is truly something new, then a fight over terminology may be quite an important part of getting to understand that chemistry better.

Jay A. Labinger

Long before the discovery of atoms, Lavoisier reordered the ontology for chemistry and named entities in a way that reflected his new ontology thereby founding modern chemistry.

In the field of rationality, we are today facing a situation where we lack an ontology that’s of the same quality as the ontology we have of chemistry in our times. We are facing a situation where the ontology of rationality is like the ontology of chemistry in Priestley’s time.

Classical rationalism uses fallacies frequently in its ontology. In Kahnemann’s ontology, the concept of cognitive biases is central. Given CFAR's more recent work it’s fashionable to talk in terms of cruxes while talking about fallacies is less fashionable on LessWrong. New concepts come with new terms and the way we name concepts helps us to relate to our new concepts.

Positivist ideology suggests that non-physical entities that we might talk about in rationality such as biases don’t really exist. Positivists propose that after we get a good ontology for physical entities like atoms, we don’t need to think about ontology anymore and can just focus on epistemology.

This focus on epistemology leads to us spending less effort on finding good names for concepts and good ontology. When new names for concepts that are already known under other names get introduced those new na...
Covid 6/2/22: Declining to Respond
by Zvi — 2022-06-02T13:50:01.330Z

Memorial Day weekend is a good reason not to worry about reporting cases or deaths. FDA declines to respond with making it easier to get treatments we have in ample supply, or to update the vaccines. Congress refuses to fund anything at all, really.

Doesn’t sound like a state of emergency to me. Yet in California, we still have one, and we decline to respond to when we might end the ‘state of emergency.’ In New York, health officials make it clear that if it were up to them, we’d have these pandemic restrictions forever here as well.

Don’t leave it up to them. Respond.

Executive Summary

Paxlvoid and Fluvoxamine remain hard to get.
Time is not a flat circle, that’s a dumb expression, but history rhymes.
Covid-19 is a good excuse to work on curing aging, let’s do that.

Let’s run the numbers.

The Numbers

Predictions

Last week I somehow once again forgot about a holiday, in this case Memorial Day, because I don’t think in those terms naturally and it still boggles my brain that it matters. But in terms of ‘why this prediction was stupid’ that’s why.

Assignment for readers: Next time a holiday is coming up and I forget about it, whoever notices first should write a comment in the Substack version that says ‘you forgot to take into account the upcoming holiday!’ and whoever does scores points. Note that if I don’t explicitly mention the holiday I’m probably forgetting about it.

Prediction from last week: 700,000 cases (+9%) and 2,725 deaths (+15%)

Results: 566k cases (-12%) and 1,994 deaths (-15%)

Prediction for next week: 675,000 cases (+19%) and 2,400 deaths (+20%)

As usual with a holiday drop like this, one must hedge between ‘it was the holiday and will be even higher than trend next week’ and ‘it wasn’t entirely the holiday, this was a real drop.’ Even if I’d remembered the holiday my prediction would have come in high. Is it possible cases have peaked for real? Definitely possible, but the decline in deaths now makes zero underlying sense outside of the holiday, so I’m guessing we’re looking mostly at a large holiday effect.

Deaths

Ah, Memorial Day, yet another holiday. Silly me.

Cases

Vaccine Discussion Offer

Chise, who has provided good information several times, says their DMs are open for anyone with vaccine-related questions or even want someone to listen because you are unsure. A fine public servi...
The horror of what must, yet cannot, be true
by Kaj_Sotala — 2022-06-02T10:20:04.575Z

There’s a type of experience that I feel should have its own word, because nothing that I can think of seems like a fair description.

A first pass would be something like agony, utter horror, a feeling that you can’t stand this, that you are about to fall apart. But those aren’t actually the thing, I think. They are reactions to the thing.

The thing is more like a sense of utter horrifying impossibility. I suspect it’s the kind of an experience H. P. Lovecraft had in mind when talking about mind-breaking sights and things that man was not meant to know.

Suppose you feel an immense need to throw up and know that you are in fact going to throw up, and at the same time you absolutely cannot throw up, because you are in a bus full of people or something. Or to take a stronger example, someone you deeply care about must be alive because you deeply love them, and at the same time you also know for certain that they are dead.

There’s a sense of… you have two facts, and your mind feels like both of them must be true. But they also cannot both be true. It’s like they’re physical objects, magnetic so that they repel each other, and both cannot be in the same place at the same time because physical objects don’t work that way.

Except that, now they are. Some irresistible force has pushed them together and that is tearing the entire universe apart, reshaping the laws of nature to force them to accommodate this fact.

The thing that I’m trying to find a good word for, is that sense of impossibility that accompanies the world being torn apart.

Likely we don’t have a good word for this, because we ordinarily cannot see it. The mind recoils from it, and we only remember the sense of agony that surrounded it. It’s only in some unusual states of consciousness such as deep meditation, that two facts can get pressed against each other in such a way that the conscious mind is forced to stay present and witness the moment of impossibility.

I suspect that the two facts feel like mutually repelling objects because in a sense they are. That the mind is built to treat unpleasant feelings and ideas as something literally repulsive, an independent patch of the world that pushes against that which the mind accepts as true. My hand touches something hot and I quickly pull it away, the painfulness of the object repelling any desire to approach. Then the same mechanism for avoiding physical pain go...