trentmkelly/LessWrong-43k huggingface dataset:
Formulating the AI Doom Argument for Analytic Philosophers
by JonathanErhardt — 2023-05-12T07:54:03.346Z

Recently, David Chalmers has asked on Twitter for a canonical formulation of the AI Doom Argument. After some back and forth with EY, DC came up with a suggestion himself.

After reading the conversation I couldn't help but think that EY and DC had partially talked past each other. DC asked a few times for a clarification of the structure of the argument ("I'm still unclear on how it all fits together into an argument", "I see lots of relevant considerations there but no clear central chain of reasoning", etc.). EY then pointed to lists of relevant considerations, but not a formulation of the argument in the specific style that analytic philosophers prefer.
This is my attempt at "providing more structure, like 1-5 points that really matter organized into an argument":
The Argument for AI-Doom

1.0 The Argument for Alignment Relevancy

1. Endowing a powerful AI with a UF (utility function) will move the world significantly towards maximizing value according to the AI's UF.

2. If we fail at alignment, then we endow a powerful AI with a UF that is hostile to human values. (Analytic truth?)

3. Therefore: If we fail at alignment, the world will move significantly towards being hostile to human values. (From (1) and (2))

2.0 The Argument for Alignment Failure as the Default - (The Doom Argument)

4. Most of all possible UFs are hostile to human values.

5. If most of all possible UFs are hostile to human values, then using an unreliable process to pick one will most likely lead to a UF that is hostile to human values.

6. Therefore: Using an unreliable process to endow an AI with a utility function will most likely endow it with a UF that is hostile to human values. (From (4) and (5))

7. We will be using gradient descent to endow AIs with utility functions.

8. Gradient descent is an unreliable process to endow agents with UFs.

9. Therefore, we will most likely endow AI with a UF that is hostile to human values. (From (6), (7) and (8))

2.1 An Argument for Hostility (4) - (Captures part of what Bostrom calls Instrumental Convergence)

10. How much a UF can be satisfied is typically a function of the resources dedicated to pursuing it.

11. Resources are limited.

12. Therefore, there are typically trade-offs between any two distinct UFs. Maximizing one UF will require resources that will prevent other UFs from being pursued.

13. Therefore, most possible UFs are ho...
Three Iterative Processes
by LoganStrohl — 2023-05-12T02:50:06.723Z

Throughout all four phases of naturalism, there are three processes that are constantly repeating in the background: Story revision, POU loops, and a few kinds of oscillation that together I think of as "breathing".

Story Revision

Story revision begins at the very start of a study. If you started with “Catching the Spark”—the orientation procedure I suggested in "Getting Started"—then articulating a story was an explicit part of the process. Otherwise, you probably encountered at least one story about your topic along the way.

I’ve presented “orientation” as something that happens only at the beginning, but that’s not really the case. Reorientation is always happening; as a mentor, I run through an abbreviated version of the orientation procedure at nearly every meeting, until I notice that a student has made a habit of asking and answering these questions on their own. In my own practice, I keep a running list of stories in my notes, a big-picture history of my studies.

There are three parts of story revision:

What's your story about what's going on?
Draw out the assumptions in your story that you might be able to check through direct observation.
Pick according to your curiosity or your sense of what would help you understand the whole space most, and try to investigate that.

A real person's story statements often don’t make a lot of sense to onlookers, presumably because they're meant to reflect intuitions or felt senses more than rigid concepts; I often find myself reminding students, "Remember that this only needs to make sense to you." But for the sake of having some kind of illustration, here is a chronological list of story statements I recorded while studying a pattern in my learning-related actions that I called “failure, smallness, and tumbling”:

Sept. 15: I think that if I focus on what I'm doing wrong, I'll learn faster from my mistakes.
Sept. 16: When I pay attention to the outcome of a goal-directed behavior, I focus in on what I believe to have been under my control.
Sept. 28: I care a lot about exerting control over myself.
Oct. 3: I am very careful to control my words and actions so that I don't attract the attention of people who would hurt me if they knew I am different.
Oct. 17: Part of me thinks that if I let go, everything will fall apart.
Oct. 20: Part of me believes that caring in front of people about things you aren't

...
Term/Category for AI with Neutral Impact?
by isomic — 2023-05-11T22:00:21.950Z

Is there any commonly known term on LessWrong to describe an AGI that does not significantly increase or decrease human value? (For instance, an AGI that stops all future attempts to build AGI, but otherwise tries to preserve the course of human history as if it had never existed).

Would such an AI be considered "aligned"? It seems that most discussions of creating aligned AI focus on making the AI share and actively promote human values, but this seems importantly different from what I described.
Thoughts on LessWrong norms, the Art of Discourse, and moderator mandate
by Ruby — 2023-05-11T21:20:52.537Z

A couple of weeks ago I asked Should LW have an official list of norms? and I appreciate the responses there. Here I want to say what I'm currently thinking following that post, and continue having a public conversation about it.

I think saying more on this topic actually gets into a bunch of interesting questions around LessWrong's purpose, userbase, de facto norms and culture, moderation mandate, etc. Without locking in things as "Officially How It Is Forever", I'll opine on my current thinking on this topics and how I relate to them in practice. It's possible that further public discussion will shift some things here, and after more back-and-forth, it'd make sense to "ratify" some of it more.

With all that said...

LessWrong and The Art of Discourse

LessWrong was founded to be a place for perfecting the Art of Human Rationality, i.e., generally thinking in ways which more reliably result in true beliefs, etc. Similarly, I think there's a closely related "Art of Discourse": communicating in ways that more reliably result in those conversing (and reading along) having more true beliefs. Perhaps it's a sub-art of the Art of Human Rationality.

The real rules of which communication most efficiently gets you towards truth lives in reality. You can choose your norms, but whether those norms are conducive to truth isn't up to you.

The LessWrong community, over its 10-15 year existence, has assembled a number of beliefs about the Art of Discourse. Things like communicating degrees of belief quantitatively, preference for asymmetric weapons, an interest in local validity, etc. We of course don't have the complete art and may be mistaken about pieces of it, but we feel strongly about some of the pieces we believe we possess of this art.

Different people in our community have somewhat different senses of the Art of Discourse, and these even form clusters. But there's a pretty solid common core set of norms on the site, such that if someone is not conforming to them, most people would want them to change their behavior or go elsewhere.

The core point I want to make here is: The Art of [Truth-seeking] Discourse lives in the territory, and we community members attempt to discover it and practice it.

Moderators moderate according to their own understanding of The Art

A thing you could imagine doing is the community comes together, writes down its sense of how you ought to beh...
Alignment, Goals, and The Gut-Head Gap: A Review of Ngo. et al.
by Violet Hour — 2023-05-11T18:06:18.962Z

(Cross-posted from my Substack. Also on the EA Forum)

If someone asks whether I know of anyone who carefully examines the arguments for AI x-risk, my answer is a pretty resounding ‘Ngo’.

Richard Ngo, specifically. Among

other

cool

pieces

, he co-authored

The Alignment Problem from a Deep Learning Perspective

. His co-authors are Lawrence Chan and Sören Mindermann, who very much deserve mentioning, even if their names are less amenable to a pun-based introduction.

Reviewers liked the paper. Despite this, I was disappointed to learn that the chair nevertheless decided to

veto acceptance of the paper

on grounds that it was “too speculative”. Disappointed because I feel like the topic is worthy of discussion. And LessWrong, for all its foibles, provides a platform for informed speculation. So it’s here that I’ll speculate.

Problem: The Difficulty of Alignment

You’re probably aware of the alignment problem, so we’ll only briefly recount the authors’ preferred framing.

Empirically, we observe ML models with ever-increasing capabilities. As we train ever-more sophisticated models, it becomes harder and harder to ensure that our models are actually trying to do the things we want. This allows us to characterize the alignment problem in terms of “three emergent properties” which could arise as a result using RL to train an AGI.

Deceptive reward hacking which exploits imperfect reward functions.
Internally represented goals which generalize beyond the training distribution.
Power-seeking behavior as a means to effectively pursue their goals.

Conceptually, the argument from here is pretty simple. Ngo. et al provide evidence in support of these mechanisms from contemporary ML. These mechanisms already exist, and we can see them. If they’re present in vastly more capable systems, we might end up with human disempowerment.

The discussion starts gently, if worryingly. Empirically, we already have systems that reward hack. You train a system to grab a ball with its claw; it learns to

place the claw between the camera and the ball

so that it appears to be grasping the ball. Also, current LLMs are – to some degree, at least – situationally aware. Bing’s Syndney

recognizes

search results about itself; ChatGPT can make (fallible)

inferences

about the source code it might be ...
Sequence opener: Jordan Harbinger's 6 minute networking
by Severin T. Seehrich — 2023-05-11T17:06:15.941Z

When I dipped my toes into becoming a coach two years ago, I asked around for resources on learning how to network - but not in the sleazy salespeople way, but in a way that fosters healthy relationships and mutual respect with the people around me.

More than one person recommended Jordan Harbinger's free course "6 minute networking".

This course is the basis most of my networking learning rests on. All the subsequent posts in this sequence are basically footnotes to Harbinger's course - though they should be able to stand on their own as well.
Advice for newly busy people
by Severin T. Seehrich — 2023-05-11T16:46:15.313Z

After writing "Advice for interacting with busy people", I was asked to write a follow-up on advice for newly busy people. So, here's a quick list of tools and mental models that help me prioritize.

This list is by no means comprehensive. It's just the tools I know and have loved. Take what's useful, and drop what doesn't fit your brain and life.

1. Prioritizing between projects

a. Apply the Tomorrow Rule.

When someone asks you to join an exciting project that's due half a year from now, it is very, very tempting to say "yes". You'll immediately have a vivid imagination of the shiny outcome, while the workload is far enough in the future to not cross your mind. To mitigate this tendency, it makes sense to apply the Tomorrow Rule. It goes as such: "Am I committed enough to this that I'd clear up time in my schedule tomorrow to make it happen?"

b. If things get too much, do a Productivity Purge.

If you already have too many projects on your plate and can't make reasonable progress on any of them, you might want to go through a round of Cal Newport's productivity purge algorithm. The steps:

"When it feels like your schedule is becoming too overwhelmed, take out a sheet of paper and label it with three columns: professional, extracurricular, and personal. Under “professional” list all the major projects you are currently working on in your professional life (if you’re a student, then this means classes and research, if you have a job, then this means your job, etc). Under “extracurricular” do the same for your side projects (your band, your blog, your plan to write a book). And under “personal” do the same for personal self-improvement projects (from fitness to reading more books).
Under each list try to select one or two projects which, at this point in your life, are the most important and seem like they would yield the greatest returns. Put a star by these projects.
Next, identify the projects that you could stop working on right away with no serious consequences. Cross these out.
Finally, for the projects that are left unmarked, come up with a 1-3 week plan for finalizing and dispatching them. Many of these will be projects for which you owe someone something before you can stop working on them. Come up with a crunch plan for the near future for shutting these down as quickly as possible.
Once you completed your crunch plan you’ll be left with only a small number

...
AI #11: In Search of a Moat
by Zvi — 2023-05-11T15:40:00.462Z

Remember the start of the week? That’s when everyone was talking about a leaked memo from a Google employee, saying that neither Google nor OpenAI had a moat and the future belonged to open source models. The author was clearly a general advocate for open source in general. If he is right, we live in a highly doomed world.

The good news is that I am unconvinced by the arguments made, and believe we do not live in such a world. We do still live in more of such a world than I thought we did a few months ago, and Meta is very much not helping matters. I continue to think ‘Facebook destroys world’ might be the most embarrassing way to go. Please, not like this.

By post time, that was mostly forgotten. We were off to discussing, among other things, constitutional AI, and Google’s new product announcements, and an avalanche of podcasts.

So it goes.

Also, I got myself a for-the-people write-up in The Telegraph (direct gated link) which I am told did well. Was a great experience to do actual word-by-word editing with the aim of reaching regular people. Start of something big?

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility Perhaps quite a lot.

Level Two Bard. Maybe soon it will be a real boy.

Writers Strike, Others Threaten to Strike Back. You might try to replace us. If you do, it will not go well.

The Tone Police. You can use AI to make everyone more polite, still won’t help.

Fun With Image Generation. MidJourney is killing it, also what you do is public.

Introducing. New tools aplenty. GPT-4 winner and still champion.

The Art of the SuperPrompt. Should you learn prompt engineering? Yes.

They Took Our Jobs. Can we get them to take only the right ones?

In Other AI News. There’s always other news.

What Would Be a Fire Alarm for Artificial General Intelligence? Some guesses.

Robotic Fire Alarms. What would be a fire alarm for robotics in particular?

OpenPhil Essay Contest: Change Their Minds. The correct goal was selected.

Quiet Speculations. Tons of different ones.

The Quest for Sane Regulation. Demis Hassabis was at the White House meeteing.

China. If China does get a frontier model, where do you think they got it from?

Do Not Open Source Your AI Development. Worst things you can do department.

Google Employees Say the Darndest Things. Doesn’t mean they’re righ

...
Bayesian update from sensationalistic sources
by houkime — 2023-05-11T15:26:57.789Z

Let's say we have statement X that we ascribe a prior probability of 0.3

Then let's say we have a news source N, presented as an infinite feed of statements.
N is sensationalistic with regards of X that are interesting to N:

If X is true N virtually always says that X is true.
But if X is not true N might still say that it is true.
N only reports if it intends to say that X is true, otherwise it says nothing and covers other topics instead.

The probability of N saying X is true when X is not true for our particular X is not known, but historically we know that N is wrong about 70% of times.

X is interesting to N and N says X is true.
How will this shift our belief in X?
I bet $500 on AI winning the IMO gold medal by 2026
by azsantosk — 2023-05-11T14:46:04.824Z

The bet was arranged on Twitter between @MichaelVassar and I (link).

Conditions are similar to this question on Metaculus, except for the open-source condition (I win even if the AI is closed-source, and in fact I would very much prefer it to be closed-source).

@Zvi has agreed to adjudicate this bet in case there is no agreement on resolution.

Michael has asked me two questions by email, and I'm sharing my answers.

Any thoughts on how to turn winning these sorts of bets into people actually updating?

Geoffrey Hinton mentioned recently that, while GPT4 can "already do simple reasoning", "reasoning is the area where we're still better" [source].

It seems to me that, after being able to beat humans at math, there won't be anything else fundamental where we're still better. I wish more people could realize this now.

For the people who disagree, I would like to get them to make their views known before that. I feel like many people just don't think enough about it and are in a "superposition state", where their belief can collapse to anything without it causing any surprise feelings or model updating on them. Maybe if they think about it and commit to their views today they'll be more surprised when it happens and therefore willing to change their mind about important matters.

Kelvin, thoughts on how you’ll update if it doesn’t at least come close?

Yes. I'll update in the following directions:

That it is much harder for search algorithms to do "amplification" right in rich (very large branching factor) environments like math (and the real-world), than in games such as Go.
That superintelligence is still further away, even if LLMs are having lots of economic use cases and replacing many human jobs.
That we're probably bottlenecked on search algorithms, rather than compute power/model-size. This would have policy implications.
Fatebook for Slack: Track your forecasts, right where your team works
by Sage Future — 2023-05-11T14:11:07.620Z

Announcing Fatebook for Slack - a Slack bot designed to help high-impact orgs build a culture of forecasting.

With Fatebook, you can ask a forecasting question in your team's Slack:

Then, everyone in the channel can forecast:

When it's time to resolve the question as Yes, No or Ambiguous, the author gets a reminder. Then everyone gets a Brier score, based on their accuracy.

It's like a tiny, private, fast Metaculus inside your team's Slack.

Why build a culture of forecasting?

Make better decisions
Communicate more clearly
Build your track record
Trust your most reliable forecasters

We built Fatebook for Slack aiming to help high-impact orgs become more effective.

See the FAQs on the website for more info. We'd really value your feedback in the comments, in our

Discord

, or at

adam@sage-future.org

.

You can add Fatebook to your workspace here.

Thanks to all our alpha testers for their valuable feedback, especially the teams at 80,000 Hours, Lightcone, EA Cambridge, and Samotsvety.
Contra Caller Signs
by jefftk — 2023-05-11T13:10:04.321Z

A contra dance stage can be a bit chaotic and noisy, especially during
a dance. It's a good fit for gestures, and over time callers and
musicians have settled on a few covering the main things they need to
communicate. I couldn't find a listing of these anywhere, so here are
the main ones:

Closed hand facing down with thumb and finger
extended, rotate wrist to waggle: change tunes.

A number of fingers: this many more times through the dance
after this one.

Lift leg or raise fist: going out at the end of this time
through the dance.

Fingers across throat ("cut"): go out immediately, not at the
end of the tune or phrase.

Palm facing up, repeatedly lift fingers: play faster.

Palm facing down, repeatedly lower fingers: play slower.

Place hand on the top of the head, palm down: the top of the
tune goes now.

The tune change sign is usually advisory, letting the band know that
if they want to change at some point they're close to running out of
time. If the caller does it early and emphatically, though, it can
mean that the music is really not working for the dance and you should
play anything else.

Holding up a number of fingers to indicate the number of times
remaining is best done in the second half of the tune (B part), to
make it clearer that you know you're not counting the current time
through.
Notes on the importance and implementation of safety-first cognitive architectures for AI
by Brendon_Wong — 2023-05-11T10:03:38.721Z

Background

I've been working on a knowledge management/representation system for ~1.5 years with the initial goal of increasing personal and collective intelligence. I discovered that this work could be highly applicable to AI safety several weeks ago through Eric Drexler's work on QNRs, which pertains to knowledge representation, and then discovered Open Agencies and the broader work that has been done on cognitive architectures and how they can be made safer. I am excited about the potential for "safety-first cognitive architectures" to help society harness AI in a safer manner.

I figured I would spent a couple hours documenting my thoughts to help people learn more about what cognitive architectures are, how they're relevant for AI safety, and how they might be designed in safer ways. It seems like this field is nascent and the resources aren't aggregated in one place, so this is my first attempt at doing so.

One-Line Summary of Safety-First Cognitive Architectures

Harness AI more safely by having intelligence emerge from separate, non-agentic systems that communicate with each other and operate in interpretable ways, rather than from a singular, agentic AI.

Implementing Safety-First Cognitive Architectures

Separate the components of cognition like planning, execution, and long-term memory. Have each component communicate with the others in a transparent, rate controlled, and human readable way (currently done with natural language, likely done with human-readable structured data in the future).
Ensure that each component can be run by some combination of non-agentic, transient, memory-constrained, and action-constrained AI models, deterministic automated systems, and/or humans.
Incorporate measures at every level of the system, from the system's goals to the outputs of AI models, to evaluate contributions and detect potentially harmful contributions.
Apply the latest alignment research to the AI models used in the architecture.

Why Cognitive Architectures Can Be Safer Than Singular, Agentic AI

When the components of the architecture are put together, the architecture can act like an agent and behave intelligently, but the architecture itself functions more like an information storage and sharing system, or an "international agency" as described in work on Open Agencies, rather than an AI agent. It is essentially fully interpretable and corrigible by design, with go...