trentmkelly/LessWrong-43k huggingface dataset:
If you’re very optimistic about ELK then you should be optimistic about outer alignment
by Sam Marks — 2022-04-27T19:30:11.785Z

[The content of this short, nontechnical post is entirely unoriginal – it’s a reframing which I’ve personally found helpful for understanding the thrust of ELK. More specifically, all of the ideas here can be found in the

ELK document

and its appendices.

I came to this reframing while in conversation with Eric Neyman. Thanks also to Ben Edelman for feedback.]

Historically, the AI alignment problem was first posed along the following lines:

Suppose you have a superintelligent AI which is optimizing some utility function. Unless that utility function is well-aligned with human values, the AI will, in the course of optimizing its own utility function, probably destroy everything we care about. Note that the issue isn’t that the AI doesn’t understand human values – after all, the AI must be able to model human behavior very well in order to achieve its own objectives, and modeling human behavior seems hard to do without understanding human values. The issue is that the AI’s model of human values lives in its world model, not in its utility function.

[ETA: to be clear, the problem described above – finding a utility function that actually reflects our values – is what we might nowadays call outer alignment, and excludes concerns like safe exploration, robustness to distributional shift, mesa-optimizers, etc. For the rest of this post, when I write "alignment," assume I'm talking about outer alignment.]

Some of the first ideas for alignment were along the lines of “maybe we can (1) get the AI to learn human values, and then (2) plug the learned model of human values in as a reinforcement learner’s utility function.” Broadly speaking, let’s lump this class of approaches together as “value learning.”

One funny thing about aligning a superintelligence via value learning is that it actually results in a redundant copy of human values: one copy, the one learned via value learning, lives in the utility function; and the other copy, the one learned by the RL agent, lives in the world model.[1]

Another funny thing about value learning is that getting an AI to selectively learn human values (and nothing else) seems harder than getting an AI to just form an accurate world model (which necessarily contains a model of human values inside of it). Part of the issue is that it’s easy to incentivize correct predictions (just train an ML model to make ...
The Game of Masks
by Slimepriestess — 2022-04-27T18:03:12.423Z
Link: https://hivewired.wordpress.com/2022/04/27/the-game-of-masks/

Epistemic Status: Endorsed
Content Warning: Antimemetic Biasing Hazard, Debiasing hazard, Commitment Hazard
Part of the Series: Open Portals
Author: Octavia

0.

So Scott has been talking about Lacan lately and I’ve been honestly pretty impressed by how hard he seemed to bounce off it. Impressed enough to get me to crawl out of my hole and write this essay that I've been putting off for the last eight months. After spending so many words of the Sadly Porn review talking about why someone might tend towards obscurantism and what sorts of things they might be gesturing at when they say things like “people think the giving tree is a mother” he somehow manages completely to miss the conceptual forest for the giving trees. Congrats Scott, you got the malformed version of the antimeme and were turned back by the confusion spell.

I think Lacan is actually very important and an understanding of Lacanian insights can have a tremendous amount of predictive power when interacting with others. It’s also something you can totally weaponize, and I think that is part of what leads the psychoanalysts to tend towards obscurantism and vaguely gesturing in the direction of what they really mean. They also just seem to like their jargon, and it’s not like the rats are ones to talk when it comes to that.

So, first: The Mother is not literally your mother, The Father is not literally your father, The Phallus is not literally your dick, this is a realm of mirrors and illusions and nothing is as it seems. It’s impolite to point out the thing that everyone hasn’t agreed not to talk about, but let's do it anyway, I’m going to strip away the metaphor and give it to you like Roshi gives it to his student and we’ll see if that takes or just confuses you further.

This is all about symbols. It’s about the effects of symbol systems on cognition and how the evolution of symbols and concepts in the mind of a child affects how they are able to perceive and engage with themselves and the world. You could think of it as an elaboration on a medium-rare Sapir-Worf hypothesis. However, words aren’t the only things that symbols in the mind can be made of. What’s going on heavily overlaps with language and involves language-like systems for organizing concepts, but is occurring within each individual in a way that we would probably call pre-linguistic from a strict words-have-definitions sta...
Law-Following AI 3: Lawless AI Agents Undermine Stabilizing Agreements
by Cullen — 2022-04-27T17:30:25.915Z

This post is written in my personal capacity, and does not necessarily represent the views of OpenAI or any other organization. Cross-posted to the Effective Altruism Forum

In the previous post of this sequence, I argued that intent-aligned AIs would, by default, have incentives to break the law. This post goes into one particularly bad consequence of that incentive: the increased difficulty of making credible pre-AGI commitments about post-AGI actions.

Image by OpenAI's DALL·E

In AGI policy and strategy, it would often be useful to adopt credible commitments about what various actors will do post-AGI. For example, it may be desirable for two leading nations in AGI to agree to refrain from racing to build AGI (at the potential cost to AGI safety) and instead split the economic upside from AGI, thereby transforming a negative-sum dynamic into a positive-sum one.[1] Nations might agree to forego their own development of militarily useful AI systems if they receive security assurances from states that do choose to pursue such systems,[2] thus reducing the number of actors pursuing potentially dangerous military AI development, and therefore reducing the risk of a catastrophic accident. One can imagine similar types of pre-AGI agreements concerning space colonization post-AGI, the welfare of digital minds, democratic control of AGI, and many more important issues in the post-AGI world.

Lawless AI poses an enormous problem for such agreements. Agreements between entities, such as states or companies, require performance by agents of the parties: a state or corporation only acts through its agents. But if one party cannot convincingly show that their agents will carry out orders to comply with such agreements, the other party will be very reluctant to enter into an agreement with them. All the more so if the principal cannot prevent its agents from working against aspects of the agreement that are detrimental to the principal.

I will use the following as a toy problem to show what I mean more concretely: X and Y are two leading AGI development firms considering agreeing to split the profits from AGI development if they both agree to common minimum implement safety measures to prevent racing.[3] How might this play out?

X might worry that Y will develop an AGI agent, A(Y), with the objective of maximizing the value of Y's shares. A(Y) can do this in ways that would under...
Law-Following AI 2: Intent Alignment + Superintelligence → Lawless AI (By Default)
by Cullen — 2022-04-27T17:27:24.210Z

This post is written in my personal capacity, and does not necessarily represent the views of OpenAI or any other organization. Cross-posted to the Effective Altruism Forum.

In the first post of this sequence, I defined "law-following AI" ("LFAI") and "intent alignment." In this post, I will begin to motivate the case for working on LFAI by showing that an AI agent A that is only intent-aligned (i.e., without additional law-following constraints directly on the agents) would in many cases break the law to advance its human principal H's interests.

If you agree that it is generally good for agents to be bound by law, then this should be a pro tanto reason to support work on LFAI in addition to intent alignment. If you need more convincing that LFAI would be generally good, the next few posts will explore some more specific motivations for LFAI.

Image by OpenAI's DALL·E

Sufficiently Intelligent Agents Can Evade Detection and Attribution

I suspect this point will need little argumentation to many of my readers. Evasion and obfuscation seems likely to be a generally promising means of ensuring self-preservation (because detection would increase the probability of punishment and constraint), which in turn is plausibly a convergent instrumental goal of optimizing AI systems.[1]

For example, to evade detection and attribution, A could:

Behave innocently only when being observed;

Orchestrate complex malevolent actions through a coordinated series of small, inscrutable or innocent-seeming steps;

Surreptitiously create and run subagents or copies of itself on new hardware, potentially very far removed from its original hardware;

Convince humans or other systems to take actions on its behalf, without revealing its true identity; and/or

Delete logs of its actions.

A sufficiently intelligent AI agent could likely consistently fool humans using these and other (unforeseen) techniques.

Other Ways to Circumvent Law

Even in the best case scenario, where the agent is detected and within the jurisdiction of a well-functioning legal system, it would be reasonable to question whether A or H could be effectively subject to normal legal processes.[2] If A had a motivation to, A could help H escape liability by, for example:[3]

"Outlawyering" counterparties.

Benefitting H in a way that would undermine recourse for creditors.

Shifting and hiding assets in ways that would make it diff

...
Law-Following AI 1: Sequence Introduction and Structure
by Cullen — 2022-04-27T17:26:57.004Z

This post is written in my personal capacity, and does not necessarily represent the views of OpenAI or any other organization. Cross-posted to the Effective Altruism Forum.

This sequence of posts will argue that working to ensure that AI systems follow laws is a worthwhile way to improve the long-term future of AI.[1]

The structure of this sequence will be as follows:

First, in this post, I will define some key terms and sketch what an ideal law-following AI ("LFAI") system might look like.

In the next few posts, I will explain why law-following might not emerge by default given the existing constellation of alignment approaches, financial objectives, and legal constraints, and explain why this is troubling.

Finally, I will propose some policy and technical routes to ameliorating these problems.

If the vision here excites you, and you would like to get funding to work on it, get in touch. I may be excited to recommend grants for people working on this, as long as it does not distract them from working on more important alignment issues.

Image by OpenAI's DALL·E.

Key Definitions

A law-following AI , or LFAI , is an AI system that is designed to rigorously comply with some defined set of human-originating rules ("laws"),[2] using legal interpretative techniques,[3] under the assumption that those laws apply to the AI in the same way that they would to a human. By "intrinsically motivated," I mean that the AI is motivated to obey those rules regardless of whether (a) its human principal wants it to obey the law,[4] or (b) disobeying the law would be instrumentally valuable.[5] (The Appendix to this post explores some possible conceptual issues with this definition of LFAI.)

I will compare LFAI with intent-aligned AI. The standard definition of "intent alignment" generally concerns only the relationship between some property of a human principal H and the actions of the human's AI agent A:

Jan Leike et al. define the "agent alignment problem" as "How can we create agents that behave in accordance with the user's intentions?"

Amanda Askell et al. define "alignment" as "the degree of overlap between the way two agents rank different outcomes."

Paul Christiano defines "AI alignment" as "A is trying to do what H wants it to do."

Richard Ngo endorses Christiano's definition.

Iason Gabriel does not directly define "intent alignment," but provides a taxonomy wherein a...
[Intro to brain-like-AGI safety] 13. Symbol grounding & human social instincts
by Steven Byrnes — 2022-04-27T13:30:33.773Z

(Last revised: July 2024. See changelog at the bottom.)

13.1 Post summary / Table of contents

Part of the “Intro to brain-like-AGI safety” post series.

In

the previous post

, I proposed that one path forward for AGI safety involves reverse-engineering human social instincts—the innate reactions in the

Steering Subsystem (hypothalamus and brainstem)

that contribute to human social behavior and moral intuitions. This post will go through some examples of how human social instincts might work.

My intention is not to offer complete and accurate descriptions of human social instinct algorithms, but rather to gesture at the kinds of algorithms that a reverse-engineering project should be looking for.

(Note: Since first writing this post, I think I've made good progress on this reverse-engineering project! See especially A theory of laughter and Neuroscience of human social instincts: a sketch. There’s still tons more work to do, of course. This post will not discuss those later developments, but rather explain and motivate the basic problem.)

This post, like Posts

#2

–

#7

but unlike the rest of the series, is pure neuroscience, with almost no mention of AGI besides here and the conclusion.

Table of contents:

Section 13.2 explains, first, why I expect to find innate, genetically-hardwired, social instinct circuits in the hypothalamus and/or brainstem, and second, why evolution had to solve a tricky puzzle when designing these circuits. Specifically, these circuits have to solve a

“symbol grounding problem”

, by taking the symbols in a

learned-from-scratch

world-model, and somehow connecting them to the appropriate social reactions.
Section 13.3 and 13.4 go through two relatively simple examples where I attempt to explain recognizable social behaviors in terms of innate reaction circuits:

filial imprinting

in Section 13.3, and fear-of-strangers in Section 13.4.
Section 13.5 discusses an additional ingredient that I suspect plays an important role in many social instincts, which I call “transient empathetic simulations”. This mechanism enables reactions where recognizing or expecting a feeling in someone else triggers a “response feeling” in oneself—for example, if I notice that my rival is suffering, it triggers the warm feelings of schadenfreude. To be clear, “transient empathetic simulations” have little in common with how the word “empathy”

...
The case for turning glowfic into Sequences
by Thomas Kwa — 2022-04-27T06:58:57.395Z

Epistemic status: serious, uncertain, moderate importance. Leaving comments is encouraged!

Recently Eliezer Yudkowsky's main writing output has been rationalist

glowfic

: role-play fiction written on an Internet forum like

glowfic.com

.[1] I think that LessWrongers, fans of rationalist fiction, and anyone interested in raising the sanity waterline should consider distilling lessons from Yudkowsky glowfic into LW posts.

Here's the basic case:

The original Sequences were extremely good at building the community and raising the sanity waterline. If you want to make the impact case, I think they plausibly get multiple percent of the entire rationality community's impact points.
The Sequences are incomplete. Despite

most of his knowledge coming from his home planet

, Eliezer has in fact learned things since 2009. Having more sequences would be great!
Eliezer's thoughts are still relevant. Recent posts like conversations with AI researchers, calling attention to underrated ideas, and short fiction have all been good.
Not everyone gets useful lessons from the Sequences, because Eliezer's writing style and tone can be annoying. Eliezer was

deliberately discourteous

towards "stupid ideas", and regrets this. Also, some people just learn better from other writing styles.
Eliezer stopped writing Sequences and probably cannot write more. This is a combination of Eliezer's chronic fatigue syndrome and being tired of trolls / bad takes in comments. The only medium he can write in without being drained is glowfic. Thus, even though it's a non-serious format, glowfic is Eliezer's main intellectual output right now.
Eliezer attempts to make his glowfic roughly as edifying as HPMOR, and among people who read glowfic, some find it

really good

at teaching rationality.
But not everyone can read glowfic and gain useful lessons.

Many people (including me) read fiction for maximum enjoyment rather than to extract maximum knowledge. I had the same problem with HPMOR, reading through it like any other novel, whereas many people I know who got more from HPMOR read it carefully, perhaps stopping after every chapter to think about the goals and motivations of each character and predict what happens next.
It's really long (>>100 hours of reading time just for the existing material in the planecrash sequence) and most of the rationality lessons are contained in a small proportion of the words

...
[Link] Evidence of Fabricated Data in a Vitamin C trial by Paul E Marik et al in CHEST
by Kenny — 2022-04-27T06:48:06.597Z

Link:

This Scattrd Corn: Evidence of Fabricated Data in a Vitamin C trial by Paul E Marik et al in CHEST

The trial paper:

Hydrocortisone, Vitamin C, and Thiamine for the Treatment of Severe Sepsis and Septic Shock - CHEST

The trial, and funding it, were discussed first (that I remember) about five years ago:

Sepsis Cure Needs An RCT | Otium

An Opportunity to Throw Money at the Problem of Medical Science – Aceso Under Glass

Update on Sepsis: Donations Probably Unnecessary | Otium
SERI ML Alignment Theory Scholars Program 2022
by Ryan Kidd — 2022-04-27T00:43:38.221Z

The Stanford Existential Risks Initiative (SERI) recently opened applications for the second iteration of the ML Alignment Theory Scholars (MATS) Program, which aims to help aspiring alignment researchers enter the field by pairing them with established research mentors and fostering an academic community in Berkeley, California over the summer. Current mentors include Alex Gray, Beth Barnes, Evan Hubinger, John Wentworth, Leo Gao and Stuart Armstrong. Applications close on May 15 and include a written response to mentor-specific selection questions, viewable on our website.

Who is this program for?

Our ideal applicant has:

an understanding of the AI alignment research landscape equivalent to having chttps://www.agisafetyfundamentals.com/ai-alignment-curriculumompleted the AGI Safety Fundamentals course;
previous experience with technical research (e.g. ML, CS, maths, physics, neuroscience, etc.);
strong motivation to pursue a career in AI alignment research.

For the first stage of the program, we asked each alignment researcher to provide a set of questions that are sufficient to select candidates they would be happy to mentor. Applicants can apply for multiple mentors, but will have to complete each mentor’s selection questions.

What will this program involve?

Over four weeks, the participants will develop an understanding of a research agenda at the forefront of AI alignment through online readings and cohort discussions, averaging 10 h/week from Jun 6 to Jul 1. After this initial upskilling period, the scholars will be paired with an established AI alignment researcher for a two-week “research sprint” to test fit from Jul 4 to Jul 15. Assuming all goes well, scholars will be accepted into an eight-week intensive research program in Berkeley, California over the US summer break (Jul 25 to Sep 16).

Participants will obtain a $6,000 grant for completing the training and research sprint and $16,000 at the conclusion of the program. Furthermore, all expenses will be covered, including accommodation, office space and networking events with the Bay Area alignment community. We are happy to continue providing funding after the two month period to promising scholars, at the discretion of our research mentors. International students can apply to the program, and will arrive in the US under a B1 visa.

We hope to run another iteration of the program in the winter, and possibl...
EU Maximizing in a Gloomy World
by David Udell — 2022-04-27T00:28:58.494Z

A thought I picked up from Ryan Greenblatt.

Even if we're probably doomed, the expected value of trying hard to improbably save Humanity Ascendant is still huge -- on the order of saving a planetary civilization in expectation, even if you're not saving a cosmic civilization in expectation.
Doomy emotions can be instrumentally useful … when you can potentially affect the thing you're being doomy about! Emotions just aren't instrumentally useful when they're about things that are out of your control. In that case, your feeling bad doesn't help anything. If you've just updated your world model in a very doomy direction, feel that hit … and then go right on EU maximizing in the world you live in, in the normal way!
(Worrying about fixed amounts of doom is like worrying about sunk costs.)
Why Copilot Accelerates Timelines
by Michaël Trazzi — 2022-04-26T22:06:19.507Z

"Say we have intelligences that are narrowly human / superhuman on every task you can think of (which, for what it’s worth, I think will happen within 5-10 years). How long before we have self-replicating factories? Until foom? Until things are dangerously out of our control? Until GDP doubles within one year? In what order do these things happen?" (source)

When discussing Takeoff Speeds, I feel the debate often gets stuck in some kind of false dichotomy between Fast and Slow, where the crux seems to be about whether some self-improving AI would be able to foom without human assistance.

Instead, we could get a Moderate Takeoff (think months or years), where AI does not self-improve (by itself). Instead, there would be a reinforcing feedback loop where progress in AI leads to AI becoming increasingly useful to make progress in AI, with humans in the loop at all times.

On top of that, things might just happen privately at some AI lab for a few months until the AI is able to foom by itself, which will look like foom for everyone outside that lab.

AI Helping Humans with AI

In Superintelligence, takeoff is defined as the period between AGI and superintelligence. In this post, I will be using as takeoff's starting point the first "AI Helping Humans with AI" (in a meaningful way), or AIHHAI for short, since it will arise before we get fully general intelligence and accelerate AI progress. Here are some examples of what I have in mind for "helping humans in a meaningful way":

GPT-N that you can prompt with "I am stuck with this transformer architecture trying to solve problem X". GPT-N would be AIHHAI if it answers along the lines of "In this arXiv article, they used trick Z to solve problems similar to X. Have you considered implementing it?", and using an implementation of Z would solve X >50% of the time.
Another example would be if some code generation tool like Copilot makes ML engineers substantially more productive in writing ML code. Making predictions about productivity is tricky and hard to measure, but it would involve something like accepting code suggestions a decent amount, like 100x more than what engineers using Copilot currently accept.

(Sidenote: My general impression from people using Copilot is that they believe it's becoming increasingly useful, and use it on a daily basis, though it rarely auto-completes the right line of code right away. Given that...