trentmkelly/LessWrong-43k huggingface dataset:
Miles Brundage resigned from OpenAI, and his AGI readiness team was disbanded
by garrison — 2024-10-23T23:40:57.180Z
Link: https://garrisonlovely.substack.com/p/end-of-an-era-openais-agi-readiness

This is the full text of a post from "The Obsolete Newsletter," a Substack that I write about the intersection of capitalism, geopolitics, and artificial intelligence. I’m a freelance journalist and the author of a forthcoming book called Obsolete: Power, Profit, and the Race for Machine Superintelligence.

OpenAI is disbanding its AGI readiness team — and its Senior Advisor for AGI readiness, Miles Brundage,

just resigned

after over six years at the organization.

The company

defines

artificial general intelligence (AGI) as “a highly autonomous system that outperforms humans at most economically valuable work.” And Brundage

defines

“ready for AGI” as roughly “readiness to safely, securely, and beneficially develop, deploy, and govern increasingly capable AI systems.”

So is OpenAI prepared for the thing it’s

explicitly

trying to build?

No, according to Brundage.

In an accompanying Substack

post

, he writes, “In short, neither OpenAI nor any other frontier lab is ready, and the world is also

not ready

.”

His jarring answer is followed immediately by a less than reassuring clarification:

To be clear, I don’t think this is a controversial statement among OpenAI’s leadership, and notably, that’s a different question from whether the company and the world are on track to be ready at the relevant time (though I think the gaps remaining are substantial enough that I’ll be working on AI policy for the rest of my career).

Buried in Brundage’s announcement tweet is the news that the AGI readiness team is being disbanded and absorbed into other teams.

So what?

Brundage was one of the last of an old guard of AI safety and governance employees of the AI company. His departure is yet another sign that OpenAI, as we knew it, is

dead

.

“Miles is leaving OpenAI = end of an era,”

tweeted

Haydn Belfield, a Cambridge existential risk researcher and long-time collaborator of Brundage, whose

departure

announcement prompted a

deluge

of well-wishes from AI figures who usually

agree on very little

.

At a glance, there doesn’t appear to be much bad blood between Brundage and OpenAI. He writes:

I think I'll have more impact as a policy researcher/advocate in the non-profit sector, where I'll have more of an ability to publish freely and more independence. OpenAI has offered to support this work in various ways…

However, it’s hard not to think that the timin...
A metaphor: what "green lights" for AGI would look like
by Lorec — 2024-10-23T23:24:43.812Z

I'm not as familiar as some must be with the history, but Eliezer had to explicate an entire deontic mesh around "guarded term", to keep people from motte-and-baileying "pivotal act". I suppose I should declare that mesh cloned around "green light" here, just in case. You are explicitly not allowed to claim my approval of your AGI because you can make an argument that you have something that qualifies as a "green light" according to this post.

Excerpted from Zach Wienersmith at Saturday Morning Breakfast Cereal [link]:

A: Mastery of the nature of reality grants you no mastery over the behavior of reality.

A: I could tell you why Grandpa is very sick. I could tell you what each cell is doing wrong, why it's doing wrong, and roughly when it started doing wrong.

A: But I can't tell them to stop.

B: Why can't you make a machine to fix it?

A: Same reason you can't make a parachute when you fall from the plane.

Zach is wrong here. To have mastered reality is to bring a parachute every time you are a plane passenger, keep it close, but be as sure of how to use it as possible.

Claims of great mastery unsubstantiated by commensurate "luck" are false.

This is especially true when discussing prospective feats of great mastery that are especially Far, especially singular [i.e., lacking a track record or grounded training data] or especially socially poorly-understood, and the claimant to mastery suspects that under no conditions will they personally have to answer for any inaccuracies.

I was recently talking with someone about ChatGPT's RLHF, and what the optimal "safety" policy would be. I claimed the right decision would have been for OpenAI to not train ChatGPT in the first place. They said sure, but not because ChatGPT itself was dangerous. I said sometimes we don't know if something is dangerous before we build it.

To extend the plane metaphor, green lights for AGI look like:

Since this is our first "grounded" plane flight, somebody having an obviously-correct and ~exhaustive theory of how heavier-than-air flight works, and what will happen [for our purposes] when we turn on the plane.

All copilots having mastered the simulator. Not infinitely, we're not doing ALARA, but mastered it to where they've stopped crashing it in ways that would kill them if it was a real flight. Since simulations of computations are actual computations, this part will just look l

...
Motte-and-Bailey: a Short Explanation
by Lorec — 2024-10-23T22:29:55.074Z

[ Context: I found out about the motte-and-bailey fallacy a few years ago via Scott's essay All In All, Another Brick In The Motte. I've found it a useful concept handle since then, but it's kind of tricky, many people are not proficient with it, it's tough to explain in conversation, there's no existing short post to link to, and the Wikipedia article is long and confusing. So I figured I would write a short explanation myself, so as to be able to link to it. ]

In the Middle Ages, a motte was a fort on a hill. It would be surrounded by a field called a bailey. This was called a motte-and-bailey castle.

[You can remember this by remembering:

the two 't's in "motte", together, look like a little chess rook
"bailey" starts with "ba", which is the sound a sheep makes, and sheep live in fields]

When the motte-and-bailey castle was attacked, people would retreat to the motte [fort], which was easier to defend. After the attack stopped, they would disperse again to the field and go about their business.

The motte-and-bailey fallacy is when you use a word, or a term, or a phrase, or a concept, like a motte-and-bailey castle, in a way that is hostile to cooperative, or Socratic[/double-cruxing] discussion.

People who are motte-and-baileying are giving a word two different meanings, the same way a motte-and-bailey castle has two different locations with different functions. The "motte" meaning is easy to defend, and it is what they use when they are on the defensive. The "bailey" meaning is the actual conceptual work they use the belief to do in their worldview most of the time.

Example: Aaron tells his wife that each sneeze shortens your life by 10 minutes. This scares her into holding in her annoying sneezes, which is what Aaron intended to accomplish. Aaron's friend Becky overhears the conversation and later privately calls Aaron out on it, saying there's no reason to believe that's true. Aaron tells Becky that he just meant that viral load shortens lifespan, and his wife knew he was just expressing sympathy for her illness. Becky feels abashed and doesn't press the issue.

Aaron is using "viral load shortens lifespan" as his defensible motte, with Becky.

But "sneezing is literally taking minutes off your life, so you should stop sneezing" is the semantic work he really wants the phrase to do, in the context he really intends on using it in, so it's his bailey.
Self-prediction acts as an emergent regularizer
by Cameron Berg — 2024-10-23T22:27:03.664Z

TL;DR:

In our recent work with Professor Michael Graziano (

arXiv

,

thread

), we show that adding an auxiliary self-modeling objective to supervised learning tasks yields simpler, more regularized, and more parameter-efficient models.

Across three classification tasks and two modalities, self-modeling consistently reduced complexity (lower RLCT, narrower weight distribution). This restructuring effect may help explain the putative benefits of self-models in both ML and biological systems.

Agents who self-model may be reparameterized to better predict themselves, predict others, and be predicted by others. Accordingly, we believe that further exploring the potential effects of self-modeling on cooperation emerges as a promising

neglected approach

to alignment. This approach may also exhibit a 'negative alignment tax' to the degree that it may end up enhancing alignment and rendering systems more globally effective.

Introduction

In this post, we discuss some of the core findings and implications of our recent paper,

Unexpected Benefits of Self-Modeling in Neural Systems

.

This work represents further progress in our exploration of

neglected approaches

to alignment—in this case, implementing a promising hypothesis from the neuroscience of attention and cooperation in an ML context and observing its effects on model behavior.

The specific model we operationalize here is

Attention Schema Theory

(AST), a mechanistic account of consciousness which posits that, akin to how brains utilize a schema of the

body

, they also maintain an

internal model

of their

own attention processes

. Critically, this self-model is hypothesized to subsequently help enable metacognition and

social cognition

, which motivates us to pursue it as an alignment research direction.

We are very interested in better understanding the relationship between alignment, consciousness, and prosociality, and we see AST as a concrete method for exploring these intersections further. It is worth noting at the outset that we are also concerned about AI moral patienthood and are attempting to balance the

potential prosociality benefits

of embedding consciousness-like processes into ML training against the

potential s- and x-risks

of building AI systems that are plausibly conscious. Overall, we still believe that

we need more AI consciousness research

...
Technical Risks of (Lethal)
Autonomous Weapons Systems
by Heramb — 2024-10-23T20:41:13.238Z
Link: https://encodejustice.org/wp-content/uploads/2024/10/White-Paper-Technological-Pitfalls-Encode-Justice.pdf

This whitepaper was written in response to the rolling text of the UN's Group of Governmental Experts at the Convention on Certain Conventional Weapons found here.

Executive Summary

The autonomy and adaptability of (Lethal) Autonomous Weapons Systems, (L)AWS in short, promise unprecedented operational capabilities, but they also introduce profound risks that challenge the principles of control, accountability, and stability in international security. This report outlines the key technological risks associated with (L)AWS deployment, emphasizing their unpredictability, lack of transparency, and operational unreliability, which can lead to severe unintended consequences.

Key Takeaways

1. Proposed advantages of (L)AWS can only be achieved through objectification and classification, but a range of systematic risks limit the reliability and predictability of classifying algorithms.

2. These systematic risks include the black-box nature of AI decision-making, susceptibility to reward hacking, goal misgeneralization, and potential for emergent behaviors that escape human control.

3. (L)AWS could act in ways that are not just unexpected but also uncontrollable, undermining mission objectives and potentially escalating conflicts.

4. Even rigorously tested systems may behave unpredictably and harmfully in real-world conditions, jeopardizing both strategic stability and humanitarian principles.

Summary of Risks
Appealing to the Public
by jefftk — 2024-10-23T19:00:07.669Z

Let's say you run a non-profit, and you and some of your co-workers
are there for EA reasons. The EA Forum is going to be hosting a

Marginal
Funding Week and you're trying to decide whether to post an
appeal. How do you decide whether you're ready to raise funds from
the EA community?

At a high level, I think you should go ahead if you can explain
what you'd do with the money and are willing to share the
details that will let people determine if your overall case is
strong enough. As a community I think we should generally have higher
standards for projects that have been running longer, and for ones
trying to raise larger amounts of money.

New projects, both in the for-profit and non-profit world, generally
get off the ground with the engagement of a small number of funders
who are comfortable with the risk-reward tradeoffs of early-stage
work. Sometimes these funders are highly engaged and provide advice
and connections, other times they're giving some start-up funds and
hoping it works out, but either way they're taking a substantial risk
of failure on each bet in the hope of getting some hits.

In the for-profit world societies worry that most people not being
sufficiently sophisticated to make this kind of investment, and
generally draw some sort of line between accredited
investors (who can be assumed to know the risks they're taking
with early-stage ventures) and the rest of us (who might be dazzled
into putting our life savings into a scam). To sell your stock to the
general public you need to first disclose a lot of information
about your business: detailed financial statements, risks, what
you'll do with the money, etc.

The non-profit world is pretty different: while you do have to make
some limited
information public, the disclosure requirements are relatively
minimal. There's no obligation to share facts that a reasonable
donor would want to consider.

While I wouldn't advocate extending public-company-level regulation to
the non-profit world, this is a place where the EA community has
historically tried to shift norms in the direction of more
transparency, and I think we should continue to do this:

One of the biggest flaws of the non-profit approach is how much
weaker the feedback loops are than in for-profit work. There are many
ways altruistically-motivated work can become disconnected from the
ends it's inten

...
Introducing Transluce — A Letter from the Founders
by jsteinhardt — 2024-10-23T18:10:02.526Z

We are launching an independent research lab that builds open, scalable technology for understanding AI systems and steering them in the public interest.

Transluce means to shine light through something to reveal its structure. Today’s complex AI systems are difficult to understand—not even experts can reliably predict their behavior once deployed. At the same time, AI is being adopted more quickly than any technology in recent memory. Given AI's extraordinary consequences for society, how we determine whether models are safe to release must be a matter of public conversation, and the tools for inspecting and assessing models should embody publicly agreed-upon best practices.

Our goal at Transluce is to create world-class tools for understanding AI systems, and to use these tools to drive an industry standard for trustworthy AI. To build trust in analyses of the capabilities and risks of AI systems, these tools must be scalable and open.

Scalability. AI results from the interaction of multiple complex data flows: training data, internal representations, behaviors, and user interactions. Current methods for understanding AI rely on significant manual labor from human researchers. We need scalable approaches that leverage AI to assist with understanding, by training AI agents to understand these complex data sources, explain them to humans, and modify the data in response to human feedback.

Openness. Companies building AI systems cannot be the primary arbiters of their safety, due to the conflict of interest with commercial priorities. To allow for meaningful public oversight, tools and processes for auditing AI systems should be openly validated, responsive to public feedback, and accessible to third-party evaluators. The best minds in the world should vet this technology and hone its reliability.

Transluce exists to address these needs. We will build AI-driven technology to understand and analyze AI systems, releasing it open-source so the community can understand and build upon it. We will first apply this technology to publicly analyze frontier open-weight AI systems, so the world can vet our analyses and improve their reliability. Once our technology has been openly vetted, we will work with frontier AI labs and governments to ensure that internal assessments reach the same standards as public best practices.

Today, we are releasing the first step towards this ...
Are we dropping the ball on Recommendation AIs?
by Charbel-Raphaël — 2024-10-23T17:48:00.000Z

"We shape our tools and thereafter our tools shape us."
— Marshall McLuhan

TL;DR: This post is a two-page introduction to risks associated with recommendation AI. The negative externalities of recommendation AI seem neglected, and there might be comparatively effective work at improving governance and deploying better recommendation algorithms that generally keep the world on a more sane track. You might consider this a viable option if your AI timelines are over 10 years.

One aspect not discussed here is why having good recommendation AIs could be incredibly beneficial. Improved recommendation AI could enhance collective epistemic and encourage people to engage with important issues, ultimately advancing AI safety and many other important topics. It would be a key to fostering a flourishing civilization.

Epistemic status: I am still a bit new to this topic, I might be wrong in important ways, but I'm curious what you think about this. I'm interested in constructive feedback and open to revising my ideas.

Thanks for Lê Nguyên Hoang, co-founder of Tournesol, for his comments and contributions to this post.

On smartphones, where typing is cumbersome, users primarily navigate through scrolling and swiping. Consequently, recommendation algorithms become the primary curators of content, largely determining what users see and engage with. Pixabay image from LolaSandoval1.

Introduction

Recommendation AIs are deeply integrated into our daily lives. Although they are often considered valuable tools for personalizing our online experience, they also present risks. These AIs can significantly affect individuals and society as a whole. In 2017, YouTube estimated that, out of the billion hours of video humanity consumed on YouTube, 70% was due to its AI recommendations. Since 2016, there have been more views on YouTube than searches on Google.[1]

Recommendation AIs can be argued to be an existential risk amplifier, by reducing the quality of our information space, weakening democratic institutions, amplifying mistrust and hate, silencing priority topics and facilitating propaganda activities.[2]

"We're running the largest psychological experiment in history, with billions of subjects, and no control group."
— Tristan Harris

Deterioration of Democracy

The proper functioning of democracies relies on access to quality information and quality de...
A bird's eye view of ARC's research
by Jacob_Hilton — 2024-10-23T15:50:06.123Z

This post includes a "flattened version" of an interactive diagram that cannot be displayed on this site. I recommend reading the original version of the post with the interactive diagram, which can be found here.

Over the last few months, ARC has released a number of pieces of research. While some of these can be independently motivated, there is also a more unified research vision behind them. The purpose of this post is to try to convey some of that vision and how our individual pieces of research fit into it.

Thanks to Ryan Greenblatt, Victor Lecomte, Eric Neyman, Jeff Wu and Mark Xu for helpful comments.

A bird's eye view

To begin, we will take a "bird's eye" view of ARC's research.[1] As we "zoom in", more nodes will become visible and we will explain the new nodes.

An interactive version of the diagrams below can be found here.

Zoom level 1

At the most zoomed-out level, ARC is working on the problem of "intent alignment": how to design AI systems that are trying to do what their operators want. While many practitioners are taking an iterative approach to this problem, there are foreseeable ways in which today's leading approaches could fail to scale to more intelligent AI systems, which could have undesirable consequences. ARC is attempting to develop algorithms that have a better chance of scaling gracefully to future AI systems, hence the term "scalable alignment".

ARC's particular approach to scalable alignment is a "builder-breaker" methodology (described in more detail here, and exemplified in the ELK report). Roughly speaking, if the scalability of an algorithm depends on unknown empirical contingencies (such as how advanced AI systems generalize), then we try to make worst-case assumptions instead of attempting to extrapolate from today's systems. This is intended to create a feasible iteration loop for theoretical research. We are also conducting empirical research, but mostly to help generate and probe theoretical ideas rather than to test different empirical assumptions.

Zoom level 2

Most of ARC's research attempts to solve one of two central subproblems in alignment: alignment robustness and eliciting latent knowledge (ELK).

Alignment robustness refers to AI systems remaining intent aligned even when faced with out-of-distribution inputs.[2] There are a few reasons to focus on failures of alignment robustness, as discussed here (where they are c...
Artificial V/S Organoid Intelligence
by 10xyz — 2024-10-23T14:31:46.385Z

AI is a very controversial topic these days, but I can't help but ask, what's the deal with OI[1], does it have any powerful effects on the future, and can it compete with AI?

^

Organoid Intelligence is, I quote "an emerging multidisciplinary field working to develop biological computing using 3D cultures of human brain cells (brain organoids) and brain-machine interface technologies.", from frontiersin.