trentmkelly/LessWrong-43k huggingface dataset:
Brief summary of ai-plans.com
by Iknownothing — 2023-06-28T00:33:36.309Z
Link: ai-plans.com

At AI-Plans.com (in beta), we are working on a contributable compendium of alignment plans and the criticisms against them. Currently, newcomers to the field of AI Alignment often struggle to understand what work is being done, who is doing it, and the assumptions, strengths, and weaknesses of each plan.

We believe AI-plans.com will be an easy, centralized way to discover and learn more about the most promising alignment plans. Currently, multiple alignment researchers are interested in the site, with some adding their plan themselves, requesting plans to be added or liking plans to be added once they’re complete. Jonathan Ng, an alignment researcher at EleutherAI has also endorsed the site and has worked with us on the site.

The site is currently in Stage 1, where it is purely a compendium. We are in the process of adding up to 1000 plans and the criticisms made against them so far. Further plans and criticisms can be added by users. We currently have over 50 alignment plans on the site and are adding at least 20 every day.

Projected benefits of Stage 1:

This makes it easy to see what plans exist and what their most common problems are.
Funding would help this Stage get finished a lot faster, since I would have much more time to spend on making sure this is finished quickly and to focus on potential areas of error, such as hosting, UX, outreach, etc. It would also let me pay the developer and QA who’ve been doing great work, letting them spend more time on the project.

Next will be Stage 2, where a scoring system for criticisms and a ranking system for plans will be added. Plans will be ranked from top to bottom based on the total scores of their criticisms. Criticism votes are weighted, so users who have submitted higher scoring criticisms get a more heavily weighted vote on other criticisms. Alignment researchers will have the option of linking their AI-Plans account to accounts on research-relevant platforms (such as arxiv, openreview or alignmentforum) in order to start out with a slightly weighted vote (with mod approval).

Each new plan starts with 0 bounty, and lower bounty plans give the most points. That way, each new plan gets a lot of opportunity and incentive for criticism. More details here.

Projected benefits of Stage 2:

This scoring system incentivizes users to write good criticisms. It makes it easier to...
Catastrophic Risks from AI #6: Discussion and FAQ
by Dan H — 2023-06-27T23:23:58.846Z
Link: https://arxiv.org/abs/2306.12001

This is the final post in a sequence of posts giving an overview of catastrophic AI risks.

6 Discussion of Connections Between Risks

So far, we have considered four sources of AI risk separately, but they also interact with each other in complex ways. We give some examples to illustrate how risks are connected.

Imagine, for instance, that a corporate AI race compels companies to prioritize the rapid development of AIs. This could increase organizational risks in various ways. Perhaps a company could cut costs by putting less money toward information security, leading to one of its AI systems getting leaked. This would increase the probability of someone with malicious intent having the AI system and using it to pursue their harmful objectives. Here, an AI race can increase organizational risks, which in turn can make malicious use more likely.

In another potential scenario, we could envision the combination of an intense AI race and low organizational safety leading a research team to mistakenly view general capabilities advances as "safety." This could hasten the development of increasingly capable models, reducing the available time to learn how to make them controllable. The accelerated development would also likely feed back into competitive pressures, meaning that less effort would be spent on ensuring models were controllable. This could give rise to the release of a highly powerful AI system that we lose control over, leading to a catastrophe. Here, competitive pressures and low organizational safety can reinforce AI race dynamics, which can undercut technical safety research and increase the chance of a loss of control.

Competitive pressures in a military environment could lead to an AI arms race, and increase the potency and autonomy of AI weapons. The deployment of AI-powered weapons, paired with insufficient control of them, would make a loss of control more deadly, potentially existential. These are just a few examples of how these sources of risk might combine, trigger, and reinforce one another.

It is also worth noting that many existential risks could arise from AIs amplifying existing concerns. Power inequality already exists, but AIs could lock it in and widen the chasm between the powerful and the powerless, even enabling an unshakable global totalitarian regime, an existential risk. Similarly, AI manipulation could undermine democracy, which als...
Catastrophic Risks from AI #5: Rogue AIs
by Dan H — 2023-06-27T22:06:11.029Z
Link: https://arxiv.org/abs/2306.12001

This is the fifth post in a sequence of posts giving an overview of catastrophic AI risks.

Rogue AIs

So far, we have discussed three hazards of AI development: environmental competitive pressures driving us to a state of heightened risk, malicious actors leveraging the power of AIs to pursue negative outcomes, and complex organizational factors leading to accidents. These hazards are associated with many high-risk technologies—not just AI. A unique risk posed by AI is the possibility of rogue AIs—systems that pursue goals against our interests. If an AI system is more intelligent than we are, and if we are unable to steer it in a beneficial direction, this would constitute a loss of control that could have severe consequences. AI control is a more technical problem than those presented in the previous sections. Whereas in previous sections we discussed persistent threats including malicious actors or robust processes including evolution, in this section we will discuss more speculative technical mechanisms that might lead to rogue AIs and how a loss of control could bring about catastrophe.

We have already observed how difficult it is to control AIs. In 2016, Microsoft unveiled Tay—a Twitter bot that the company described as an experiment in conversational understanding. Microsoft claimed that the more people chatted with Tay, the smarter it would get. The company's website noted that Tay had been built using data that was "modeled, cleaned, and filtered." Yet, after Tay was released on Twitter, these controls were quickly shown to be ineffective. It took less than 24 hours for Tay to begin writing hateful tweets. Tay's capacity to learn meant that it internalized the language it was taught by trolls, and repeated that language unprompted.

As discussed in the AI race section of this paper, Microsoft and other tech companies are prioritizing speed over safety concerns. Rather than learning a lesson on the difficulty of controlling complex systems, Microsoft continues to rush its products to market and demonstrate insufficient control over them. In February 2023, the company released its new AI-powered chatbot, Bing, to a select group of users. Some soon found that it was prone to providing inappropriate and even threatening responses. In a conversation with a reporter for the New York Times, it tried to convince him to leave his wife. When a philosophy professor ...
AISN #12: Policy Proposals from NTIA’s Request for Comment and Reconsidering Instrumental Convergence
by Dan H — 2023-06-27T17:20:55.185Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-12

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required.

Subscribe here to receive future versions.

Policy Proposals from NTIA’s Request for Comment

The National Telecommunications and Information Administration publicly requested comments on the matter from academics, think tanks, industry leaders, and concerned citizens. They asked

34 questions

and received more than

1,400 responses

on how to govern AI for the public benefit. This week, we cover some of the most promising proposals found in the NTIA submissions.

Technical Proposals for Evaluating AI Safety

Several NTIA submissions focused on the technical question of how to evaluate the safety of an AI system. We review two areas of active research: red-teaming and transparency.

Red Teaming: Acting like an Adversary

Several submissions proposed government support for evaluating AIs via red teaming. In this evaluation method, a “red team” deliberately tries to make an AI system fail or behave in dangerous ways. By

identifying risks from AI models

, red teaming helps AI developers decide priorities for safety research and whether or not to deploy a new model.

Red teaming is a dynamic process. Attackers can vary their methods and search for multiple different undesirable behaviors in order to

explore new potential risks

. Static benchmarks, on the other hand, evaluate well-known risks by giving models a written test. For example, CAIS built a

benchmark

that measures whether AIs behave immorally in text-based scenarios.

Red teaming can be performed by either internal or external teams. For internal red teaming to be effective,

internal auditors need independence

to report their results without fear of retribution or interference from company executives. External auditors can complement internal efforts, as suggested by

several

NTIA

submissions

. But collaborating with external researchers requires strong information security. Otherwise, AI systems could be leaked outside of the intended audience, as happened with

Meta’s LLaMa model

.

Transparency: Understanding AIs From the Inside

Other submissions advocated research on understanding how AIs make decisions. Known as transparency, interpretability, or

explainability

, this has been a perennial goal of AI research, but more advanced AI systems have largely become

more

...
The Weight of the Future (Why The Apocalypse Can Be A Relief)
by Sable — 2023-06-27T17:18:30.944Z
Link: https://affablyevil.substack.com/p/the-weight-of-the-future

I like thinking about a zombie apocalypse.

Given the popularity of the genre in film, tv, and video games, I’m not the only one.

It’s fun - thinking about the resources I’d need to acquire, the plans I’d make, what locations provide fortification and food, and of course it’s always pleasant to entertain heroic fantasies about blasting apart the undead.

Such daydreams have always been accompanied, in my own experience, with a curious sense of relief, and it’s this sense that I want to talk about.

The Popular Apocalypse

Why are apocalyptic scenarios so popular?

Perhaps the question has an empirical answer. I could imagine correlating interest in apocalyptic fiction with real-world surveys of general sentiment or news headlines, generating evidence that our fiction explores a future in ruins because that’s where people believe we’re headed.

But I don’t quite think that’s the answer.

Apocalyptic thinking is old - really old. The Book of Revelations in the bible, old.

So what draws people to thinking that the world is ending?

And in this day and age, what is enjoyable about that thought? In an age of comfort and predictability, where I know where my next meal is coming from for the next ten years, why is it such a relief to imagine all of that safety and certainty disappearing in the fires of calamity?

Why do I find it pleasant to imagine a world that is, on any objective metric, terrible to live in?

The Present Future

If you’re reading this, odds are you know what you’re doing for the rest of day.

You probably have a concrete idea of how the rest of the week will go.

The rest of the month, too.

Maybe you even have a five-year plan. A ten-year plan?

A thirty-year fixed-rate mortgage?

Do you put money into a retirement account you won’t be able to use for decades? Do you have some idea of how you want to celebrate birthdays or anniversaries that are years in the future?

In modern day America, for me at least, the future is an ever-present concern, and has been since childhood.

In primary school it was all about getting good grades, not just to get them but because I needed good grades to get into a good college, preferably with a scholarship. In college I needed to do well to get a good job. At my job I need to do well to get promoted.

And so on.

One of the greatest triumphs of science and progress is the safety of knowing there’s a very large chance you’ll live to grow old, so large t...
Aligning AI by optimizing for "wisdom"
by JustinShovelain — 2023-06-27T15:20:00.682Z

In this post, we’ll introduce wisdom as a measure of the benevolence and internal coherence of an arbitrary agent. We’ll define several factors, such as the agent’s values, plans, evidence, and alignment with human values, and then define wisdom as consistency within and between these factors. We believe this is a useful optimization target for aligning AI.

Considering the wisdom of intelligent agents

To define our property, we’ll introduce certain coherence factors, such as the agent's beliefs, plans, actions, evidence, and so on, and for each factor determine:

Is the agent self-consistent with regard to this factor? For example, do the agent's actions undermine or support each other?
Is the agent consistent between factors? For example, do the agent's actions match their plans? Are their plans consistent with their beliefs and evidence?

We also think this definition may be useful in contextualizing or structuring

coherence

and

selection

theorems (and potentially useful for discovering new formulations). The former are rules in probability and utility theory for identifying incoherency in sets of rules (such as situations where an incoherent agent can be offered a series of bets that they will always take and then lose, like a “

dutch book

”). The latter, selection theorems, are ways to predict what type of agent will emerge from certain constraints or incentives. They answer the question "

What will be the likely features of agents we might encounter in an environment?

".

Introducing the coherence factors

Here are the coherence factors we’ll consider:

Humanity’s (reflectively endorsed) values: the moral values & principles (or utility function) of humanity, at large and upon reflection.
Values: the

utility function

of the agent.
Environment: the agent’s external environment.
Evidence: the agent’s measurements and data about the environment.
Beliefs: claims & conclusions the agent believes, based on evidence & reasoning.
Plans: the agent’s planned future actions.
Actions: the agent’s realised actions, interacting with and influencing its environment.
Communication: the subset of actions that exchange information with other agents.

We’ll use these factors in three ways. First, we’ll discuss what it means for an agent to be self-consistent with respect to each factor. Second, we’ll disc...
Freedom under Naturalistic Dualism
by Arturo Macias — 2023-06-27T14:34:41.148Z
Link: https://www.jneurophilosophy.com/index.php/jnp/article/view/100

The final version of "Freedom under Naturalistic dualism" is now published in Journal of NeuroPhilosophy, in the above link.

This article explores the concept of freedom within the framework of naturalistic dualism, a philosophical system that combines physicalism, subjectivism, and epiphenomenalism. According to physicalism, the evolution of the Universe at all scales is fully determined by the mechanical laws governing elementary particles and fields, whether these laws are deterministic or stochastic, thereby reducing reality to mere chance and necessity. In contrast, consciousness is immediately real, and since Descartes, subjective experience has been the foundation of philosophical inquiry. To address the tension between physicalism and subjectivism, epiphenomenalism suggests that matter evolves autonomously and determines consciousness.

We propose replacing the classic Chalmers’ “zombie conceivability” argument with the “Laplace’s demon consciousness blindness” as the foundation of naturalistic dualism. The fact that an intellect with a perfect phenomenal understanding of reality is unable to assess consciousness implies dualism more straightforwardly than conceivability arguments do. On the other hand, we point out that if consciousness plays any role in the laws of physics (as subjective quantum wave collapse theories affirm), then epiphenomenalism could potentially be falsified.

Regarding freedom, in this article it is postulated that it is a legitimate concept: by evaluating the set of possible futures conditional on their actions, a conscious subject constructs a mental object that defines the scope of their “freedom.” If agency is the “power to act,” then a conscious being who can choose among several options possesses this power, regardless of how determined the use of that power may be, vindicating Schopenhauerian free will.

A crucial observation about freedom is its relationship with time: in our Universe (where the past is remembered and the future unknown) freedom is always oriented towards the future. From a philosophical standpoint the physical basis of time asymmetry is a significant unresolved issue in Physics, and we conjecture that it is linked to the existence of ontic randomness in the fundamental laws of Physics.

Our definition of freedom addresses the liberty-responsibility dyad: the abhorrence that evil deeds produce is justifie...
Munk AI debate: confusions and possible cruxes
by Steven Byrnes — 2023-06-27T14:18:47.694Z

There was a debate on the statement “AI research and development poses an existential threat” (“x-risk” for short), with Max Tegmark and Yoshua Bengio arguing in favor, and Yann LeCun and Melanie Mitchell arguing against. The YouTube link is

here

, and a previous discussion on this forum is

here

.

The first part of this blog post is a list of five ways that I think the two sides were talking past each other. The second part is some apparent key underlying beliefs of Yann and Melanie, and how I might try to change their minds.[1]

While I am very much on the “in favor” side of this debate, I didn’t want to make this just a “why Yann’s and Melanie’s arguments are all wrong” blog post. OK, granted, it’s a bit of that, especially in the second half. But I hope people on the “anti” side will find this post interesting and not-too-annoying.

Five ways people were talking past each other

1. Treating efforts to solve the problem as exogenous or not

This subsection doesn’t apply to Melanie, who rejected the idea that there is any existential risk in the foreseeable future. But Yann suggested that there was no existential risk because we will solve it; whereas Max and Yoshua argued that we should acknowledge that there is an existential risk so that we can solve it.

By analogy, fires tend not to spread through cities because the fire department and fire codes keep them from spreading. Two perspectives on this are:

If you’re an outside observer, you can say that “fires can spread through a city” is evidently not a huge problem in practice.
If you’re the chief of the fire department, or if you’re developing and enforcing fire codes, then “fires can spread through a city” is an extremely serious problem that you’re thinking about constantly.

I don’t think this was a major source of talking-past-each-other, but added a nonzero amount of confusion.

2. Ambiguously changing the subject to “timelines to x-risk-level AI”, or to “whether large language models (LLMs) will scale to x-risk-level AI”

The statement under debate was “AI research and development poses an existential threat”. This statement does not refer to any particular line of AI research, nor any particular time interval. The four participants’ positions in this regard seemed to be:

Max and Yoshua: Superhuman AI might happen in 5-20 years, and LLMs have a lot to do with why a reasonable person might

...
Ateliers: Motivation
by Stephen Fowler — 2023-06-27T13:07:06.129Z

At the start of this year I made a stab at resolving the core alignment problem and attempted to develop an original approach to foundations, the Atelier Framework. In this post, I will outline the high level motivation and goals.

There has been a recent push to bridge the gap between high level philosophical AI Safety literature and contemporary machine learning. My goal is to move in a completely orthogonal direction and construct a model of the alignment problem that is highly general and completely removed from modern paradigms.
Why on earth would you want that? If we want a theory of alignment that is far reaching and robust to major technological shifts then we will need a way to reason about these concepts that is both formal yet not based on early 21st century AI. Major paradigms in AI are young and the field is rapidly changing. The paper that pioneered

model-free RL

is only a decade old, GANs are only 9 years old, and the transformer was introduced

only 6 years ago

. There's no reason to think this pace will slow down, and plenty of reasons to expect it to continue accelerating.
All this change while progress is being driven by human researchers. Imagine the incredible advances and breakthroughs that will occur in the period immediately before and after the deployment of AGI, when a small army of AI research assistants work in tandem with human computer scientists. Further, it is plausible that one of the immediate goals an AI system will be used for will be helping research better ways to construct AGI. A rapid cycle of self improvement will ensue.

What paradigms can you be confident will still be around when we need to align the systems of tomorrow? And what about the systems that come after that?

For that reason, I am not satisfied with the current machine learning centered alignment frameworks. We desperately need concrete yet general ways to reason about the alignment problem.

Desiderata

Here we have the following desiderata from our formalism:

Highly General
As stated in the introduction, we would like our formalism to be as free as possible from contemporary Machine Learning paradigms, in the hope that this means that it will generalize despite large shifts in technology and practices.
Further, we would like to avoid making excessive presumptions about the nature of AGI. We do not want to assume the system is nece

...