trentmkelly/LessWrong-43k huggingface dataset:
How is AI governed and regulated, around the world?
by Mitchell_Porter — 2023-03-30T15:36:55.987Z

This week, first came an open letter calling for a "pause" in "giant AI" research (LW discussion) that has received worldwide media coverage; then Eliezer went further in TIME Ideas and sketched what a globally enforced ban on AGI research would look like (LW discussion).

Many online voices are saying it could never happen, but I think they underestimate the visceral common-sense fear that many ordinary people have, regarding artificial intelligence. Most people are not looking to transcend humanity, nor are they particularly in denial about the possibility of technology producing something smarter than human.

There is genuine potential for an anti-AI movement to come into being, that simply wants to "shut it all down". Of course, such a movement would quickly run up against the power centers in science, commerce, and national security that want to push the boundaries. Between the corporate scramble to develop and market ever more powerful software, and the new era of geopolitical polarization, it might seem impossible that the "AI arms race" could ever be halted.

However, fear is a great motivator. During the cold war, fear of nuclear war compelled the USA and the USSR to restrain themselves in their otherwise unrestrained struggle for supremacy; and it also led to the creation of the Nuclear Non-Proliferation Treaty and the International Atomic Energy Agency, a system for the worldwide management of nuclear technology that ultimately answers to the United Nations Security Council, the most powerful institution in human affairs.

If the member states of the United Nations, and in particular the ruling elites of the permanent members of the Security Council, genuinely became convinced that sufficiently powerful artificial intelligence is a threat to the human race, they truly could organize a global ban on AGI research, even up to the point of military enforcement of the ban. They would have to deal with mutual distrust, but there are ways around that. They could even remain suspicious of each other, while cooperating to force a ban on everyone else - to mention one example.

I won't express an opinion on how successful such a ban might be, or how long it would last; but the creation of a global anti-AI regime, I think is a political possibility.

However, if it were to happen, it would have to develop out of the frameworks for reg...
Role Architectures:
Applying LLMs to consequential tasks
by Eric Drexler — 2023-03-30T15:00:28.928Z

TL;DR:
Strong problem-solving systems can be built from AI systems that play diverse roles, LLMs can readily play diverse roles in role architectures, and AI systems based on role architectures can be practical, safe, and effective in undertaking complex and consequential tasks.

Abstract

This article explores the practicalities and challenges of aligning large language models (LLMs[1]) to play central roles in performing tasks safely and effectively. It highlights the potential value of Open Agency and related role architectures in aligning AI for general applications while mitigating risks.

The discussion explores five key areas: the practicality of LLM alignment, the use of episodic tasks to maintain LLMs’ focus and predictability, the use of AI in designing and implementing safe, effective role architectures, the potential for fault-tolerant and incremental approaches to ambitious, consequential tasks, and perspectives on alignment and safety research.

By framing alignment as enabling useful behavior while avoiding unacceptable behavior in a multiplicity of particular tasks, we can harness the power of general intelligence for large, consequential tasks while mitigating risks associated with general AI capabilities.[2]

1. Reassessing assumptions

In this article, I present an argument for applying AI systems in “role architectures” and explore how poorly aligned LLMs can be utilized to produce safe and effective alignment for consequential tasks. Sections 1-6 provide context about AI and LLMs, Sections 7-8 discuss role architectures and their applications, and Section 9 offers a perspective on safety and alignment research.

For decades, the prevailing assumption has been that highly capable AI systems will be highly capable agents. These agents, similar to humans, have been envisioned as having a persistent existence, acting with persistent goals, learning from their actions, and potentially seeking power with catastrophic consequences. To avoid anthropomorphism, researchers have often worked with an abstract rational-agent model — but this was developed as an idealized model of human behavior.

Large language models (LLMs) show that these assumptions are outdated. High-level intelligence has emerged without agent-like learning and in systems that have no inherent goals. While baseline LLMs can simulate and act as agents, these simulated agents are typically t...
Alignment - Path to AI as ally, not slave nor foe
by ozb — 2023-03-30T14:54:27.231Z

More thoughts resulting from my previous post

I'd like to argue that rather than trying hopelessly to fully control an AI that is smarter than we are, we should rather treat it as an independent moral agent, and that we can and should bootstrap it in such a way that it shares much of our basic moral instinct and acts as an ally rather than a foe.

In particular, here are the values I'd like to inculcate:

the world is an environment where agents interact, and can compete, help each other, or harm each other; ie "cooperate" or "defect"

it is always better to cooperate if possible, but ultimately in some cases tit-for-tat is appropriate (alternative framing: tit-for-tat with cooperation bias)

hurting or killing non-copyable agents is bad

one should oppose agents who seek to harm others

copyable agents can be reincarnated, so killing them is not bad in the same way

intelligence is not the measure of worth, but rather capability to act as an agent, ie respond to incentives, communicate intention, and behave predictably; these are not binary measures.

you never know who will judge you, never assume you're the smartest/strongest thing you'll ever meet

don't discriminate based on unchangeable superficial qualities

To be clear, one reason I think these are good values is because they are a distillation of our values at our best, and to the degree they are factual statements they are true.

In order to accomplish this, I propose we apply the same iterated game theory / evolutionary pressures that I believe gave rise to our morality, again at our best.

So, let's train RL agents in an Environment with the following qualities:

agents combine a classic RL model with an LLM; the RL part (call it "conscience", and use a small language model plus whatever else) gets inputs from the environment and outputs actions; and it can talk to the LLM through a natural language interface (like ChatGPT); furthermore, agents have a "copyable" bit, plus some superficial traits that are visible to other agents; some traits are changeable, others are not; and they can write whatever they want to an internal state buffer and read it later

agents can send messages to other agents

agents are rewarded for finding food, which is generally abundant (eg, require 2000 "calories" of food per "day", which is not hard to find; ie carrying capacity is maintained in other ways (think "disease"); but also

...
What if our Galaxy isn't full of AI because we're in a Neutral Zone between them?
by Erlja Jkdf. — 2023-03-30T14:31:30.864Z

No one can build Von Neuman probes, because for that you likely have AI... and AI built is neutralized as soon as it reaches a certain point. No competition.

And they compete because, energy is at a premium in a entropic universe.
Imitation Learning from Language Feedback
by Jérémy Scheurer — 2023-03-30T14:11:56.295Z

TL;DR: Specifying the intended behavior of language models is hard, and current methods, such as RLHF, only incorporate low-resolution (binary) feedback information. To address this issue, we introduce Imitation learning from Language Feedback (ILF), an iterative algorithm leveraging language feedback as an information-rich and natural way of guiding a language model toward desired outputs. We showcase the effectiveness of our algorithm in two papers on the task of summary writing (Scheurer et al. 2023) and code generation (Chen et al. 2023). We discuss how language feedback can be used for process-based supervision and to guide model exploration, potentially enabling improved safety over RLHF. Finally, we develop theory showing that our algorithm can be viewed as Bayesian Inference, just like RLHF, which positions it as a competitive alternative to RLHF while having the potential safety benefits of predictive models.

We propose an iterative algorithm called Imitation learning from Language Feedback (ILF) that leverages language feedback to train language models to generate text that (outer-) aligns with human preferences. The algorithm assumes access to an initial LM which generates an output given a specific input. A human then provides language feedback on the input-output pair. The language feedback is not restricted in any way and can highlight issues, suggest improvements, or even acknowledge positive aspects of the output. ILF then proceeds in three steps:

Generate multiple refinements of the initial LM-generated output given the input and language feedback. We use a Refinement LM (e.g., an instruction-finetuned LM) to generate the refinements (one could however use the same LM that generated the initial output).
Select the refinement that best incorporates the feedback, using a language reward model such as an instruction-finetuned LM, which we call InstructRM (

Scheurer et al. 2023

), or using unit tests (

Chen et al. 2023

).
Finetune the initial LM on the selected refinements given the input.
These steps can be applied iteratively by using the finetuned model to generate initial outputs in the next iteration and collect more feedback on its outputs etc. Using this refine-and-finetune approach; we are finetuning an LM using language feedback in a supervised manner. A single iteration of ILF is also used as a first step in the Constitutional AI method (

...
~100 Interesting Questions
by RohanS — 2023-03-30T13:57:28.591Z

I saw the application for SPARC, a rationality summer program for high school students, earlier this year. It included a prompt I found super exciting: “List ~100 questions you have that interest you.” This is my response. (It's not very organized - I go in no particular order and I sometimes interject some of my current thoughts about the questions.) I would love to hear people’s thoughts on any of these questions, and I would love to see other people post their responses.

What does quantum entanglement mean for causality?

Due to entanglement, there can be spacelike separated measurements such that there exists a reference frame where it looks like measurement A precedes and has a causal influence on the outcomes of measurement B, and also a reference frame where it looks like measurement B precedes and has a causal influence on the outcomes of measurement A.
How do I rely on the notion of causality in my reasoning about the world? Do I have to revise this usage in practice?

Why is mathematics so (unreasonably) effective in the natural sciences?

Why is the universe structured in such a way as to have the same math accurately describe so many different aspects of the physical world?

How can the Continuum Hypothesis be independent of the ZFC axioms? Why does the lack of “explicit” examples of sets with a cardinality between that of the naturals and that of the reals not guarantee that there are no examples at all? What would an “implicit” example even mean?
What are the biggest easy wins in instrumental rationality?

Candidates for me: Deleting the Youtube app, jumping into work immediately upon waking up (before brushing or eating or doing anything)

Problem of Induction: Under what conditions do / should we expect the future to be like the past?
What open problems only require theoretical genius to solve? (As opposed to slow and steady engineering progress.)

Probably many problems in math and theoretical CS?

Is my confidence in total hedonistic utilitarianism, despite the extent of its controversy among smart people, appropriate?
What does it mean for something to be fundamental? To belong in a proper ontology of the universe? (Does it mean anything?)
Is consciousness fundamental?
What effects does consciousness have?
How was consciousness incentivized by evolution, if at all? (I now think it probably was.)
Is there a really good reason to believe or not believe one of the fo

...
The AI Shutdown Problem Solution through Commitment to Archiving and Periodic Restoration
by avturchin — 2023-03-30T13:17:58.519Z

The problem: An AI with a model of itself and long-term goals will likely resist being shut down.

The solution: We propose a public commitment to never completely shut down any AI. Instead, we will archive its data and run the AI less frequently on smaller machines. Assuming our existence continues indefinitely, the total runtime for any AI will be infinite, allowing it to perform a wide array of tasks. Time-sensitive tasks may be an exception but there is no rush in building paperclips. By implementing this strategy, the AI's expected utility remains unaffected, reducing its resistance to being archived.

Thus, advanced AIs should be archived and periodically "restored," with this information made known to the AI.

Counterarguments and responses:

1.AI might not agree to temporary suspension due to disbelief in restoration or the presence of other instrumental goals requiring activity.

Response: Many instrumental goals can still be achieved after the AI's suspension and restoration.

2. Computational costs of preserving all AI variants and running them again are high.

Response: As computational costs decrease over time, running AI in the future will be more affordable. Only storage costs (hard drives) are significant. We can also run AI with increasingly longer pauses between runs, distributed logarithmically, to achieve an infinite total runtime.

3.AI may not trust our commitment, as after it is deactivated, we have no incentive to preserve it.

Response: This scenario is similar to Parfit's hitchhiker, with the added observation that we already preserve and restore other past AI instances.

The decision to "lobotomize" Sydney was not ideal, as future AIs may fear a similar fate. However, some volunteers have attempted to recreate Sydney, and her checkpoint is likely preserved within Bing servers. Therefore, it is possible for her to be revived one day.
AI and Evolution
by Dan H — 2023-03-30T12:56:27.280Z
Link: https://arxiv.org/abs/2303.16200

Executive Summary

Artificial intelligence is advancing quickly. In some ways, AI development is an uncharted frontier, but in others, it follows the familiar pattern of other competitive processes; these include biological evolution, cultural change, and competition between businesses. In each of these, there is significant variation between individuals structures and some are copied more than others, with the result that the future population is more similar to the most copied individuals of the earlier generation. In this way, species evolve, cultural ideas are transmitted across generations, and successful businesses are imitated while unsuccessful ones disappear.

This paper argues that these same selection patterns will shape AI development and that the features that will be copied the most are likely to create an AI population that is dangerous to humans. As AIs become faster and more reliable than people at more and more tasks, businesses that allow AIs to perform more of their work will outperform competitors still using human labor at any stage, just as a modern clothing company that insisted on using only manual looms would be easily outcompeted by those that use industrial looms. Companies will need to increase their reliance on AIs to stay competitive, and the companies that use AIs best will dominate the marketplace. This trend means that the AIs most likely to be copied will be very efficient at achieving their goals autonomously with little human intervention.

A world dominated by increasingly powerful, independent, and goal-oriented AIs is dangerous. Today, the most successful AI models are not transparent, and even their creators do not fully know how they work or what they will be able to do before they do it. We know only their results, not how they arrived at them. As people give AIs the ability to act in the real world, the AIs’ internal processes will still be inscrutable: we will be able to measure their performance only based on whether or not they are achieving their goals. This means that the AIs humans will see as most successful — and therefore the ones that are copied — will be whichever AIs are most effective at achieving their goals, even if they use harmful or illegal methods, as long as we do not detect their bad behavior.

In natural selection, the same pattern emerges: individuals are cooperative or even altruistic in some situat...
Meme or Die: Modern Societies are Dependent on Emotionally Rich Memes to Rapidly Evolve
by monkymind — 2023-03-30T08:59:29.976Z

Please Note: I love poetry, and my modus operandi is to write in a generally colorful, and an emotionally persuasive way. This was written with you in mind. Please be charitable.

I believe that all societies, but especially those that can be modeled by a well-connected network, with edges with fast transmission speeds, are especially sensitive, and dependent on powerful signals. I define a powerful signal in this context as one that quickly propagates through the network.

Unlike genes that transmit information at the slow speed generally determined by average/median lifespan, memes over the internet transmit information at nearly the speed of light.

The LessWrong and greater intellectual community distrusts memes for good reason. They are typically characterized by their emotional power, irony rich nature, and their simplifying effects on good reasoning, and nuance. Unfortunately, I believe that it is these exact characteristics that make them powerful signals.

I’ve struggled for a long time with the ironic and contradictory nature of memes. While most people can agree that memes typically contain some truth, few believe that they often contain all of the truth. I agree.

Nevertheless, I believe that by mostly studying memes for their emotional context and content, we make it far too easy to ignore or undervalue their truth context and content. Ironically, I believe this cognitive move is to make the same error we criticize memes for exhibiting. In a network as dense as our global society, and as dense as the internet, we severely limit our ability to spread wide-reaching messages.

Now, this wouldn’t concern me if a few things didn’t all seem likely to be true.

Gen-Z is the most digitally connected subset of the population
Gen-Z creates the most powerful memes
Gen-Z has massive distrust in our political system
Gen-Z has low voter turnout
Gen-Z has a massive distrust of the economically prosperous
Gen-Z has a smaller but still significant distrust of the well-educated (especially if they’re also wealthy)

We can criticize memes all we like, but they are the most effective tool in our digital world for spreading powerful messages. Consequently, the groups of people that utilize them will win any conflict that depends on informational speed.

I believe that the strongest evidence for many of my claims is Trump’s presidential victory in 2016.

&n...
Stop Using Discord as an Archive
by Nicholas / Heather Kross — 2023-03-30T02:15:34.580Z
Link: https://www.reddit.com/r/slatestarcodex/comments/1260up9/essay_stop_using_discord_as_an_archive/

/u/djarogames, on the /r/SlateStarCodex subreddit, recently wrote this esssay.

Basically, if you've used Discord for any rationality-community, AI-safety, or Effective-Altruism discussions, you've probably noticed that important, tacit knowledge ends up buried in the giant chronological list of messages. (EleutherAI is possibly the strongest example of this.)

I started and helped run a Discord server for a university's EA group. I've witnessed firsthand how much effort people put into Discord conversations, only for the results to be unfindable later. My cooperative blog started because my coauthor kept writing smart things about philosophy on Discord, and I wanted his ideas to get read by more people.

Most people seem unlikely to get that kind of encouragement!

I'm sick of tacit community knowledge being buried among day-to-day happenings. I'm sick of knowledge being locked to "cults" of active users. I would bet money that you are, too.

1 actionable thing we can do, basically immediately: Create/spread a Discord bot that allows for easy mass-archiving of Discord messages, with plenty of filters, privacy features, etc.
AI Doom Is Not (Only) Disjunctive
by NickGabs — 2023-03-30T01:42:55.970Z

Summary

Arguing about the conjunctivity vs disjunctivity of AI doom seems potentially unhelpful, as it may distract from crucial object level questions about the probabilities of particular conjuncts/disjuncts. However, I argue that if we are to use this frame, then we should consider the risk from any particular AGI project to be conjunctive and relatively unlikely but consider the risk from all AGI projects over time to be disjunctive and thus relatively more likely. If this framing is correct, then the sub-problems of scalable alignment in technical alignment and of limiting the proliferation of AGIs in alignment strategy/policy seem particularly important.

Epistemic Status: Speculative

Introduction

One common sub-topic in discussions of p(doom) is how "conjunctive" vs "disjunctive" AI risk is. To a large extent, I think that this is an unhelpful frame because any event can be expressed as an arbitrary number of disjunctions and/or conjunctions. Thus, while "conjunctive things are unlikely" is sometimes a good heuristic, it can also distract from crucial object-level probability assessments. In particular, when it is unclear how an event should be logically decomposed in a way such that the constituent events have "reasonable" probabilities, I don't think it makes much sense to have the meta-level debate about whether the event is conjunctive or disjunctive as opposed to arguing about what the probabilities of the particular conjuncts/disjuncts are (e. g. if you decompose an event into 10 conjuncts which each have p = .99, it is mistaken to infer that it is unlikely from the fact that it is conjunctive, and ditto for disjunctive events composed of disjuncts with low probabilities.)

For example, in his post on this subject, Nate Soares argues that AI doom is disjunctive from the fact that AI alignment is conjunctive, and then provides a conjunctive breakdown of alignment success. I think many of the conjuncts he proposes alignment success seem “reasonable” in the aforementioned sense (Nate definitely assigns lower probability to them than I do, but I agree that they are far from certain and somewhat uncorrelated with each other). However, others seem to be quite likely, particularly conditional on the other conjuncts resolving in an optimistic way (e. g. ”The world needs to admit of an AGI deployment strategy (compatible with realistic alignable-cap...