trentmkelly/LessWrong-43k huggingface dataset:
How to Control an LLM's Behavior (why my P(DOOM) went down)
by RogerDearnaley — 2023-11-28T19:56:49.679Z

This is a link-post for a paper I recently read: Pretraining Language Models with Human Preferences, followed by my reactions to this paper.

Reading this paper has significantly reduced my near-term P(DOOM), and I'd like to explain why. Thus, this is also an alignment proposal. While I don't think what I'm proposing here is a complete solution to aligning a superintelligent ASI, I think it might work well up to at least around a human-level AGI, and even be a useful basis to build on at ASI level (at that level, I'd advocate adding on value learning). It can achieve some of the simpler things that people have been hoping we might get from Interpretability (and for more complex things might also combine well with and even simplify Interpretability, if that can be made to work at scale.) It's also simple, immediately actionable, has a fairly low alignment tax, and best of all, also has lots of useful capabilities effects, so that even a superscalar not very concerned about x-risk might well still want to implement it. [I'm fully aware that many people on the Alignment Forum/Less Wrong disapprove of pointing out anything that helps capabilities — however, in this case fortunately the alignment and capabilities advantages are heavily and obviously entangled, and I see this as the sugar that may help the medicine go down, so I am going to briefly mention it.]

The Paper

Let's start with the paper. The authors experiment with a number of different ways you might train an LLM not to do some form of undesired behavior. For the paper, they chose three simple, well-defined bad behaviors for which they had low-computational cost, high-accuracy classifiers, and which were behaviors simple enough that a fairly small, economical-to-pretrain LLM could reasonably be expected to understand them. They demonstrate that, unlike the common approach of first training a foundation model on the task "learn to autocomplete a large chunk of the web, which includes both good and bad behavior", followed by fine-tuning/RLHF on "now learn to recognize and only do good behavior, not bad", it is a lot more effective to build this control training in from the start during the pretraining (they estimate by around an order of magnitude). So they evaluate five different methods to do that (plus standard pretraining as a control).

The simplest behavior training approach they try is just prefilt...
Is there a word for discrimination against A.I.?
by Aaron Bohannon — 2023-11-28T19:03:42.161Z

We've all heard of racism. We've all heard of sexism. Many of us have heard of speciesism. Other than "speciesism", is there a word for discrimination against the intelligence in systems running on computing machinery? Do you think that we're going to need one? Or do you think that we can get by without one?
Update #2 to "Dominant Assurance Contract Platform": EnsureDone
by moyamo — 2023-11-28T18:02:50.367Z

This is the second update to The Economics of the Asteroid Deflection Problem (Dominant Assurance Contracts).

It took a bit longer than I expected but I finally launched a platform for raising money using Dominant Assurance Contracts: EnsureDone.

Why is it called EnsureDone? Well, we ran a naming contest on manifold.markets, and EnsureDone had the most Keynesian Beauty.

We've launched with three projects

Building a Platform and Organization to Foster Logic Tournaments Around the World By Sebastian Garren $1800. Closing 2024-02-08
Keep tabs on your local council By Parth Pasnani $550. Closing 2023-12-30
Reflective clouds in the stratosphere to cool the Earth By Make Sunsets $1000. Closing 2023-12-26

If any of these projects sound interesting, remember that

If they don't raise enough money you get all your money back + a refund bonus
If they do raise enough money you will get the public good.

Pledging is a win-win situation.

If you are interested in producing public goods and using a Dominant Assurance Contract to raise funds apply to have your project featured on EnsureDone!

What's next?

Since I have a product I can sell to people. I'm going to focus the next few months on sales.

I'm unlikely to post again on LessWrong, so if you are interested in this project join our Discord.
Ethicophysics II: Politics is the Mind-Savior
by MadHatter — 2023-11-28T16:27:19.233Z
Link: https://bittertruths.substack.com/p/ethicophysics-ii-affilliation-economics

We present an ethicophysical treatment on the nature of truth and consensus reality, within the framework of a historical approach to ludic futurecasting modeled on the work of Hegel and Marx. We prove an ethicophysical approximate conservation law: the Conservation of Bullshit. We conclude with a lengthy list of open questions, homework problems, and assigned readings, focused on many large and weighty questions about history and politics. We do not presume to know the answers to any of these weighty questions, but we invite any interested readers to submit answers to these questions, either as comments on this post, comments on my substack, direct messages on LessWrong, or direct emails to my personal email.

We hope that, by providing a mathematically rigorous treatment of naturalistic game theory and hooking it up to these weighty political and historical questions, we can encourage people to take history and morality more seriously without starting any unproductive flamewars. I will be setting the moderation settings on this post to Reign of Terror to encourage the LessWrong moderators to enforce these norms to the limits of their ability and discretion.

We provide the list of open questions, homework exercises, and assigned reading from the PDF, in order to facilitate productive discussion on LessWrong. Please share your answers or work-in-progress for any of the following questions in the comments:

Read Keat’s Ode on a Grecian Urn, reflect on the story of Dr. Faust by Goethe, and then formulate a short description of Keat’s Fallacy.
Read Friedrich Schiller’s “On the Aesthetic Education of Man”. Why would Keat’s Fallacy be relevant to moral actions in the world?
Read “Avant-Garde and Kitsch” by Clement Greenberg (1939). Read any honest description of Adolf Hitler’s artistic style and level of skill. Then formulate working definitions of active kitsch, potential kitsch, describe how to measure both, and prove that total kitsch is a conserved quantity. Further, reflect on the question of whether the art school that denied Hitler admission made the right or the wrong decision for German society.
Read T. H. White’s “The Sword in the Stone”. Then explain the concept of the “might makes right makes might” cycle. Prove that “might” is a conserved quantity, and come up with an empirical framework for measuring active and potential might. If a war is won b

...
Neither EA nor e/acc is what we need to build the future
by jasoncrawford — 2023-11-28T16:04:16.803Z
Link: https://rootsofprogress.org/neither-ea-nor-e-acc

Over the last few years, effective altruism has gone through a rise-and-fall story arc worthy of any dramatic tragedy.

The pandemic made them look prescient for warning about global catastrophic risks, including biosafety. A masterful book launch put them on the cover of TIME. But then the arc reversed. The trouble started with FTX, whose founder Sam Bankman-Fried claimed to be acting on EA principles and had begun to fund major EA efforts; its collapse tarnished the community by association with fraud. It was bad for EA if SBF was false in his beliefs; it was worse if he was sincere. Now we’ve just watched a major governance battle over OpenAI that seems to have been driven by concerns about AI safety of exactly the kind long promoted by EA.

SBF was willing to make repeated double-or-nothing wagers until FTX exploded; Helen Toner was apparently willing to let OpenAI be destroyed because of a general feeling that the organization was moving too fast or commercializing too much. Between the two of them, a philosophy that aims to prevent catastrophic risk in the future seems to be creating its own catastrophes in the present. Even Jaan Tallinn is “now questioning the merits of running companies based on the philosophy.”

On top of that, there is just the general sense of doom. All forms of altruism gravitate towards a focus on negatives. EA’s priorities are the relief of suffering and the prevention of disaster. While the community sees the potential of, and earnestly hopes for, a glorious abundant technological future, it is mostly focused not on what we can build but on what might go wrong. The overriding concern is literally the risk of extinction for the human race. Frankly, it’s exhausting.

So I totally understand why there has been a backlash. At some point, I gather, someone said, hey, we don’t want effective altruism, we want “effective accelerationism”—abbreviated “e/acc” (since of course we can’t just call it “EA”). This meme has been frequent in my social feeds lately.

I call it a meme and not a philosophy because… well, as far as I can tell, there isn’t much more to it than memes and vibes. And hey, I love the vibe! It is bold and ambitious. It is terrapunk. It is a vision of a glorious abundant technological future. It is about growth and progress. It is a vibe for the builder, the creator, the discoverer, the inventor.

But…...
Agentic Growth
by Logan Kieller — 2023-11-28T15:45:54.055Z
Link: https://logankieller.substack.com/p/agentic-growth

Personal growth is paramount. A north star to chase indefinitely.

However, most of this growth is reactionary. In the same way, we wouldn’t commend a knee for jerking at a doctor’s strike, we shouldn't see this reactionary response as growth.

The connotation of growth is overwhelmingly positive, yet we only decide that this change is positive from the perspective of the person doing the ‘growing’, which seems to be greatly biased.

For change to lead to self-improvement, I suggest a few alterations to our conception of growth:

First, I want to acknowledge that a significant amount of growth only exists as an agent’s response to the environment.

Second, I want to recognize growth as a neutral word, rather than allow it to be charged with positive or negative connotations. I want to recognize growth as something much more simple: change.

With this model, we can consider how change generally emerges from reactions to an external environment, often out of our control.

The platitude to respond to this is: “It's not what happens to you, but how you react to it that matters.” While I recognize the merits of this notion, it warrants investigation.

Yes, in the face of adversity, stoically bettering oneself is ideal, but we must question whether this change would have occurred at all without adversity. If we assume that external factors drive this change, we resign to our life's path being determined by the series of adversarial events that we choose to survive.

We are all worth a lot more than defining our growth as mere reactions to adversity.

If you put a broken car on a hill and it rolls, it is simply reacting to its surroundings: gravity is the only thing moving it.

As one moves through life, the topology along the road changes, and the car will continue to roll. Looking back, the car may be far from where it started, but not once was it turned on.

Some would describe it as growth, I would describe it as reactionary change. Agentic growth is the process of fixing the engine, turning the key, and speeding off in the direction of your choosing.

Progressing past metaphor, agentic growth involves creating an engine for change that enables one to self-instantiate momentum without needing any exterior adversity at all.

The first important step: break free from the sway of topological changes in life. Rather than letting the rapidly shifting values of the world be forced upon you, und...
AISC project: How promising is automating alignment research? (literature review)
by Bogdan Ionut Cirstea — 2023-11-28T14:47:29.372Z
Link: https://docs.google.com/document/d/1cXU-DoE2O2vLhVBRWYFBcXyyOuvwVQx97wwtMroJzZU/edit?usp=sharing

Apply to work on this project with me at AI Safety Camp 2024 before 1st December 2023.
Two sources of beyond-episode goals (Section 2.2.2 of “Scheming AIs”)
by Joe Carlsmith — 2023-11-28T13:49:49.175Z

This is Section 2.2.2 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?”. There’s also a summary of the full report here (audio here). The summary covers most of the main points and technical terms, and I’m hoping that it will provide much of the context necessary to understand individual sections of the report on their own.

Audio version of this section here, or search "Joe Carlsmith Audio" on your podcast app.

Two sources of beyond-episode goals

Our question, then, is whether we should expect models to have goals
that extend beyond the time horizon of the incentivized episode – that
is, beyond the time horizon that training directly pressures the model
to care about. Why might this happen?

We can distinguish between two different ways.

On the first, the model develops beyond-episode goals for reasons
independent of the way in which beyond-episode goals motivate
instrumental training-gaming. I'll call these
"training-game-independent" beyond-episode goals.

On the second, the model develops beyond-episode goals specifically
because they result in instrumental training-gaming. That is, SGD
"notices" that giving the model beyond-episode goals would cause it
to instrumentally training-game, and thus to do better in training,
so it explicitly moves the model's motives in the direction of
beyond-episode goals, even though this wouldn't have happened
"naturally." I'll call these "training-gaming-dependent"
beyond-episode goals.

These have importantly different properties – and I think it's worth
tracking, in a given analysis of scheming, which one is at stake. Let's
look at each in turn.

Training-game-independent beyond-episode goals

My sense is that the most common story about how schemers arise is via
training-game-independent beyond-episode goals.[1] In particular, the
story goes: the model develops some kind of beyond-episode goal, pursuit
of which correlates well enough with getting reward-on-the-episode that
the goal is reinforced by the training process. Then at some point, the
model realizes that it can better achieve this goal by playing the
training game – generally, for reasons to do with "goal guarding" that
I'll discuss below. So, it turns into a full-fledged schemer at that
point.

On one version of this story, the model specifically learns the
beyond-episode goal prior to situational awareness. Thus, for ex...
How can I use AI without increasing AI-risk?
by Yoav Ravid — 2023-11-28T10:05:44.321Z

I barely use AI tools, and the main reason is that I developed some sort of revulsion to them, because associate them with the harms they cause and the risks they bring.

On the other hand, it seems that more and more, whoever doesn't adopt AI tools will get become far less productive and will be left behind.

And also, if the people who worry about the risk limit themselves, and the people who don't worry about the risk don't limit themselves, it creates a personal responsibility vortex which tilts the balance of power away from safety.

But, perhaps it's possible to get the best of both words - use AI tools , but in a responsible manner that doesn't increase the harms and risks from it.

How does one do that?
A Reading From The Book Of Sequences
by Screwtape — 2023-11-28T06:45:57.806Z

Summary: A speaker will read excerpts from a text the community finds important, weaving the quotes in with experiences from their life or stories.

Tags: Large, Investment, maybe Experimental?

Purpose: This creates common knowledge around the subject of the reading. Everyone knows, and everyone knows everyone else knows. A small number of lessons, repeated regularly, also ensures that newcomers learn the things the community wants everyone to know.

Materials: Copies of the text to hand out to readers. If you do this regularly, it may be nice to have a bunch of copies of a book that contains the things readings will be from.

Announcement Text: “We’ll have a reading from Rationality: From AI to Zombies, a text that drew many of us into the rationality community. Today there will be a reading of _____, read by speaker ______. The general format will be a mix of short speeches on the topics of the reading and songs sung in a group, followed by a potluck luncheon.”

Description: Wait for the audience to take their seats, introduce your speaker if necessary, and then let them speak. This text is written as though there’s one speaker, but I’d strongly recommend getting multiple speakers if you can.

In some detail:

Write the program

Choose the readings.
Write speech notes for each reading, leaving slots for announcements and songs. Have a clear cue to connect them: “...and now we’ll turn to page twelve and sing this song.”
Community announcements
Songs.

Find speakers and songleaders

A few days to a week before the meetup, ask around in your community. It helps to ask specific people directly, but you can also pair the announcement with “we’d like to have extra speakers for parts of this. Please get in touch if you’re interested in helping!”
For speakers without much skill or practice, make sure they have copies of what they’re saying ahead of time.
If time allows, have each person rehearse their part at least once or twice. People usually sound much smoother the second time.

Follow the program

You’re the one who announces when it’s time to start.
A key role is tracking where we are in the program, and calling for the next step.
A short connective speech, even as simple as “thank you John Smith so much for that last reading. I enjoyed it. In a moment, we’ll have the next reading by Jane Davis,” can give people enough time to get up or to sit down. This can also help put

...
Anthropic Fall 2023 Debate Progress Update
by Ansh Radhakrishnan — 2023-11-28T05:37:30.070Z

This is a research update on some work that I’ve been doing on Scalable Oversight at Anthropic, based on the original

AI safety via debate

proposal and a more recent agenda developed at NYU and Anthropic. The core doc was written several months ago, so some of it is likely outdated, but it seemed worth sharing in its current form.

I’d like to thank Tamera Lanham, Sam Bowman, Kamile Lukosiute, Ethan Perez, Jared Kaplan, Amanda Askell, Kamal Ndousse, Shauna Kravec, Yuntao Bai, Alex Tamkin, Newton Cheng, Buck Shlegeris, Akbir Khan, John Hughes, Dan Valentine, Kshitij Sachan, Ryan Greenblatt, Daniel Ziegler, Max Nadeau, David Rein, Julian Michael, Kevin Klyman, Bila Mahdi, Samuel Arnesen, Nat McAleese, Jan Leike, Geoffrey Irving, and Sebastian Farquhar for help, feedback, and thoughtful discussion that improved the quality of this work and write-up.

1. Anthropic’s Debate Agenda

In this doc, I’m referring to the idea first presented in

AI safety via debate

(

blog post

). The basic idea is to supervise future AI systems by pitting them against each other in a debate, encouraging them to argue both sides (or “all sides”) of a question and using the resulting arguments to come to a final answer about the question. In this scheme, we call the systems participating in the debate debaters (though usually, these are actually the same underlying system that’s being prompted to argue against itself), and we call the agent (either another AI system or a human, or a system of humans and AIs working together, etc.) that comes to a final decision about the debate the judge.

For those more or less familiar with the original OAI/Irving et al. Debate agenda, you may wonder if there are any differences between that agenda and the agenda we’re pursuing at Anthropic, and indeed there are!

Sam Bowman

and

Tamera

Lanham

have written up a working

Anthropic-NYU Debate Agenda

draft which is what the experiments in this doc are driving towards. [1]

To quote from there about the basic features of this agenda, and how it differs from the original Debate direction:

Here are the defining features of the base proposal:

Two-player debate on a two-choice question: Two debaters (generally two instances of an LLM) present evidence and arguments to a judge (generally a human or, in some cases, an LLM) to persuade the judge to choose their assigned answer to a question with

...
Apocalypse insurance, and the hardline libertarian take on AI risk
by So8res — 2023-11-28T02:09:52.400Z

Short version: In a saner world, AI labs would have to purchase some sort of "apocalypse insurance", with premiums dependent on their behavior in ways that make reckless behavior monetarily infeasible. I don't expect the Earth to implement such a policy, but it seems worth saying the correct answer aloud anyway.

Background

Is

advocating for AI shutdown

contrary to

libertarianism

? Is advocating for AI shutdown like arguing for markets that are free except when I'm personally uncomfortable about the solution?

Consider the old adage "your right to swing your fists ends where my nose begins". Does a libertarian who wishes not to be punched, need to add an asterisk to their libertarianism, because they sometimes wish to restrict their neighbor's ability to swing their fists?

Not necessarily! There are many theoretical methods available to the staunch libertarian who wants to avoid getting punched in the face, that don't require large state governments. For instance: they might believe in private security and arbitration.

This sort of thing can get messy in practice, though. Suppose that your neighbor sets up a factory that's producing quite a lot of lead dust that threatens your child's health. Now are you supposed to infringe upon their right to run a factory? Are you hiring mercenaries to shut down the factory by force, and then more mercenaries to overcome their counter-mercenaries?

A staunch libertarian can come to many different answers to this question. A common one is: "internalize the externalities".[1] Your neighbor shouldn't be able to fill your air with a bunch of lead dust unless they can pay appropriately for the damages.

(And, if the damages are in fact extraordinarily high, and you manage to bill them appropriately, then this will probably serve as a remarkably good incentive for finding some other metal to work with, or some way to contain the spread of the lead dust. Greed is a powerful force, when harnessed.)

Now, there are plenty of questions about how to determine the size of the damages, and how to make sure that people pay the bills for the damages they cause. There are solutions that sound more state-like, and solutions that sound more like private social contracts and private enforcement. And I think it's worth considering that there are lots of costs that aren't worth billing for, because the cost of the infrastructure t...