trentmkelly/LessWrong-43k huggingface dataset:
NYU Debate Training Update: Methods, Baselines, Preliminary Results
by samarnesen — 2024-07-06T18:28:54.053Z

[This writeup reflects work done jointly with David Rein and Julian Michael at NYU's Alignment Research Group]

Introduction

In the past year, there have been a number of projects aimed at validating the basic premises behind debate as a mechanism for scalable oversight (see here, here, and here). One important next step would be to actually train models to debate, as this would let us directly test how models adapt to a debate training objective and whether the debate protocol can withstand optimization pressure. For the last few months at NYU, we’ve been trying to do just that. Our hope is that by doing so with open-sourced models and code, we can help support scalable oversight research being conducted outside of the major labs.

In this write-up, we wanted to share our experimental setup, training procedure, and some of our preliminary results, with the hope of receiving feedback on our approach before we extend our work to more complicated domains.

TL;DR

Like previous work, we are using a hidden-information setup with questions from the QuALITY dataset. For now, our preliminary results are limited to single-turn, simultaneous debates, however we are currently working on extending this to the more interesting multi-turn setting.
Also like previous work, we test the robustness of the debate protocol by measuring the judge’s accuracy when evaluating rounds between debaters of varying skill levels. However, we also introduce two new baselines (both extensions of consultancy from Michael et al., 2023) to more precisely isolate the impact of the debate protocol itself.
We construct debaters of differing skill levels by varying the model type, supervised finetuning data, best-of-N value, and use of Direct Preference Optimization.
We use a finetuned version of GPT-4-Turbo as our judge, after observing that open-source models were too error-prone and that the standard GPT-4-Turbo struggled to produce calibrated results.
Our preliminary experiments show a clear positive relationship between debater skill and judge accuracy, providing some early signal that judge accuracy improves as the debaters improve, despite the judge staying fixed. However, we also find that this same effect can be replicated in our baseline settings that lack debate’s distinctive attributes.

Motivation and Background

Brief Motivation for Debate as a Scalable Oversight Technique

As originally proposed by ...
Scalable oversight as a quantitative rather than qualitative problem
by Buck — 2024-07-06T17:42:41.325Z

[Many of these ideas were developed in conversation with Ryan Greenblatt and Ansh Radhakrishnan; a lot of this isn’t original but I haven’t seen it written up]

A lot of the time when people talk about scalable oversight, which I'll define for the moment as "the problem of developing techniques that allow you to assess whether an action taken or proposed by an AI is good (oversee it), in a way that works even if the AI is very smart and knowledgeable", the main application they describe is the following situation: you have an extremely powerful AI that's taking actions that might have subtle properties that humans are unable to understand, and you want to be oversee them (e.g. to train it, or to determine whether the actions will have catastrophic consequences).

I think that this is indeed part of the value proposition for scalable oversight. But in my opinion, it's missing the more central application of these techniques: situations where the AIs are taking many actions, where humans would eventually understand any particular action if they spent a whole lot of time investigating it, but where that amount of time taken to oversee any action is prohibitively large. In such cases, the point of scalable oversight is to allow them to oversee actions at a much lower cost in terms of human time--to push out the Pareto frontier of oversight quality vs cost.

Of course, when you're thinking about arbitrarily subtle actions, this distinction doesn't matter: there are actions subtle enough that overseeing them would take more than the whole budget of the AI project, and for those actions there's no difference between "we can't afford to oversee it" and "we'd never be able to oversee it even with infinite budget".

But I think that this distinction will be very important in practice, and I often think that people make reasoning errors when thinking about scalable oversight due to neglecting it. The basic reason to think that it's important is that we probably need to oversee many model actions, and so we probably need to decide on a per-action oversight budget (e.g. the amount of human labor spent on ) that will determine what subtlety of actions we will or won't be able to resolve, but we probably won't choose a per-action oversight budget that is close to the whole AI project budget.

Another reason this feels important to me is that, as discussed here, I think a lot of AI r...
An AI Manhattan Project is Not Inevitable
by Maxwell Tabarrok — 2024-07-06T16:42:35.920Z
Link: https://www.maximum-progress.com/p/an-ai-manhattan-project-is-not-inevitable

Early last month, Leopold Aschenbrenner released a

long essay

and

podcast

outlining his projections for the future of AI. Both of these sources are full of interesting arguments and evidence, for a comprehensive summary see

Zvi’s post here

. Rather than going point by point I will instead accept the major premises of Leopold’s essay but contest some of his conclusions.

So what are the major premises of his piece?

There will be several orders of magnitude increase in investment into AI. 100x more spending, 100x more compute, 100x more efficient algorithms, and an order of magnitude or two gains from some form of “learning by doing” or “unhobbling” on top.
This investment scale up will be sufficient to achieve AGI. This means the models on the other side of the predicted compute scale up will be able to automate all cognitive jobs with vast scale and speed.
These capabilities will be essential to international military competition.

All of these premises are believable to me and well-argued for in Leopold’s piece.

Leopold contends that these premises imply that the national security state will take over AI research and the major data centers, locking down national secrets in a race against China, akin to the Manhattan project.

Ultimately, my main claim here is descriptive: whether we like it or not, superintelligence won’t look like an SF startup, and in some way will be primarily in the domain of national security.

By late 26/27/28 … the core AGI research team (a few hundred researchers) will move to a secure location; the trillion-dollar cluster will be built in record-speed; The Project will be on.

The main problem is that Leopold’s premises can be applied to conclude that other technologies will also inevitably lead to a Manhattan project, but these projects never arrived. Consider electricity. It's an incredibly powerful technology with rapid scale up, sufficient to empower those who have it far beyond those who don’t and it is essential to military competition. Every tank and missile and all the tech to manufacture them relies on electricity. But there was never a Manhattan project for this technology. It’s initial invention and spread was private and decentralized. The current sources of production and use are mostly private.

This is true of most other technologies with military uses: explosives, steel, computing, the internet, etc. All of these technol...
[Linkpost] A Case for AI Consciousness
by cdkg — 2024-07-06T14:52:21.704Z
Link: https://philpapers.org/archive/GOLACF-2.pdf

Just wanted to share a new paper on AI consciousness with Simon Goldstein that members of this community might be interested in. Here's the abstract:

It is generally assumed that existing artificial systems are not phenomenally conscious, and that the construction of phenomenally conscious artificial systems would require significant technological progress if it is possible at all. We challenge this assumption by arguing that if Global Workspace Theory (GWT) — a leading scientific theory of phenomenal consciousness — is correct, then instances of one widely implemented AI architecture, the artificial language agent, might easily be made phenomenally conscious if they are not already. Along the way, we articulate an explicit methodology for thinking about how to apply scientific theories of consciousness to artificial systems and employ this methodology to arrive at a set of necessary and sufficient conditions for phenomenal consciousness according to GWT.
Can agents coordinate on randomness without outside sources?
by Mikhail Samin — 2024-07-06T13:43:44.633Z

There are situations where two agents that can read each other’s source code want to have a bit of random information (e.g., they want to cooperate and split an indivisible thing by deciding randomly who’ll have it and having half of it in expectation).

If these agents don’t have access to unpredictable sources of randomness (e.g., it’s acausal trade through very detailed simulations and they can perfectly predict parts of each other’s environment), is there any way for them to coordinate on generating a random bit in a way that can’t be manipulated/exploited?

I feel like the answer should be “no”, as schemes like “both transparently don’t look at a part of the other’s source code when they coordinate on the scheme, then look and generate the bit from these parts” fail due to the possibility of something else looking at the whole of the agent A’s source code first and then designing a successor/a simulated agent B which won’t look there but will produce the needed bit.

But maybe I’m missing something?
AI Alignment Research Engineer Accelerator (ARENA): Call for applicants v4.0
by James Fox — 2024-07-06T11:34:57.227Z

TL;DR

We are excited to announce the fourth iteration of ARENA (Alignment Research Engineer Accelerator), a 4-5 week ML bootcamp with a focus on AI safety! ARENA’s mission is to provide talented individuals with the skills, tools, and environment necessary for upskilling in ML engineering, for the purpose of contributing directly to AI alignment in technical roles. ARENA will be running in-person from LISA from 2nd September - 4th October (the first week is an optional review of the fundamentals of neural networks).

Apply

here

before 23:59 July 20th anywhere on Earth!

Summary

ARENA has been successfully run three times, with alumni going on to become MATS scholars and LASR participants; AI safety engineers at Apollo Research, Anthropic, METR, and OpenAI; and even starting their own AI safety organisations!

This iteration will run from 2nd September - 4th October (the first week is an optional review of the fundamentals of neural networks) at the London Initiative for Safe AI (LISA) in Old Street, London. LISA houses small organisations (e.g., Apollo Research, BlueDot Impact), several other AI safety researcher development programmes (e.g., LASR Labs, MATS extension, PIBBS, Pivotal), and many individual researchers (independent and externally affiliated). Being situated at LISA, therefore, brings several benefits, e.g. facilitating productive discussions about AI safety & different agendas, allowing participants to form a better picture of what working on AI safety can look like in practice, and offering chances for research collaborations post-ARENA.

The main goals of ARENA are to:

Help participants skill up in ML relevant for AI alignment.
Produce researchers and engineers who want to work in alignment and help them make concrete next career steps.
Help participants develop inside views about AI safety and the paths to impact of different agendas.

The programme's structure will remain broadly the same as ARENA 3.0 (see below); however, we are also adding an additional week on evaluations.

For more information, see our website.

Also, note that we have a Slack group designed to support the independent study of the material (join link here).

Outline of Content

The 4-5 week program will be structured as follows:

Chapter 0 - Fundamentals

Before getting into more advanced topics, we first cover the basics of deep learning, including ba...
Links and brief musings for June
by Kaj_Sotala — 2024-07-06T10:10:03.344Z

Links in English

Schrödinger’s Ursula

Apparently the concept of Schrödinger’s cat got popularized thanks to Ursula Le Guin.

Schrödinger originally invented the cat image as a gag. If true believers in quantum mechanics are right that the microworld’s uncertainties are dispelled only when we observe it, Schrödinger felt, this must also sometimes happen in the macroworld – and that’s ridiculous. Writing in a paper published in 1935 in the German-language journal Naturwissenschaften (23 807), he presented his famous cat-in-a-box image to show why such a notion is foolish.

For a while, few paid attention. […] there were no citations of the phrase “Schrödinger’s cat” in the literature for almost 20 years. […] The American philosopher and logician Hilary Putnam (1926–2016) first learned of Schrödinger’s image around 1960. […] In his 1965 paper “A philosopher looks at quantum mechanics” Putnam called it “absurd” to say that human observers determine what exists. But he was unable to refute the idea. […]

It was to be another decade before the cat and its bizarre implications jumped into popular culture. In 1974 Le Guin published The Dispossessed (1974), an award-winning book about a physicist whose new, relativistic theory of time draws him into the politics of the pacifist-anarchist society in which he lived. […] she read up on relativity theory to make her character’s “theory of simultaneity” sound plausible.

Le Guin, it appears, seems to have read Putnam’s article in about 1972. “The Cat & the apparatus exist, & will be in State 0 or State 1, IF somebody looks,” Le Guin wrote in a note to herself. “But if he doesn’t look, we can’t say they’re in State 0, or State 1, or in fact exist at all.” […]

In “Schrödinger’s cat”, which Le Guin finished in September 1972 but didn’t publish for another two years, an unnamed narrator senses that “things appear to be coming to some sort of climax”. […] Le Guin’s story was soon followed by other fictional and non-fictional treatments of quantum mechanics in which Schrödinger’s cat is a major figure. Examples include the Schrödinger’s Cat Trilogy (Robert Anton Wilson, 1979); Schrödinger’s Baby: a Novel (H R McGregor, 1999); Schrödinger’s Ball (Adam Felber, 2006); Blueprints of the Afterlife (Ryan Budinot, 2012).

Parenting advice

Thing of Things: Non-Parents Give Crappy Parenting Advice. Ways in which the common thought of “I’ve be...
Indecision and internalized authority figures
by Kaj_Sotala — 2024-07-06T10:10:02.528Z

A trauma book I was reading had an interesting claim that indecision is often because the person looks for the approval of an internalized authority figure (the writer is a Jungian therapist so attributed it to looking for the approval of an internalized parent, but I think it can be broader) but is unable to predict what action they would approve of.

I feel like that has some intuitive truth to it, in that when I don’t care about anyone’s opinion (or if nobody ever finds out) then it’s much easier to just pick one action and commit to it even if it might go badly. But one of the main reasons why I might struggle with that is if I fear that anyone would judge me for doing things incorrectly.

Or it can be a conflict between different internalized authority figures. “If I do this then X will be angry at me but if I do the other thing, then Y will be angry at me”. Or just the expectation that X will be angry at me no matter what I do.

This also reminds me of the way I think a big part of the appeal of various ideologies and explicit decision-making systems is that they give people a clear external ruleset that tells them what to do. Then if things go wrong, people can always appeal (either explicitly or just inside their own mind) to having followed The Right Procedure and thus being free of blame.

The most obvious external example of this is people within a bureaucracy following the rules to the letter and never deviating from them in order to avoid blame. Or more loosely, following what feels like the common wisdom – “nobody ever got fired for buying IBM”.

But those are examples of people trying to avoid blame from an existing, external authority. I think people also do a corresponding move to avoid blame from internalized authority figures – such as by trying to follow a formalized ethical rule system such as utilitarianism or deontology.

Of course, if the system is one that easily drives people off a cliff when followed (e.g. extreme utilitarianism demanding infinite self-sacrifice), this isn’t necessarily helpful. Now what was supposed to give relief from the pressures of constant inner judgment, turns into a seemingly-rigorous proof for why the person has to constantly sacrifice everything for the benefit of others.

At one point I also wondered why it is that being very confident about what you say makes you very persuasive to many people. Wh...
Free Will, Determinism, And Choice
by Zero Contradictions — 2024-07-06T06:34:41.495Z
Link: https://thewaywardaxolotl.blogspot.com/2020/05/free-will-determinism-and-choice.html

Rough Summary:

Free will is not "free" in the sense of being uncaused. Free will is "free" in the sense that you are the cause. You are free to choose between A or B if your choice will determine the outcome.
Free will and determinism are both assumptions that are implicit in everything that we do. They both depend on each other. The conflict between free will and determinism arises due to Subject | Object Dissonance.
A choice is not the creation of new causality out of nothing. It is the causality of the universe flowing through you.

Although I'm not the author of this post (a friend of mine wrote it), I have created a PDF version of the essay that has a table of contents and headers to make it even easier to read.
Travel Buffer
by jefftk — 2024-07-06T02:20:02.723Z

I recently finished a

nine-day
road trip, playing a series of dances. This combined driving long
distances with arrival deadlines: you don't want to arrive
late. Not only is it unprofessional, but it's stressful rushing to
set up and you'll likely have a bad night from skipping some nice-to-have
steps. A concept I found pretty useful was thinking about allocating
"buffer".

Let's say it's 8am in Pittsburgh PA and the hall in Bloomington IN
opens for setup at 5:30pm. GPS says its a 6.5hr drive, so we have 3hr
of buffer to spend. Some of the buffer we'll need to spend stopping
for gas and restrooms. We might choose to spend additional buffer on
a relaxed breakfast, stopping at interesting places along the way, or
exploring Bloomington. Or we might be unlucky with traffic (or,
heaven forfend, the car) and lose some buffer to bad luck.

Since the risk of things going wrong or taking longer than you expect
is roughly proportional to distance, it's pretty risky to front-load
your buffer consumption. You don't want to spend all by 30min early
on and then run into a 1hr traffic jam. But the most enjoyable ways
of spending buffer are probably distributed along the route, so the
safest option of reserving it all for the destination isn't very
pleasant. It's much nicer to spend the marginal half hour with your
toes in a shady stream than waiting around in the parking lot outside
the hall.

While this was somewhat useful in my own planning, the place where it
really demonstrated its value was in talking with my
tourmates. Getting close to the hall I might ask if anyone had
anything they wanted to spend buffer on; playing at a park I could use
it to explain to my kids why we should leave soon; it avoided people
mistaking the GPS arrival time for our actual arrival; any proposed
activity had a nice currency for considering its cost.

I don't remember thinking about this explicitly on past tours, or
missing it before. I think the main reason is that we scheduled this
tour much more tightly. On days when we had a lot of driving we still
wanted to take nice breaks (one park had a serious
zipline and a 90ft slide) and on days when we had less driving we
did a lot of seeing things (boating on an underground
river). I expect it to continue to be rare for me to be in a
situation where I need to do this kind of collaborative planning
around a deadline, ...
What progress have we made on automated auditing?
by LawrenceC — 2024-07-06T01:49:43.714Z

One use case for model internals work is to perform automated auditing of models:

https://www.alignmentforum.org/posts/cQwT8asti3kyA62zc/automating-auditing-an-ambitious-concrete-technical-research

That is, given a specification of intended behavior, the attacker produces a model that doesn't satisfy the spec, and the auditor needs to determine how the model doesn't satisfy the spec. This is closely related to static backdoor detection: given a model M, determine if there exists a backdoor function that, for any input, transforms that input to one where M has different behavior.[1]

There's some theoretical work (Goldwasser et al. 2022) arguing that for some model classes, static backdoor detection is impossible even given white-box model access -- specifically, they prove their results for random feature regression and (the very similar setting) of wide 1-layer ReLU networks.

Relatedly, there's been some work looking at provably bounding model performance (Gross et al. 2024) -- if this succeeds on "real" models and "real" specification, then this would solve the automated auditing game. But the results so far are on toy transformers, and are quite weak in general (in part because the task is so difficult).[2]

Probably the most relevant work is Halawi et al. 2024's Covert Malicious Finetuning (CMFT), where they demonstrate that it's possible to use finetuning to insert jailbreaks and extract harmful work, in ways that are hard to detect with ordinary harmlessness classifiers.[3]

As this is machine learning, just because something is impossible in theory and difficult on toy models doesn't mean we can't do this in practice. It seems plausible to me that we've demonstrated non-zero empirical results in terms of automatically auditing model internals. So I'm curious: how much progress have we made on automated auditing empirically? What work exists in this area? What does the state-of-the-art in automated editing look like?

^

Note that I'm not asking about mechanistically anomaly detection/dynamic backdoor detection; I'm aware that it's pretty easy to distinguish if a particular example is backdoored using baseline techniques like "fit a Gaussian density on activations and look at the log prob of the activations on each input" or "fit a linear probe on a handful of examples using logistic regression".

^

I'm also aware of some of the work in t

...