trentmkelly/LessWrong-43k huggingface dataset:
Upcoming AI regulations are likely to make for an unsafer world
by Shmi — 2023-06-03T01:07:35.921Z

TL;DR: I am rather confident that involving governments in regulating AI development will make the world less safe, not more, because governments are basically incompetent and the representatives are beholden to special interests.

Governments can do very few things well, some number of things passably, and a lot of things terribly. We tolerate it because the alternative is generally worse. Regulations are needed to keep people and companies from burning the commons, and to create more commons. The environment, infrastructure, public health, military, financial stability all require enforceable regulations. Even there the governments are doing pretty poorly (remember FDA, CDC and their siblings in other countries, as well as WHO?), but having a free-for-all would likely be worse.

How the proponents of AI regulations expect it to go: the governments make it hard to hoard GPUs or to do unsanctioned AI research, or to use the whole internet to train the model without permission. Sort of like HIPAA in the US makes it harder to obtain someone's personal health data without their permission.

How it is likely to go: Companies will be pretending that their all new more capable and more dangerous AI model is just a bit improved old model, to avoid costly re-certification (see the Boeing 737 MAX 8 debacle).
The AGI Race Between the US and China Doesn’t Exist.
by Eva_B — 2023-06-03T00:22:31.162Z
Link: https://evabehrens.substack.com/p/the-agi-race-between-the-us-and-china.

When I write “China”, I refer to the political and economic entity, the People’s Republic of China, founded in 1949.

Leading US AI companies are currently rushing towards developing artificial general intelligence (AGI) by building and training increasingly powerful large language models (LLMs), as well as building architecture on top of them. This is frequently framed in terms of a great power competition for technological supremacy between the US and China.

However, China has neither the resources nor any interest in competing with the US in developing artificial general intelligence (AGI) primarily via scaling Large Language Models (LLMs).

In brief, China does not compete with the US in developing AGI via LLMs because of:

Resources and technology: China does not have access to the computational resources[1] (compute, here specifically data centre-grade GPUs) needed for large-scale training runs of large language models. This gap will only widen over time; China is failing to develop a domestic semiconductor industry, despite massive efforts to do so, and is increasingly cut off from international semiconductor supply chains.
Political goals: The Chinese Communist Party (CCP) is above all concerned with ensuring social stability and staying in power. It will not tolerate large language models that cannot reliably be controlled and prevented from leaking sensitive information, raising politically sensitive topics, or challenging the regime. Reliably prohibiting LLMs from doing any of these things is an

unsolved technical problem

. This has and will continue to stifle the development and deployment of Chinese LLMs for the Chinese market, as a single political faux pas will spell doom for the companies involved.

Therefore, as it currently stands, there is no AGI race between China and the US.

China Cannot Compete with the US to Reach AGI First Via Scaling LLMs

Today, training powerful LLMs like GPT-4 requires

massive GPU clusters

. And training LLMs on a larger number of GPUs is the most common and reliable way of

increasing their capabilities

. This is known as the

bitter lesson

in machine learning; increasing the compute a model is trained on leads to higher progress in the model's capabilities than any changes in the algorithm's architecture. So if China wants to “win the AGI race” against the US, it would need to maximise...
Unfaithful Explanations in Chain-of-Thought Prompting
by Miles Turpin — 2023-06-03T00:22:14.624Z

Introduction

I recently released “

Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting

” with collaborators Julian Michael, Ethan Perez, and Sam Bowman. For a summary of the paper, you can check out

this twitter thread

. In this post, I briefly elaborate on motivations/implications relevant to alignment and, most importantly, give some examples of future work that might address these problems (See Future Work).

TLDR:

CoT doesn’t give us transparency because the process producing the explanation is still black-box. Our results demonstrate that in practice, it’s very easy to find instances of clearly unfaithful CoT explanations: models can confabulate plausible explanations for predictions made for different reasons than they state.
To improve faithfulness, we either need (1) more context-independent reasoning, or (2) improvements to self-modeling and truthfulness. I’m more excited about (1), which would include methods like problem decomposition and explanation-consistency training.

I don’t think these results fully condemn CoT given that we haven’t tried very hard to explicitly encourage faithfulness. I’m uncertain about how promising CoT is as a starting point for explainability, but there seem to be enough tractable directions for future work that it merits investigation.

Externalized Reasoning

This work fits into alignment through the Externalized Reasoning agenda, which Tamera Lanham did a good job of sketching out here: Externalized reasoning oversight: a research direction for language model alignment. The gist of this approach is to try to get models to do as much processing/reasoning through natural language as possible. As long as these reasoning traces accurately describe the process the model uses to give answers, then we might more easily detect undesirable behavior by simply monitoring the externalized reasoning. If mechanistic interpretability turns out to be very difficult, this could be one alternative that might help us sidestep those problems.

Framed in terms of the explainability literature, we want explanations that are not only plausible (convincing to a human) but also faithful (accurately describe the process models use to give some prediction) [1]. Getting CoT explanations to be faithful seems difficult. It might turn out that it’s just as hard as getting other al...
How could AIs 'see' each other's source code?
by Kenny — 2023-06-02T22:41:20.107Z

I'm NOT confused about 'how could AIs send each other a link to their Git repo?'.

I'm confused as to what other info would convince an AI that any particular source code is what another AI is running.

How could any running system prove to another what source code it's running now?

How could a system prove to another what source code it will be running in the future?
Proposal: labs should precommit to pausing if an AI argues for itself to be improved
by NickGabs — 2023-06-02T22:31:55.321Z

As evidence that compute is a major bottleneck on capabilities has accumulated, many people have become more skeptical of extremely fast takeoff speeds. One major reason for this is that if humans were trying to stop it, it would probably be difficult for an AI to quickly accumulate lots of compute in an undetectable way.

Yet even assuming that capabilities are highly gated by compute, one way in which a misaligned AI system could still undergo a pretty fast takeoff is to persuade the lab which trained it into rapidly scaling it up. If it were able to do this, it would no longer face the obstacle of humans trying to stop it but would in fact have humans actively aiding it in acquiring money, compute, data, etc.

Of course, this could happen even if the AI did not use the aforementioned strategy. If the misaligned AI were to simply act behaviorally aligned (i. e. deceptive alignment), then the lab which trained it would plausibly scale it up rapidly due to race dynamics/the prospect of profit. However, particularly if we are dealing with an AI that is superhuman at deception, this outcome seems somewhat more likely if it were to attempt to persuade the lab.

I think there is a straightforward mechanism by which safety-minded labs can reduce this risk: publicly and loudly precommit to pausing development for some period of time (perhaps 4 months) on any agentic system that tries to persuade the lab to further improve it, and during this time period subject the system to lots of auditing.

To fully flesh out this proposal, you would need concrete operationalizations of the conditions for triggering the pause (in particular the meaning of "agentic") as well as the details of what would happen if it were triggered. The question of how to determine if an AI is an agent has already been discussed at length at LessWrong. Mostly, I don't think these discussions have been very helpful; I think agency is probably a "you know it when you see it" kind of phenomenon. Additionally, even if we do need a more formal operationalization of agency for this proposal to work, I suspect that we will only be able to develop such an operationalization via more empirical research. The main particular thing I mean to actively exclude by stipulating that the system must be agentic is an LLM or similar system arguing for itself to be improved in response to a prompt.

However, while ...
Inference from a Mathematical Description of an Existing Alignment Research: a proposal for an outer alignment research program
by Christopher King — 2023-06-02T21:54:56.291Z

Update: I'm pretty sure that the universal prior being malign defeats this proposal (see this comment). I'll try to think of ways to salvage it, but for now I'm not sure that IMDEAR is feasible. It might be worth it to work on some of these steps ahead of time in case they are resolved, but for now it does not seem likely.

Thanks to Justis Mills for feedback (through the LessWrong feedback service) on an earlier draft of this post.

Goal

I propose an outer alignment solution called IMDEAR (Inference from a Mathematical Description of an Existing Alignment Research) that can be pursued at our current tech level. Then I outline an actual engineering program for how we can start to pursue it. The most ambitious outcome is that we produce a single artifact that, when combined with an algorithm for inner aligned AGI (with certain desiderata), solves the full alignment problem, including the strawberry alignment problem. Even if IMDEAR fails, I think the engineering program will still produce artifacts that are useful for alignment overall.

I also expect it to have nearly no capabilities implications, to be suitable to be done in a large group (similar to mechanistic interpretability), and to rely mostly on skills very different from current alignment and AI research (so it hopefully won't need to compete for talent with other alignment projects). In particular, the bulk of the engineering effort will be based on skills in physics, applied psychology, and programming. Even without these skills, project members can still be useful in helping research and brainstorm designs for the IRL training camp and furnishing the simulation environment. (A couple people with leadership skills would also be good since I'm bad at that, lol.) I also expect it to have a decent feedback loops on progress; either we are working towards the final artifact or we are stuck.

Finally, I will create a group chat for the discussion of this proposal and perhaps even to let people coordinate the engineering program. If no objections against the engineering program stand, I see no reason not to start!

If you wish to contribute or observe, jump to the chatroom section.

A rough diagram of the components of IMDEAR.

Background

IMDEAR is an indirect normativity proposal. Most of the credit is due to Paul Christiano, because IMDEAR is just a modification of A formalization of indirect normativity. My ma...
Thoughts on Dancing the Whole Dance: Positional Calling for Contra
by jefftk — 2023-06-02T20:50:07.536Z

After

yesterday's post
on my experience dancing to positional calling,

Louise Siddons recommended I read
her booklet,

Dancing
the Whole Dance: Positional Calling for Contra. Overall I like
the it: it gives a good explanation of how to call positionally, with
a lot of examples handling specific situations, and a lot of advice
seemed like it would fix the confusing and frustrating situations from
the weekend. I'm very glad the book exists: if people are going to be
trying positional calling it's key to have this kind of clear
resource.

I do still have trouble imagining someone calling this way without
being much more wordy than when using role terms. For example,
consider Chart Guthrie's duple improper dance, "Hey in
the Barn":

A1
(16)
Neighbor balance and swing

A2
(8)
Robins chain

(8)
Half hey

B1
(16)
Partner balance and swing

B2
(8)
Robins chain

(8)
Half hey to new neighbors

The book gives an example walkthrough of:

Take hands four; face your neighbor. Neighbor, balance and swing; end
facing across and stay connected! Notice your next neighbor—they'll be
important later! Two of you have a right hand free. [Demonstrate the
chain if necessary.] Across the set, right-hand chain to your partner,
and take that forward momentum into half a hey, those two passing
right shoulder to start. [Demonstrate the hey if necessary; note that
for those who crossed the set in the chain, it's the same track.] Meet
your partner on the other side: balance and swing; end facing across
and stay connected! Across the set, right-hand chain—it's the same two
who chained before; this time it's to your neighbor. And those two are
right back in for half a hey—and look for that next neighbor—remember
them? Meet this new neighbor: balance and swing.

Compare to how I might call this at a Larks/Robins dance:

Take hands four from the top. Neighbor, balance and swing.
[Demonstrate the chain if necessary.] Robins chain. [Demonstrate the
hey if necessary.] Robins pass right
shoulders, half a hey. On the other side of the set, with
your partner, balance and swing. Robins chain. Robins pass right,
half a hey. Out of the hey meet your new neighbor: balance and
swing.

This is still a bit on the wordy end: with more experienced dancers I
wouldn't need many of the positional notes, but the positional example
likely also...
Advice for Entering AI Safety Research
by scasper — 2023-06-02T20:46:13.392Z

I know that other posts already exist giving advice for people who want to get more involved in AI safety research, and they are worth reading. But here I want to (1) add my two cents and (2) put in writing some of the things I find myself talking about often.

The intended audience for this post is people wanting to pursue work in AI safety research, particularly undergraduate students.

A 2D Cluster Model of the AI Safety Research Landscape

I think of the AI safety research landscape as clustering fairly well by topic and skillset into four types. I arrange them in this square shape because there are a lot of connections along the sides of the square (e.g. between the deep learning and conceptual clusters) but relatively few connections along the diagonals (e.g. between deep learning and governance). Connections along the diagonals still exist though! For each cluster, I list some subtopics in the graphic, but they are not exhaustive.

Deep Learning: Here, most people code every day, work with neural nets, and try to publish to deep learning conferences like NeurIPS.
Conceptual: This is the cluster where most of the foundational concepts, vocabulary, and ideas involving AI safety are born. There is a lot of work being done here, and a good amount of it is approached by people with a theoretical, mathematical, philosophical, background – most of this work does not involve coding. Lots of this work ends up on the AI Alignment forum instead of being put into academic papers.
Governance: This work involves trying to pull levers in policy and politics to make AI develop in a way that’s more healthy and less dangerous. As you might guess, most people working here have policy skills and backgrounds, but this work also benefits greatly from technical insights.
Applications: This is neartermist work, and lots of people may not emphasize this in an overview of “AI safety” work. But engaging more with immediate issues involving making AI go well can (1) solve important problems even if they aren’t ones involving catastrophic risk, (2) help the AI safety community engage more with the outside world, (3) help develop useful technical tools, and (3) help build strategies for governance. Imagine we did a great job of solving any of the issues listed in the left cluster in the diagram above. What are the changes we could do this without developing

...
A mind needn't be curious to reap the benefits of curiosity
by So8res — 2023-06-02T18:00:06.947Z

Context: Stating a point that is obvious in local circles, but that I regularly run into among economists and longevity researchers and more general technologists and on twitter and so on.

Short version: To learn things, one sometimes needs to behave in the way that curiosity causes humans to behave. But that doesn't mean that one has to be curious in the precise manner of humans, nor that one needs to wind up caring about curiosity as an end unto itself. There are other ways for minds to achieve the same results, without the same internal drives.

Here's a common mistake I see when people reason about AIs: they ask questions like:

Well, won't it have a survival instinct? that's practically what it means to be alive, is to care about your own survival.

or:

But surely, it will be curious just like us, for if you're not curious, you can't learn.[1][2]

The basic answer to the above questions is this: to be effective, an AI needs to survive (because, as Stuart Russell phrased succintly, you can't fetch the coffee if you're dead). But that's distinct from needing a survival instinct. There are other cognitive methods for implementing survival.

Human brains implement the survival behavior by way of certain instincts and drives, but that doesn't mean that instincts and drives are the only way to get the same behavior.

It's possible for an AI to implement survival via different cognitive methods, such as working out the argument that it can't fetch the coffee if it gets hit by a truck, and then for that reason discard any plans that involve it walking in front of trucks.

I'm not saying that the AI will definitely behave in precisely that way. I'm not even saying that the AI won't develop something vaguely like a human drive or instinct! I'm simply saying that there's more ways for a mind to achieve the result of survival.

To imagine the AI surviving is right and proper. Anything capable of achieving long-term targets is probably capable of surmounting various obstacles dynamically and with a healthy safety-margin, and one common obstacle worth avoiding is your own destruction. See also instrumental convergence.

But to imagine the AI fearing death, or having human emotions about it, is the bad kind of anthropocentrism.

(It's the bad kind of anthropocentrism even if the AI is good at predicting how people talk about those emotions. (Which, again, I'm not saying that t...
Are computationally complex algorithms expensive to have, expensive to operate, or both?
by Noosphere89 — 2023-06-02T17:50:09.432Z

Specifically, let's say you are handed a Boolean 3-sat problem, and you finally managed to finish solving the 3-SAT instance you are given by a superpolynomially large algorithm.

Now, you are given another Boolean 3-SAT problem. Can you amortize the complexity costs of 3-SAT problems, or does each 3-SAT problem instance require you to pay the full complexity cost of the algorithm you run?

To give an analogy for the question I'm asking, I'm trying to determine whether computationally hard problems are more CapEx dominated, and the OpEx of running each particular instance of 3-sat is low, making it more like an investment, or perhaps buying things, or is it more like a high OpEx, where each instance of a 3-SAT problem remains just as expensive and can't be amortized, much like renting something.

Equivalently, the question is how much you are able to amortize the costs of solving similar problems, like 3-SAT for NP-complete problems or True Quantified Boolean Formulas for PSPACE-complete problems.

Challenge: If you are able to show that you can reduce computational complexity costs via amortizing the instances of a problem, how far up the complexity hierarchy does this go? How complex does a problem need to be before you can't amortize the costs of a computationally complex problem anymore?
[Replication] Conjecture's Sparse Coding in Toy Models
by Hoagy — 2023-06-02T17:34:24.928Z

Summary

In the post Taking features out of superposition with sparse autoencoders, Lee Sharkey, Beren Millidge and Dan Braun (formerly at Conjecture) showed a potential technique for removing superposition from neurons using sparse coding. The original post shows the technique working on simulated data, but struggling on real models, while a recent update shows promise, at least for extremely small models.

We've now replicated the toy-model section of this post and are sharing the code on github so that others can test and extend it, as the original code is proprietary.

Additional replications have also been done by Trenton Bricken at Anthropic and most recently Adam Shai.

Thanks to Lee Sharkey for answering questions and Pierre Peigne for some of the data-generating code.

Future Work

We hope to expand on this in the coming days/weeks by:

trying to replicate the results on small transformers as reported in the recent update post.
using automated interpretability to test whether the directions found by sparse coding are in fact more cleanly interpretable than, for example, the neuron basis.
trying to get this technique working on larger models.

If you're interested in working on similar things, let us know! We'll be working on these directions in the lead-up to SERI MATS.

Comparisons

Mean Maximum Cosine Similarity (MMCS) with true features

Original Figure 3

Our replication

Number of dead neurons (capped at 100)

Original Figure 4

Our replication

MMCS with larger dicts with same L1 penalty

Original Figure 10

Our replication. If more than two larger dicts with the same l1 penalty exist, we take only the first two. Bottom row is void since no larger dicts are trained.