trentmkelly/LessWrong-43k huggingface dataset:
Slaying the Hydra: toward a new game board for AI
by Prometheus — 2023-06-23T17:04:39.119Z

AI Timelines as a Hydra

Think of current timelines as a giant hydra. You can’t exactly see where the head is, and you don’t know exactly if you’re on the neck of the beast or the body. But you do have some sense of what a hydra is, and the difficulty of what you’re in for. Wherever the head of the beast is, that’s the part where you get eaten, so you want to kill it before that point. Say, you saw a head approaching. Perhaps it was something being created by Facebook AI, perhaps Google, it doesn’t matter. You see an opportunity, and you prevent the apocalypse from happening.

Looks like Deepmind was about to destroy the world, but you somehow prevented it. Maybe you convinced some people at the company to halt training, it doesn't matter. Congratulations! You’ve cut the head off the hydra! However, you seem to remember reading something about the nature of how hydras work, and as everyone is celebrating, you notice something weird happening.

So it looks like you didn’t prevent the apocalypse, but just prevented an apocalypse. Maybe Google held off on destroying the world, but now FAIR and OpenAI are on a race of their own. Oh, well, now it’s time to figure out which path to turn, and cut the head off again. Looks like FAIR is a little more dangerous, and moving a bit faster, better prevent that happening now.

Well, this isn’t working. Now there are three of them. You now have two new AI tech companies companies to deal with, and you still haven’t taken care of that bad OpenAI timeline. Better keep cutting. But it seems the further you go down, the more heads appear.

This is the key problem of finding a good outcome. It’s not the odds of Facebook, or OpenAI, or anyone else creating the head of the hydra, it’s that the head appears whichever path you go down. And the further down you go, the harder it will be to foresee and cut off all of the many heads.

AI as a Chess Game

AI is an unstable game. It involves players continuously raising stakes until, inevitably, someone plays a winning move that ends the game. As time goes on, the players increase in possible moves. Controlling players becomes increasingly more and more difficult as the game becomes more and more unstable.

The only way stability is brought back is the game ending. This happens by one player executing a winning move. And, given our current lack of control or understanding, that winning...
When is correlation transitive?
by Ege Erdil — 2023-06-23T16:09:56.369Z

It's a well-known property of correlation that it's not transitive in general. If X,Y,Z are three real-valued random variables such that ρ(X,Y)>0 and ρ(Y,Z)>0, it doesn't have to be the case that ρ(X,Z)>0.

Nevertheless, there are some circumstances under which correlation is transitive. I will focus on two such cases in this post.

Primer: correlation as an inner product

For what follows, some background knowledge is necessary that we can regard correlations of real-valued random variables with finite second moments as inner products in an appropriate Hilbert space.

Specifically, if X,Y are two such random variables with zero mean and unit standard deviation, which is a simplification we can always make as correlation is invariant under translation and scalar multiplication, then we can compute

ρ(X,Y)=cov(X,Y)σXσY=E[XY]−E[X]E[Y]=E[XY]

The pairing (A,B)→E[AB] defines an inner product on the space of random variables with finite second moments where two random variables are considered equivalent if they are equal with probability 1 (almost surely). The properties that we expect out of an inner product are easy to check: the pairing is obviously bilinear and positive definite.

Furthermore, it turns out this inner product turns the space of random variables with finite second moments into a Hilbert space: the vector space turns out to be complete under the induced norm ∥X∥=√E[X2]. Roughly speaking, this means that we can take orthogonal projections onto closed subspaces with impunity.

Now that we have this framework, we can move on to the main results of this post.

Correlation is transitive when the correlations are sufficiently strong

I'll first prove the following:

Claim 1: If ρ(X,Y)=a and ρ(Y,Z)=b, then

ab−√(1−a2)(1−b2)≤ρ(X,Z)≤ab+√(1−a2)(1−b2)

Moreover, these bounds are tight: for any a,b, there is a combination X,Y,Z for which we can make either the right or the left inequality into an equality.

Proof

We can assume X,Y,Z have mean zero and unit variance without loss of generality. Taking orthogonal projections of X,Z onto the one-dimensional subspace spanned by Y, we can write

X=aY+√1−a2HXYZ=bY+√1−b2HZY

where E[YHXY]=E[YHZY]=0 and the random variables HXY,HZY have mean zero and variance 1. Taking inner products gives

E[XZ]=ab+√(1−a2)(1−b2)E[HXYHZY]

Using the Cauchy-Schwarz inequality for our inner product finishes the proof: |E[HXYHZY]|≤∥HXY∥⋅∥HZY∥=1....
Park Toys
by jefftk — 2023-06-23T16:00:01.482Z

The park closest to our house has a lot of "park toys". These are
toys that people brought to the park for everyone to play with.
They're usually somewhat broken when they show up, and eventually get
thrown out when they're so broken that there's not much interesting to
do with them anymore (which is hard!) This isn't something I
experienced growing up, and I see more of it this park than other
parks in the area. Walking around the park today, here's the current
assortment.

A fire truck. It used to have a ladder, but that fell off a couple
weeks ago:

A smaller fire truck. Also used to have a ladder. I think it used to
have doors too:

A very small bike. Julia pumped the tires up a few weeks ago.
Probably the nicest toy here? Mildly surprised no one has wandered
off with it:

Some sort of standing trike? Maybe for teaching someone the leg
motions before they're ready for a balance bike?

A Minnie Mouse themed riding airplane. Missing a wing and part of the
flight controls. Nora still loves riding it, though!

A basketball that won't stay inflated (we've tried):

A tricycle. It's designed to have the option for an adult to push it,
and the pedals freewheel when you're going faster than you're
pedaling. The handlebars are loose and older kids like turning them
around so it's a reverse-only trike, but today it's in its designed
forward-only configuration:

A digger. I think this is a front-end
loader, or at least it was before it lost its bucket:

A haul truck:

Another haul truck, with better build quality. Currently transporting
fake icecream:

Less sure about this one. Maybe a skid-steer?

A car. The front wheels are clearly the least sturdy component. At
one point Lily had a
go at fixing one of them. Since then it broke again, someone
replaced them, and it broke again. Bonus: a ladder from one of the
fire trucks in the background:

These all get a lot of use, and I'm really glad we have a culture of
people bringing their old toys to the park!

Comment via: facebook, mastodon
How tall is the Shard, really?
by philh — 2023-06-23T08:10:02.124Z

According to Wikipedia, the Shard (the tallest building in the UK) stands "309.6 meters (1,016 feet) high". I put this in my Anki deck as "Height of the Shard / 310m", but I was saying "height" to mean "tallness" (because I don't much like that word) and I had assumed Wikipedia was using it the same way. So I thought the Shard was 310 m tall.

But according to Bron Maher in Londonist,

most sources put the building’s tallness - that is, its length base-to-tip - at about 306 metres. The last three metres come from the height of the ground on which the Shard is built. So while its height is indeed 309 metres above sea level, the Shard is only 306 metres tall.

Is that so? I was curious enough to investigate1, but it turns out I don't really know how to.

(Also: Maher says the Shard is 306 m, not 309 m, not "not 310 m". And he gets 309 from Wikipedia, quoting it on the list of highest points in London, which talks about "the 309 m (1,014 ft) tall Shard". Where did those extra 60 cm go? The article on the Shard itself also has that number, in the "records" section near the bottom, but the infobox on the right has 309.6/1,016. So is it 306 m, 309 m, or 309.6 m?)

The boring investigation

The first thing to do2 is to look at Wikipedia's sources. It doesn't have one on the list of highest points, and it doesn't have one in the "records" section. The infobox does have a source for the height, which is on skyscraperpage.com. It lists the spire at 1,016 ft or 309.7 m, yet another number! There are various drawings, all giving the roof at 304.2 m and the spire at 309.7 m. (1,016 ft is 309.68 m, but 309.6 m is 1015.75 ft and 309.7 m is 1,016.08 ft. So if you convert 309.6 m to feet and back to meters, you could get 309.7 m. But wikipedia shouldn't be taking 309.6 m from that page.)

Skyscraperpage lists the source for the height as http://www.the-shard.com, and the about page on that site says: "How tall is The Shard? The Shard is 309.6 metres, or 1,016 feet, high and is Western Europe's tallest building." You'd certainly expect the Shard's official webpage to be right about this, but also it says "high" not "tall" so it could be a misdirect.

Next3, does Maher have any sources? For 306 m tallness he just says "most sources" give that figure. For 3 m above sea level, I think that comes from its height at the top. He quotes someone at Ordnance Survey saying the top of the S...
Are vaccine safe enough, that we can give their producers liability?
by ChristianKl — 2023-06-23T02:47:06.804Z

I listened to a few interviews from RFK jr. One story that he tells is that during the Trump administration he sat in a meeting with Fauci and claimed "The vaccines we use didn't go through placebo-blind trials". In RFK recounting Fauci claimed that the vaccine were placebo tested and promised to send Fauci studies showing so.

It turns out, that they frequently are not placebo tested and RFK was right with that claim. When looking at the claim at skeptics.stackexchange, the establishment answer is basically "Running placebo-blind trials would be unethical and we know that the vaccine are safe". The question whether or not you really need placebo blind trials is complex and not easy to answer.

Factual issues like that seems complex enough that Peter Hotez and other people in the field are afraid of actually debating RFK on the issues.

One point RFK makes for both nuclear energy and for vaccines is that if those technologies are completely safe, then the companies that deploy them should have no problem with being liable for negative consequences and buy insurance policies to shield themselves from the risk. There would be no need for the government to have laws that limit the liability.

Asking vaccine companies to accept liability for their vaccines is basically asking them to make a bet that their vaccines are safe or to let an insurance company make a bet that they are safe.

If someone would want to increase trust in vaccines, a good step would be to write a law that says: "Because of the advance in vaccine technology, the 1986 vaccine act that declared vaccines unavoidably unsafe is outdated, and from now on vaccine producers are responsible for significant side-effect resulting from vaccines."
Why Not Subagents?
by johnswentworth — 2023-06-22T22:16:55.249Z

Alternative title for economists: Complete Markets Have Complete Preferences

The justification for modeling real-world systems as “agents” - i.e. choosing actions to maximize some utility function - usually rests on

various coherence theorems

. They say things like “either the system’s behavior maximizes some utility function, or it is throwing away resources” or “either the system’s behavior maximizes some utility function, or it can be exploited” or things like that. [...]

Now imagine an agent which prefers anchovy over mushroom pizza when it has anchovy, but mushroom over anchovy when it has mushroom; it’s simply never willing to trade in either direction. There’s nothing inherently “wrong” with this; the agent is not necessarily executing a dominated strategy, cannot necessarily be exploited, or any of the other bad things we associate with inconsistent preferences. But the preferences can’t be described by a utility function over pizza toppings.

… that’s what I (John) wrote four years ago, as the opening of

Why Subagents?

. Nate

convinced me

that it’s wrong: incomplete preferences do imply a dominated strategy. A system with incomplete preferences, which can’t be described by a utility function, can contract/precommit/self-modify to a more-complete set of preferences which perform strictly better even according to the original preferences.

This post will document that argument.

Epistemic Status: This post is intended to present the core idea in an intuitive and simple way. It is not intended to present a fully rigorous or maximally general argument; our hope is that someone else will come along and more properly rigorize and generalize things. In particular, we’re unsure of the best setting to use for the problem setup; we’ll emphasize some degrees of freedom in the High-Level Potential Problems (And Potential Solutions) section.

Context

The question which originally motivated my interest was: what’s the utility function of the world’s financial markets? Just based on a loose qualitative understanding of coherence arguments, one might think that the inexploitability (i.e. efficiency) of markets implies that they maximize a utility function. In which case, figuring out what that utility function is seems pretty central to understanding world markets.

In

Why Subagents?

, I argued that a market of utility-maximizing traders is inexploitable, but n...
Catastrophic Risks from AI #2: Malicious Use
by Dan H — 2023-06-22T17:10:08.374Z
Link: https://arxiv.org/abs/2306.12001

This is the second post in a sequence of posts giving an overview of catastrophic AI risks.

2 Malicious Use

On the morning of March 20, 1995, five men entered the Tokyo subway system. After boarding separate subway lines, they continued for several stops before dropping the bags they were carrying and exiting. An odorless, colorless liquid inside the bags began to vaporize. Within minutes, commuters began choking and vomiting. The trains continued on toward the heart of Tokyo, with sickened passengers leaving the cars at each station. The fumes were spread at each stop, either by emanating from the tainted cars or through contact with people's clothing and shoes. By the end of the day, 13 people lay dead and 5,800 seriously injured. The group responsible for the attack was the religious cult Aum Shinrikyo [5]. Its motive for murdering innocent people? To bring about the end of the world.

Powerful new technologies offer tremendous potential benefits, but they also carry the risk of empowering malicious actors to cause widespread harm. There will always be those with the worst of intentions, and AIs could provide them with a formidable tool to achieve their objectives. Moreover, as AI technology advances, severe malicious use could potentially destabilize society, increasing the likelihood of other risks.

In this section, we will explore the various ways in which the malicious use of advanced AIs could pose catastrophic risks. These include engineering biochemical weapons, unleashing rogue AIs, using persuasive AIs to spread propaganda and erode consensus reality, and leveraging censorship and mass surveillance to irreversibly concentrate power. We will conclude by discussing possible strategies for mitigating the risks associated with the malicious use of AIs.

Unilateral actors considerably increase the risks of malicious use. In instances where numerous actors have access to a powerful technology or dangerous information that could be used for harmful purposes, it only takes one individual to cause significant devastation. Malicious actors themselves are the clearest example of this, but recklessness can be equally dangerous. For example, a single research team might be excited to open source an AI system with biological research capabilities, which would speed up research and potentially save lives, but this could also increase the risk of malicious use if the A...
Catastrophic Risks from AI #1: Introduction
by Dan H — 2023-06-22T17:09:40.883Z
Link: https://arxiv.org/abs/2306.12001

This is the first post in a sequence of posts giving an overview of catastrophic AI risks.

Abstract

Rapid advancements in artificial intelligence (AI) have sparked growing concerns among experts, policymakers, and world leaders regarding the potential for increasingly advanced AI systems to pose catastrophic risks. Although numerous risks have been detailed separately, there is a pressing need for a systematic discussion and illustration of the potential dangers to better inform efforts to mitigate them. This paper provides an overview of the main sources of catastrophic AI risks, which we organize into four categories: malicious use, in which individuals or groups intentionally use AIs to cause harm; AI race, in which competitive environments compel actors to deploy unsafe AIs or cede control to AIs; organizational risks, highlighting how human factors and complex systems can increase the chances of catastrophic accidents; and rogue AIs, describing the inherent difficulty in controlling agents far more intelligent than humans. For each category of risk, we describe specific hazards, present illustrative stories, envision ideal scenarios, and propose practical suggestions for mitigating these dangers. Our goal is to foster a comprehensive understanding of these risks and inspire collective and proactive efforts to ensure that AIs are developed and deployed in a safe manner. Ultimately, we hope this will allow us to realize the benefits of this powerful technology while minimizing the potential for catastrophic outcomes.

1 Introduction

The world as we know it is not normal. We take for granted that we can talk instantaneously with people thousands of miles away, fly to the other side of the world in less than a day, and access vast mountains of accumulated knowledge on devices we carry around in our pockets. These realities seemed far-fetched decades ago, and would have been inconceivable to people living centuries ago. The ways we live, work, travel, and communicate have only been possible for a tiny fraction of human history.

Yet, when we look at the bigger picture, a broader pattern emerges: accelerating development. Hundreds of thousands of years elapsed between the time Homo sapiens appeared on Earth and the agricultural revolution. Then, thousands of years passed before the industrial revolution. Now, just centuries later, the artificial intelligence (AI) re...
AI #17: The Litany
by Zvi — 2023-06-22T14:30:11.203Z

The biggest news this week was on the government front.

In the UK, Ian Hogarth of ‘We Must Slow Down the Race to Godlike AI’ fame was put in charge of a 100 million pound taskforce for AI safety.

I am writing up a separate post on that. For now, I will say that Ian is an excellent choice, and we have reason to believe this taskforce will be our best shot at moving for real towards solutions that might actually mitigate extinction risk from AI. If this effort succeeds we can build upon it. If it fails, hope in future similar approaches seems mostly gone.

As I said last week, the real work begins now. If you are in position to help, you can fill out this Google Form here. Its existence reflects a startup government mindset that makes me optimistic.

In the USA, we have Biden taking meetings about AI, and we have Majority Leader Chuck Schumer calling for comprehensive regulatory action from Congress, to be assembled over the coming months. In both cases, this seems clearly focused on mundane harms that are already frequently harped upon by such types with regard to things like social media, along with concerns over job losses, with talk of bias and democracy and responsibility, as well as jobs, while being careful to speak in support of innovation and the great promise of AI.

In other words, it’s primed to be the worst case, where we stifle mundane utility while not mitigating extinction risks. There is still time. The real work beings now.

The good news is that despite the leadership of Biden and Schumer here, the issue remains for now non-partisan. Let’s do our best to keep it that way.

I do wish I had a better model or better impact plan for such efforts.

Capabilities advances continue as well. Biggest news there was on the robotics front.

Table of Contents

Introduction. UK task force, USA calls for regulation.

Table of Contents.

Language Models Offer Mundane Utility. For great justice or breaking captcha.

Language Models Don’t Offer Mundane Utility. Bard especially doesn’t, not yet.

GPT-4 Does Not Ace MIT’s EECS Curriculum Without Cheating. Whoops.

What GPT-4 Is Doing Under the Hood. Some hints have been dropped.

The Art of the Jailbreak. The only thing we have to fear is the litany of fear.

Not only LLMs. Atari benchmarks defeated yet again.

Fun With Image Generation. Read minds, mess with metaphysics.

Deepfaketown and Botpocal

...
[Research Update] Sparse Autoencoder features are bimodal
by Robert_AIZI — 2023-06-22T13:15:00.207Z
Link: https://aizi.substack.com/p/research-update-sparse-autoencoder

Overview

The

sparse autoencoders project

is a mechanistic interpretability effort to algorithmically find semantically meaningful “features” in a language model. A

recent update

hints that features learned by this approach separate into two types depending on their maximum cosine similarity (MCS) score against a larger feature dictionary:

High-MCS features that reoccur across hyperparameters (speculatively, the “real” features that would be helpful for mechanistic interpretability)
Low-MCS features that do not reoccur (speculatively, dead neurons or artifacts of random noise)

Figure 1: Figure 3 from the

replication

, showing that MCS scores are bimodal, with peaks near MCS=.3 and MCS=1.

In this post, we:

Demonstrate that the MCS distribution of the low-MCS features matches the distribution of random vectors.
Present data show that a feature’s MCS against two larger models are highly correlated.
Show that as the size of the feature dictionary increases, the number of features which are high-MCS increases, even as their proportion decreases.

Thanks to Hoagy Cunningham for providing the model weights, answering my questions, and feedback on this draft, and to Dan Braun, Beren Millidge, Logan Riggs, and Lee Sharkey for producing the work that motivates this post.

Background

[Feel free to skip or skim this section if you’re already familiar with the project. Original sources: the

Interim Research Report

,

my summary

,

the replication

.]

Our goal is to extract features from the internal activations of a transformer language model. We used this small language model, as described in

the replication

:

We used

Andrej Karpathy's nanoGPT

to train a small transformer, with 6 layers, a 32 dimensional residual stream, MLP widths of 128 and 4 attention heads with dimension 8. We trained on a node of 4xRTX3090s for about 9h, reaching a loss of 5.13 on OpenWebText.

The internal activations of this model’s MLP in layer 2 (d=128) are used to train an autoencoder as described in

my summary

. In fact, six autoencoders are trained, each with L^1 penalty coefficient α=0.001, but with different “dictionary size” J, ranging across the set {256, 512, 1024, 2048, 4096, 8192}, and different random initializations.

Figure 2: An overview of the sparse autoencoder architecture, the loss function used in its training, and its (hyper)parameters.

The product of interest from this training process is the resulting ...