trentmkelly/LessWrong-43k huggingface dataset:
Why I find Davidad's plan interesting
by Paul W — 2024-05-20T08:13:15.950Z

Abstract

The aim of this text is to give an overview of Davidad’s safety plan, while also outlining some of the limitations and challenges. Additionally, I’ll explain why I would like to contribute to this field.

Disclaimers

I am not Davidad, I tried to convey his ideas as I understand them. While my interpretation may not be exact, I hope it still holds value. Also, this post does not focus on the technical details; I might write another one later, with a deeper technical discussion[1].

Epistemic status

I began exploring these questions during a weekend-long research sprint. Then, for a couple of months, I kept thinking about them, reading related posts (Davidad’s List of AI Safety Problems). Additionally, I engaged in discussions with Charbel, who co-authored this post on Davidad's plan. While empirical evidence may be limited at this stage, I anticipate more to emerge as efforts progress[2]. I'm currently learning potentially relevant mathematical theory. However, the aim here is to offer a broad explanation of key ideas rather than an in-depth analysis of the technical problems.

Acknowledgements

Thanks to Charbel-Raphaël Segerie for discussing these questions with me. Thanks to Jeanne S. for useful suggestions.

Outline

0/ Definitions

I/ The plan, and why it might yield positive results

II/ Technical questions, risks, objections

III/ Why I'm interested in contributing

0/Definitions

In what follows, I make a distinction between “Artificial General Intelligence” and “Artificial Super Intelligence”, the latter having strongly superhuman performance levels, while the former can be compared to a human being (if we ignore the speed and duplicability factors).

I shall use “AI Safety” in a loose way[3], essentially relying on the “notkilleveryoneism” idea: the point is to limit the overall amount of harm [4]caused by AIs (and humans), while keeping some of the benefits of powerful AIs.

A “formalized world-model” is one inside which formal proofs (that accept relevant forms of nondeterminism) can be derived, typically regarding the consequences of some actions. A key underlying assumption, of course, is that the model is close enough to the real world, and that it contains descriptions of the phenomena that are relevant to AI safety[5].

By “meaningful formal guarantees”, I mean safety-relevant prop...
Anthropic: Reflections on our  Responsible Scaling Policy
by Zac Hatfield-Dodds — 2024-05-20T04:14:44.435Z
Link: https://www.anthropic.com/news/reflections-on-our-responsible-scaling-policy

Last September we published our first Responsible Scaling Policy (RSP) [LW discussion], which focuses on addressing catastrophic safety failures and misuse of frontier models. In adopting this policy, our primary goal is to help turn high-level safety concepts into practical guidelines for fast-moving technical organizations and demonstrate their viability as possible standards. As we operationalize the policy, we expect to learn a great deal and plan to share our findings. This post shares reflections from implementing the policy so far. We are also working on an updated RSP and will share this soon.

We have found having a clearly-articulated policy on catastrophic risks extremely valuable. It has provided a structured framework to clarify our organizational priorities and frame discussions around project timelines, headcount, threat models, and tradeoffs. The process of implementing the policy has also surfaced a range of important questions, projects, and dependencies that might otherwise have taken longer to identify or gone undiscussed.

Balancing the desire for strong commitments with the reality that we are still seeking the right answers is challenging. In some cases, the original policy is ambiguous and needs clarification. In cases where there are open research questions or uncertainties, setting overly-specific requirements is unlikely to stand the test of time. That said, as industry actors face increasing commercial pressures we hope to move from voluntary commitments to established best practices and then well-crafted regulations.

As we continue to iterate on and improve the original policy, we are actively exploring ways to incorporate practices from existing risk management and operational safety domains. While none of these domains alone will be perfectly analogous, we expect to find valuable insights from nuclear security, biosecurity, systems safety, autonomous vehicles, aerospace, and cybersecurity. We are building an interdisciplinary team to help us integrate the most relevant and valuable practices from each.

Our current framework for doing so is summarized below, as a set of five high-level commitments.

Establishing Red Line Capabilities. We commit to identifying and publishing "Red Line Capabilities" which might emerge in future generations of models and would present too much risk if stored or deployed under our current safety and secu

...
The consistent guessing problem is easier than the halting problem
by jessicata — 2024-05-20T04:02:03.865Z
Link: https://unstableontology.com/2024/05/20/the-consistent-guessing-problem-is-easier-than-the-halting-problem/

The halting problem is the problem of taking as input a Turing machine M, returning true if it halts, false if it doesn't halt. This is known to be uncomputable. The consistent guessing problem (named by Scott Aaronson) is the problem of taking as input a Turing machine M (which either returns a Boolean or never halts), and returning true or false; if M ever returns true, the oracle's answer must be true, and likewise for false. This is also known to be uncomputable.

Scott Aaronson inquires as to whether the consistent guessing problem is strictly easier than the halting problem. This would mean there is no Turing machine that, when given access to a consistent guessing oracle, solves the halting problem, no matter which consistent guessing oracle (of which there are many) it has access too. As prior work, Andrew Drucker has written a paper describing a proof of this, although I find the proof hard to understand and have not checked it independently. In this post, I will prove this fact in a way that I at least find easier to understand. (Note that the other direction, that a Turing machine with access to a halting oracle can be a consistent guessing oracle, is trivial.)

First I will show that a Turing machine with access to a halting oracle cannot in general determine whether another machine with access to a halting oracle will halt. Suppose M(O, N) is a Turing machine that returns true if N(O) halts, false otherwise, when O is a halting oracle. Let T(O) be a machine that runs M(O, T), halting if it returns false, running forever if it returns true. Now M(O, T) must be its own negation, a contradiction.

In particular, this implies that the problem of deciding whether a Turing machine with access to a halting oracle halts cannot be a Σ01 statement in the arithmetic hierarchy, since these statements can be decided by a machine with access to a halting oracle.

Now consider the problem of deciding whether a Turing machine with access to a consistent guessing oracle halts for all possible consistent guessing oracles. If this is a Σ01 statement, then consistent guessing oracles must be strictly weaker than halting oracles. Since, if there were a reliable way to derive a halting oracle from a consistent guessing oracle, then any machine with access to a halting oracle can be translated to one making use of a consistent guessing oracle, that halts for all consistent gu...
A poem titled 'Tick Tock'.
by Krantz — 2024-05-20T03:52:09.003Z

Inspired by a collective intelligence project that I've been working on in the GOFAI space for over a decade. Hoping to share more at less online if I can afford to make it.

1st prediction:

My second prediction will be true.

2nd prediction:

My first prediction was false.

42nd prediction:

This is true iff prediction 24 NAND 70 are true.

43rd prediction:

This is true iff prediction 25 NAND 71 are true.

24th prediction:

The 43rd prediction will be true.

25th prediction:

The 42nd prediction will be true.

70th prediction:

This is true iff prediction 3 NAND 255 are true.

71st prediction:

This is true iff prediction 3 NAND 256 are true.

Prediction 3:

Today is Wednesday.

Prediction 86:

Prediction 7 is more likely to be true than 13.

Prediction 7:

Prediction 42 is more valuable to learn than 86.
Testing for parallel reasoning in LLMs
by meemi — 2024-05-19T15:28:03.771Z

Summary

We study language models' capability to perform parallel reasoning in one forward pass. To do so, we test GPT-3.5's ability to solve (in one token position) one or two instances of algorithmic problems. We consider three different problems: repeatedly iterating a given function, evaluating a mathematical expression, and calculating terms of a linearly recursive sequence.

We found no evidence for parallel reasoning in algorithmic problems: The total number of steps the model could perform when handed two independent tasks was comparable to (or less than) the number of steps it could perform when given one task.

Motivation

Broadly, we are interested in AI models' capability to perform hidden cognition: Agendas such as scalable oversight and AI control rely (to some degree) on our ability to supervise and bound models' thinking. Correspondingly, these approaches would be less promising if models could perform vast amounts of computation that humans couldn't oversee.

As our understanding of current models' internals is very lacking, the computation a model performs in one forward pass is by default hidden. This motivates the study and evaluation of model capabilities in one forward pass and, in transformer-based language models, the capabilities at one token position.

Here we test language models' capability to perform parallel reasoning at one token position. A part of the motivation is that the number of layers and thus the number of serial steps a model can perform is known and bounded, whereas it is not a priori clear how much parallel processing fits into one token position.

Setup

To evaluate a model's parallel reasoning capabilities, we compare its performance when handed one vs. two instances of a given task. We choose these tasks so that they require multiple serial steps to solve. For example, one of our tasks is about computing iterates such as f(f(f(f(3)))) of a given function f:{1,2,...,30}→{1,2,...,30}. The number of serial steps is a parameter we vary, obtaining easier and harder versions of the task.

We thus can measure the total number of steps a model can perform at one token position. This experiment could provide evidence of parallel reasoning: hypothetically, it could be the case that for one task instance the model can perform just 6 serial steps, but for two task instances it can perform 5 and 5 serial steps in parallel,...
Hot take: The AI safety movement is way too sectarian and this is greatly increasing p(doom)
by O O — 2024-05-19T02:18:53.524Z

The movement to reduce AI x-risk is overly purist. This is leading to a lot of sects to maintain each individual sect's platonic level of purity and is actively (greatly) harming the cause.

How the Safety Sects Manifest

People suggest not publishing AI research
More recently, Jan and his team leaving OpenAI
Less recently, Paul Christiano leaving OpenAI to form METR[1]
Even less recently, Anthropic forming off of OpenAI
A suggestion to blacklist anyone who decided to give $30 million (a paltry sum of money for a startup) to OpenAI.

I think these were all legitimate responses to a perceived increase in risk, but ultimately did or will do more harm than good. Disclaimer: I am the least sure that the formation Anthropic increases p(doom) but I speculate, post AGI, it will be seen as such.

The Safetyists Played Their Hands Too Early

To a fundamentalist, it's unethical to ignore the causes of those actions, but the world is a messy and unpredictable place. It isn't possible to get anything done without cooperating with some actors who may be deceitful or even harmful. As an example, most corporations are filled with people who don't care about the mission and would hop for a higher paying job. Despite this apparent mess of conflicting incentives, most corporations are very good at making a lot of money. Maybe it isn't possible to align incentives for non-monetary goals but I doubt this. (Paying an employee more hurts the company's profits).

The ideal response to each of these examples is to wait until we're far closer to AGI to ring the alarm bells. If prediction markets are right, we still have ~8 years until we have something that meets their relatively weak definition of AGI. There is no momentum in being 8 years early, and instead the doom claims lose credibility the same way the Earth going underwater predictions of the 70s fell flat.[2] This behavior has happened with GPT-2 as well.

I get race conditions are a factor in those decisions, but hardware is probably the key limiting factor, and they are already follow an exponential curve.[3] If there is no global race to make AGI, it's far more likely that Google builds a bunch of bigger datacenters to train their content and ads algorithms. Then someone at DeepMind stumbles across it with little international scrutiny. Google leadership realizes this will make them a lo...
Some "meta-cruxes" for AI x-risk debates
by Aryeh Englander — 2024-05-19T00:21:18.800Z

[Epistemic status: As I say below, I've been thinking about this topic for several years and I've worked on it as part of my PhD research. But none of this is based on any rigorous methodology, just my own impressions from reading the literature.]

I've been thinking about possible cruxes in AI x-risk debates for several years now. I was even doing that as part of my PhD research, although my PhD is currently on pause because my grant ran out. In particular, I often wonder about "meta-cruxes" - i.e., cruxes related to debates or uncertainties that are more about different epistemological or decision-making approaches rather than about more object-level arguments.

The following are some of my current top candidates for "meta-cruxes" related to AI x-risk debates. There are some others I might list, but I think these are probably the biggest ones. (Of course, these cruxes influence lots of debates, not just AI x-risk debates. But I've mostly focused on AI x-risk debates for my PhD research so I'll focus on that here as well.)

Hypothetical vs. empirical arguments

In many AI x-risk debates, it often feels like those who are more worried about the risks are essentially saying, "here's this chain of logic and analysis that leads to an all-things-considered conclusion that AI x-risk is something we should be very concerned about." And then those who aren't so worried often respond with something like, "well, if you can give me empirical evidence, or perhaps proven theorems, that clearly demonstrate the problem then I'll pay attention, but until then it's pie in the sky philosophizing and speculation." The former approach seems particularly common among philosophy or rationalist types, while the latter approach seems most common among engineers and practicing scientist types - although there are of course lots of exceptions on both sides.

This also feels closely related to Bayesianism vs. Frequentist or Popperian approaches in philosophy of science.

Object-level arguments vs. reference class forecasting

Even among those who take more abstract, less-completely-demonstrated arguments seriously, many seem to give much more weight to broad economic models, reference class forecasting using broadly-construed reference classes, or the like, over more gears-level arguments, narrow trend extrapolations that conflict with broader trends, or similar.

(Note that I'm calling this object-...
On Privilege
by Shmi — 2024-05-18T22:36:25.430Z

The forum has been very much focused on AI safety for some time now, thought I'd post something different for a change. Privilege.

Here I define Privilege as an advantage over others that is invisible to the beholder. [EDIT: thanks to JenniferRM for pointing out that "beholder" is a wrong word.] This may not be the only definition, or the central definition, or not how you see it, but that's the definition I use for the purposes of this post. I also do not mean it in the culture-war sense as a way to undercut others as in "check your privilege". My point is that we all have some privileges [we are not aware of], and also that nearly each one has a flip side.

In some way this is the inverse of The Lens That Does Not See Its Flaws: The lens that does not see its strengths.

On to the examples, as non-polarizing as I could think of, so no focus on race or gender:

Intelligence privilege. This one ought to be familiar to the regulars, but is often a blind spot. One manifestation of it being a privilege (i.e. being unaware that you have something others do not) is saying something like "anyone can learn calculus". No, not anyone.
Health privilege. Most healthy people don't give a second thought to how lucky they are compared to those who struggle physically or mentally to just get through the day.
Conventional beauty privilege. Everything is just invisibly easier when you look good. The flip side, of course, is that you have to deal with harassment a lot more. Or the judgment "you got where you are because of your looks".
A reasonably happy childhood privilege. Those who grew up in "normal" families rarely appreciate what it is like to be a child with a high ACE score. The flip side is the determination to get ahead that some high-adversity survivors develop.
Having an instinctive social "game". Not just for dating, but in general social interactions where a person is naturally likable and relatable, at least on the surface level. If you have that, you probably can't understand why others do not. Saying "I just go and talk to them, why don't you do the same, just be yourself" is an indicator of having this privilege.
Wealth privilege. Not, like, billionaire-level, but middle- and upper middle class vs, say, working poor.
A host of others, like education, location, culture, background...

One lesson to take from this is learning to notice this blind spo...
Fund me please - I Work so Hard that my Feet start Bleeding and I Need to Infiltrate University
by Johannes C. Mayer — 2024-05-18T19:53:10.838Z

Thanks to Taylor Smith for doing some copy-editing this.

In this article, I tell some anecdotes and present some evidence in the form of research artifacts about how easy it is for me to work hard when I have collaborators. If you are in a hurry I recommend skipping to the research artifact section.

Bleeding Feet and Dedication

During AI Safety Camp (AISC) 2024, I was working with somebody on how to use binary search to approximate a hull that would contain a set of points, only to knock a glass off of my table. It splintered into a thousand pieces all over my floor.

A normal person might stop and remove all the glass splinters. I just spent 10 seconds picking up some of the largest pieces and then decided that it would be better to push on the train of thought without interruption.

Sometime later, I forgot about the glass splinters and ended up stepping on one long enough to penetrate the callus. I prioritized working too much. A pretty nice problem to have, in my book.

[Edit 2024-05-19] The point is that this is irrational, and I have the problem of working too much. But this is a problem that's much easier to solve than "I have trouble making myself do anything". More details here.

Collaboration as Intelligence Enhancer

It was really easy for me to put in over 50 hours per week during AISC[1] (where I was a research lead). For me, AISC mainly consisted of meeting somebody 1-on-1 and solving some technical problem together. Methylphenidate helps me with not getting distracted when I am on my own, though Methylphenidate is only the number 2 productivity enhancer. For me, the actual ADHD cure seems to be to take methylphenidate while working 1-on-1 with somebody.

But this productivity enhancement is not just about the number of hours I can put in. There is a qualitative difference. I get better at everything. Seriously. Usually, I am bad at prioritization, but when I work with somebody, it usually feels, in retrospect, like over 75% of the time was spent working on the optimal thing (given our state of knowledge at the time). I've noticed similar benefits for my abilities in writing, formalizing things, and general reasoning.

Hardcore Gamedev University Infiltration

I don't quite understand why this effect is so strong. But empirically, there is no doubt it's real. In the past, I spent 3 years making video games. This was always done in teams of 2-4 people. We wou...
To Limit Impact, Limit KL-Divergence
by J Bostock — 2024-05-18T18:52:39.081Z

TL;DR

Run a potentially-harmful model alongside a known-harmless model, such that their action-spaces (e.g. output token sets) are equivalent. Combine the output probabilities so as to limit the KL-divergence between the resulting token probabilities and the harmless model's probabilities. This provides a mathematical ceiling on the impact of the resulting combined policy.

Unfortunately, the probability of catastrophe (q) scales linearly with the allowed DKL, whereas depending on the distribution, reward may only scale with √DKL (though for unbounded rewards it often scales with DKL. Bounds on probability of catastrophe compared to the probability of the weaker model causing a catastrophic outcome (q0) are also very weak. Overall the probability of catastrophe scales as q=DKLln1/q0.

Long Version

We will use the following model of AI interactions:

In this case, P represents the past, A0 the output of an AI (which we will represent as having policy π0) and F0 the future after this AI's actions have occcurred. For a "base" case, we will consider using an AI which is both relatively powerful, but that we think has a probability p0=ϵ of being catastrophic (like GPT-4) to calculate Pπ0→A0. We must consider the model as running at a temperature T≥0 for this to make sense.

Now imagine using a powerful AI with policy π1, which is of unknown safety. If we do the same thing here, the results could be catastrophic!

What we want is a way to use some of π1's power, but not all. Let's consider a modified case, where we only change the relation PπΠ→AΠ. We will generate AΠ by interpolating between A0 and A1 according to some rule Π.

For example we might use the rule A(b)=softmax(logits(A0)+b×logits(A1)), which smoothly interpolates between A(b)=A0 and A(∞)=argmax(A1). What is important is that we choose the maximum value of b=bmax such that:

maxp∈P[DKL(A(b)|P=p∥A|P=p)]≤Π2

Then we can define AΠ=A(bmax) for some allowed "power level" Π. This then limits the impact of the combined system on the future:

DKL(FΠ∥F0)≤Π2

This lemma is proved at the end of this post.

Limiting Takeover Probability

If, as stated, our model π0 has a probability ϵ of taking over, then what's the probability ...
Are There Other Ideas as Generally Applicable as Natural Selection
by Amin Sennour — 2024-05-18T16:37:22.918Z

I've noticed that the principles of Evolution / Natural Selection apply to a lot of things besides the context they were initially developed for (Biology).

Examples are things like ideas / culture (memetics), technological progress, and machine learning (sort of).

Reasoning about things like history, politics, companies, etc in terms of natural selection has helped me understand the world much better than I did when I thought that natural selection applied only to Biology.

So, I'm asking here for any other ideas that are generally applicable in a similar way.

(Sorry if this has been asked before. I tried searching for it and didn't find anything, but it's possible that my phrasing was off and I missed it).