trentmkelly/LessWrong-43k huggingface dataset:
Avoiding the Bog of Moral Hazard for AI
by Nathan Helm-Burger — 2024-09-13T21:24:34.137Z

Imagine if you will, a map of a landscape. On this map, I will draw some vague regions. Their boundaries are uncertain, for it is a new and under-explored land. This map is drawn as a graph, but I want to emphasize that the regions are vague guesses, and the true borders could be very convoluted.

So here's the problem. We're making these digital minds, these entities which are clearly not human and process the world in different ways from human minds. As we improve them, we wander further and further into this murky fog covered bog of moral hazard. We don't know when these entities will become sapient / conscious / valenced / etc to such a degree that they have moral patient-hood. We don't have a good idea of what patterns of interaction with these entities would be moral vs immoral. They operate by different rules than biological beings. Copying, merging, pausing and resuming, inference by checkpoints with frozen weights... We don't have good moral intuitions for these things because they differ so much from biological minds.

Once we're all in agreement that we are working with an entity on the right hand side of the chart, and we act accordingly as a society, then we are clear of the fog. Many mysteries remain, but we know we aren't undervaluing the beings we are interacting with.

While we are very clearly on the left hand side of the chart, we are also fine. These are entities without the capacity for human-like suffering, who don't have significant moral valence according to most human ethical philosophies.

Are you confident you know where to place Claude Opus 3 or Claude Sonnet 3.5 on this chart? If you are confident, I encourage you to take a moment to think carefully about this. I don't think we have enough understanding of the internals of these models to be confident.

My uncertain guess would place them in the Bog of Moral Hazard, but close to the left hand side. In other words, probably not yet moral patients but close to the region where they might become such. I think that we just aren't going to be able to clear up the murk surrounding the Bog of Moral Hazard anytime soon. I think we need to be very careful as we proceed with developing AI to deliberately steer clear of the Bog. Either we make a fully morally relevant entity, with human-level moral patient-hood and treat it as equal to humans, or we deliberately don't make int...
If I ask an LLM to think step by step, how big are the steps?
by ryan_b — 2024-09-13T20:30:50.558Z

I mean big in terms of number of tokens, and I am thinking about this question specifically in the context of training windows vs context windows. This question is inspired by an Andrew Mayne tweet covered in AI #80: Never Have I Ever:

Most AI systems are trained on less than 2,000 words per sample. They can generalize across all of its training but no system to date trains on 80,000 words per sample. This will change with more memory and processing power. When that happens AI will be able to actually “read” entire novels and understand story structure on a fundamental level.

Gwern says context windows solve this. Pretty sure I am behind the curve on this one, but concretely, if LLMs cannot write a good novel because they are constrained by a training window of 2000 tokens, would they also under step-by-step prompting tend to produce steps with that same constraint?
On the flip side of the coin, how does adding a context window resolve this, especially when drawing on information outside of what was provided in the context window?
Estimating Tail Risk in Neural Networks
by Mark Xu — 2024-09-13T20:00:06.921Z

Machine learning systems are typically trained to maximize average-case performance. However, this method of training can fail to meaningfully control the probability of tail events that might cause significant harm. For instance, while an artificial intelligence (AI) assistant may be generally safe, it would be catastrophic if it ever suggested an action that resulted in unnecessary large-scale harm.

Current techniques for estimating the probability of tail events are based on finding inputs on which an AI behaves catastrophically. Since the input space is so large, it might be prohibitive to search through it thoroughly enough to detect all potential catastrophic behavior. As a result, these techniques cannot be used to produce AI systems that we are confident will never behave catastrophically.

We are excited about techniques to estimate the probability of tail events that do not rely on finding inputs on which an AI behaves badly, and can thus detect a broader range of catastrophic behavior. We think developing such techniques is an exciting problem to work on to reduce the risk posed by advanced AI systems:

Estimating tail risk is a conceptually straightforward problem with relatively objective success criteria; we are predicting something mathematically well-defined, unlike instances of eliciting latent knowledge (ELK) where we are predicting an informal concept like "diamond".

Improved methods for estimating tail risk could reduce risk from a variety of sources, including central misalignment risks like deceptive alignment.

Improvements to current methods can be found both by doing empirical research, or by thinking about the problem from a theoretical angle.

This document will discuss the problem of estimating the probability of tail events and explore estimation strategies that do not rely on finding inputs on which an AI behaves badly. In particular, we will:

Introduce a toy scenario about an AI engineering assistant for which we want to estimate the probability of a catastrophic tail event.

Explain some deficiencies of adversarial training, the most common method for reducing risk in contemporary AI systems.

Discuss deceptive alignment as a particularly dangerous case in which adversarial training might fail.

Present methods for estimating the probability of tail events in neural network behavior that do not rely on evaluating behavior on concret

...
If-Then Commitments for AI Risk Reduction [by Holden Karnofsky]
by habryka — 2024-09-13T19:38:53.194Z
Link: https://carnegieendowment.org/research/2024/09/if-then-commitments-for-ai-risk-reduction?lang=en

Holden just published this paper on the Carnegie Endowment website. I thought it was a decent reference, so I figured I would crosspost it (included in full for convenience, but if either Carnegie Endowment or Holden has a preference for just having an excerpt or a pure link post, happy to change that)

If-then commitments are an emerging framework for preparing for risks from AI without unnecessarily slowing the development of new technology. The more attention and interest there is these commitments, the faster a mature framework can progress.

Introduction

Artificial intelligence (AI) could pose a variety of catastrophic risks to international security in several domains, including the proliferation and acceleration of cyberoffense capabilities, and of the ability to develop chemical or biological weapons of mass destruction. Even the most powerful AI models today are not yet capable enough to pose such risks,[1] but the coming years could see fast and hard-to-predict changes in AI capabilities. Both companies and governments have shown significant interest in finding ways to prepare for such risks without unnecessarily slowing the development of new technology.

This piece is a primer on an emerging framework for handling this challenge: if-then commitments. These are commitments of the form: If an AI model has capability X, risk mitigations Y must be in place. And, if needed, we will delay AI deployment and/or development to ensure the mitigations can be present in time. A specific example: If an AI model has the ability to walk a novice through constructing a weapon of mass destruction, we must ensure that there are no easy ways for consumers to elicit behavior in this category from the AI model.

If-then commitments can be voluntarily adopted by AI developers; they also, potentially, can be enforced by regulators. Adoption of if-then commitments could help reduce risks from AI in two key ways: (a) prototyping, battle-testing, and building consensus around a potential framework for regulation; and (b) helping AI developers and others build roadmaps of what risk mitigations need to be in place by when. Such adoption does not require agreement on whether major AI risks are imminent—a polarized topic—only that certain situations would require certain risk mitigations if they came to pass.

Three industry leaders—Google DeepMind, OpenAI, and Anthropic—have publis...
Can startups be impactful in AI safety?
by Esben Kran — 2024-09-13T19:00:33.306Z

With

Lakera's

strides in securing LLM APIs,

Goodfire AI's

path to scaling interpretability, and 20+

model

evaluations

startups

among

much

else

, there's a rising number of technical startups attempting to secure the model ecosystem.

Of course, they have varying levels of impact on superintelligence containment and security and even with these companies, there's a lot of potential for aligned, ambitious and high-impact startups within the ecosystem. This point isn't new and has been made in our

previous

posts

and by

Eric Ho

(Goodfire AI CEO).

To set the stage, our belief is that these are the types of companies that will have a positive impact:

Startups with a profit incentive completely aligned with improving AI safety;
that have a deep technical background to shape AGI deployment and;
do not try to compete with AGI labs.

Piloting AI safety startups

To understand impactful technical AI safety startups better, Apart Research joined forces with collaborators from Juniper Ventures, vectorview (alumni from the latest YC cohort), Rudolf (from the upcoming def/acc cohort), Tangentic AI, and others. We then invited researchers, engineers, and students to resolve a key question “can we come up with ideas that scale AI safety into impactful for-profits?”

The hackathon took place during a weekend two weeks ago with a keynote by

Esben Kran

(co-director of Apart) along with ‘HackTalks’ by

Rudolf Laine

(def/acc) and

Lukas Petersson

(YC / vectorview). Individual submissions were a 4 page report with the problem statement, why this solution will work, what the key risks of said solution are, and any experiments or demonstrations of the solution the team made.

This post details the top 6 projects and excludes 2 projects that were made private by request (hopefully turning into impactful startups now!). In total, we had 🙌 101 signups and 🏆 11 final entries. Winners were decided by an LME model conditioned on reviewer bias. Watch the authors' lightning talks

here

.

Dark Forest: Making the web more trustworthy with third-party content verification

By Mustafa Yasir (AI for Cyber Defense Research Centre, Alan Turing Institute)

Abstract: ‘DarkForest is a pioneering Human Content Verification System (HCVS) designed to safeguard the authenticity of online spaces in the face of increasing AI-generated content. By l...
I just can't agree with AI safety. Why am I wrong? 
by Ya Polkovnik — 2024-09-13T17:48:22.245Z

Take, for example, AISafety.info, that tries to explain.

"In recent years, AI has exceeded people’s expectations in a wide variety of domains — including playing Go, composing human-like text, writing code, and modeling protein folding"

Not a single current "AI" can do all of it simultaneously. All of them are neuros, who can't even learn and perform over 1 task, to say nothing of escaping the power of alt+f4.

"Advanced AI could provide great benefits, but it could also cause unprecedented disasters, and even human extinction, unless difficult technical safety problems are solved and humanity cooperates to deploy advanced AI wisely."

As if other pieces of technology weren't just as "autonomous" and dangerous.

"Rapid progress in the capabilities of AI systems has recently pushed the topic of existential risk from AI into the mainstream. The abilities of systems like GPT-4 used to seem out of reach in the foreseeable future."

Ah, GPT-4. This neuro is recognising text. The text may be sent from an image recognition neuro, and be sent to an image generation neuro.

It still sees the world as text, and doesn't even learn from user's inputs, nor it acts without an input button being pressed. How is it dangerous? Especially since...

"The leading AI labs today are aiming to create “artificial general intelligence” in the not-too-distant future"

Now my words start to sound blatant and I look like an overly confident noob, but... This phrase... Most likely is an outright lie. GPT-4 and Gemini aren't even two-task neuros. Both cannot take a picture and edit it. Instead, an image recognition neuro gives the text to the blind texting neuro, that only works with text and lacks the basic understanding of space. That neuro creates a prompt for an image generator neuro, that can't see the original image.

Talking about the general intelligence, (with ability to learn any task), when the largest companies only lie about a TWO-task one, is what initially made me ragequit the theme.

To say nothing about that "AI cashier" fraud, and the fact openAI doesn't seem to consider AI safety (safety of a hypothetical project that isn't being developed?) all that important.

Guardian News Article
Washington Post News Article
Reddit Post

"AI alignment researchers haven't figured out how to take an objective and ensure that a powerful AI will reliably pursue that exact objective. The way the most capable sys

...
Keeping it (less than) real: Against 
ℶ₂ possible people or worlds
by quiet_NaN — 2024-09-13T17:29:44.915Z

Epistemic status and trigger warnings: Not rigorous in either math or physics. Not proof-read by any third party yet. May contain original research. Lengthy. Some sections are tongue-in-cheek. Anthrophics. Cosmology. Theism.

TL;DR: There are good reasons why both the number of possible distinct human-like conscious experiences and the number of possible physics-like observable universes are countable.

Introduction

Catching up The Bayesian Conspiracy, I have recently listened to the episode from a month ago where they talked with Bentham's Bulldog about his argument for God, which was also recently linked by Scott, as an example of an anthropic mistake which is much more interesting.

Unsurprisingly, like Eneasz and Steve, I was also not convinced by their guest's arguments.

However, my difficulties with these arguments start even earlier. Bentham's Bulldog asserts:

But the number of possible people is

at least Beth 2

—that’s

an infinite way bigger

than aleph null. In fact, Beth 2 is infinitely bigger than something that’s infinitely bigger than aleph null. Beth 2 is more than the numbers of numbers that exist—and there’s no plausible atheistic account of reality on which Beth 2 people come to exist.

His citation can be tracked back to D. Lewis: Counterfactuals (1973). The gist of it seems to be that you start with an Euklidian spacetime R4, and consider mappings of that to {0,1} which indicate if a particle is present at that location and time[1]. Finally you do some pruning of equivalent states and arrive at ℶ2 possible universes.

From this, Bentham's Bulldog then argues that there are ℶ2 possible people.

I will first argue against that second proposition, then present another lengthy armchair argument against ℶ2 possible universes, and how to build universes like this one with finite resources. Finally I will try to shed some light on the unfortunate implications of x∈R in general.

Counting people

People, that is, the subjective experience of sentient, sapient, possibly conscious beings are their brains. Compared to universes, brains have a few key advantages when trying to determine how many of them could possibly exist: they have a finite size and a finite amount of matter in them.

Let us consider the question how many distinct brain states there are. As an upper limit, we could consider the wave fun...
Why I'm bearish on mechanistic interpretability: the shards are not in the network
by tailcalled — 2024-09-13T17:09:25.407Z

Once upon a time, the sun let out a powerful beam of light which shattered the world. The air and the liquid was split, turning into body and breath. Body and breath became fire, trees and animals. In the presence of the lightray, any attempt to reunite simply created more shards, of mushrooms, carnivores, herbivores and humans. The hunter, the pastoralist, the farmer and the bandit. The king, the blacksmith, the merchant, the butcher. Money, lords, bureaucrats, knights, and scholars. As the sun cleaved through the world, history progressed, creating endless forms most beautiful.

It would be perverse to try to understand a king in terms of his molecular configuration, rather than in the contact between the farmer and the bandit. The molecules of the king are highly diminished phenomena, and if they have information about his place in the ecology, that information is widely spread out across all the molecules and easily lost just by missing a small fraction of them. Any thing can only be understood in terms of the greater forms that were shattered from the world, and this includes neural networks too.

But through gradient descent, shards act upon the neural networks by leaving imprints of themselves, and these imprints have no reason to be concentrated in any one spot of the network (whether activation-space or weight-space). So studying weights and activations is pretty doomed. In principle it's more relevant to study how external objects like the dataset influence the network, though this is complicated by the fact that the datasets themselves are a mishmash of all sorts of random trash[1].

Probably the most relevant approach for current LLMs is Janus's, which focuses on how the different styles of "alignment" performed by the companies affect the AIs, qualitatively speaking. Alternatively, when one has scaffolding that couples important real-world shards to the interchangeable LLMs, one can study how the different LLMs channel the shards in different ways.

Admittedly, it's very plausible that future AIs will use some architectures that bias the representations to be more concentrated in their dimensions, both to improve interpretability and to improve agency. And maybe mechanistic interpretability will work better for such AIs. But we're not there yet.

^

Possibly clustering the data points by their network gradients would be a way to put some order into this mes

...
Increasing the Span of the Set of Ideas
by Jeffrey Heninger — 2024-09-13T15:52:39.132Z

Epistemic Status: I wrote this back in January, and have been uncertain about whether to publish it. I expect that most people who read this here will be unconvinced. But I still want to express my intuition.

In the last month, these ideas have come up in conversation again. Then, Toby Ord published a paper on arxiv a few days ago with substantial overlap with ideas presented here, so it seems like it is time to post it. I considered rewriting this post to better respond to & engage with that paper, but decided to leave it as is.

Introduction

I have previously stated that I am skeptical that AGI is possible on a (classical) computer.[1] It seems virtuous to try to turn this into specific predictions about what things can and cannot be done using neural nets, the current leading model of AI development.

I am aware that similar sorts of predictions by other people have failed. Nevertheless, I will make predictions.

Here are some things that I would be surprised if AI were able to do:

A language model trained exclusively on a single language[3] invents a new language.
An image classifier trained to associate words with images invents grammar.
An image model trained exclusively on images without text invents writing.
An image model trained exclusively on photographs and naturalistic art invents cubism.[4]
An AI trained to fit classical theories of physics to data invents quantum mechanics.
A model trained to prove theorems in Euclidean geometry invents hyperbolic geometry.

The key thing unifying all of these examples is an invention which is not merely a recombination of the things the AI was trained on. If you have additional examples that you think reflect what I am saying in this post, suggest them in the comments, and I might add them to the list.

The Span of the Set of Ideas

Consider the set of all thoughts anyone has ever had. Now consider all possible recombinations of these ideas or parts of these ideas. I will call this the span of the set of ideas. The analogy is to vector spaces: the span of a set of vectors is the set of all linear combinations of those vectors. A set of ideas might not be a vector space and recombining ideas might not be vector addition (and scalar multiplication), but I think the analogy works well enough.

It also might be worthwhile to think of the span of various subsets of ideas. One subset could be the set of all thoughts e...
How difficult is AI Alignment?
by Sammy Martin — 2024-09-13T15:47:10.799Z

This article revisits and expands upon the AI alignment difficulty scale, a framework for understanding the increasing challenges of aligning artificial intelligence systems with human values.

We explore how alignment difficulties evolve from simple goal misalignment to complex scenarios involving deceptive alignment and gradient hacking as we progress up the scale. We examine the changing dynamics across different levels, including the growing difficulty of verifying alignment and the increasing disconnect between alignment and capabilities research.

By introducing the concept of High Impact Tasks (HITs) and discussing architectural influences on alignment, we are able to explain precisely what determines AI alignment difficulty.

This work was funded by Polaris Ventures

There is currently no consensus on how difficult the AI alignment problem is. We have yet to encounter any real-world, in the wild instances of the most concerning threat models, like deceptive misalignment. However, there are compelling theoretical arguments which suggest these failures will arise eventually.

Will current alignment methods accidentally train deceptive, power-seeking AIs that appear aligned, or not? We must make decisions about which techniques to avoid and which are safe despite not having a clear answer to this question.

To this end, a year ago, we introduced the AI alignment difficulty scale, a framework for understanding the increasing challenges of aligning artificial intelligence systems with human values.

This follow-up article revisits our original scale, exploring how our understanding of alignment difficulty has evolved and what new insights we've gained. This article will explore three main themes that have emerged as central to our understanding:

The Escalation of Alignment Challenges: We'll examine how alignment difficulties increase as we go up the scale, from simple reward hacking to complex scenarios involving deception and gradient hacking. Through concrete examples, we'll illustrate these shifting challenges and why they demand increasingly advanced solutions. These examples will illustrate what observations we should expect to see "in the wild" at different levels, which might change our minds about how easy or difficult alignment is.
Dynamics Across the Difficulty Spectrum: We'll explore the factors that change as we progress up the scale, including th

...