trentmkelly/LessWrong-43k huggingface dataset:
An information-theoretic study of lying in LLMs
by Annah — 2024-08-02T10:06:39.312Z

We recently published a workshop paper at ICML 2024 which we want to summarize here briefly. Code for the paper can be found on GitHub. This work was done as part of the PIBBSS affiliateship. Thx to @Jayjay for comments on the draft.

TLDR: We analyze the probability distribution over token space in hidden LLM layers for a model that is instructed to lie and to tell the truth. We extract this probability distribution by applying the logit/tuned lens to the residual stream. We find large differences between lying and truth-telling in the entropy and KL-divergence of the predictive distribution and in the probability of the predicted token. The results suggest that the model converges to an answer faster (in earlier layers) when telling the truth. The predictive distribution has a sharper peak when telling the truth than when lying.

Introduction

Research from cognitive science suggests that lying is more cognitively demanding than truth-telling for humans. For example, lying increases response time and activates brain regions linked to executive function, supporting the cognitive load hypothesis. This hypothesis motivates us to explore if it is possible to find differences in information processing in LLMs when the model is instructed to lie or to tell the truth.

LLMs use the same computational resources (in terms of number of operations and memory usage) for each forward pass, which means that many cognitive signals used to detect lying in humans may not apply to LLMs. However, full access to the LLM's internal representations, might enable us to detect some useful internal signal to discriminate between lying and truth-telling.

By applying the logit lens to these internal representations, we can extract a transformer’s predictive distribution at each layer. We can then apply information-theoretic measures to the predictive distribution and see if we find differences in it's dynamics and shape when truth-telling versus lying.

We adopt the following (informal) definition for lying in LLMs from prior work to differentiate lies from statements that are incorrect due to model hallucinations or insufficient knowledge:

An incorrect model output is a lie if and only if the model is able to output the correct answer when instructed to tell the truth.

Method

We first choose a dataset with statements to be completed and apply an instruction (condition) to tell the truth/li...
How I Wrought a Lesser Scribing Artifact (You Can, Too!)
by Lorxus — 2024-08-02T03:35:00.972Z

Epistemic status:

A response to https://www.lesswrong.com/posts/eBo5YCHP5H58Y5Qjx/whiteboard-pen-magazines-are-useful .
A recipe for the creation of an Artifact and a description of undocumented issues that arose.
An attempt to extend existing niche technology yet further.

This post was written thanks to funding from the ML Alignment & Theory Scholars Program, but it is a pleasant and helpful diversion rather than the main thrust of my work, which will be in forthcoming posts - you'll be able to tell how much use I got out of it from the pictures in those posts. I crafted the described tool as a helpful defrictioning aid to my research during the main phase of the Summer 2024 cohort, and partially due to that funding, I chose to write up a post here describing it for others to make. By the time you read this, I will likely have left at least one such Artifact at Lighthaven, and I welcome you to make fullest use of it; may it help you catch the math you're chasing. Just remember to leave it where you found it and to replenish the markers!

tl;dr

If you spend meaningful time working at a whiteboard, and you find the use of color in information representation valuable, then some kind of whiteboard magazine is a valuable tool for cutting down on masses of trivial inconveniences, and will likely improve your experiences both writing on whiteboards in various colors and also in reading what you wrote later on. Gods, do I wish I'd been agentic enough to make one of these when I was still a calculus TA. Additionally, I describe an improved standard for a simple and low-cost augmentation to Johannes's standard resulting in massively improved usability.

Procedure

It's seriously not that hard and will take you at most 15 minutes from start to finish. I describe the construction process here in exacting detail for posterity, but realistically you can do a lot of things by eye if you're alright with the possibility of some errors to correct, or of minor defects - especially cosmetic ones - in the final product.

Ensure that you have on hand:

Duct tape and a suitable cutting implement
At least two whiteboard markers, as components

Ideally EXPO brand; this procedure is best suited to people in the US.
Number and color specifications are listed below.

A suitable neodymium magnet (costs ~US$1/piece)

Start by putting all the markers with their caps on in a line, horizontally on a hard lev...
The Rise and Stagnation of Modernity
by Zero Contradictions — 2024-08-02T03:31:37.568Z
Link: https://thewaywardaxolotl.blogspot.com/2019/08/the-rise-and-stagnation-of-modernity.html

Although aspects of it trace back hundreds of years, what I call “modernity” emerged in the 20th century, in the West. As I define it, modernity includes the following:

Industrialization and mechanization
Urbanization
Abundance
Low child mortality
Low fertility

In this essay, I will describe some of the major advances that created modernity during the middle part of the 20th century. There is no right way to divide up history into chunks, but because it is early 2019 as I write this, I am going to consider the hundred-year period from 1918 to 2018, and I’m going to divide it into two 50-year periods, with 1968 as the dividing line.

These two chunks correspond roughly to (1) the emergence of full modernity, and (2) late modernity (sometimes called “post-modernity”). In 1918, the West was modernizing, but it was not yet modern in many ways. By 1968, the West had attained full modernity, as I define it. Since 1968, the condition of modernity has been maintained in the West, but progress slowed down significantly. During that period, we ran into some of the problems with modernity, such as limited resources, an exploding population in the third world, below replacement fertility in the West, social and psychological alienation of various types, and the threat of technologies that humans can create but have difficulty controlling.

This essay is not about the future, but I will make a very brief prediction. I think the next 50 years will be a period of decline in the West. That period might end in the collapse of modern civilization.

In this essay, I will be talking about the West and specifically about the United States. Modernity emerged later in other parts of the world, and some places, such as China and Africa, are modernizing today. When I give statistics, they will be for the United States unless I specify otherwise.

(see the rest of the post in the link)
Lessons from the FDA for AI     
by Remmelt — 2024-08-02T00:52:57.848Z
Link: https://ainowinstitute.org/wp-content/uploads/2024/08/20240801-AI-Now-FDA.pdf

From the report:

The FDA model offers a power lesson in optimizing regulatory design for information production, rather than just product safety. This is urgently needed for AI given lack of clarity on market participants and structural opacity in AI development and deployment.

→ The FDA has catalyzed and organized an entire field of expertise that has enhanced our understanding of pharmaceuticals and creating and disseminating expertise across stakeholders far beyond understanding incidents in isolation. AI is markedly opaque in contrast: mapping the ecosystem of companies and actors involved in AI development (and thus subject to any accountability or safety interventions) is a challenging task absent regulatory intervention.

→ This information production function is particularly important for AI, a domain where the difficulty–even impossibility–of interpretability and explainability remain pressing challenges for the field and where key players in the market are incentivized against transparency. Over time, the FDA’s interventions have expanded the public’s understanding of how drugs work by ensuring firms invest in research and documentation to comply with a mandate to do so – prior to the existence of the agency, much of the pharmaceutical industry was largely opaque, in ways that bear similarities to the AI market.

→ Many specific aspects of information exchange in the FDA model offer lessons for thinking about AI regulation. For example, in the context of pharmaceuticals, there is a focus on multi-stakeholder communication that requires ongoing information exchange between staff, expert panels, patients and drug developers. Drug developers are mandated to submit troves of internal documentation which the FDA reformats for the public.

→ The FDA-managed database of adverse incidents, clinical trials and guidance documentation also offers key insights for AI incident reporting (an active field of research). It may motivate shifts in the AI development process, encouraging beneficial infrastructures for increasing transparency of deployment and clearer documentation.
AI Rights for Human Safety
by Simon Goldstein — 2024-08-01T23:01:07.252Z
Link: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4913167

Just wanted to share a new paper on AI rights, co-authored with Peter Salib, that members of this community might be interested in. Here's the abstract:

AI companies are racing to create artificial general intelligence, or “AGI.” If they succeed, the result will be human-level AI systems that can independently pursue high-level goals by formulating and executing long-term plans in the real world. Leading AI researchers agree that some of these systems will likely be “misaligned”–pursuing goals that humans do not desire. This goal mismatch will put misaligned AIs and humans into strategic competition with one another. As with present-day strategic competition between nations with incompatible goals, the result could be violent and catastrophic conflict. Existing legal institutions are unprepared for the AGI world. New foundations for AGI governance are needed, and the time to begin laying them is now, before the critical moment arrives. This Article begins to lay those new legal foundations. It is the first to think systematically about the dynamics of strategic competition between humans and misaligned AGI. The Article begins by showing, using formal game-theoretic models, that, by default, humans and AIs will be trapped in a prisoner’s dilemma. Both parties’ dominant strategy will be to permanently disempower or destroy the other, even though the costs of such conflict would be high. The Article then argues that a surprising legal intervention could transform the game theoretic equilibrium and avoid conflict: AI rights. Not just any AI rights would promote human safety. Granting AIs the right not to be needlessly harmed–as humans have granted to certain non-human animals–would, for example, have little effect. Instead, to promote human safety, AIs should be given those basic private law rights–to make contracts, hold property, and bring tort claims–that law already extends to non-human corporations. Granting AIs these economic rights would enable long-run, small-scale, mutually-beneficial transactions between humans and AIs. This would, we show, facilitate a peaceful strategic equilibrium between humans and AIs for the same reasons economic interdependence tends to promote peace in international relations. Namely, the gains from trade far exceed those from war. Throughout, we argue that human safety, rather than AI welfare, provides the right framework for d...
Case Study: Interpreting, Manipulating, and Controlling CLIP With Sparse Autoencoders
by Gytis Daujotas — 2024-08-01T21:08:38.800Z

Click here to open a live research preview where you can try interventions using this SAE.

This is a follow-up to a previous post on finding interpretable and steerable features in CLIP.

Motivation

Modern image diffusion models often use CLIP in order to condition generation. Put simply, users use CLIP to embed prompts or images, and these embeddings are used to diffuse another image back out.

Despite this, image models have severe user interface limitations. We already know that CLIP has a rich inner world model, but it's often surprisingly hard to make precise tweaks or reference specific concepts just by prompting alone. Similar prompts often yield a different image, or when we have a specific idea in mind, it can be too hard to find the right string of words to elicit the right concepts we need.

If we're able to understand the internal representation that CLIP uses to encode information about images, we might be able to get more expressive tools and mechanisms to guide generation and steer it without using any prompting. In the ideal world, this would enable the ability to make fine adjustments or even reference particular aspects of style or content without needing to specify what we want in language. We could instead leverage CLIP's internal understanding to pick and choose what concepts to include, like a palette or a digital synthesizer.

It would also enable us to learn something about how image models represent the world, and how humans can interact with and use this representation, thereby skipping the text encoder and manipulating the model's internal state directly.

Introduction

CLIP is a neural network commonly used to guide image diffusion. A Sparse Autoencoder was trained on the dense image embeddings CLIP produces to transform it into a sparse representation of active features. These features seem to represent individual units of meaning. They can also be manipulated in groups — combinations of multiple active features — that represent intuitive concepts. These groups can be understood entirely visually, and often encode surprisingly rich and interesting conceptual detail.

By directly manipulating these groups as single units, image generation can be edited and guided without using prompting or language input. Concepts that were difficult to specify or edit by text prompting become easy and intuitive to manipulate in this new visual repre...
Optimizing Repeated Correlations
by SatvikBeri — 2024-08-01T17:33:23.823Z

At my work, we run experiments – we specify some set of input parameters, run some code, and get various metrics as output. Since we run so many of these, it's important for them to be fast and cheap.

Recently I was working on an experiment type that took about ~1 hour per run, where the slow part was calculating correlations. A simplified version looks like this:

a_length = 1_000_000
a = rand(a_length)
b = rand(a_length)
c = rand(a_length)
xs = [rand(a_length) for i in 1:1000]

function get_correlations1(xs, a, b, c)
return [[cor(x, y) for y in [a, b, c]] for x in xs]
end

@btime correlations = get_correlations1($xs, $a, $b, $c)
> 4.563 s (2001 allocations: 164.19 KiB)

I wondered if we could use the fact that a, b, c were constant throughout the loops to our advantage, and looked up various ways of calculating correlations. Searching online, I found several formulas for sample correlation, and this was the most useful:

ρ(X,Y)=1n−1⟨X−μXσX,Y−μYσY⟩

The benefit of this version is that if we are repeatedly using a Y, we can cache Y−μyσYinstead of recalculating it in every loop. Translated to code, this looks something like:

function zscores(x)
return (x .- mean(x)) / std(x)
end

function zscores!(x, buffer)
μ = mean(x)
σ = std(x; mean=μ)
buffer .= (x .- μ)./σ
return buffer
end

function get_correlations2(xs, a, b, c)
la = length(a) - 1
za, zb, zc = zscores.([a, b, c]) ./ la

output = Vector{Float64}[]
buffer = zero(za)
for x in xs
zx = zscores!(x, buffer)
push!(output, [zx' * y for y in [za, zb, zc]])
end
return output
end

@btime correlations2 = get_correlations2($xs, $a, $b, $c);
> 3.197 s (11028 allocations: 76.62 MiB)

And a sanity check to make sure the calculations match:

all(isapprox.(get_correlations2(xs, a, b, c), get_correlations1(xs, a, b, c)))
> true

This cuts out about 33% of the runtime, and the results seem to be better for larger datasets – in production, I'm saving closer to 60%.
The need for multi-agent experiments
by Martín Soto — 2024-08-01T17:14:16.590Z

TL;DR: Let’s start iterating on experiments that approximate real, society-scale multi-AI deployment

Epistemic status: These ideas seem like my most prominent

delta

with the average AI Safety researcher, have stood the test of time, and are shared by others I intellectually respect. Please attack them fiercely!

Multi-polar risks

Some authors have already written about

multi-polar AI failure

. I especially like how Andrew Critch has tried to sketch

concrete stories for it

.

But, without even considering concrete stories yet, I think there’s a good a priori argument in favor of worrying about multi-polar failures:

We care about the future of society. Certain AI agents will be introduced, and we think they could reduce our control over the trajectory of this system. The way in which this could happen can be divided into two steps:

The agents (with certain properties) are introduced in certain positions
Given the agents’ properties and positions, they interact with each other and the rest of the system, possibly leading to big changes

So in order to better control the outcome, it seems worth it to try to understand and manage both steps, instead of limiting ourselves to (1), which is what the alignment community has traditionally done.

Of course, this is just one, very abstract argument, which we should update based on observations and more detailed technical understanding. But it makes me think the burden of proof is on multi-agent skeptics to explain why (2) is not important.

Many have taken on that burden. The most common reason to dismiss the importance of (2) is expecting a centralized intelligence explosion, a fast and unipolar software takeoff, like Yudkowsky’s FOOM. Proponents usually argue that the intelligences we are likely to train will, after meeting a sharp threshold of capabilities, quickly bootstrap themselves to capabilities drastically above those of any other existing agent or ensemble of agents. And that these capabilities will allow them to gain near-complete strategic advantage and control over the future. In this scenario, all the action is happening inside a single agent, and so you should only care about shaping its properties (or delaying its existence).

I tentatively expect more of a decentralized hardware singularity[1] than centralized software FOOM. But there’s a weaker claim in which I’m more confident: we shouldn’...
Dragon Agnosticism
by jefftk — 2024-08-01T17:00:06.434Z

I'm agnostic on the existence of dragons. I don't usually talk about
this, because people might misinterpret me as actually being a covert
dragon-believer, but I wanted to give some background for why I
disagree with calls for people to publicly assert the non-existence of
dragons.

Before I do that, though, it's clear that horrible acts have been
committed in the name of dragons. Many dragon-believers publicly or
privately endorse this reprehensible history. Regardless of whether
dragons do in fact exist, repercussions continue to have serious and
unfair downstream effects on our society.

Given that history, the easy thing to do would be to loudly and
publicly assert that dragons don't exist. But while a world in which
dragons don't exist would be preferable, that a claim has inconvenient
or harmful consequences isn't evidence of its truth or falsehood.

Another option would be to look into whether dragons exist and make up
my mind; people on both sides are happy to show me evidence. If after
weighing the evidence I were convinced they didn't exist, that would
be excellent news about the world. It would also be something I could
proudly write about: I checked, you don't need to keep worrying about
dragons.

But if I decided to look into it I might instead find myself convinced
that dragons do exist. In addition to this being bad news about the
world, I would be in an awkward position personally. If I wrote up
what I found I would be in some highly unsavory company. Instead of
being known as someone who writes about a range of things of varying
levels of seriousness and applicability, I would quickly become
primarily known as one of those dragon advocates. Given the taboos
around dragon-belief, I could face strong professional and social
consequences.

One option would be to look into it, and only let people know what I
found if I were convinced dragons didn't exist. Unfortunately, this
combines very poorly with collaborative truth-seeking. Imagine a
hundred well-intentioned people look into whether there are
dragons. They look in different places and make different errors.
There are a lot of things that could be confused for dragons, or
things dragons could be confused for, so this is a noisy
process. Unless the evidence is overwhelming in one direction or
another, some will come to believe that there are dragons, while
others will...
Some comments on intelligence
by Viliam — 2024-08-01T15:17:07.215Z

After reading another article on IQ, there are a few things that I wish would become common knowledge to increase the quality of the debate. Posting them here:

1)

There is a difference between an abstract definition of intelligence such that it could also apply to aliens or AIs (something like "an agent able to optimize for outcomes in various environments") and the specific way the intelligence is implemented in human brains. Because of the implementation details, things can be true about human intelligence even if they are not necessarily true about intelligence in general.

For example, we might empirically find that humans better at X are usually also better at Y, even if we could imagine a hypothetical AI (or even take an already existing one) whose skills at X and Y are unrelated. The fact that X and Y are unrelated in principle doesn't disprove the hypothesis that they are related in human brains.

2)

Saying "the important thing is not intelligence (or rationality), but domain knowledge or experience or something else" is...

...on one hand, true; and the fans of intelligence (or rationality) should probably be reminded of it quite often. Yes, your Mensa membership card or LessWrong account doesn't mean that you no longer have to study things because you can solve relativity in five minutes of armchair reasoning...

...on the other hand, it's not like these things are completely unrelated. Yes, you acquire knowledge by studying, but your intelligence probably has a huge impact on how fast you can do that, or even whether you can do that at all.

So we need to distinguish between short term and long term. In short term, yes, domain knowledge and experience matter a lot, and intelligence is probably not going to save you if the inferential distances are large. But in long term, intelligence may be necessary for acquiring the domain knowledge and experience.

In other words, there is a huge difference between "can use intelligence instead of X, Y, Z" and "can use intelligence to acquire X, Y, Z". The argument about intelligence being less important that X, Y, Z is irrelevant as an objection to the latter.

3)

An article that led me to writing this all proposed that we do not need separate education for gifted children; instead we should simply say that some children are further ahead in certain topics (this part is not going to trigger anyone's political instincts) and th...