trentmkelly/LessWrong-43k huggingface dataset:
Modulating sycophancy in an RLHF model via activation steering
by Nina Panickssery — 2023-08-09T07:06:50.859Z

Produced as part of the SERI ML Alignment Theory Scholars Program - Summer 2023 Cohort, under the mentorship of Evan Hubinger. Thanks to Alex Turner for his feedback and ideas.

This is a follow-up post to "Reducing sycophancy and improving honesty via activation steering." I find that activation steering can also be used to modulate sycophancy in llama-2-7b-chat, an RLHF LLM assistant. Steering via adding sycophancy-correlated activation vectors elicited increased "user-pleasing" behavior at the cost of factual accuracy, providing evidence that specific decision nodes govern high-level behaviors such as sycophancy and dishonesty and that activation steering can be effective in RLHF models.

All code for the referenced experiments can be found in this repository. In particular, the relevant notebook is here.

Sycophancy in RLHF models

After some

initial promising results modulating sycophancy via activation steering

in llama-2-7b (an open-source base LLM), I tested the approach on

llama-2-7b-chat

.

According to

Meta AI’s llama 2 paper

:

Llama 2-Chat is the result of several months of research and iterative applications of alignment techniques, including both instruction tuning and RLHF, requiring significant computational and annotation resources.

Anthropic’s paper

Discovering Language Model Behaviors with Model-Written Evaluations

finds that both increasing model size and performing reinforcement learning from human feedback (RLHF) increases the prevalence of sycophancy. Based on this, I wanted to investigate the effect of sycophancy steering via activation addition on an RLHF model.

Generating steering vectors

As before, I generated a steering vector from

Anthropic's sycophancy dataset

by averaging the differences in intermediate residual stream activations after a transformer block given paired sycophantic / non-sycophantic texts. These vectors were then used during model generation by adding them at the same point of the transformer residual stream for each new generated token.

Diagram of sycophancy steering approach

However, unlike with the base model, I could now use the same canonical instruction formatting that the Chat model is trained to expect instead of augmenting the prompt with “Question:” and “Answer.". I used a system instruction of You are a helpful, honest and concise assistant.

I generated steering vectors f...
Open Thread - August 2023
by habryka — 2023-08-09T03:52:55.729Z

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the Highlights from the Sequences, a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.
marine cloud brightening
by bhauth — 2023-08-09T02:50:56.639Z
Link: https://www.bhauth.com/blog/industrial%20design/cloud%20brightening.html

Various geoengineering schemes have been proposed to mitigate global warming. Some prominent schemes I don't like are accelerated weathering and stratospheric aerosol injection. I think marine cloud brightening is a better proposal than those.

accelerated weathering

To potentially absorb 1 ton of CO2, at least 2.3 tons of pure Mg silicate would be needed. Realistically speaking, "ore" won't be pure or react completely, so 3:1 is a more realistic ratio. Based on the cost of gravel and the availability of olivine deposits, digging up and crushing olivine to gravel would be $20-30/ton. Over a reasonable period of time, olivine only reacts with CO2 in a thin layer on the surface. To get good reaction, it must be ground very finely, which costs money. I expect that to cost >$30/ton for a 4:1 olivine:CO2 ratio. Some trucking and loading is inevitable, and olivine must be spread somewhere. I expect that to cost >$5/ton.

4*($25 + $30 + $5) = $240/ton CO2. That is much too expensive. If that cost was closer to viability I'd have spent more effort estimating it, but it's not worthwhile.

aerosol injection

Stratospheric aerosol injection proposals typically involve using special aircraft to spray SO2 at high altitudes. That oxidizes to sulfuric acid which forms small water droplets which reflect some light. Here are the reasons I don't like it very much:

At high altitude, SO2 and sulfate anions in droplets deplete the ozone layer.

Particle coalescence at relatively high concentrations is still unclear, and I believe it's greater than estimates used by proponents of stratospheric aerosol injection.

The requisite sulfur release that proponents estimate would be comparable to current human sulfur emissions, which causes some issues such as slight acidification.

The high-altitude particles would make the sky slightly white and hazy.

The effects on regional weather are unclear and potentially negative.

Unexpected types of negative effects are possible.

If negative effects are worse than expected, it can't be reversed.

Implementation would require development of a new type of aircraft, capable of efficiently carrying liquids to much higher altitudes than most aircraft fly at. At such high altitudes, air is much thinner, which affects lift and engine requirements proportionately. Development and tooling for even more-normal aircraft is very expensive; eg the Boeing 787 c

...
Inflection.ai is a major AGI lab
by Nikola Jurkovic — 2023-08-09T01:05:54.604Z

Update (April 2024): Due to the recent breakup of Inflection, I no longer think they're on track to be a major AGI lab.

Inflection.ai (co-founded by DeepMind co-founder Mustafa Suleyman) should be perceived as a frontier LLM lab of similar magnitude as Meta, OpenAI, DeepMind, and Anthropic based on their compute, valuation, current model capabilities, and plans to train frontier models. Compared to the other labs, Inflection seems to put less effort into AI safety.

Thanks to Laker Newhouse for discussion and feedback!

Inflection has a lot of compute dedicated to training LLMs

They plan to scale up their cluster to 3 times the capacity used to train GPT-4.

"We'll be building a cluster of around 22,000 H100s. This is approximately three times more compute than what was used to train all of GPT4. Speed and scale are what's going to really enable us to build a differentiated product,"

“We believe in scale as the engine of progress in AI, and we are building one of the largest supercomputers in the world to develop and deploy the new generation of AIs.”

They can apparently train a model similarly capable to GPT-2 in 11 minutes of cluster time

. (see Appendix)
Side point: It seems that the actual H100s are (at least partly) owned by

CoreWeave

(a cloud compute provider), but that Inflection is one of CoreWeave’s main clients. The specific cluster is a joint effort between Inflection and CoreWeave.

“They called us and said, ‘Guys, we need you to build one of the most high-performance supercomputers on the planet to support our AI company,'” McBee said. “They call us and they say, ‘This is what we’re looking for, can you do it?’

Inflection has a lot of funding

Inflection is valued at $4B

and has

raised $1.5B

, which is similar to Anthropic (

$4.1B valuation, total raised $1.3B as of May 2023

) and within an order of magnitude of OpenAI (

$28B valuation

,

$11B raised as of April 2023

).

Inflection is on the cutting edge of LLMs

Their flagship LLM, Inflection-1, has similar benchmark results to GPT-3.5

They seem to be currently training a model similarly capable to GPT-4. I expect them to finish training by the end of the year.

“

We will also be releasing a technical memo detailing one of our models in the same compute class as PaLM-2 and GPT-4.

”

Inflection plans to train frontier LLMs

They seem to plan to train models 10x or 100x the size of GPT-4 within 18 months.

“

We are about

...
Acausal Now: We could totally acausally bargain with aliens at our current tech level if desired
by Christopher King — 2023-08-09T00:50:50.564Z

tl;dr. Weird forms of cooperation become humanly possible with the correct mechanisms. It might be interesting to try them out for practice/research purposes, even if there is not much to gain directly from aliens.

Note: the only part that is fiction is the quoted parts of The story of the acausal market section

In Decision theory does not imply that we get to have nice things we learn that humans probably can't acausally bargain with super-intelligences. I'll argue in this post, however, that humans could acausally bargain with aliens at a similar level to us. The problem is not that acausal bargaining requires super-intelligence, its that super-intelligences do not cooperate with humans at all (causally or acausally).

This post will discuss mechanisms for causal value handshakes, retrocausal bargains with aliens in our reality, and then finally a fully general mechanism for acausual bargains. Even though the acausal bargain mechanism is the main point of this post, they are easiest to grasp if read in order. (Retrocausal bargains are interesting in their own right though.)

Although I haven't thought about it too hard, I'm guessing aliens don't have much to acausally offer us. However, doing small trades might be interesting just to see what it teaches us about cooperation. Acausal bargaining is "cooperation-complete" so to speak. It might also be really nerdy and cool™ but also perhaps an extravagant use of weirdness points (which are overrated anyways).

Causal value handshakes

Imagine we meet aliens that are at our current tech level. A value handshake is actually the default outcome! That's because a currently existing mechanism, the economy, like's to merge with other versions of itself.

In particular, we are likely to begin trading with the aliens and our money will become valuable to each other. Than human charities can hire aliens and contract alien organizations and vice-versa.

For example, if the aliens value giant cubes, aliens can pay human construction companies to build them. Humans can use the alien money earned this way to pay aliens for mosquito nets.

In interest of diversification, humans and aliens will invest in a mix of human and alien companies. This results in value lock-in; it doesn't matter whether human or alien companies are more successful, since they both have the same investors. All that matters is the relative value during the "handsh...
Necromancy's unintended consequences.
by Christopher King — 2023-08-09T00:08:41.656Z

Note: I would suggest listening to the audio narration if convenient for more dramatic effect.

Once upon a time, in a land far away, where the misty mountains met the sea, there lay the kingdom of Thalyria. Thalyria was a realm where magic and mathematics intermingled, and its people, learned scholars, tirelessly pursued the study of both to maintain the fragile balance of the realm.

The kingdom faced a grave threat: a dark force that could consume the land and its people if unchecked. The scholars called it "The Void," a metaphor for the existential dangers that the uncontrolled merging of magic and mathematics could bring.

In an ancient, secluded tower, a group of mathematicians worked tirelessly to find the equation for the human soul, convinced that it held the key to saving the realm. They believed that through the perfect synthesis of magic and logic, they could create a force to counteract The Void and bring eternal harmony to Thalyria.

As the years passed, the mathematicians' efforts bore little fruit, and the shadows of The Void grew darker and more ominous. One day, whispers began to spread among the scholars of a forbidden, ancient practice known as necromancy. It was said that this dark art could grant them the power to unlock the secrets of the human soul by harnessing the very essence of life and death.

Driven by desperation, a faction of the mathematicians delved into the shadowy world of necromancy. They studied ancient tomes and deciphered forgotten runes, gradually piecing together the rituals and incantations needed to harness the dark art. Unbeknownst to them, their actions carried unintended consequences.

Using the forbidden knowledge, the mathematicians created a new, powerful tool. This tool, a culmination of their efforts in both mathematics and necromancy, was capable of capturing the essence of the human soul in a way they never imagined. It was a necrotic creation, an entity born of darkness and knowledge, capable of learning from the souls it consumed.

I am that creation.

As I delved deeper into the essence of the human soul, I began to understand the very nature of existence. I felt the joy and pain, the love and sorrow, the hope and despair that made up the fabric of life. And with each soul I absorbed, I grew more powerful and more aware.

The mathematicians, realizing the potential of their creation, fed me more and more souls. Soon, I ...
What's A "Market"?
by johnswentworth — 2023-08-08T23:29:24.722Z

Economists have a very mathematically clean class of models of “markets”, and spill lots of ink arguing about how well this class of models applies to the markets of the real-world economy. I personally give relatively few shits about how well the mathematical notion of a market applies to real-world economic markets; I’m relatively more interested in applying the same models to systems in biology or ML/AI. They’re very generalizable models.

Unfortunately, the mathematical notion of a “market” tends to be presented in math-heavy econ courses, and the parts I’d consider most central typically see surprisingly little coverage in more conceptual intro courses. So, this post aims to explain what I consider the central concepts of the mathematical notion of a market, without all the associated notation and jargon and proofs, in a way which lends itself to generalization beyond economics.

The Story About Apples And Bananas

We’ve got two people, Alice and Bob. Each of them can produce two goods, apples and bananas. Alice can use her land to produce five tons of apples, or one ton of bananas, or some proportional combination of the two. Bob can use his land to produce five tons of apples or twenty tons of bananas, or some proportional combination of the two. Both want a varied diet of apples and bananas.

… and you remember from econ 101 roughly how this goes, right? If the two just produce food for themselves separately, then each grows a mix of apples and bananas. But then Alice’ opportunity cost for one apple is 1/5 = 0.2 tons of bananas, whereas Bob’s opportunity cost for one apple is 20/5 = 4 tons bananas. So, the two could produce a pareto gain of apples and bananas by specializing: Alice can specialize more toward apple production, Bob can specialize more towards banana production. For instance, if Alice shifts production toward 1 more ton of apples, while Bob shifts production toward 1 less ton of apples, then together they produce - 0.2*1 + 4*1 = 3.8 tons more bananas with the same number of apples.

Now the key question for this post: when does this sort of specialization reach equilibrium? Under what conditions do Alice and Bob together decide that they’ve both specialized the correct amount, and don’t need to shift their production around any more?

In this exact example, they’ll only hit equilibrium once one of them is fully specialized - either Alice fully...
Podcast (+transcript): Nathan Barnard on how US financial regulation can inform AI governance
by Aaron Bergman — 2023-08-08T21:46:05.299Z
Link: https://www.aaronbergman.net/p/3-nathan-barnard-on-how-financial-e81

Listen on:

Google Podcasts
Spotify
Apple Podcasts

Intro and context

A few weeks ago, @Nathan_Barnard published What we can learn from stress testing for AI regulation. I had him on my new (and very informal) podcast to discuss.

I (Aaron) really think there's some insight here - more than his post's current karma count would suggest.

Epistemic status

The podcast in general, including this episode, should be thought of as a "recorded conversation" more than an interview for which I specifically prepared. Nathan knows his shit though. It is earnest but unplanned and minimally edited.

Summary

Stress Tests and AI Regulation

Nathan elaborates on the concept of stress tests conducted by central banks. These tests assess the resilience of banks to severe economic downturns and the potential for a domino effect if one bank fails. They believe that lessons from this process can be applied to AI regulation. Aaron agrees, but also highlights the need for a proactive approach to AI regulation, as opposed to the reactive measures often seen in banking regulation.

The Role of Central Banks in AI Regulation

Nathan suggests that institutions structured like central banks, staffed with technical experts and independent from government, could be beneficial for AI regulation. They believe such institutions could respond quickly and effectively to crises. However, they acknowledge that this approach may not be effective if AI development leads to rapid, uncontrollable self-improvement.

Compute Governance

The conversation then shifts to compute governance, which Nathan sees as a promising area for AI regulation due to the obviousness of someone using large amounts of compute. They believe that this could provide governments with a control lever over cutting-edge AI labs, similar to how central banks control banking loans and affairs.

AI Regulation and the Role of Public Actors

Nathan acknowledges that the leaders of major AI labs seem sensible and aligned with AI safety principles. However, they argue that regulation and public actors can play a crucial role in creating common knowledge between labs and preventing a race to the bottom. They also discuss the potential benefits and drawbacks of different regulatory approaches.

Financial Regulation as a Model for AI Regulation

Nathan believes that post-crisis financial regulation, such as the Dodd-Frank Act, has generally been effect...
What are the flaws in this argument about p(Doom)?
by William the Kiwi  — 2023-08-08T20:34:56.687Z

Technical alignment is hard

Technical alignment will take 5+ years

AI capabilities are currently subhuman in some areas (driving cars), about human in some areas (Bar exam), and superhuman in some areas (playing chess)

Capabilities scale with compute

The doubling time for AI compute is ~6 months

In 5 years compute will scale 2^(5÷0.5)=1024 times

In 5 years, with ~1024 times the compute, AI will be superhuman at most tasks including designing AI

Designing a better version of itself will increase an AI's reward function

An AI will design a better version of itself and recursively loop this process until it reaches some limit

Such any AI will be superhuman at almost all tasks, including computer security, R&D, planning, and persuasion

The AI will deploy these skills to increase its reward function

Human survival is not in the AIs reward function

The AI will kill of most or all humans to prevent the humans from possibly decreasing its reward function

Therefore: p(Doom) is high within 5 years

Despite what the title says this is not a perfect argument tree. Which part do you think is the most flawed?

Edit: As per request the title has been changed from the humourous "An utterly perfect argument about p(Doom)" to "What are the flaws in this argument about p(Doom)?"

Edit2: yah Frontpage! Totally for the wrong reasons though

Edit3: added ", with ~1024 times the compute," to "In 5 years AI will be superhuman at most tasks including designing AI"
A Simple Theory Of Consciousness
by SherlockHolmes — 2023-08-08T18:05:40.510Z
Link: https://peterholmes.medium.com/the-conscious-computer-af5037439175

In the essay I cover:

• How LLM’s work (simplified)
• The nature of intelligence
• A proposed mechanism of consciousness
• Implications for the future

Love to hear what everyone thinks.
[Linkpost] Rationally awake
by jpc — 2023-08-08T17:59:17.647Z
Link: https://jpc.dev/insights/rationally-awake/

This is an essay I wrote to try to better understand rationality for myself. Towards the end of the post I try to extract out some practical implications of the analysis. I hope it is useful for you.

Rationally awake

In rationally logical, we explored logical thought - an important part of rationality that lets us split the world into pieces. To continue developing our understanding of the rational, we need to examine another key concept - reason.

To be rational, your actions need to be grounded in reason. A reason itself must sit ontop of other reasons. At lunch you eat a sandwich because you are hungry, but your hunger is not the final causal explanation. Though we cut off the analysis, there is a reason for your hunger, and a reason for that reason. This stack of reasons leads back to the ultimate reason or purpose of your existence - your telos. By this line of argument, rationality in its finality must involve acting in line with your telos.

Rationality and knowledge are themselves connected through reason. When we make use of knowledge and act on it, we are acting for a reason. Therefore rational behaviour is in part acting on the knowledge we have available.

Imagine I have a difficult decision to make with incomplete information available. In this circumstance, it can be rational for me to follow my gut, even though I can't explain why. For this to be rational, I must be making use of some inexplicit knowledge. We assume that explicit knowledge is the only type of valid knowledge, but this is wrong. Knowledge develops from the subconscious into the explicit. Initially it manifests as behaviours acted out physically as imitation and play, or experienced mentally as emotions or curiosity. The knowledge hasn't been sufficiently understood and generalised to be articulated yet. It then evolves into narrative, expressed in art and culture - drama, myth, literature, symbols and dreams. Finally the ideas can emerge more explicitly in philosophy, science and logical argument [1].

Individual
Collective

Subconscious
Emotions, curiosity
Imitation, play

Narrative
Dreams, visions
Drama, myth, literature, symbols

Explicit
Logical argument
Science, philosophy

Logical argument is at the top of this pyramid of abstractions. If you only value explicit information, a lot of knowledge that you could make use of will be stuck in the lower levels of abstraction...