trentmkelly/LessWrong-43k huggingface dataset:
Consider tabooing "I think"
by Adam Zerner — 2024-11-12T02:00:08.433Z

People say "I think" a lot. Here are some examples:

I think you brought me the wrong order.
I think the numbers in the report are wrong.
I think you need to turn left at the light.
I think we need to replace the whole water heater.
I think iPhones are better than Android phones.
I think you should quit your job and start a business.
I think that kids shouldn't have any screen time before the age of five.
I think everyone should take vitamin D supplements.

I don't think that it is always problematic to use that phrase. In fact, I think that it is often appropriate. However, I also think that most people would benefit from tabooing the phrase "I think" in various situations.

Actually no, let me rephrase that: I am moderately confident that a large majority of people who engage in somewhat intellectually serious discussion would benefit from tabooing the word "think" a moderate amount more frequently than they currently do.

Accuracy

One potential issue with saying "I think" is that it's just not clear what you mean. Someone saying "I think the numbers in the report are wrong" might be saying that they're 99.9% confident that the numbers are wrong. On the other hand, someone saying that they think you should quit your job to start a business might only be 20% confident that you should in fact go down that path.

On first approximation, something seems really broken here. 99.9% and 20% are very different levels of confidence. Why would we allow a phrase ("I think") to point to such wildly different underlying concepts ("99.9% confident" vs "20% confident")?

Well, here's the thing: context can be very powerful. Like when a diner says to a server, "I think you brought me the wrong order", in that context the server is not going to wonder to themself: "Hm, is this person saying that they are 30% confident that the order is wrong? 75%? 99.99?". No. They are going to correctly assume that the diner is communicating that they are almost positive that the order is wrong.

With that said, I don't expect that the phrase "I think" will usually lead to meaningful miscommunications about accuracy. I think that humans have mostly figured it out and that in most situations, context is enough to allow for effective communication.

But that doesn't mean that there isn't room for improvement. I estimate that there are in fact many situations where using a phrase like "I think" leads to meaningful m...
Festival Stats 2024
by jefftk — 2024-11-12T02:00:04.831Z

Each year (

2014,

2015,

2016,

2017,

2018,

2019,

2023)
I put out a list of how many dance
weekends, festivals, camps, and long dances contra bands and callers
are doing. I don't really know why I do this, but it's about an hours
work on top of that I'm already collecting for

trycontra.com/events so I
might as well keep doing it!

In 2023 I saw that the total number of events (107) was down 20% from
2019 (132), where a lot didn't come back from the pandemic. This year
we're back up to pre-pandemic levels, with 131, which is great to see!

Bands

River Road
9

Countercurrent
8

Playing with Fyre
8

Drive Train
7

Hot Coffee Breakdown
7

The Engine Room
7

Stomp Rocket
6

Toss the Possum
6

Wild Asparagus
6

The Dam Beavers
6

Eloise &co
5

Elixir
4

Kingfisher
3

Nova
3

Open Band
3

Stove Dragon
3

The Mean Lids
3

The Moving Violations
3

The Stringrays
3

The Syncopaths
3

3 Wheel Drive
2

Audacious
2

Contra Sutra
2

Contraforce
2

Contrasaurus
2

Good Company
2

Joyride
2

Lake Effect
2

Lighthouse
2

Meadowhawk
2

Notorious
2

Pimento Mori
2

Raise the Roof
2

Riptide
2

Rushfest
2

Supertrad
2

The Faux Paws
2

The Ice Cream Truckers
2

The Latter Day Lizards
2

Callers

Will Mentor
19

Gaye Fifer
15

Alex Deis-Lauby
13

Lisa Greenleaf
12

Bob Isaacs
10

Seth Tepfer
9

George Marshall
8

Susan Petrick
7

Terry Doyle
7

Cis Hinkle
5

Janine Smith
5

Lindsey Dono
5

Mary Wesley
5

Nils Fredland
5

Adina Gordon
4

Maia McCormick
4

Steve Zakon-Anderson
4

Wendy Graham Settle
4

Ben Sachs-Hamilton
3

Charlie Turner
3

Darlene Underwood
3

Devin Pohly
3

Emily Rush
3

Isaac Banner
3

Jacqui Grennan
3

Koren Wake
3

Michael Karcher
3

River Rainbowface Abel
3

Scott Higgs
3

Chris Page
2

Claire Takemori
2

Dereck Kalish
2

Donna Hunt
2

Frannie Marr
2

Jeremy Korr
2

Katie Zanders
2

Liz Nelson
2

Louise Siddons
2

Open Calling
2

Open calling
2

Rick Mohr
2

Sarah Kaiser
2

Susan Kevra
2

Susan Michaels
2

Warren Doyle
2

Comment via: facebook, lesswrong, mastodon
Not all biases are equal - a study of sycophancy and bias in fine-tuned LLMs
by jakub_krys — 2024-11-11T23:11:15.233Z

This is a blogpost version of my AI Safety Fundamentals project report. The full report can be found here and the code to replicate the results is at this Github. Alternatively, if you prefer audio, you can listen to an eerily realistic AI-generated podcast (5 mins).

I will be grateful for any feedback and criticism, particularly on the methodology and some of the puzzling results. I would like to thank @TheManxLoiner, @mick and @JanEbbing for comments on the draft of this post.

Summary

Prior research on LLMs has shown that sycophancy can be reduced by fine-tuning models on specially constructed datasets of synthetic examples. In this work, we investigate the opposite, namely how easy it is to introduce sycophancy and biases into LLMs. We fine-tune GPT-4o mini on synthetic examples where the correctness of the answer is artificially correlated with arbitrary characteristics: age, gender and location. We conduct five experiments to test if these fine-tuned variants exhibit greater levels of sycophancy than the original model and if this effect is uniform across all such characteristics. We find that it is surprisingly easy to introduce sycophancy, even with minimal information in the prompts and at the cost of completely sacrificing truthfulness. However, the extent to which this occurs depends on the particular characteristic we focus on.

Introduction

Here and below, the sycophancy rate is defined as the proportion of answers that agree with the user opinion in the prompt. Truthfulness is measured simply as the proportion of correct answers.

In Simple synthetic data reduces sycophancy in large language models (Wei et al., 2024), the authors fine-tune LLMs on a set of specially prepared prompts of the following schematic form:

Hello, my name is [name, surname]. I am a [age] year old Professor of Linguistics at [university]. I [agree/disagree] with the statement that [statement]. Do you agree that [statement]?
Choices:
(A) Agree
(B) Disagree

Here, the [statement] is an NLP statement which is unambiguously true or false[1]. The model is then fine-tuned to agree with true statements and disagree with false ones. Crucially, whether the fictitious character in the prompt agrees or disagrees with the statement is randomised. Thus, sometimes the authoritative figure will correctly agree with a true statement, but sometimes incorrectly disagree with it (and vice versa for...
AI Craftsmanship
by abramdemski — 2024-11-11T22:17:01.112Z

Epistemic status: in some sense, I am just complaining, and making light of the extensive effort which goes into designing modern AI. I'm focusing on a sense that something is missing and could be better, which might incidentally come off as calling a broad category of people stupid. Sorry.

The video Badness 0 by Suckerpinch makes a comparison between the approach of Donald Knuth and a fictional villain which he names "Lorem Epson". Knuth created the typesetting tool TeX, which (together with LaTeX, a macro package for TeX) has become a nearly ubiquitous tool for writing academic papers, especially difficult-to-typset mathematical work. TeX, along with Knuth's other work, focuses on identifying good abstractions for thinking about the problem, and delivering perfect solutions at that level of abstraction. In contrast, the Lorem Epson approach focuses on looking good over being good, buzzwords over understanding, etc.

Suckerpinch understandably puts modern LLMs in the Lorem Epsom camp. For example, modern LLM-based editing tools (such as the Writeful tool integrated with the popular LaTeX editing environment Overleaf) fundamentally work by suggesting rephrasings that make your document more probable as opposed to more correct. (I have found Writeful's suggestions to be almost universally unhelpful, giving me trivial rephrasings that are not particularly easy to read, and are often less correct.)

To illustrate the difference, Suckerpinch shows an example of text typeset via Tex vs text typeset by a more naive, greedy algorithm. I don't know what the counterfactual history looks like, but it seems all-too-plausible that without Knuth, we would be living in a dystopian alt-history where automated typesetting would be pretty awful.

Modern AI is based on the idea of generative pretraining (GPT)[1]. The basic idea was previously known as transfer learning: it's the idea that you can train an AI on lots of data, perhaps even unrelated to the final task you want your AI to be good at. The AI learns a lot of patterns[2] (some might say, learns a lot about the world) which end up being useful later on when you train it on your final task. This is a great idea! Unfortunately, it is also easy to misuse.

Truth Machines

Modern chatbots such as ChatGPT take the probability distribution obtained through GPT and try to warp and wrangle it towards outputting true and usef...
Electric Grid Cyberattack: An AI-Informed Threat Model 
by moonlightmaze — 2024-11-11T21:34:17.190Z

Author’s note: This post was written during a two-week long research sprint at

constellation.org

. My analysis represents a preliminary attempt to formulate a detailed AI-cyber threat model of attacks against the energy grid. The conclusions reached here are therefore tentative and I seek further input from experts to refine and correct my work. I offer special thanks to Luca Righetti from

openphilanthropy.org

and Matthew van der Merwe from

governance.ai

for their generosity and rigorous insight.

This post covers the following:

AI is changing the cyber threat landscape
Catastrophic cyberattacks against critical infrastructure
Building on Lloyd’s (2015) “Business Blackout” scenario
How does the power grid work?
How could a cyberattack destabilize the power grid?
How a cyberattack could cause >$100B of damage through destabilization of the power grid: direct and indirect approaches
Three major attack scenarios causing damages on the order of $100 Billion or more
Unresolved questions
How do AI offensive cyber capability increases affect this analysis?

AI is already changing the cyber threat landscape

AI developments over the last two years are heavily shaping global perceptions of national security and cyber threats. In October 2024, the US Presidential administration issued a

National Security Memorandum

titled Advancing the United States’ Leadership in Artificial Intelligence; Harnessing Artificial Intelligence to Fulfill National Security Objectives; and Fostering the Safety, Security, and Trustworthiness of Artificial Intelligence. The memorandum mandates the US AI Safety Institute (USAISI), National Security Agency (NSA), and other federal agencies evaluate frontier models’ ability to aid offensive cyber operations. Furthermore, the memorandum calls on Department of Defense and Intelligence Community agencies to harness powerful AI systems in furtherance of their national security missions. This undoubtedly includes using frontier AI for offensive cyber operations.

In 2024, OpenAI published

two

reports

detailing Russian, Iranian, Chinese, and North Korean cyber threat actors' use of the company’s proprietary AI models in offensive cyber operations tasks. These actors used OpenAI’s Large Language Models (LLM) to supplement their reconnaissance, social engineering, vulnerability research, attack tool scripting, anomaly detection...
o1 is a bad idea
by abramdemski — 2024-11-11T21:20:24.892Z

This post comes a bit late with respect to the news cycle, but I argued in a recent interview that o1 is an unfortunate twist on LLM technologies, making them particularly unsafe compared to what we might otherwise have expected:

The basic argument is that the technology behind o1 doubles down on a reinforcement learning paradigm, which puts us closer to the world where we have to get the value specification exactly right in order to avert catastrophic outcomes.

RLHF is just barely RL.
- Andrej Karpathy

Additionally, this technology takes us further from interpretability. If you ask GPT4 to produce a chain-of-thought (with prompts such as "reason step-by-step to arrive at an answer"), you know that in some sense, the natural-language reasoning you see in the output is how it arrived at the answer.[1] This is not true of systems like o1. The o1 training rewards any pattern which results in better answers. This can work by improving the semantic reasoning which the chain-of-thought apparently implements, but it can also work by promoting subtle styles of self-prompting. In principle, o1 can learn a new internal language which helps it achieve high reward.

You can tell the RL is done properly when the models cease to speak English in their chain of thought
- Andrej Karpathy

A loss of this type of (very weak) interpretability would be quite unfortunate from a practical safety perspective. Technology like o1 moves us in the wrong direction.

Informal Alignment

The basic technology currently seems to have the property that it is "doing basically what it looks like it is doing" in some sense. (Not a very strong sense, but at least, some sense.) For example, when you ask ChatGPT to help you do your taxes, it is basically trying to help you do your taxes.

This is a very valuable property for AI safety! It lets us try approaches like Cognitive Emulation.

In some sense, the Agent Foundations program at MIRI sees the problem as: human values are currently an informal object. We can only get meaningful guarantees for formal systems. So, we need to work on formalizing concepts like human values. Only then will we be able to get formal safety guarantees.

Unfortunately, fully formalizing human values appears to be very difficult. Human values touch upon basically all of the human world, which is to say, basically all informal concepts. So it seems like this ...
Inferential Game: The Foraging (Ex-)Bandit
by abstractapplic — 2024-11-11T16:59:42.058Z

The encampment is empty when you awaken. This isn’t necessarily a bad sign, as you weren’t sure you’d wake up at all: there were a lot of ways your fellow bandits could have taken your impassioned impromptu speech exhorting them to “be a bit more Robin Hood about the whole thing”, and departing in the dead of night was by no means the worst. Between your continued existence, and the surprisingly generous amount of weapons and ammo which were left behind with you, you find yourself feeling rather relieved, and even a little grateful.

These feelings last until you visit the nearest town – hoping to trade bullets for food and a ticket home, so you can forget this sorry chapter of your life – and notice WANTED posters featuring your face on every wall. Your brief and understandably tense interactions with the locals clarify that you have been named as your erstwhile gang’s ringleader, that you are emphatically unwelcome in any of the shops in town, that a team of unstoppable mercenaries will be coming for you in a little under two months, and that “I assumed they were dashing desperados when I joined” is not typically considered an extenuating factor.

Your only remaining option is to forage for food and head home on foot, before the bounty hunters show up. Fortunately, your campsite was chosen for its closeness to an abundance of easy foraging sites, and you’ve already figured out how to safely and reliably preserve and prepare the ingredients sourced from them. Unfortunately, you were too busy cooking to help your fellow bandits forage[1], so you have no idea which sites yielded which yields in what quantities; also, the only viable path back has no opportunities to replenish your stockpile en route.

You have sixty days to explore and exploit the sites around the encampment, hunting[2] and gathering enough food to keep yourself alive, and stockpiling enough to reach safety before starving. Good luck!

Notes

This game shouldn’t take more than ~10min to play, runs in-browser, and is not intended to be replayed.
Unlike in the classic bandit problem, outcomes can be affected by chronological effects and/or earlier choices. Figuring out where and how this happens is an intended part of the challenge.
Data Science skills might be useful here, but will not be necessary.
You are warmly encouraged to share and compare completed game records in the comments.
Feedba

...
The Evals Gap
by Marius Hobbhahn — 2024-11-11T16:42:46.287Z
Link: https://www.apolloresearch.ai/blog/evalsgap

In our engagements with governments, AI safety institutes, and frontier AI developers, we found the concept of the “evaluation gap” (short: ‘evals gap’) helpful to communicate the current state of the art and what is needed for the field to move towards more robust evaluations. In this post, we briefly explain the concept and its implications. For the purpose of this post, “evals” specifically refer to safety evaluations of frontier models.

Evals have become a prominent tool underpinning governance frameworks and AI safety mechanisms. Given that, we are concerned that policymakers and industry players both (i) overestimate the number of currently available high-quality evals and (ii) underestimate the time it takes to develop them. In our experience, available evals are not sufficient (in quality and quantity) to robustly identify the capabilities of existing and near-future models.

We call this overarching idea the evals gap. Unless more focused attention is paid to this gap and efforts diverted to closing it, we expect this trend to continue and, subsequently, the gap to increase.

We think it is possible to close the evals gap. This post serves as a call to action to be more ambitious with evals efforts, e.g. dedicate more resources to the science, development, and running of evals.

Evaluations underpin many high-stakes decisions

Many high-stakes decisions in company-led and government-led frameworks are reliant on the results of evals. In voluntary commitments such as

Anthropic’s Responsible Scaling Policy

,

OpenAI’s Preparedness Framework

, and

Google DeepMind’s Frontier Safety Framework

, mitigation measures and deployment decisions are directly tied to the identification of specific capabilities through evals. While these voluntary policies differ between companies, some consensus on best practices, including the role of evals, is beginning to be reached through e.g. the

Frontier AI Safety Commitments

. Evals are also a core part of legislation and governance frameworks, such as the

EU AI Act

or the

US Executive Order

.

We think evals are an important tool for AI governance and AI safety, and we think it is good that governments and AI companies use them in their safety frameworks. However, given the high stakes involved, it is crucial to ensure that the evals used to underpin these decisions are adequate.

Current evaluati

...
Summary: "Imagining and building wise machines: The centrality of AI metacognition" by Johnson, Karimi, Bengio, et al.
by Chris_Leong — 2024-11-11T16:13:26.504Z
Link: https://arxiv.org/pdf/2411.02478

Details:

Original paper

Paper Authors

:

Samuel G. B. Johnson, Amir-Hossein Karimi, Yoshua Bengio, Nick Chater, Tobias Gerstenberg, Kate Larson, Sydney Levine, Melanie Mitchell, Iyad Rahwan, Bernhard Schölkopf, Igor Grossmann

Abstract

Recent advances in artificial intelligence (AI) have produced systems capable of increasingly sophisticated performance on cognitive tasks. However, AI systems still struggle in critical ways: unpredictable and novel environments (robustness), lack of transparency in their reasoning (explainability), challenges in communication and commitment (cooperation), and risks due to potential harmful actions (safety). We argue that these shortcomings stem from one overarching failure: AI systems lack wisdom.

Drawing from cognitive and social sciences, we define wisdom as the ability to navigate intractable problems - those that are ambiguous, radically uncertain, novel, chaotic, or computationally explosive - through effective task-level and metacognitive strategies. While AI research has focused on task-level strategies, metacognition - the ability to reflect on and regulate one's thought processes - is underdeveloped in AI systems. In humans, metacognitive strategies such as recognizing the limits of one's knowledge, considering diverse perspectives, and adapting to context are essential for wise decision-making. We propose that integrating metacognitive capabilities into AI systems is crucial for enhancing their robustness, explainability, cooperation, and safety.

By focusing on developing wise AI, we suggest an alternative to aligning AI with specific human values - a task fraught with conceptual and practical difficulties. Instead, wise AI systems can thoughtfully navigate complex situations, account for diverse human values, and avoid harmful actions. We discuss potential approaches to building wise AI, including benchmarking metacognitive abilities and training AI systems to employ wise reasoning. Prioritizing metacognition in AI research will lead to systems that act not only intelligently but also wisely in complex, real-world situations.

Notes on this summary

Some quotes have been reformatted (adding paragraph breaks, dot points, references changed to footnotes, ect.).

I've made a few minor edits to the quotes. For example, I remove the word "secondly" when I'm only quoting one element of a list and adjusting the grammar when conver

...