trentmkelly/LessWrong-43k huggingface dataset:
Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought
by Miles Turpin — 2024-03-11T23:46:18.041Z
Link: https://arxiv.org/abs/2403.05518

[Twitter thread]

I'm not going to add much additional commentary at the moment and will just let people check out the paper!

But to give a bit more context: This paper is building off prior work we have done showing that chain-of-thought explanations can be misleading, which I wrote about on the alignment forum here. Broadly, this work fits into the process-based oversight agenda (e.g., here). Consistency training/evaluation also fits into scalable oversight: Evaluating consistency may be easier than evaluating a model's reasoning or final predictions directly (e.g., also explored here).

Abstract:

While chain-of-thought prompting (CoT) has the potential to improve the explainability of language model reasoning, it can systematically misrepresent the factors influencing models' behavior--for example, rationalizing answers in line with a user's opinion without mentioning this bias. To mitigate this biased reasoning problem, we introduce bias-augmented consistency training (BCT), an unsupervised fine-tuning scheme that trains models to give consistent reasoning across prompts with and without biasing features. We construct a suite testing nine forms of biased reasoning on seven question-answering tasks, and find that applying BCT to GPT-3.5-Turbo with one bias reduces the rate of biased reasoning by 86% on held-out tasks. Moreover, this model generalizes to other forms of bias, reducing biased reasoning on held-out biases by an average of 37%. As BCT generalizes to held-out biases and does not require gold labels, this method may hold promise for reducing biased reasoning from as-of-yet unknown biases and on tasks where supervision for ground truth reasoning is unavailable.
AI Safety Action Plan - A report commissioned by the US State Department
by agucova — 2024-03-11T22:14:24.370Z
Link: https://www.gladstone.ai/action-plan

This morning, Gladstone AI published a report they made that was commissioned by the US State Department. The report not only acknowledges catastrophic risks from AI, but also includes a very ambitious set of proposals designed for the US to mitigate catastrophic risks arising from advanced AI, going from export controls to a new federal regulatory agency, and even suggesting the creation of multilateral treaties for a new international regulatory agency.

I've only read the executive summary and skimmed the full report, but it looks pretty great. It's detailed and concrete, it's written very well, and includes some pretty useful diagrams to explain their implementation proposals. It's surprisingly polished for a report made by just a few authors.

The report also got pretty extensive coverage by Time Magazine today (which published a general article, as well as one zooming in on the expert interviews). You can also find a good Twitter thread of the lead author explaining the context for the report here.
A discussion of AI risk and the cost/benefit calculation of stopping or pausing AI development
by DuncanFowler — 2024-03-11T21:41:09.934Z

I've been interested in AI risk for a very long time, but I've never written that much about it because I always felt other people were better at discussing the situation.
I have begun to revise this opinion, and decided to write down the basic points of disagreement I have, which rather ran one longer than I thought it would and now stands at 18000 words, and I still feel like many of the points I make could be expanded upon.
I attach a link below.
I try to make the main points that:
a) The current dominant paradigm of AI is likely not be hostile by default
b) While there are many things that could go wrong there are also many possible solutions
c) Multipolar scenarios are probably more stable then everyone here seems to think
d) Any pause, whether from a few years to indefinitely, probably makes things worse because our leaders are not capable of organising it
And conclude that our chances of survival are over 50% (not including anthropics, aliens saving us etc.) and that we should not pause. I do not mean that this should promote complacency.
Anyway, I hope this gives people a modicum of hope. Please let me know if you think this is a waste of time, making points others have already made better, of whether it is of any value.
https://docs.google.com/document/d/e/2PACX-1vRHYihtx2QUXH54RXNJRN2q7jjhwAcBqhPAhzJhjmw3Yv546vsXfv3pvNM9NUIoybpM-IgplyQWelCW/pub
Among the A.I. Doomsayers - The New Yorker
by agucova — 2024-03-11T21:35:27.638Z
Link: https://www.newyorker.com/magazine/2024/03/18/among-the-ai-doomsayers

The New Yorker just realized a feature article on AI Safety, rationalism, and effective altruism, and… it's surprisingly good? It seems honest, balanced, and even funny. It doesn't take a position about AI Safety, but IMO it paints it in a good way.

Paul Crowley (mentioned in the article) did have some criticism, and put up a response in a post. Most of his concerns seem to be with how the article takes a focus on the people interviewed, rather than the ideas being discussed.
Be More Katja
by Nathan Young — 2024-03-11T21:12:14.249Z

Katja is widely respected amongst the rationalists and, according to Hive, she is one of the most followed/respected EA accounts[1]. But she doesn't give off the same vibe as many impact olympians. She doesn’t have iron self-will, nor does she manage a huge team. She's hasn't got all the facts at her fingertips. But she has got something, I’m confident of that. How can I be more like her?

To understand her impact, let's consider the top things she's done:

She ran surveys on AI researchers well before they were needed and has continued to run them
She wrote an early blog on how we could slow down AI. This blog, I've heard, played a part in encouraging the Musk AI letter, which in turn inspired the “Existential Risks” AI letter.
She thought about AI long before it was vogue, since about 2010
She has a large track record of predictions

These actions seem impactful to me. And I guess someone should have paid $10mn in hindsight for the first 2, maybe more.

To me, Katja has a very low tolerance for incomplete stories. When she sees something that she doesn’t quite understand or that seems a bit off she struggles to pretend otherwise, so she says “how does that work?”. She doesn’t accept handwaving when discussing something, whether it be the simulation argument, how efficient flight is or the plot of Dune, part 2[2]. She wants an unbroken chain of arguments she can repeat[3].

She also doesn’t mind admitting she doesn’t know the answer. In her living room she will turn to her friend Joe Carlsmith and ask “Wait, why are we worried about AI, again?” even though she’s been thinking about this for 15 years. Because at that moment it doesn’t fit for her and she has a high tolerance for embarrassed[4] when it comes to truth. There is an deep resolve here - she doesn't get it, so she will ask until she does.

She works on the most important thing, slowly. If you are Elon Musk, maybe you can just work all the time. But I am not. And much as I love her, neither is Katja. She does not get an abnormal amount of work done per day. Instead, month in, month out, Katja works on what she thinks is most important. And eventually she gets the survey done, years ahead of when it’s needed.

There are lessons we can take from this. Just as we often talk about learning to code, or task management, I can become better at saying “wait that doesn’t work”. Here are some st...
AI Incident Reporting: A Regulatory Review
by Deric Cheng — 2024-03-11T21:03:02.036Z

This article is the first in a series of ~10 posts comprising a 2024 State of the AI Regulatory Landscape Review, conducted by the

Governance Recommendations Research Program

at

Convergence Analysis

. Each post will cover a specific domain of AI governance (e.g. incident reporting, safety evals, model registries, etc.). We’ll provide an overview of existing regulations, focusing on the US, EU, and China as the leading governmental bodies currently developing AI legislation. Additionally, we’ll discuss the relevant context behind each domain and conduct a short analysis.

This series is intended to be a primer for policymakers, researchers, and individuals seeking to develop a high-level overview of the current AI governance space. We’ll publish individual posts on our website and release a comprehensive report at the end of this series.

Let us know in the comments if this format is useful, if there are any topics you'd like us to cover, or if you spy any key errors / omissions!

Context

AI incident reporting refers to an emerging series of voluntary practices or regulatory requirements for AI labs to document any unexpected events, malfunctions, or adverse effects that arise from the deployment of AI systems. Such mechanisms are designed to capture a wide range of potential issues, from privacy breaches and security vulnerabilities to biases in decision-making.

The rationale behind incident reporting is to create a feedback loop where regulators, developers, and the public can learn from past AI deployments, leading to continuous improvement in safety standards and compliance with legal frameworks. By systematically documenting incidents, stakeholders can identify patterns, understand the root causes of failures, and implement corrective measures to prevent recurrence.

Historically, incident reporting has been a highly effective tool used across a variety of industries for decades to mitigate risk from still-developing technologies.

For instance, the

Aviation Safety Reporting System

(ASRS) has been noted for its effectiveness at drastically reducing the fatality rate in US aviation during its launch. Its success has been attributed to its confidential, voluntary, and non-punitive approach: anybody can submit a confidential incident report of a near-miss or an abuse of safety standards to a neutral third-party organization (in this

...
Results from an Adversarial Collaboration on AI Risk (FRI)
by Josh Rosenberg — 2024-03-11T20:00:24.642Z
Link: https://forecastingresearch.org/s/AIcollaboration.pdf

Authors of linked report: Josh Rosenberg, Ezra Karger, Avital Morris, Molly Hickman, Rose Hadshar, Zachary Jacobs, Philip Tetlock[1]

Today, the Forecasting Research Institute (FRI) released “Roots of Disagreement on AI Risk: Exploring the Potential and Pitfalls of Adversarial Collaboration,” which discusses the results of an adversarial collaboration focused on forecasting risks from AI.

In this post, we provide a brief overview of the methods, findings, and directions for further research. For much more analysis and discussion, see the full report: https://forecastingresearch.org/s/AIcollaboration.pdf

(This report is cross-posted to the EA Forum.)

Abstract

We brought together generalist forecasters and domain experts (n=22) who disagreed about the risk AI poses to humanity in the next century. The “concerned” participants (all of whom were domain experts) predicted a 20% chance of an AI-caused existential catastrophe by 2100, while the “skeptical” group (mainly “superforecasters”) predicted a 0.12% chance. Participants worked together to find the strongest near-term cruxes: forecasting questions resolving by 2030 that would lead to the largest change in their beliefs (in expectation) about the risk of existential catastrophe by 2100. Neither the concerned nor the skeptics substantially updated toward the other’s views during our study, though one of the top short-term cruxes we identified is expected to close the gap in beliefs about AI existential catastrophe by about 5%: approximately 1 percentage point out of the roughly 20 percentage point gap in existential catastrophe forecasts. We find greater agreement about a broader set of risks from AI over the next thousand years: the two groups gave median forecasts of 30% (skeptics) and 40% (concerned) that AI will have severe negative effects on humanity by causing major declines in population, very low self-reported well-being, or extinction.

Extended Executive Summary

In July 2023, we released our Existential Risk Persuasion Tournament (XPT) report, which identified large disagreements between domain experts and generalist forecasters about key risks to humanity (Karger et al. 2023). This new project—a structured adversarial collaboration run in April and May 2023—is a follow-up to the XPT focused on better understanding the drivers of disagreement about AI risk.

Methods

We recruited participants to join “AI skeptic...
The Astronomical Sacrifice Dilemma
by Matthew McRedmond — 2024-03-11T19:58:12.179Z

Epistemic Status: I think the dilemma as outlined in Section 1 follows from well-established ideas about Astronomical Waste. However, given that I have not seen it anywhere before I might have made some oversight I am unaware of. You don't know what you don't know but maybe someone on LessWrong does.

UPDATE: I have found a reference to this scenario as a footnote in Kaj Sotala's Chapter in "Artificial Intelligence Safety and Security" on

Disjunctive Scenarios of Catastrophic AI Risk

on page 318:

"An AI which was simply seeking to maximize human happiness in general might be willing to sacrifice all currently living humans, if it thought that this allowed it to create more happy humans later on."

Introduction

The potential dangers posed by a misaligned superintelligence have been extensively explored on this forum. However, in this short post, I will outline a moral dilemma that introduces the possibility that beings who are living on Earth when a superintelligence emerges would be at risk even if the superintelligence is value-aligned[1]. In one sentence; once it can be done safely, the immense benefit of colonising the accessible universe quickly may vastly outweigh the welfare of beings living on Earth at the time colonisation becomes possible, potentially justifying extreme disregard for their welfare.

Now, this is not the life-or-death question facing humanity, but it is potentially a life-or-death question that a generation of humans might have to consider. If nothing else, it is a scenario that may challenge readers to consider some of the critical assumptions that they use to justify longtermism and/or their motivations for working on the alignment problem.

In this post, I will simply outline the dilemma. If the dilemma turns out to be of interest to the community I will write a follow-up post with a more extensive examination of the scenario.

The AI loves you, but you are made of atoms which it can use to help create an inconceivably large number of flourishing transhumans

Consider this: we succeed in creating a superintelligence that carries out actions that are aligned with whatever objective we give it. After thorough consideration, we choose an objective along the lines of “use the available energy in the accessible universe to create the greatest amount of 'human value’". I have a complete theory of human value but she lives in Cana...
Epiphenomenalism leads to eliminativism about qualia
by Clément L — 2024-03-11T19:53:25.082Z

Introduction

In this post I will explain why I think that certain forms of realism about conscious experience face some issues that ultimately lead them to the conclusion that our belief that consciousness exists is not reliable, and thus consciousness may not exist at all, as counterintuitively as it seems. This conclusion is similar to the one given by the illusionist theory of consciousness, a kind of eliminativism about qualia, which entails that conscious experience does not exist.

A brief presentation of epiphenomenalism and illusionism

Epiphenomenalism

First of all, we need to briefly define what these two views consists in. This is the definition of epiphenomenalism found of Wikipedia :

Epiphenomenalism is a position on the mind–body problem which holds that subjective mental events are completely dependent for their existence on corresponding physical and biochemical events within the human body, yet themselves have no influence over physical events. The appearance that subjective mental states (such as intentions) influence physical events is merely an illusion, consciousness being a by-product of physical states of the world. For instance, fear seems to make the heart beat faster, but according to epiphenomenalism the biochemical secretions of the brain and nervous system (such as adrenaline)—not the experience of fear—is what raises the heartbeat.[1] Because mental events are a kind of overflow that cannot cause anything physical, yet have non-physical properties, epiphenomenalism is viewed as a form of property dualism.

Indeed, epiphenomenalism supposes a property dualism which holds that although the world is constituted of one kind of substance that is the physical kind, or matter, there exists two kinds of properties : first, physical properties (mass, electric charge etc.), which are described by physics. They can be measured and are publicly observable. Second, there are mental properties, which constitute our subjective conscious experience. These are essentially private and not publicly observable ; no measuring device or no one else than myself could ever access my conscious experience itself, apart from its behavioral or neurological correlates, which are physical states.

Eliminative materialism and illusionism

Eliminative materialism, or simply eliminativism, is the view that most mental states as we refer to them in our commo...
Open Thread Spring 2024
by habryka — 2024-03-11T19:17:23.833Z

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the Highlights from the Sequences, a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.
New social credit formalizations
by KatjaGrace — 2024-03-11T19:00:06.201Z

Here are some classic ways humans can get some kind of social credit with other humans:

Do something for them such that they will consider themselves to ‘owe you’ and do something for you in future

Be consistent and nice, so that they will consider you ‘trustworthy’ and do cooperative activities with you that would be bad for them if you might defect

Be impressive, so that they will accord you ‘status’ and give you power in group social interactions

Do things they like or approve of, so that they ‘like you’ and act in your favor

Negotiate to form a social relationship such as ‘friendship’, or ‘marriage’, where you will both have ‘responsibilities’, e.g. to generally act cooperatively and favor one another over others, and to fulfill specific roles. This can include joining a group in which members have responsibilities to treat other members in certain ways, implicitly or explicitly.

Presumably in early human times these were all fairly vague. If you held an apple out to a fellow tribeswoman, there was no definite answer as to what she might owe you, or how much it was ‘worth’, or even whether this was an owing type situation or a friendship type situation or a trying to impress her type situation.

We have turned the ‘owe you’ class into an explicit quantitative system with such thorough accounting, fine grained resolution and global buy-in that a person can live in prosperity by arranging to owe and to be owed the same sliver of an overseas business at slightly different evaluations, repeatedly, from their bed.

My guess is that this formalization causes a lot more activity to happen in the world, in this sphere, to access the vast value that can be created with the help of an elaborate rearrangement of owings.

People buy property and trucks and licenses to dig up rocks so that they can be owed nonspecific future goods thanks to some unknown strangers who they expect will want gravel someday, statistically. It’s harder to imagine this scale of industry in pursuit entirely of social status say, where such trust and respect would not soon cash out in money (e.g. via sales). For instance, if someone told you about their new gravel mine venture, which was making no money, but they expected it to grant oodles of respect, and therefore for people all around to grant everyone involved better treatment in conversations and negotiations, that would be prett...
How disagreements about Evidential Correlations could be settled
by Martín Soto — 2024-03-11T18:28:25.669Z

Since beliefs about Evidential Correlations don't track any direct ground truth, it's not obvious how to resolve disagreements about them, which is very relevant to acausal trade.
Here I present what seems like the only natural method (Third solution below).
Ideas partly generated with Johannes Treutlein.

Say two agents (algorithms A and B), who follow EDT, form a coalition. They are jointly deciding whether to pursue action a. Also, they would like an algorithm C to take action c. As part of their assessment of a, they’re trying to estimate how much evidence (their coalition taking) a would provide for C taking c. If it gave a lot of evidence, they'd have more reason to take a. But they disagree: A thinks the correlation is very strong, and B thinks it’s very weak.

This is exactly the situation in which researchers in acausal trade have many times found themselves: they are considering whether to take a slightly undesirable action a (spending a few resources on paperclips), which could provide evidence for another agent C (a paperclip-maximizing AI in another lightcone) taking an action c (the AI spending a few resources on human happiness) that we'd like to happen. But different researchers A and B (within the coalition of "humans trying to maximize human happiness") have different intuitions about the strength of the correlation.

A priori, there could exist the danger that, by thinking more, they would unexpectedly learn the actual output of C. This would make the trade no longer possible, since then taking a would give them no additional evidence about whether c happens. But, for simplicity, assume that C is so much more complex and chaotic than what A and B can compute, that they are very certain this won’t happen.

First solution: They could dodge the question by just looking for different actions to take they don't disagree on. But that’s boring.

Second solution: They could aggregate their numeric credences somehow. They could get fancy on how to do this. They could even get into more detail, and aggregate parts of their deliberation that are more detailed and informative than a mere number (and that are upstream of this probability), like different heuristics or reference-class estimates they've used to come up with them. They might face some credit assignment problems (which of my heuristics where most important in setting this probability?). This is not bo...