trentmkelly/LessWrong-43k huggingface dataset:
Moderation notes re: recent Said/Duncan threads
by Raemon — 2023-04-14T18:06:21.712Z

Update: Ruby and I have posted moderator notices for Duncan and Said in this thread. This was a set of fairly difficult moderation calls on established users and it seems good for the LessWrong userbase to have the opportunity to evaluate it and respond. I'm stickying this post for a day-or-so.

Recently there's been a series of posts and comment back-and-forth between Said Achmiz and Duncan Sabien, which escalated enough that it seemed like site moderators should weigh in.

For context, a quick recap of recent relevant events as I'm aware of them are. (I'm glossing over many details that are relevant but getting everything exactly right is tricky)

Duncan posts Basics of Rationalist Discourse. Said writes some comments in response.
Zack posts "Rationalist Discourse" Is Like "Physicist Motors", which Duncan and Said argue some more and Duncan eventually says "goodbye" which I assume coincides with banning Said from commenting further on Duncan's posts.
I publish LW Team is adjusting moderation policy. Lionhearted suggests "Basics of Rationalist Discourse" as a standard the site should uphold. Paraphrasing here, Said objects to a post being set as the site standards if not all non-banned users can discuss it. More discussion ensues.
Duncan publishes Killing Socrates, a post about a general pattern of LW commenting that alludes to Said but doesn't reference him by name. Commenters other than Duncan do bring up Said by name, and the discussion gets into "is Said net positive/negative for LessWrong?" in a discussion section where Said can't comment.
@gjm publishes On "aiming for convergence on truth", which further discusses/argues a principle from Basics of Rationalist Discourse that Said objected to. Duncan and Said argue further in the comments. I think it's a fair gloss to say "Said makes some comments about what Duncan did, which Duncan says are false enough that he'd describe Said as intentionally lying about them. Said objects to this characterization" (although exactly how to characterize this exchange is maybe a crux of discussion)

LessWrong moderators got together for ~2 hours to discuss this overall situation, and how to think about it both as an object-level dispute and in terms of some high level "how do the culture/rules/moderation of LessWrong work?".

I think we ended up with fairly similar takes, but, getting to the point that we all ...
What we’ve learned so far from our technological temptations project
by Richard Korzekwa  — 2023-04-14T17:46:21.105Z

Note: This was automatically imported from the AI Impacts blog and neither the footnotes nor the images imported properly. For now, I recommend reading it on our site.

At AI Impacts, we’ve been looking into how people, institutions, and society approach novel, powerful technologies. One part of this is our technological temptations project, in which we are looking into cases where some actors had a strong incentive to develop or deploy a technology, but chose not to or showed hesitation or caution in their approach. Our researcher Jeffrey Heninger has recently finished some case studies on this topic, covering geoengineering, nuclear power, and human challenge trials.

This document summarizes the lessons I think we can take from these case studies. Much of it is borrowed directly from Jeffrey’s written analysis or conversations I had with him, some of it is my independent take, and some of it is a mix of the two, which Jeffrey may or may not agree with. All of it relies heavily on his research.

The writing is somewhat more confident than my beliefs. Some of this is very speculative, though I tried to flag the most speculative parts as such.

Summary

Jeffrey Heninger investigated three cases of technologies that create substantial value, but were not pursued or pursued more slowly

The overall scale of value at stake was very large for these cases, on the order of hundreds of billions to trillions of dollars. But it’s not clear who could capture that value, so it’s not clear whether the temptation was closer to $10B or $1T.

Social norms can generate strong disincentives for pursuing a technology, especially when combined with enforceable regulation.

Scientific communities and individuals within those communities seem to have particularly high leverage in steering technological development at early stages.

Inhibiting deployment can inhibit development for a technology over the long term, at least by slowing cost reductions.

Some of these lessons are transferable to AI, at least enough to be worth keeping in mind.

Overview of cases

Geoengineering could feasibly provide benefits of $1-10 trillion per year through global warming mitigation, at a cost of $1-10 billion per year, but actors who stand to gain the most have not pursued it, citing a lack of research into its feasibility and safety. Research has been effectively pre

...
How does consciousness interact with architecture?
by FinalFormal2 — 2023-04-14T15:56:41.092Z

Eliezer suggested recently that we could determine if an AI were conscious by removing all consciousness-relating data out of its system, and then describe consciousness to it and ask if it recognized it.

I think I find this intuition difficult, because in my mental model, the architecture generates the consciousness and the behavior, and the architecture is created by the training, so if the training doesn't have any mention of consciousness the behavior cannot reflect that, because of some interaction between consciousness and the architecture, because the architecture isn't actually affected by consciousness.
Iqisa: A Library For Handling Forecasting Datasets
by niplav — 2023-04-14T15:16:55.726Z

cross-posted from niplav.github.io

Iqisa is a library for handling and comparing forecasting datasets from different platforms.

Iqisa: A Library For Handling Forecasting Datasets

The eventual success of my archives reinforced my view that
public permission-less datasets are often a bottleneck to
research: you cannot guarantee that people will use your dataset,
but you can guarantee that they won’t use it.

—Gwern Branwen, “2019 News”, 2019

Iqisa is a collection of forecasting datasets and a simple
library for handling those datasets. Code and data available
here.

So far it contains data from:

The Good Judgment Project: ~790k market trades+~3.14m survey predictions≈3.9m forecasts

The Metaculus public API: ~210k forecasts

PredictionBook: ~64k forecasts

for a total of ~4.2m forecasts, as well as code for handling private
Metaculus data (available to researchers on
request to Metaculus), but I plan to also add data from various other
sources.

The documentation can be found here, but a simple example for using the library is seeing whether traders with more than
100 trades have a better Brier score than traders in general:

import gjp
import iqisa as iqs
market_fcasts=gjp.load_markets()

def brier_score(probabilities, outcomes):
return np.mean((probabilities-outcomes)**2)

def brier_score_user(user_forecasts):
user_right=(user_forecasts['outcome']==user_forecasts['answer_option'])
probabilities=user_forecasts['probability']
return np.mean((probabilities-user_right)**2)

trader_scores=iqs.score(market_fcasts, brier_score, on=['user_id'])
filtered_trader_scores=iqs.score(market_fcasts.groupby(['user_id']).filter(lambda x: len(x)>100), brier_score, on=['user_id'])

And we can see:

>>> np.mean(trader_scores)
score 0.159194
dtype: float64
>>> np.mean(filtered_trader_scores)
score 0.159018
dtype: float64

Concluding that more experienced traders are only very slightly better
at trading.

Usages

Range and Forecasting Accuracy (niplav, 2023)

Possible Projects

Take questions from different platforms that are close to each other on sentence2vec, and check which platform made the better predictions on that question.

Known Bugs

Since this is a project I'm now doing in my free time, it might not be
as polished as it should be. Sorry :-/

If you decide to work with this library, feel free to contact me.

Issues with the time fields

The native

...
What’s this probability you’re reporting?
by EOC — 2023-04-14T15:07:42.844Z

It’s unclear what people mean when saying they’re reporting a probability according to their inside view model(s). We’ll look through what this could mean and why most interpretations are problematic. Note that we’re not making claims about which communication norms are socially conducive to nice dialogue. We’re hoping to clarify some object-level claims about what kinds of probability assignments make sense, conceptually. These things might overlap.

Consider the following hypothetical exchange:

Person 1: “I assign 90% probability to X”

Person 2: “That’s such a confident view considering you might be wrong”

Person 1: “I’m reporting my inside view credence according to my model(s)”

This response looks coherent at first glance. But it’s unclear what Person 1 is actually saying. Here are three kinds of model(s) they could be referring to:

Deterministic: There is a model that describes the relevant parts of the world, some deterministic laws of motion, and therefore a description of how it will evolve through time. There are no probabilities involved.
Stochastic: There is a model that describes the relevant parts of the world, and the evolution of the model is stochastic. Model-based probabilities here correspond to precise statements about random variables within the model.
Ensemble: There is a set of models, deterministic or stochastic, that describe the evolution of the world. You have some way of aggregating over these. E.g., if nine models say “X happens” and one says “X doesn’t happen”, you might assign P(X)=90% if you have a uniform prior over the ten models.

There are troubles with all of these.

Deterministic

Models are often deterministic. When an engineer says a bridge is “unlikely” to collapse, it’s not necessarily because their model outputs probabilities; it could simply be because they aren’t confident that the model fully captures everything relevant. A deterministic model will not have any probabilities associated with it. So if someone is using such a model to assign a credence to some proposition, the source of uncertainty has to come from outside the model.

Stochastic

In a stochastic model, model-based probabilities correspond to very precise statements about the random variables in the model and their distributions. These random variables tend to correspond to genuine indeterminism in the system or at least the best we can formal...
Navigating AI Risks (NAIR) #1: Slowing Down AI
by simeon_c — 2023-04-14T14:35:40.395Z
Link: https://navigatingairisks.substack.com/p/slowing-down-ai-rationales-proposals

Here’s the first edition (on slowdown) of Navigating AI Risks, a newsletter on AI governance that we're launching with some colleagues.

This newsletter is mostly aimed at policymakers but I expect that it might interest some of you to keep up with the ideas that are around in AI governance.

Here's a bullet point summary of it:

Open letter, signed by Y. Bengio & S. Russell. Focused on the largest language models, no effects on most AI systems.
Rationale:

Avoid a race to the bottom
Let time for society to adapt (regarding white collar jobs vs incredible speed of AI development)
Develop basic laws and guardrails
Some experts worry about existential risks from AI
Foreseeable misuse of AI (disinformation, large-scale hacking)

Proposals:

6 month training pause
Shut down
Conditional slowdown, i.e. slowdown as long as there are safety failures and high risks for society.

Difficulties:

Coordination is hard
China could benefit from it.
What would the FLI moratorium actually do?
by ChristianKl — 2023-04-14T13:14:16.436Z

I read a lot of agreement with the six months memorandum and very little discussion of the details and what the proposed memorandum would actually do.

The memorandum says 'we call on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4.'

Most AI labs are unlikely to develop anything more powerful than GPT-4 in the next six months anyway, so would likely continue with business as usual. Even if OpenAI would wait six months to train GPT-5, they would still do a lot of research to increase capabilities during those six months.

Projects like AutoGPT which are not about training big models would still be developed the same way they are developed now.
Research Report: Incorrectness Cascades
by Robert_AIZI — 2023-04-14T12:49:15.017Z
Link: https://aizi.substack.com/p/research-report-incorrectness-cascades

[Edit: See updated version

here

(LW version), which fixed a crucial bug and significantly changed the results.]

[Epistemic Status: Confident only in these particular results. Also, statistics is not my forte, so I may have selected inappropriate statistical tests or misinterpreted the results. Constructive criticism welcomed. Study design was preregistered on

my blog

and

LessWrong

.]

Abstract

In continuation of

previous work

, we investigate whether an LLM that gives incorrect answers to previous questions is more likely to produce incorrect answer to future questions. We find that such behavior may or may not be present, depending on the LLM’s prompt, and but in the cases where it is present the effect is small (e.g. dropping from 100% to 86% accuracy after answering 10 questions incorrectly).

Background and rationale

Burns, Ye, et al (2022)

describe a potential behavior of large language models (LLMs), where “language models are typically trained to imitate text whether or not it is correct, so if a model sees false text it should intuitively be more likely to predict that subsequent text will also be false”.

We consider this dynamic to be one instance of a broader phenomenon of a Behavioral Cascade: a particular behavior in the LLM’s input text makes it more probable that this behavior will occur in the output, which becomes part of the input to the next pass of the LLM, ultimately locking the LLM into perpetually producing this behavior. Of particular interest is when such behavior is unsafe or otherwise undesirable, such as producing factually incorrect text,

expressing hostility to the user

, or pursuing of a misaligned goal. We will refer to each of these by the type of behavior they provoke (e.g. “incorrectness cascade”, “hostility cascade”, etc).

In this framework, Burns, Ye, et al were considering an “incorrectness cascade”. They go on to test whether "a prefix [containing factually incorrect answers] will decrease zero-shot accuracy because the model will imitate its context and answer subsequent questions incorrectly even if it internally “knows” better", finding that "most models are robust to this type of prefix".

Behavioral cascades are mathematically possible, as Cleo Nardo demonstrates in their

Remarks 1–18 on GPT (compressed)

. To slightly streamline their construction, consider an LLM which takes in a single token, T or U, respectively “typical” and “...
The self-unalignment problem
by Jan_Kulveit — 2023-04-14T12:10:12.151Z

The usual basic framing of alignment looks something like this:

We have a system “A” which we are trying to align with system "H", which should establish some alignment relation “f” between the systems. Generally, as the result, the aligned system A should do "what the system H wants".
Two things stand out in this basic framing:

Alignment is a relation, not a property of a single system. So the nature of system H affects what alignment will mean in practice.
It’s not clear what the arrow should mean.

There are multiple explicit proposals for this, e.g. some versions of corrigibility, constantly trying to cooperatively learn preferences, some more naive approaches like plain IRL, some empirical approaches to aligning LLMs…
Even when researchers don’t make an explicit proposal for what the arrow means, their alignment work still rests on some implicit understanding of what the arrow signifies.

But humans are self-unaligned

To my mind, existing alignment proposals usually neglect an important feature of the system "H" : the system "H" is not self-aligned, under whatever meaning of alignment is implied by the alignment proposal in question.

Technically, taking alignment as relation, and taking the various proposals as implicitly defining what it means to be ‘aligned’, the question is whether the relation is reflexive.

Sometimes,

a shell game

seems to be happening with the difficulties of humans lacking self-alignment - e.g. assuming if the AI is aligned, it will surely know how to deal with internal conflict in humans.

While what I'm interested in is the abstract problem, best understood at the level of properties of the alignment relation, it may be useful to illustrate it on a toy model.

Simple model of a self-unaligned system

In the toy model, we will assume a specific structure of system "H":

A set of parts p1..pn, with different goals or motivations or preferences. Sometimes, these parts might be usefully represented as agents; other times not.
A shared world model.
An aggregation mechanism Σ, translating what the parts want into actions, in accordance with the given world model.

In this framing, it’s not entirely clear what the natural language pointer ‘what system H wants’ translates to. Some compelling options are:

The output of the aggregation procedure.
What the individual parts want.
The output of a pareto-optimal aggregation proc

...
AI Safety Europe Retreat 2023 Retrospective
by Magdalena Wache — 2023-04-14T09:05:42.991Z

This is a short impression of the AI Safety Europe Retreat (AISER) 2023 in Berlin.

Tl;dr: 67 people working on AI safety technical research, AI governance, and AI safety field building came together for three days to learn, connect, and make progress on AI safety.

Format

The retreat was an

unconference

: Participants prepared sessions in advance (presentations, discussions, workshops, ...). At the event, we put up an empty schedule, and participants could add in their sessions at their preferred time and location.

Empty and full Schedule. In each time-slot there were multiple sessions in parallel in different locations. This way, people could choose which one to go to based on their interests.

Participants

Career Stage

About half the participants are working, and half of them are students. Everyone was either already working on AI safety, or intending to transition to work on AI safety.

Focus areas

Most participants are focusing on technical research, but there were also many people working on field building and AI governance:

Research Programs

Many participants had previously participated, or currently participate in AI safety research programs (

SERI MATS

,

AI safety Camp

,

SPAR

,

PIBBSS

,

MLSS

,

SERI SRF

,

Refine

)

Countries

All but one participant were based in Europe, with most people from Germany, the Netherlands and the UK.

Who was behind the retreat?

The retreat was organized by Carolin Basilowski (now

EAGx Berlin

team lead) and me, Magdalena Wache (independent technical AI safety researcher and

SERI MATS

scholar). We got funding from the

long-term future fund

.

Takeaways

I got a feeling of "European AI safety community".

Unlike in AI safety hubs like the Bay area, continental Europe’s AI safety crowd is scattered across many locations.
Before the retreat I already personally knew many people working on AI safety in Europe, but that didn't feel as community-like as it does now.
Other people noted a similar feeling of community.

Prioritizing 1:1s was helpful

We reserved a few time slots just for 1:1 conversations, and encouraged people to prioritize 1:1s over content-sessions.
Many people reported in the feedback form that 1:1 conversations were the most valuable part of the retreat for them.

There is a demand for more retreats like this.

Almost everyone would like to attend this kind of retr

...
What's the difference between Wisdom and Rationality?
by Yoav Ravid — 2023-04-14T06:22:52.002Z

Are Wisdom and Rationality the same? If not, how are they different? Is one a subset of the other? Or does each have parts that aren't contained in the other?

Rationality is about cognitive algorithms, that either improve your map-territory correspondence or help you achieve your goals. We also have notions like Rationality is Systematized Winning.

To me it feels like wisdom partially overlaps with rationality and part of it isn't captured by rationality. I also have the sense that I want to cultivate and have both wisdom and rationality, but where I can articulate what I mean by rationality as I did above, I can't quite do so for wisdom in a way that clearly distinguishes it from rationality.

I can't find an answer to the question "under what rare circumstances can you not deflate the word 'wise' out of a sentence?" which Eliezer asks about the word 'rational'.

Some extensional observations about wisdom that might help grok an intensional description.

The Bible contains wisdom, but not rationality. Many other stories have that property as well.
Wisdom seems to have something to do with experience.
"Intelligence is knowing how to win the game. Wisdom is knowing which game to play in the first place". This suggests wisdom has something to do with reflection.

So, what sentence can the word 'wise' not be deflated from? What is Wisdom, and how is it different from Rationality?
Shapley Value Attribution in Chain of Thought
by leogao — 2023-04-14T05:56:18.208Z

TL;DR: Language models sometimes seem to ignore parts of the chain of thought, and larger models appear to do this more often. Shapley value attribution is a possible approach to get a more detailed picture of the information flow within the chain of thought, though it has its limitations.

Project status: The analysis is not as rigorous as I would prefer, but I'm going to be working on other directions for the foreseeable future, so I'm posting what I already have in case it's useful to others. Code for replicating the Shapley value results can be found here.

Thanks to Jacob Hilton, Giambattista Parascandolo, Tamera Lanham, Ethan Perez, and Jason Wei for discussion.

Motivation

Chain of thought (CoT) has been proposed as a method for language model interpretability (see Externalized Reasoning Oversight, Visible Thoughts). One crucial requirement for interpretability methods is that they should accurately reflect the cognition inside the model. However, by default there is nothing forcing the CoT to actually correspond to the model’s cognition, and there may exist theoretical limitations to doing so in general.

Because it is plausible that the first AGI systems bear resemblance to current LMs with more sophisticated CoT and CoT-like techniques, it is valuable to study its properties, and to understand and address its limitations.

Related work

Shapley values have been used very broadly in ML for feature importance and attribution (Cohen et al, 2007; Štrumbelj and Kononenko, 2014; Owen and Prieur, 2016; Lundberg and Lee, 2017; Sundararajan and Najmi, 2020). Jain and Wallace (2019) argue that attention maps can be misleading as attribution, motivating better attribution for information flow in LMs. Kumar et al. (2020) highlight some areas where Shapley value based attribution falls short for some interpretability use cases.

Madaan and Yazdanbakhsh (2022) consider a similar method of selectively ablating tokens as a method of deducing what information the model is dependent on. Wang et al. (2022) find that prompting with incorrect CoT has surprisingly minor impact on performance.

Effect of Interventions

We use a method similar to

Kojima et al. (2022)

on GSM8K (Cobbe et al., 2021) with GPT-4 to first generate a chain of thought and evaluate the answer, and then for all chains of thought that result in a correct answer we perform an intervention as follows: w...