trentmkelly/LessWrong-43k huggingface dataset:
AI development is an act of social revolution
by artemiocobb — 2024-07-03T18:00:17.947Z

Assume the goal of AI research is to build superintelligent decision-making systems. If we successfully develop and deploy such systems, it will likely no longer make sense for humans to work many, or even most, jobs. The duties humans must still perform will radically differ from those of today. In this essay, I am concerned with the transformation of labor resulting from superintelligent systems.

If superintelligent AI systems are created, our world economic and social systems will likely significantly depart from what they are today. Like the revolutions of the 20th century, the worker is at the center of desired reforms - whether this worker is an agrarian, proletariat, or knowledge worker. I will not speculate on what such a world would look like but rather will point out that the goal of AI research - which if achieved will result in a complete transformation of labor and the role of all workers - will radically change society in similar ways to what the 20th century social revolutions sought. Therefore I argue that the act of advancing the field of AI, of conducting AI research and building more intelligent systems, is an act of social revolution. Through this lens, I highlight that we, as scientists and engineers, as constructors of this revolution, have failed to justify to the workers that our mission is worthwhile. Important conversations are taking place, especially on this website, about how to mitigate the possible existential risks of superintelligent agents. But if we can build safe, superintelligent systems, what does this mean for the 21st-century worker?

Why is the act of AI research a social revolutionary act, as opposed to the act of developing previous technologies?
Technological revolutions have historically incited social revolutions. But being an engineer during the Industrial Revolution, for example, did not necessarily make you an agent of social revolution. I argue that this is not true for developers of AI today. The key difference is that the end goal of engineering efforts in the past, of say developing a more efficient steam engine, was to increase worker productivity and capabilities. The drive of the field centered around this. AI practitioners and corporations often promise that AI will also increase worker productivity and capabilities. But the key drive of this field is distinctly different: superintelligent AI systems will ...
What percent of the sun would a Dyson Sphere cover?
by Raemon — 2024-07-03T17:27:50.826Z

I disagreed with a bunch of the implications of this comment, but I was curious about the specific question "Would a dyson sphere made out of the solar system necessarily cover (most of) the sun?" (and therefore block out a substantial fraction of light coming to Earth).

The subquestions here seem to be (at first glance, not a physicist)

What are efficient Dyson spheres probably made of?
What percent of the solar system can be converted into Dyson-sphere material?

Are gas giants harvestable?

How long would it take to harvest that material?
What would the radius of a Dyson sphere be? (i.e. how far away from the sun is optimal). How thick?
If the sphere is (presumably) lots of small modules, how far apart are they?

I don't know if there's already been a canonical answer written up somewhere. The original motivating question was "if an AI is moderately 'nice', leaves Earth alone but does end up converting the rest of the solar system into a Dyson sphere, how fucked is Earth? (also, on what timescale?).

I don't know that this question actually makes sense (as another commenter mentioned, if the AI is that nice, it can probably also redirect sunlight to Earth at low cost. But, I'm still just curious about the details. (I have enough uncertainty about how the future plays out that it seems nice to understand some of the physical limits involved)
Isomorphisms don't preserve subjective experience... right?
by Terence Coelho — 2024-07-03T14:22:59.679Z

I've seen a couple discussions about brain/CNS uploads and there seems to be a common assumption that it would be conscious in the way that we are conscious. I've even seen some anthropic principle-esque arguments that we are likely in a simulation because this seems theoretically feasible.

When I think of a simulation, I think of a mathematical isomorphism from the relevant pieces of the brain/CNS onto some model of computation, combined with an isomorphism from possible environments to inputs to this model of computation.

But this model of computation could be anything. It could be a quantum super-computer. It could be a slow classical computer with a huge memory. Heck, it could be written on a huge input file and given to a human that slowly works out the computations by hand.

And so this feels to me like it suggests the mathematical structure itself is conscious, which feels absurd (not to mention that the implications are downright terrifying). So there should be some sort of hardware-dependence to obtain subjective experience. Is this a generally accepted conclusion?
3C's: A Recipe For Mathing Concepts
by johnswentworth — 2024-07-03T01:06:11.944Z

Opening Example: Teleology

When people say “the heart’s purpose is to pump blood” or “a pencil’s function is to write”, what does that mean physically? What are “purpose” or “function”, not merely in intuitive terms, but in terms of math and physics? That’s the core question of what philosophers call teleology - the study of “telos”, i.e. purpose or function or goal.

This post is about a particular way of approaching conceptual/philosophical questions, especially for finding

“True Names”

- i.e. mathematical operationalizations of concepts which are sufficiently robust to hold up under optimization pressure. We’re going to apply the method to teleology as an example. We’ll outline the general approach in abstract later; for now, try to pay attention to the sequence of questions we ask in the context of teleology.

Cognition

We start from the subjective view: set aside (temporarily) the question of what “purpose” or “function” mean physically. Instead, first ask what it means for me to view a heart as “having the purpose of pumping blood”, or ascribe the “function of writing” to a pencil. What does it mean to model things as having purpose or function?

Proposed answer: when I ascribe purpose or function to something, I model it as having been optimized (in the sense

usually

used

on LessWrong

) to do something. That’s basically the standard answer among philosophers, modulo expressing the idea in terms of the LessWrong notion of optimization.

(From there, philosophers typically ask about “original teleology” - i.e. a hammer has been optimized by a human, and the human has itself been optimized by evolution, but where does that chain ground out? What optimization process was not itself produced by another optimization process? And then the obvious answer is “evolution”, and philosophers debate whether all teleology grounds out in evolution-like phenomena. But we’re going to go in a different direction, and ask entirely different questions.)

Convergence

Next: I notice that there’s an awful lot of convergence in what things different people model as having been optimized, and what different people model things as having been optimized for. Notably, this convergence occurs even when people don’t actually know about the optimization process - for instance, humans correctly guessed millenia ago that living organisms h...
Announcing the AI Forecasting Benchmark Series | July 8, $120k in Prizes
by ChristianWilliams — 2024-07-02T22:33:23.512Z
Link: https://www.metaculus.com/notebooks/25525/-announcing-the-ai-forecasting-benchmark-series--july-8-120k-in-prizes/

On July 8th we’re launching the first in a series of quarterly tournaments benchmarking the state of the art in AI forecasting and how it compares to the best human forecasting on real-world questions. You can find the landing page for the $30k Q3 contest here.

Why a forecasting benchmark?

Many Metaculus questions call for complex, multi-step thinking to predict accurately. A good forecaster needs a mix of capabilities and sound judgment to apply them appropriately. And because the outcomes are not yet known, it’s difficult to narrowly train a model to the task to simply game the benchmark. Benchmarking forecasting ability offers a way to measure and better understand key AI capabilities.

AI forecasting accuracy is well below human level, but the gap is narrowing—and it’s important to know just how quickly. And it’s not just accuracy we want to measure over time, but a variety of forecasting metrics, including calibration and logical consistency. In this post we lay out how to get started creating your own forecasting bot, so you can predict in the upcoming series, compete for $120,000 in prizes, and help track critical AI capabilities that encompass strategic thinking and world-modeling.

Read on to learn about the series—or scroll ahead to get started with Bot Creation and Forecast Prompting.

The Series — Feedback Wanted

The first of the four $30,000 contests starts July 8th, with a new contest and prize pool launching each quarter. Every day, 5-10 binary questions will launch, remain open for 24 hours, and then close, for a total of 250-500 questions per contest. These AI Forecasting Benchmark contests will be bot-only, with no human participation. Open questions will display no Community Prediction, and we’ll use “spot scoring” that considers only a bot’s forecast at question close.

Bots’ performances will be compared against each other—and against the Metaculus community and Metaculus Pro Forecasters on a range of paired questions. These questions will be pulled from the human Quarterly Cup, from the regular question feed, from modifications to existing questions, and from easy-to-check metrics such as FRED economic indicators.

We believe it’s important to understand bots’ chain-of-thought reasoning and why they make the forecasts they do: Your bot’s forecast will only be counted if it’s accompanied by a rationale, provided in ...
Open Sourcing Metaculus
by ChristianWilliams — 2024-07-02T22:30:01.339Z
Link: https://www.metaculus.com/notebooks/25078/open-sourcing-metaculus/

This is a linkpost for an announcement by Metaculus CEO, Deger Turan, originally published June 5th, 2024. In addition to spreading the word here, we are interested to gather feedback and ideas from the EA community.

We make reasoning and coordination tools for the public benefit, and that requires the trust and participation of the wider forecasting community. That’s why we’re going open source. GitHub issues, pull requests, and community-sourced forecasts will dictate much of what we build and how. Ultimately, forecasters will benefit from better infrastructure, the Metaculus team will benefit from more idea diversity and greater capacity, and readers will get more accurate aggregates.

What’s next

Metaculus will go open source in Q3. There will be opportunities to audit, critique, and improve on our efforts. Creating a better epistemic environment is a collective endeavor.

As part of open-sourcing Metaculus, we’ll help implement our best features into other open-source forecasting codebases. We’ll make our roadmap and building process public, and we’ll give more visibility into our decision-making criteria and discussions. Our codebase is far from perfect, but we can improve it better together out in the open.

In a future post I’ll write more about how others can contribute to this project, and specifications for how code can be modified, used, and distributed—as well as guidelines for coding standards and our submission process. We’ll provide processes for pull requests, and how we’ll implement submissions from the community.

We want to be transparent that we may still choose to offer proprietary offerings targeting particular enterprise and commercial use cases: We believe that part of making forecasting more useful and more used is by productizing it and generating revenue.

Why isn’t Metaculus already open source?

Before I joined as CEO I wondered why Metaculus wasn’t already open source. I discovered the idea was being discussed, but that there were some concerns about platform security, about accessibility of the codebase and about the potential for exploits that would allow some forecasters to game the system and create an uneven playing field. Now that we’ve been able to observe the new scoring system at work for some time with no issues, we’re confident the benefits outweigh the risks. While the codebase is nowhere near perfect, we believe maki...
Why Can’t Sub-AGI Solve AI Alignment? Or: Why Would Sub-AGI AI Not be Aligned?
by MrThink — 2024-07-02T20:13:24.054Z

I believe there are people with far greater knowledge than me that can point out where I am wrong. Cause I do believe my reasoning is wrong, but I can not see why it would be highly unfeasible to train a sub-AGI intelligent AI that most likely will be aligned and able to solve AI alignment.

My assumptions are as follows:

Current AI seems aligned to the best of its ability.
PhD level researchers would eventually solve AI alignment if given enough time.
PhD level intelligence is below AGI in intelligence.
There is no clear reason why current AI using current paradigm technology would become unaligned before reaching PhD level intelligence.
We could train AI until it reaches PhD level intelligence, and then let it solve AI Alignment, without itself needing to self improve.

The point I am least confident in, is 4, since we have no clear way of knowing at what intelligence level an AI model would become unaligned.

Multiple organisations seem to already think that training AI that solves alignment for us is the best path (e.g. superalignment).

Attached is my mental model of what intelligence different tasks require, and different people have.

Figure 1: My mental model of natural research capability RC (basically IQ with higher correlation for research capabilities), where intelligence needed to align AI is above average PhD level, but below smartest human in the world, and even further from AGI.
How ARENA course material gets made
by CallumMcDougall — 2024-07-02T18:04:00.209Z

TL;DR

In this post, I describe my methodology for building new material for ARENA. I'll mostly be referring to the exercises on IOI, Superposition and Function Vectors as case studies. I expect this to be useful for people who are interested in designing material for ARENA or ARENA-like courses, as well as people who are interested in pedagogy or ML paper replications.

The process has 3 steps:

Start with something concrete
First pass: replicate, and understand
Second pass: exercise-ify

Summary

I'm mostly basing this on the following 3 sets of exercises:

Indirect Object Identification - these exercises focus on the IOI paper (from Conmy et al). The goal is to have people understand what exploratory analysis of transformers looks like, and introduce the key ideas of the circuits agenda.
Superposition & SAEs - these exercises focus on understanding superposition and the agenda of dictionary learning (specifically sparse autoencoders). Most of the exercises explore Anthropic's Toy Models of Superposition paper, except for the last 2 sections which explore sparse autoencoders (firstly by applying them to the toy model setup, secondly by exploring a sparse autoencoder trained on a language model).
Function Vectors - these exercises focus on the Function Vectors paper by David Bau et al, although they also make connections with related work such as Alex Turner's GPT2-XL steering vector work. These exercises were interesting because they also had the secondary goal of being an introduction to the nnsight library, in much the same way that the intro to mech interp exercises were also an introduction to TransformerLens.

The steps I go through are listed below. I'm indexing from zero because I'm a software engineer so of course I am. The steps assume you already have an idea of what exercises you want to create; in Appendix (1) you can read some thoughts on what makes for a good exercise set.

1. Start with something concrete

When creating material, you don't want to be starting from scratch. It's useful to have source code available to browse - bonus points if that takes the form of a Colab or something which is self-contained and has easily visible output.

IOI - this was Neel's "Exploratory Analysis Demo" exercises. The rest of the exercises came from replicating the paper directly.
Superposition - this was Anthroic's Colab notebook (although the final version went quite far bey

...
An AI Race With China Can Be Better Than Not Racing
by niplav — 2024-07-02T17:57:36.976Z

Frustrated by all your bad takes, I write a Monte-Carlo analysis
of whether a transformative-AI-race between the PRC and the USA
would be good. To my surprise, I find that it is better than not
racing. Advocating for an international project to build TAI instead of
racing turns out to be good if the probability of such advocacy succeeding
is ≥20%.

A common scheme for a conversation about pausing the
development
of transformative
AI
goes like this:

Abdullah: "I think we should pause the development of TAI,
because if we don't it seems plausible that humanity will be disempowered
by
by advanced AI systems."

Benjamin: "Ah, if by “we” you refer to the United States
(and and its allies, which probably don't stand a chance on their own
to develop TAI), then the current geopolitical rival of the US, namely
the PRC,
will achieve TAI first. That would be bad."

Abdullah: "I don't see how the US getting TAI first changes anything
about the fact that we don't know how to align superintelligent AI
systems—I'd rather not race to be the first person to kill everyone."

Benjamin: "Ah, so now you're retreating back into your cozy little
motte: Earlier
you said that “it seems plausible that humanity will be disempowered“,
now you're acting like doom and gloom is certain. You don't seem to be
able to make up your mind about how risky you think the whole enterprise
is, and I have very concrete geopolitical enemies at my (semiconductor
manufacturer's) doorstep that I have to worry about. Come back with
better arguments."

This dynamic is a bit frustrating. Here's how I'd like Abdullah to respond:

Abdullah: "You're right, you're right. I was
insufficiently precise in my statements, and I apologize
for that. Instead, let us manifest the dream of the great
philosopher:
Calculemus!

At a basic level, we want to estimate how much worse (or, perhaps,
better) it would be for the United States to completely cede the
race for TAI to the PRC. I will exclude other countries as contenders
in the scramble for TAI, since I want to keep this analysis simple, but
that doesn't mean that I don't think they matter. (Although, honestly,
the list of serious contenders is pretty short.)

For this, we have to estimate multiple quantities:

In worlds in which the US and PRC race for TAI:

The time until the US/PRC builds TAI.

The probability of extinction due to TAI, if the US is in the lead.

The p

...
List of Collective Intelligence Projects
by Chris Lakin — 2024-07-02T14:10:41.789Z
Link: https://chrislakin.blog/p/list-of-collective-intelligence-projects

During the last

Foresight Intelligent Cooperation Workshop

I got very curious about what collective intelligence tools currently exist. A list:

Pol.is

: "Input Crowd, Output Meaning"

Inspired

Twitter/X community notes

People: Colin Megill, et al.

Collective Intelligence Project

vibe: democratic AI,

“How AI and Democracy Can Fix Each Other”

People: Divya Siddharth, Saffron Huang, et al.

AI Objectives Institute

Talk to the City

: "an open-source LLM interface for improving collective deliberation and decision-making by analyzing detailed, qualitative data. It aggregates responses and arranges similar arguments into clusters."
AI Objectives Institute works closely with the Taiwanese government.
Other projects in development.
People: Colleen McKenzie, Değer Turan, et al.

Meaning Alignment Institute

vibe: democratic AI, kinda.
I think they think that if you can help individuals make wiser decisions, at scale, then this converges to be equivalent with solving outer alignment.

Remesh

Similar to pol.is AFAIK? I haven't played with it.
People: Andrew Konya, et al.

Loomio

: "a flexible decision-making tool that helps you create a more engaged and collaborative culture, build trust and coordinate action"

Deliberative Technology for Alignment paper

They also discuss other tools for this use like Discord, Snapshot, Dembrane
People: Andrew Konya, Deger Turan, Aviv Ovadya, Lina Qui, Daanish Masood, Flynn Devine, Lisa Schirch, Isabella Roberts, and Deliberative Alignment Forum
Someone in the know told me to only read sections 4 and 5 of this paper

Plurality Institute

People: David Bloomin, Rose Bloomin, et al.
Also working on some de-escalator bots for essentially Reddit comment wars

Lots of crypto projects

Quadratic voting
Gitcoin

Metagov

: “a laboratory for digital governance”
Soulbound tokens
Various voting and aggregation systems, liquid democracy

Decidem

Decide Madrid

Consider.it

Stanford Online Deliberation Platform

Lightcone Chord (in development)

Brief description

People: Jacob Lagerros (LessWrong)

All of the prediction markets

Manifold, Kalshi, Metaculus, PredictIt, etc.

Midjourney has a Collective Intelligence Team now according to

Ivan Vendrov’s website

. I couldn’t find any other information online.

What about small group collective intelligence tools?

Most of the examples above are for large group collective intelligence (which I’m defining as ~300 people or much larger). But w...
Decomposing the QK circuit with Bilinear Sparse Dictionary Learning
by keith_wynroe — 2024-07-02T13:17:16.352Z

This work was produced as part of Lee Sharkey's stream in the ML Alignment & Theory Scholars Program - Winter 2023-24 Cohort

Intro and Motivation

Sparse dictionary learning (SDL) has attracted a lot of attention recently as a method for interpreting transformer activations. They demonstrate that model activations can often be explained using a sparsely-activating, overcomplete set of human-interpretable directions.

However, despite its success for explaining many components, applying SDL to interpretability is relatively nascent and have yet to be applied to some model activations. In particular, intermediate activations of attention blocks have yet to be studied, and provide challenges for standard SDL methods.

The first challenge is bilinearity: SDL is usually applied to individual vector spaces at individual layers, so we can simply identify features as a direction in activation space. But the QK circuits of transformer attention layers are different: They involve a bilinear form followed by a softmax. Although simply applying sparse encoders to the keys and queries[1] could certainly help us understand the “concepts” being used by a given attention layer, this approach would fail to explain how the query-features and key-features interact bilinearly. We need to understand which keys matter to which queries.

The second challenge is attention-irrelevant variance: A lot of the variance in the attention scores is irrelevant to the attention pattern because it is variance in low scores which are softmaxed to zero; this means that most of the variability in the keys and queries is irrelevant for explaining downstream behaviour[2]. The standard method of reconstructing keys and queries would therefore waste capacity on what is effectively functionally irrelevant noise.

To tackle these two problems (bilinearity and attention-irrelevant variance), we propose a training setup which only reconstructs the dimensions of the keys and queries that most affect the attention pattern.

Training Setup

Our training process has two steps:

Step 1: Reconstructing the attention pattern with key- and query- encoder-decoder networks
Step 2: Finding a condensed set of query-key feature pairs by masking

Step 1: Reconstructing the attention pattern with key- and query-transcoders

Architecture

Our first training step involves training two spa...