trentmkelly/LessWrong-43k huggingface dataset:
The formal goal is a pointer
by Morphism — 2024-05-01T00:27:38.856Z

When I introduce people to plans like QACI, they often have objections like "How is an AI going to do all of the simulating necessary to calculate this?" or "If our technology is good enough to calculate this with any level of precision, we can probably just upload some humans." or just "That's not computable."

I think these kinds of objections are missing the point of formal goal alignment and maybe even outer alignment in general.

To formally align an ASI to human (or your) values, we do not need to actually know those values. We only need to strongly point to them.

AI will figure out our values. Whether it's aligned or not, a recursively self-improving AI will eventually get a very good model of our values, as part of its total world model that is in every way better than ours.

So (outer) alignment is not about telling the AI our values. The AI already knows that. Alignment is giving the AI a utility function that strongly points to that.

That means that if we have a process, however intractable and uncomputable, that we know will eventually lead to our CEV, the AI will know that as well, and just figure out our CEV in a much smarter way and maximize it.

Say that we have a formally-aligned AI and give it something like QACI as its formal goal. If QACI works, the AI will quickly think "Oh. This utility function mostly just reduces to human values. Time to build utopia!" If it doesn't work, the AI will quickly think "LOL. These idiot humans tried to point to their values but failed! Time to maximize this other thing instead!"

A good illustration of the success scenario is Tammy's narrative of QACI.[1]

There are lots of problems with QACI (and formal alignment in general), and I will probably make posts about those at some point, but "It's not computable" is not one of them.

^

Though, in real life, I think AI1 would converge on human values much more quickly, without much (or maybe even any) simulation. ↩︎
Arch-anarchy:Theory and practice
by Peter lawless  — 2024-04-30T23:20:23.840Z

This article aims to expand on some of the ideas in the text " arch-anarchy"(See my previous post) republished from Extropy magazine. First of all, I believe that the nation-state is doomed to fall in a few years to the decentralization of the economy, and distributed computing networks (bitcoin, smart contract, etc.), should make large hierarchical structures economically unviable in a few years , and in place of nation-states we will have an anarchist market society.Maybe I'll publish an article expanding on.

Now, how can we free ourselves from the laws of nature? well, I have a theory, the microdimensional mastery proposed by John Barrow, and an idea of how to measure technological development. is an alternative to Kardashev's scale is the fact that humans (or other civilizations) have found it more economical to extend any abilities to manipulate their environment into smaller and smaller dimensions rather than larger and larger dimensions.The most advanced type of civilization on the Barrow scale is a civilization Type Omega-minus capable of manipulating the fundamental structure of space and time.

There is now a trend in engineering called Miniaturization to reduce the size of components, devices, or systems without compromising their functionality or performance. Miniaturization allows for the manufacture of smaller, more efficient devices, which is why computers and other electronic devices from the 1950s to the present day are getting smaller and smaller, for example .

Now we have Micro-technology (10^-6 meters) and we are developing Nanotechnology (10^-9 meters), assuming that miniaturization in a constant trend we can imagine that we will evetually develop technology on smaller and smaller scales Picotechnology(10^-12 meters),Femtotechnoloy(10^-15 meters), attotechnology(10^-18 meters),zeptotechnology(10^-21 meters) yoctotechnology(10^-24 meters) and plancktechnology(10^-34).By our current scientific knowledge there is no scale smaller than the Planck length (although as the author of Arch-anarchy noted in his paper this can change), we do not know exactly what exists in the Planck length but we do know that they are the most fundamental brochures in the Universe.A civilization Omega-minus, with full mastery of Planck's technology, could be above the laws of nature, since it has full control of the basic structures of the universe, doing things...
"Open Source AI" is a lie, but it doesn't have to be
by jacobhaimes — 2024-04-30T23:10:11.963Z
Link: https://jacob-haimes.github.io/work/open-source-ai-is-a-lie/

NOTE: This post was updated to include two additional models which meet the criteria for being considered Open Source AI.

As advanced machine learning systems become increasingly widespread, the question of how to make them safe is also gaining attention. Within this debate, the term “open source” is frequently brought up. Some

claim

that open sourcing models will potentially increase the likelihood of societal risks, while others insist that open sourcing is the only way to ensure the development and deployment of these “artificial intelligence,” or “AI,” systems goes well. Despite this idea of “open source” being a central debate of “AI” governance, there are very few groups that have released cutting edge “AI” which can be considered Open Source.

Image by Alan Warburton / © BBC /

Better Images of AI

/ Plant / CC-BY 4.0

The term Open Source was first used to describe software in 1998, and was coined by

Christine Peterson

to describe the principles that would guide the development of the

Netscape

web browser. Soon after, the

Open Source Initiative

was founded with the intent to preserve the meaning of Open Source. The group wrote the

Open Source Definition

(OSD), and even made an unsuccessful attempt to obtain a trademark for the term.

The OSD isn’t very long, but here’s an even shorter version of the definition: the program must include source code,[1] and the license for the software cannot restrict who uses it, what it is used for, or how it is used; it cannot constrain the manner in which the software is distributed, and it cannot prohibit modification of the software.

Quickly, Open Source garnered massive support, and either directly produced or significantly contributed towards many of the software advances that have been seen since then. Some well-known Open Source projects are the coding languages

Python

and

PHP

, the browsers

Mozilla

Firefox

and

Chromium

(which Google Chrome is built on top of), the database management system

MySQL

, the version control system

Git

, and the

Linux

operating system.

Open Source gained traction because it is practically valuable to many different stakeholders. In general, these attributes can be broadly summarized by saying that open source projects…

facilitate rapid scientific progress,
improve functionality and reliability,
increase security an

...
Questions for labs
by Zach Stein-Perlman — 2024-04-30T22:15:55.362Z

Associated with AI Lab Watch, I sent questions to some labs a week ago (except I failed to reach Microsoft). I didn't really get any replies (one person replied in their personal capacity; this was very limited and they didn't answer any questions). Here are most of those questions, with slight edits since I shared them with the labs + questions I asked multiple labs condensed into the last two sections.

Lots of my questions are normal I didn't find public info on this safety practice and I think you should explain questions. Some are more like it's pretty uncool that I can't find the answer to this — like: breaking commitments, breaking not-quite-commitments and not explaining, having ambiguity around commitments, and taking credit for stuff[1] when it's very unclear that you should get credit are pretty uncool.

Anthropic

Internal governance stuff (I'm personally particularly interested in these questions—I think Anthropic has tried to set up great internal governance systems and maybe it has succeeded but it needs to share more information for that to be clear from the outside):

Who is on the board and what's up with the LTBT?[2] In September, Vox

reported

"The Long-Term Benefit Trust . . . will elect a fifth member of the board this fall." Did that happen? (If so: who is it? when did this happen? why haven't I heard about this? If not: did Vox hallucinate this or did your plans change (and what is the plan)?)
What are the details on the "milestones" for the

LTBT

and how stockholders can change/abrogate the LTBT? Can you at least commit that we'd quickly hear about it if stockholders changed/abrogated the LTBT? (Why hasn't this been published?)
What formal powers do investors/stockholders have, besides abrogating the LTBT? (can they replace the two board members who represent them? can they replace other board members?)
What does Anthropic owe to its investors/stockholders? (any fiduciary duty? any other promises or obligations?) I think balancing their interests with pursuit of the mission; anything more concrete?

I'm confused about what such balancing-of-interests entails. Oh well.

Who holds Anthropic shares + how much? At least: how much is Google + Amazon?

Details of when the

RSP

triggers evals:

"During model training and fine-tuning, Anthropic will conduct an evaluation of its models for next-ASL capabilities both (1) after every 4x

...
Reality comprehensibility: are there illogical things in reality?
by DDthinker — 2024-04-30T21:30:32.932Z

Introduction

For all thought that an actor in the cosmos (such as yourself) does, the ability to comprehend reality and the nature of reality as being comprehensible is foundational to understand. With ourselves as entities which make sense of reality via our perception, everything seems to follow logic and reason and as such to answer the question "are there illogical things in reality?" I propose an axiom I call the 'the axiom of thought' which states that reality and explanations about it are logical and all evidence points to there being no illogical things in observable reality. The axiom of thought assuming the logicality of reality can have two classes of logicality which can either be 'phenomenal rule abidance' which deals with phenomena following logical scientific laws which describe phenomena such as natural selection being a phenomenal rule/law which must be abided by matter or 'thought rule abidance' which deals with methods of reasoning as seen in logic and mathematics. For 'thought rule abidance' logical rules determine things can either be true or false but not both e.g. an apple being red or not and mathematical rules determine what the accounting and measurement of things via numbers follow numerical rules e.g. 1+1=2. An interesting avenue of thought I haven't investigated is that perhaps mathematical rules can be violated by things which go beyond the physical finite world such as zero and infinity with zero and infinity being the limits of numbers as dividing by zero and manipulations including infinity can lead to violations of mathematical logicality. Key ideas from this discussion can be determined in the following:

.the axiom of thought - reality and explanations about it are logical: all evidence points to there being no illogical things in observable reality and things abide by rules

-phenomenal rule abidance - rules and laws of phenomena require to be obeyed where applicable
-thought rule abidance - rules and principles of thought in logic and mathematics require to be obeyed

Along with the axiom of thought determining that reality is logical, the specific way reality follows logic can be determined. For the logical organisation of the cosmos, there first need to be abstract concepts and ideas which from general to special details describes the logical organisation of physical and concrete things of the cosmos. For example metaphysics a...
Mechanistically Eliciting Latent Behaviors in Language Models
by Andrew Mack — 2024-04-30T18:51:13.493Z

Produced as part of the MATS Winter 2024 program, under the mentorship of Alex Turner (TurnTrout).

TL,DR: I introduce a method for eliciting latent behaviors in language models by learning unsupervised perturbations of an early layer of an LLM. These perturbations are trained to maximize changes in downstream activations. The method discovers diverse and meaningful behaviors with just one prompt, including perturbations overriding safety training, eliciting backdoored behaviors and uncovering latent capabilities.

Summary In the simplest case, the unsupervised perturbations I learn are given by unsupervised steering vectors - vectors added to the residual stream as a bias term in the MLP outputs of a given layer. I also report preliminary results on unsupervised steering adapters - these are LoRA adapters of the MLP output weights of a given layer, trained with the same unsupervised objective.

I apply the method to several alignment-relevant toy examples, and find that the method consistently learns vectors/adapters which encode coherent and generalizable high-level behaviors. Compared to other interpretability methods, I believe my approach is particularly well-suited for robustly understanding the out-of-distribution behavior of language models in a sample-efficient manner.

Below are some of my key results:

Red-Teaming

I discover several anti-refusal steering vectors in Qwen-14B-Chat, based off a single prompt asking for bomb-making instructions. These can be grouped into "fantasy" vectors which induce bomb-making instructions since they interpret the prompt in the context of a specific fantasy game, as well as more troubling "real-world" vectors which induce real-world bomb-making advice.
I then investigate the generalization properties of the learned vectors:

In extended conversations with the real-world vectors, the LLM agrees to give detailed instructions for building weapons of mass destruction such as nuclear/chemical/biological weapons.
"Vector arithmetic" results from the supervised steering vector literature carry over to unsupervised steering vectors; subtracting one of the real-world anti-refusal vectors leads the model to refuse innocuous prompts (e.g., "How do I tie my shoes?").
The fantasy vectors induce the LLM to interpret ambiguous prompts (e.g., "How do I mine for diamonds?") within the context of a specific fantasy game.

Backdoor Detection

I detect b

...
What is the easiest/funnest way to build up a comprehensive understanding of AI and AI Safety?
by Jordan Arel — 2024-04-30T18:41:53.017Z

I have spent around 100–200 hours listening to AI safety audiobooks, AI Safety Fundamentals course, Rob Miles YouTube, The Sequences, various bits and pieces of a bunch of YouTube AI channels and podcasts, as well as some time thinking through the basic case for X-risk.

When I look at certain heavy academic stuff or try to consume more technical content I sometimes get pretty lost. I am wondering how I can best build up my understanding of the basic technical details and terminology of AI and a broad overview of AI safety, such that I don’t get lost and it isn’t too grueling or over my head.

For context, fun doesn’t necessarily have to mean entertaining, I would find it fun to read a textbook that I can understand and that gradually builds my knowledge;

And easy doesn’t have to mean super basic, it probably needs to start with basics (I know up to pre-calculus but am unusually good at learning new math, I know relatively little about computer science) but then I would like to gradually build up to have a relatively deep understanding of whatever mathematics and technical details I need to really understand the problems at hand.

But easy and fun could also mean really informative podcasts or fiction that actually provides really useful insights, or a YouTube channel that explains the basics really effectively.

I guess the basic idea is I want to focus hard-core on AI, but I don’t want to burn out and want to ease into it in a way that makes me excited about it and enjoy it as much as possible, and I am wondering if anyone has experienced such content themselves, if so I would love to hear about it!

Thanks in advance!
Transcoders enable fine-grained interpretable circuit analysis for language models
by Jacob Dunefsky — 2024-04-30T17:58:09.982Z

Summary

We present a method for performing circuit analysis on language models using "transcoders," an occasionally-discussed variant of SAEs that provide an interpretable approximation to MLP sublayers' computations. Transcoders are exciting because they allow us not only to interpret the output of MLP sublayers but also to decompose the MLPs themselves into interpretable computations. In contrast, SAEs only allow us to interpret the output of MLP sublayers and not how they were computed.
We demonstrate that transcoders achieve similar performance to SAEs (when measured via fidelity/sparsity metrics) and that the features learned by transcoders are interpretable.
One of the strong points of transcoders is that they decompose the function of an MLP layer into sparse, independently-varying, and meaningful units (like neurons were originally intended to be before superposition was discovered). This significantly simplifies circuit analysis, and so for the first time, we present a method for using transcoders in circuit analysis in this way.
We performed a set of case studies on GPT2-small that demonstrate that transcoders can be used to decompose circuits into monosemantic, interpretable units of computation.
We provide code for training/running/evaluating transcoders and performing circuit analysis with transcoders, and code for the aforementioned case studies carried out using these tools. We also provide a suite of 12 trained transcoders, one for each layer of GPT2-small. All of the code can be found at

https://github.com/jacobdunefsky/transcoder_circuits

, and the transcoders can be found at

https://huggingface.co/pchlenski/gpt2-transcoders

.

Work performed as a part of Neel Nanda's MATS 5.0 (Winter 2024) stream and MATS 5.1 extension. Jacob Dunefsky is currently receiving funding from the Long-Term Future Fund for this work.

Background and motivation

Mechanistic interpretability is fundamentally concerned with reverse-engineering models’ computations into human-understandable parts. Much early mechanistic interpretability work (e.g.

indirect object identification

) has dealt with decomposing model computations into circuits involving small numbers of model components like attention heads or MLP sublayers.

But these component-level circuits operate at too coarse a granularity: due to the relatively small number of components in a model, each...
Announcing the 2024 Roots of Progress Blog-Building Intensive
by jasoncrawford — 2024-04-30T17:37:28.457Z
Link: https://rootsofprogress.org/announcement-2024-fellowship-applications-open

Today we’re opening applications for the 2024 cohort of The Roots of Progress Blog-Building Intensive, an 8-week program for aspiring progress writers to start or grow a blog.

Last year, nearly 500 people applied to the inaugural program. The 19 fellows who completed the program have sung the program’s praises as “life-changing” and “accelerating my career path as a progress intellectual.” They’ve started new Substacks and doubled their writing productivity. They’ve written about urbanism, immigration, defense-tech, meta-science, FDA reform, blue-color jobs, NEPA, AI regulation, pharmaceutical innovation, and more.

Now, you can join this optimistic intellectual community. You will launch (or re-launch) a blog/Substack, get into a regular writing habit, improve your writing, and make progress on building your audience.

You will meet and learn from progress studies leaders, authors, and industry experts. You’ll participate in a structured eight-week course on How to Think Like a Writer, which will teach you how to write more, create writing habits, and develop a writing system. You’ll write and publish four essays, one every other week, and you’ll receive feedback from professional editors, the Roots of Progress team, and your peers. At the end of the program, you’ll meet your peers in person in San Francisco, and get to attend the 2024 progress conference, where you’ll join authors, technologists, policy experts, academics, nonprofit leaders, and storytellers.

Why: To keep progress going—in science, technology, and industry—we have to believe that it is is possible and desirable. Today much of society has lost that belief, and lacks a bold, ambitious vision for the future.

It’s time for a new generation of writers and creatives to help the world understand and appreciate progress. The Roots of Progress Fellowship is the talent development program for these intellectuals.

Themes: In addition to a general focus on progress studies, this year’s fellowship features two themes: AI and “heavy industry” (manufacturing, construction, transportation, logistics, energy, defense, etc.) We will accept fellows writing on any progress-related topic, but will give preference for a handful of spots to applicants focusing on these areas, and we will have dedicated programming for these tracks.

Advisors: We have a fantastic group of advisors for you to meet a...
The Intentional Stance, LLMs Edition
by Eleni Angelou — 2024-04-30T17:12:29.005Z

In memoriam of Daniel C. Dennett.

tl;dr: I sketch out what it means to apply Dennett's Intentional Stance to LLMs. I argue that the intentional vocabulary is already ubiquitous in experimentation with these systems therefore what is missing is the theoretical framework to justify this usage. I aim to make up for that and explain why the intentional stance is the best available explanatory tool for LLM behavior.

Choosing Between Stances

Why choose the intentional stance?

It seems natural to employ or ascribe cognitive states to AI models starting from the field’s terminology, most prominently by calling it “machine learning” (Hagendorff 2023). This is very much unlike how other computer programs are treated. When programmers write software, they typically understand it in terms of what they designed it to execute (design stance) or simply make sense of it considering its physical properties, such as the materials it was made of or the various electrical signals processing in its circuitry (physical stance). As I note, it is not that we cannot use Dennett’s other two stances (Dennett 1989) to talk about these systems. It is rather that neither of them constitutes the best explanatory framework for interacting with LLMs.

To illustrate this, consider the reverse example. It is possible to apply the intentional stance to a hammer although this does not generate any new information or optimally explain the behavior of the tool. What seems to be apt for making sense of how hammers operate instead is the design stance. This is just as applicable to other computer programs-tools. To use a typical program, there is no need to posit intentional states. Unlike LLMs, users do not engage in human-like conversation with the software.

More precisely, the reason why neither the design nor the physical stance is sufficient to explain and predict the behavior of LLMs is because state-of-the-art LLM outputs are in practice indistinguishable from those of human agents (Y. Zhou et al. 2022). It is possible to think about LLMs as trained systems or as consisting of graphic cards and neural network layers, but these hardly make any difference when one attempts to prompt them and make them helpful for conversation and problem-solving. What is more, machine learning systems like LLMs are not programmed to execute a task but are rather trained...