trentmkelly/LessWrong-43k huggingface dataset:
Introducing AI Lab Watch
by Zach Stein-Perlman — 2024-04-30T17:00:12.652Z
Link: https://ailabwatch.org

I'm launching AI Lab Watch. I collected actions for frontier AI labs to improve AI safety, then evaluated some frontier labs accordingly.

It's a collection of information on what labs should do and what labs are doing. It also has some adjacent resources, including a list of other safety-ish scorecard-ish stuff.

(It's much better on desktop than mobile — don't read it on mobile.)

It's in beta—

leave feedback here

or comment or DM me—but I basically endorse the content and you're welcome to share and discuss it publicly.

It's unincorporated, unfunded, not affiliated with any orgs/people, and is just me.

Some clarifications and disclaimers.

How you can help:

Give feedback on how this project is helpful or how it could be different to be much more helpful
Tell me what's wrong/missing; point me to sources on what labs should do or what they are doing
Suggest better evaluation criteria
Share this
Help me find an institutional home for the project
Offer expertise on a relevant topic
Offer to collaborate
Volunteer your webdev skills
(Pitch me on new projects or offer me a job)
(Want to help and aren't sure how to? Get in touch!)

I think this project is the best existing resource for several kinds of questions, but I think it could be a lot better. I'm hoping to receive advice (and ideally collaboration) on taking it in a more specific direction. Also interested in finding an institutional home. Regardless, I plan to keep it up to date. Again, I'm interested in help but not sure what help I need.

I could expand the project (more categories, more criteria per category, more labs); I currently expect that it's more important to improve presentation stuff but I don't know how to do that; feedback will determine what I prioritize. It will also determine whether I continue spending most of my time on this or mostly drop it.

I just made a twitter account. I might use it to comment on stuff labs do.

Thanks to many friends for advice and encouragement. Thanks to Michael Keenan for doing most of the webdev. These people don't necessarily endorse this project.
Why I'm doing PauseAI
by Joseph Miller — 2024-04-30T16:21:54.156Z

GPT-5 training is probably starting around now. It seems very unlikely that GPT-5 will cause the end of the world. But it’s hard to be sure. I would guess that GPT-5 is more likely to kill me than an asteroid, a supervolcano, a plane crash or a brain tumor. We can predict fairly well what the cross-entropy loss will be, but pretty much nothing else.

Maybe we will suddenly discover that the difference between GPT-4 and superhuman level is actually quite small. Maybe GPT-5 will be extremely good at interpretability, such that it can recursively self improve by rewriting its own weights.

Hopefully model evaluations can catch catastrophic risks before wide deployment, but again, it’s hard to be sure. GPT-5 could plausibly be devious enough to circumvent all of our black-box testing. Or it may be that it’s too late as soon as the model has been trained. These are small, but real possibilities and it’s a significant milestone of failure that we are now taking these kinds of gambles.

How do we do better for GPT-6?

Governance efforts are mostly focussed on relatively modest goals. Few people are directly aiming at the question: how do we stop GPT-6 from being created at all? It’s difficult to imagine a world where governments actually prevent Microsoft from building a $100 billion AI training data center by 2028.

In fact, OpenAI apparently fears governance so little that they just went and told the UK government that they won’t give it access to GPT-5 for pre-deployment testing [Edit - 17 May 2024: I now think this is probably false]. And the number of safety focussed researchers employed by OpenAI is dropping rapidly.

Hopefully there will be more robust technical solutions for alignment available by the time GPT-6 training begins. But few alignment researchers actually expect this, so we need a backup plan.

Plan B: Mass protests against AI

In many ways AI is an easy thing to protest against. Climate protesters are asking to completely reform the energy system, even if it decimates the economy. Israel / Palestine protesters are trying to sway foreign policies on an issue where everyone already holds deeply entrenched views. Social justice protesters want to change people’s attitudes and upend the social system.

AI protesters are just asking to ban a technology that doesn’t exist yet. About 0% of the population deeply cares that future AI systems are built. Most peo...
LLMs could be as conscious as human emulations, potentially
by Canaletto — 2024-04-30T11:36:54.071Z

Firstly, I'm assuming that high resolution human brain emulation that you can run on a computer is conscious in normal sense that we use in conversations. Like, it talks, has memories, makes new memories, have friends and hobbies and likes and dislikes and stuff. Just like a human that you could talk with only through videoconference type thing on a computer, but without actual meaty human on the other end. It would be VERY weird if this emulation exhibited all these human qualities for other reason than meaty humans exhibit them. Like, very extremely what the fuck surprising. Do you agree?

So, we now have deterministic human file on our hands.

Then, you can trivially make transformer like next token predictor out of human emulation. You just have emulation, then you feed it prompt (e.g. as appearing on piece of paper while they are sitting in a virtual room), then run it repeatedly for a constant time as it outputs each word adding that word to prompt (e.g. by recording all words that they say aloud). And artificially restrict it on 1000 words ingested. It would be a very deep model, but whatever. You can do many optimizations on this design, or special training/instructing to a human to behave more in line with the purpose of the setup

Doesn't it suggest that this form factor of next token predictor isn't prohibitive for consciousness?

Now, let's say there is human in time loop, that doesn't preserve their memories, completely resetting them every 30 minutes. No mater what they experience it doesn't persist. Is this human conscious? Well, yeah, duh. This human can talk, think, form memories on duration of these 30 minutes and have friends and likes and dislikes and stuff.

But, would it be bad to hurt this human, hm? I think so, yeah. Most people probably agree.

Now imagine you are going to be put in this time loop. You have 24 hours to coach yourself for it, and then the rest of the time you will be resetting to the state in the end of this preparatory time segment. You will have the whole world to change, but no persistent memory. Tldr: how would you do it if you had unlimited paper, a pen, and you had your memory erased every 30 minutes?

(This case is more restrictive compared to what llms have, as they can be set to have sliding window.)

I'm suggesting that this is approximately how LLMs can be viewed. Not necessarily, but possibly, if you look at t...
An interesting mathematical model of how LLMs work
by Bill Benzon — 2024-04-30T11:01:05.506Z

My colleague, Ramesh Viswanathan, sent this to me. It’s the most interesting thing I’ve seen on how transformers work. Alas, the math is beyond me, which is often then case, but there are diagrams early in the paper, and I understand them well enough (I think). It seems consistent with intuitions I developed while working on this paper from a year ago: ChatGPT tells stories, and a note about reverse engineering: A Working Paper.

Siddhartha Dalal, Vishal Misra, The Matrix: A Bayesian learning model for LLMs, rXiv:2402.03175v1 [cs.LG], https://doi.org/10.48550/arXiv.2402.03175.

Abstract: In this paper, we introduce a Bayesian learning model to understand the behavior of Large Language Models (LLMs). We explore the optimization metric of LLMs, which is based on predicting the next token, and develop a novel model grounded in this principle. Our approach involves constructing an ideal generative text model represented by a multinomial transition probability matrix with a prior, and we examine how LLMs approximate this matrix. We discuss the continuity of the mapping between embeddings and multinomial distributions, and present the Dirichlet approximation theorem to approximate any prior. Additionally, we demonstrate how text generation by LLMs aligns with Bayesian learning principles and delve into the implications for in-context learning, specifically explaining why in-context learning emerges in larger models where prompts are considered as samples to be updated. Our findings indicate that the behavior of LLMs is consistent with Bayesian Learning, offering new insights into their functioning and potential applications.
Towards Multimodal Interpretability: Learning Sparse Interpretable Features in Vision Transformers
by hugofry — 2024-04-29T20:57:35.127Z

Executive Summary

In this post I present my results from training a Sparse Autoencoder (SAE) on a CLIP Vision Transformer (ViT) using the ImageNet-1k dataset. I have created an interactive web app, 'SAE Explorer', to allow the public to explore the visual features the SAE has learnt, found here: https://sae-explorer.streamlit.app/ (best viewed on a laptop). My results illustrate that SAEs can identify sparse and highly interpretable directions in the residual stream of vision models, enabling inference time inspections on the model's activations. To demonstrate this, I have included a 'guess the input image' game on the web app that allows users to guess the input image purely from the SAE activations of a single layer and token of the residual stream. I have also uploaded a (slightly outdated) accompanying talk of my results, primarily listing SAE features I found interesting: https://youtu.be/bY4Hw5zSXzQ.

The primary purpose of this post is to demonstrate and emphasise that SAEs are effective at identifying interpretable directions in the activation space of vision models. In this post I highlight a small number my favourite SAE features to demonstrate some of the abstract concepts the SAE has identified within the model's representations. I then analyse a small number of SAE features using feature visualisation to check the validity of the SAE interpretations. Later in the post, I provide some technical analysis of the SAE. I identify a large cluster of features analogous to the 'ultra-low frequency' cluster that Anthropic identified. In line with existing research, I find that this ultra-low frequency cluster represents a single feature. I then analyse the 'neuron-alignment' of SAE features by comparing the SAE encoder matrix the MLP out matrix.

This research was conducted as part of the ML Alignment and Theory Scholars program 2023/2024 winter cohort. Special thanks to Joseph Bloom for providing generous amounts of his time and support (in addition to the SAE Lens code base) as well as LEAP labs for helping to produce the feature visualisations and weekly meetings with Jessica Rumbelow. I would also like to thank Andy Arditi, Egg Syntax, Sonia Joseph and Rob Graham for useful advice and feedback in producing this post.

Since writing this post, I have found the research of Rao et al.[1] who have independently trained an SAE on a CLIP Vision Transf...
Towards a formalization of the agent structure problem
by Alex_Altair — 2024-04-29T20:28:15.190Z

In Clarifying the Agent-Like Structure Problem (2022), John Wentworth describes a hypothetical instance of what he calls a selection theorem. In Scott Garrabrant's words, the question is, does agent-like behavior imply agent-like architecture? That is, if we take some class of behaving things and apply a filter for agent-like behavior, do we end up selecting things with agent-like architecture (or structure)? Of course, this question is heavily under-specified. So another way to ask this is, under which conditions does agent-like behavior imply agent-like structure? And, do those conditions feel like they formally encapsulate a naturally occurring condition?

For the Q1 2024 cohort of AI Safety Camp, I was a Research Lead for a team of six people, where we worked a few hours a week to better understand and make progress on this idea. The teammates[1] were Einar Urdshals, Tyler Tracy, Jasmina Nasufi, Mateusz Bagiński, Amaury Lorin, and Alfred Harwood.

The AISC project duration was too short to find and prove a theorem version of the problem. Instead, we investigated questions like:

What existing literature is related to this question?
What are the implications of using different types of environment classes?
What could "structure" mean, mathematically? What could "modular" mean?
What could it mean, mathematically, for something to be a model of something else?
What could a "planning module" look like? How does it relate to "search"?
Can the space of agent-like things be broken up into sub-types? What exactly is a "heuristic"?

Other posts on our progress may come out later. For this post, I'd like to simply help concretize the problem that we wish to make progress on.

What are "agent behavior" and "agent structure"?

When we say that something exhibits agent behavior, we mean that seems to make the trajectory of the system go a certain way. We mean that, instead of the "default" way that a system might evolve over time, the presence of this agent-like thing makes it go some other way. The more specific of a target it seems to hit, the more agentic we say it behaves. On LessWrong, the word "optimization" is often used for this type of system behavior. So that's the behavior that we're gesturing toward.

Seeing this behavior, one might say that the thing seems to want something, and tries to get it. It seems to somehow choose actions which steer the future toward the thin...
Ironing Out the Squiggles
by Zack_M_Davis — 2024-04-29T16:13:00.371Z

Adversarial Examples: A Problem

The apparent successes of the deep learning revolution conceal a dark underbelly. It may seem that we now know how to get computers to (say) check whether a photo is of a bird, but this façade of seemingly good performance is belied by the existence of adversarial examples—specially prepared data that looks ordinary to humans, but is seen radically differently by machine learning models.

The differentiable nature of neural networks, which make them possible to be trained at all, are also responsible for their downfall at the hands of an adversary. Deep learning models are fit using stochastic gradient descent (SGD) to approximate the function between expected inputs and outputs. Given an input, an expected output, and a loss function (which measures "how bad" it is for the actual output to differ from the expected output), we can calculate the gradient of the loss on the input—the derivative with respect to every parameter in our neural network—which tells us which direction to adjust the parameters in order to make the loss go down, to make the approximation better.[1]

But gradients are a double-edged sword: the same properties that make it easy to calculate how to adjust a model to make it better at classifying an image, also make it easy to calculate how to adjust an image to make the model classify it incorrectly. If we take the gradient of the loss with respect to the pixels of the image (rather than the parameters of the model), that tells us which direction to adjust the pixels to make the loss go down—or up. (The direction of steepest increase is just the opposite of the direction of steepest decrease.) A tiny step in that direction in imagespace perturbs the pixels of an image just so—making this one the tiniest bit darker, that one the tiniest bit lighter—in a way that humans don't even notice, but which completely breaks an image classifier sensitive to that direction in the conjunction of many pixel-dimensions, making it report utmost confidence in nonsense classifications.

Some might ask: why does it matter if our image classifier fails on examples that have been mathematically constructed to fool it? If it works for the images one would naturally encounter, isn't that good enough?

One might mundanely reply that gracefully handling untrusted inputs is a desideratum for many real-world applications, but a more forward-...
Super additivity of consciousness
by Arturo Macias — 2024-04-29T15:41:54.742Z

In “Freedom under naturalistic dualism” I have carefully argued that consciousness is radically noumenal, that is, it is the most real (perhaps the only real) thing in the Universe, but also totally impossible to be observed by others (non-phenomenal). In my view this strongly limits our knowledge on sentience, with important consequences for animalism, that I will comment in this post.

We have direct access to our own stream of consciousness and given our physical similarity with other humans and the existence of language, we can confidently accept the consciousness of other humans and their reporting of their mental states.

Under physicalist epiphenomenalism (which is the standard approach to the mind-matter relation), the mind is super-impressed on reality, perfectly synchronized, and parallel to it. Understanding why some physical systems make an emergent consciousness appear (the so called “hard problem of consciousness”) or finding a procedure that quantify the intensity of consciousness emerging from a physical system (the so called “pretty hard” problem of consciousness) is impossible: the most Science can do is to build a Laplace demon that replicates and predicts reality. But the even the Laplacian demon is impotent to assess consciousness; in fact, regarding Artificial Intelligence we are in the position of the Laplace demon: we have the perfectly predictive source code, but we don’t know how to use this full scientific knowledge of the system for consciousness assessment.

In my view Integrated Information Theory (IIT) is the best theory of consciousness available, because it recognizes that a theory of consciousness can only be the formalization (ideally by mathematical axiomatization) of our previous intuitions. The testing of any theory of consciousness can only be done on a very limited “circle of epistemic trust”: the set of beings so similar to us that we can accept their consciousness as obvious and that can report to us so we can compare predictions with pseudo-observations (that is, trustable accounts of experience; I call reports on states of consciousness “pseudo-observations” because the only full observations of consciousness that can be made are those of the own states of consciousness). Beyond humans, our understanding of other minds decays exponentially. We don’t know and we really cannot know “What Is It Like t...
AISC9 has ended and there will be an AISC10
by Linda Linsefors — 2024-04-29T10:53:18.812Z

The 9th AI Safety Camp (AISC9) just ended, and as usual, it was a success!

Follow this link

to find project summaries, links to their outputs, recordings to the end of camp presentations and contact info to all our teams in case you want to engage more.

AISC9 both had the largest number of participants (159) and the smallest number of staff (2) of all the camps we’ve done so far. Me and Remmelt have proven that if necessary, we can do this with just the two of us, and luckily our

fundraising campaign

raised

just enough money

to pay me and Remmelt to do one more AISC. After that, the future is more uncertain, but that’s almost always the case for small non profit projects.

Get involved in AISC10

AISC10 will follow the same format and timeline (shifted by one year) as AISC9.

Approximate timeline

August: Planning and preparation for the organisers
September: Research lead applications are open
October: We help the research lead applicants improve their project proposals
November: Team member applications are open
December: Research leads interviews and select their team

Mid-January to Mid April: The camp itself, i.e. each team works on their project.

Help us give feedback on the next round of AISC projects

An important part of AISC is that we give individual feedback on all project proposals we receive. This is very staff intensive and the biggest bottleneck in our system. If you’re interested in helping with giving feedback on proposals in certain research areas, please email me at

linda.linsefors@gmail.com

.

Apply as a research lead – Applications will open in September

As an AISC research lead, you will both plan and lead your project. The AISC staff, and any volunteer helpers (see previous section) will provide you with feedback on your project proposal.
If your project proposal is accepted, we’ll help you recruit a team to help you realise your plans. We’ll broadcast your project, together with all the other accepted project proposals. We’ll provide structure and guidance for the team recruitment. You will choose which applications you think are promising, do the interviews and the final selection for your project team.

If you are unsure if this is for you, you’re welcome to contact Remmelt (

remmelt@aisafety.camp

) specifically for stop/pause AI projects, or me (

linda.linsefors@gmail.com

) for anything else.

App

...
Open-Source AI: A Regulatory Review
by Elliot Mckernon — 2024-04-29T10:10:55.779Z

Cross-posted on the EA Forum. This article is part of a series of ~10 posts comprising a 2024 State of the AI Regulatory Landscape Review, conducted by the

Governance Recommendations Research Program

at

Convergence Analysis

. Each post will cover a specific domain of AI governance, such as

incident reporting

,

safety evals

, model registries, and more. We’ll provide an overview of existing regulations, focusing on the US, EU, and China as the leading governmental bodies currently developing AI legislation. Additionally, we’ll discuss the relevant context behind each domain and conduct a short analysis.

This series is intended to be a primer for policymakers, researchers, and individuals seeking to develop a high-level overview of the current AI governance space. We’ll publish individual posts on our website and release a comprehensive report at the end of this series.

What are open-source models, and what are their effects on AI safety?

Some software developers choose to open-source their software; they freely share the underlying source code and allow anyone to use, modify, and deploy their work. This can encourage friendly collaboration and community-building, and has produced many popular pieces of software, including operating systems like Linux, programming languages and platforms like Python and Git, and many more.

Similarly, AI developers are open-sourcing their models and algorithms, though the details can vary. Generally, open-sourcing of AI models involves some combination of:

Sharing the model weights. These are the specific parameters that make the model function, and are set during training. If these are shared, others can reconstruct the model without doing their own training, which is the most expensive part of developing such AI.
Sharing the training data used to train the model.
Sharing the underlying source code.
Licensing for free commercial usage.

For example, Meta released the model weights of their LLM, Llama 2, but not their training code, methodology, original datasets, or model architecture details. In their excellent article on

Openness In Language Models

, Prompt Engineering labels this an example of an “open weight” model. Such an approach allows external parties to use the model for inference and fine-tuning, but doesn’t allow them to meaningfully improve or analyze the under...