trentmkelly/LessWrong-43k huggingface dataset:
What's new at FAR AI
by AdamGleave — 2023-12-04T21:18:03.951Z
Link: https://far.ai/post/2023-12-far-overview/

Summary

We are FAR AI: an AI safety research incubator and accelerator. Since our inception in July 2022, FAR has grown to a team of 12 full-time staff, produced 13 academic papers, opened the coworking space FAR Labs with 40 active members, and organized field-building events for more than 160 ML researchers.

Our organization consists of three main pillars:

Research. We rapidly explore a range of potential research directions in AI safety, scaling up those that show the greatest promise. Unlike other AI safety labs that take a bet on a single research direction, FAR pursues a diverse portfolio of projects. Our current focus areas are building a science of robustness (e.g. finding vulnerabilities in superhuman Go AIs), finding more effective approaches to value alignment (e.g. training from language feedback), and model evaluation (e.g. inverse scaling and codebook features).

Coworking Space. We run FAR Labs, an AI safety coworking space in Berkeley. The space currently hosts FAR, AI Impacts, MATS, and several independent researchers. We are building a collaborative community space that fosters great work through excellent office space, a warm and intellectually generative culture, and tailored programs and training for members. Applications are open to new users of the space (individuals and organizations).

Field Building. We run workshops, primarily targeted at ML researchers, to help build the field of AI safety research and governance. We co-organized the International Dialogue for AI Safety bringing together prominent scientists from around the globe, culminating in a public statement calling for global action on AI safety research and governance. We will soon be hosting the New Orleans Alignment Workshop in December for over 140 researchers to learn about AI safety and find collaborators.

We want to expand, so if you’re excited by the work we do, consider donating or working for us! We’re hiring research engineers, research scientists and communications specialists.

Incubating & Accelerating AI Safety Research

Our main goal is to explore new AI safety research directions, scaling up those that show the greatest promise. We select agendas that are too large to be pursued by individual academic or independent researchers but are not aligned with the interests of for-profit organizations. Our structure allows us to both (1) explore a portfolio of agenda...
n of m ring signatures
by DanielFilan — 2023-12-04T20:00:06.580Z

A normal cryptographic signature associated with a message and a public key lets you prove to the world that it was made by someone with access to the private key associated with the known public key, without revealing that private key. You can read about it on Wikipedia here.

A ring signature associated with a message and a set of public keys lets you prove to the world that it was made by someone with access to the message and one private key associated to one of the public keys in the set, but nobody will be able to tell which public key it was. This lets you say something semi-anonymously, which is neat. It’s also used in the private cryptocurrency Monero. You can read about them on Wikipedia here.

Here’s a thing that would be better than a ring signature: a signature that proved that it was made by a subset of public keys of a certain size. In my head, I was calling this an n of m ring signature for a while. But when I googled “n of m ring signature”, nothing came up. It turns out this is because in the literature, it’s called a “threshold ring signature”, a “k of n ring signature”, or a “t of n ring signature” instead. I think perhaps the first paper about it is this one, but I haven’t checked very hard.

Anyway: I would like to make it so that when you search for n-of-m ring signatures online, you find a thing telling you that you should instead search for “threshold ring signature”. Hence this post.
Mechanistic interpretability through clustering
by Alistair Fraser — 2023-12-04T18:49:26.777Z

How are activations in a transformer clustered together and what can we learn?

There has been a lot of progress using unsupervised methods (such as sparse autoencoders) to find monosemantic features in LLMs. However, is there a way that we can interpret activations without breaking them down into features?

The answer is yes. I contend that clustering activations (from a large dataset of examples) reveals strong interpretability. The process is simple. Choose any example and then find a set of activations that are L2 closest at your favorite activation stage. Semantic meaning of that stage can then be inferred by recognizing similarities in the examples that make up that set.

In some ways, cluster analysis is like the logit lens in that it focuses on what the transformer believes after each step. However cluster analysis reveals additional insight and detail.

Interactive Demo

Consider a simple case. Using a transformer trained to do 4 digit addition, we can examine particular stages (e.g. transformer lens: “pattern”, “result” , “attn_out” ) at all layers, heads and words to see the examples that are grouped tougher. This grouping can be interpreted to see how the transformer solves the problem.

To add two numbers (e.g. L + R = Sum), the transformer has to both select and sum the correct digit from L and R while also factoring in the appropriate carry.

Using clustering it is easy to see the specific steps where the transformer separately focuses on each of these tasks.

Click here to view an interactive demo.
Agents which are EU-maximizing as a group are not EU-maximizing individually
by Mlxa — 2023-12-04T18:49:08.708Z

Introduction

Why Subagents? and Why Not Subagents? explore whether a group of expected utility maximizers is itself a utility maximizer. Here I want to discuss the converse: if a group wants to maximize some utility function as a whole, what can be said about the individual agents? Of course, if they could make decisions together, they will just compute what each agent needs to do, but what if the only thing they have is a common algorithm that each of them uses independently?

It seems that such agents, in general, don't make decisions by multiplying utilons with probabilities and instead they need to consider the whole distribution of outcomes to evaluate a choice. A similar idea was already presented in Against Expected Utility, though without the focus on the number of agents.

Specific example

Imagine two traders, who select trades independently, but pool their returns together and optimize for the expected logarithm of their total wealth (as in Kelly betting). Also I will assume for simplicity that they select the same trade for both of them, though the outcomes are still sampled independently.

So if a trade multiplies the wealth by (a random variable) X, utility for one trader would be E[logX]. But for the described group of two traders it becomes E[log(X1+X22)], where X1,X2 are independent random variables with the same distribution as X. It is not linear in terms of the outcome probabilities anymore:

U(p)=∫x1,x2(logx1+x22)p(x1)p(x2)dx1dx2

Increasing number of agents

Qualitatively, as the number of agents in the group increases, the agents can afford more risky actions, thanks to the aggregation of returns. So their decision will be somewhere between what an individual agent would do to maximize E[logX] and what it would do to maximize E[X]. Even more specific example to support this intution: there is a fair coin, and the agent can bet fraction f of the wealth available to them on a certain side, which will turn into 3f if the coin lands this side and 0 otherwise, i.e. their wealth will be multiplied by either 1+2f or 1−f.

Expected increase in utility (after aggregation) as a function of f for different number of agents.

So as the number of agents increases, each agent becomes closer to maximizing E[X], but for any fi...
Planning in LLMs: Insights from AlphaGo
by jco — 2023-12-04T18:48:57.508Z

Introduction

Talk of incorporating planning techniques such as Monte Carlo tree search (MCTS) into LLMs has been bubbling around the AI sphere recently, both in relation to Google's Gemini and OpenAI's Q*. Much of this discussion has been in the context of AlphaGo, so I decided to go back and read through AlphaGo and some subsequent papers (AlphaGo Zero and AlphaZero). This post highlights what these papers did in the context of LLMs and some thoughts I had while reviewing the papers.

When I say LLMs in this post I am referring to causal/decoder/GPT LLMs.

AlphaGo

Overview

AlphaGo trains two supervised learning (SL) policy networks, a reinforcement learning (RL) policy network, a SL value network, and uses MCTS for planning. It learns to play the game of Go.

SL Policy Networks

Two SL policy networks are trained:

A slow policy network using a CNN. Used to compute prior probability of state-action pairs during MCTS rollouts.
A fast policy network using hand-crafted linear features. Used for full game simulations during MCTS rollouts.

Both networks are trained to predict the next move from a data set of expert games. This is the same self-supervised target as LLMs trained to predict the next token from a data set of internet text; AlphaGo learns a softmax over next moves & LLMs learn a softmax over next tokens.

This use of a fast and slow model reminds me of speculative decoding. I haven't thought through the implications of this much and it is far from a perfect analogy, but it could be a useful insight.

RL Policy Network

The RL policy network is trained by fine-tuning the SL policy network with REINFORCE on games between the current RL policy network and randomly selected previous RL policy networks. Rewards are +1 for winning and -1 for losing.

This RL policy network, using no search, won 85% of games against Pachi, a Go program that used 100,000 MCTS simulations per move. This shows that pure RL can outperform pure search, but usually a combination of the two gives the best performance.

The RL policy network was not actually used in the final version of AlphaGo; it was only used to generate data for training the value network. The authors noted that the SL policy network performed better than the RL policy network "presumably because humans select a diverse beam of promising moves, whereas RL optimizes for the single best move."

This lack of diversity is reminisce...
Non-classic stories about scheming (Section 2.3.2 of "Scheming AIs")
by Joe Carlsmith — 2023-12-04T18:44:32.825Z

This is Section 2.3.2 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?”. There’s also a summary of the full report here (audio here). The summary covers most of the main points and technical terms, and I’m hoping that it will provide much of the context necessary to understand individual sections of the report on their own.

Audio version of this section [here[(https://www.buzzsprout.com/2034731/13984902), or search for "Joe Carlsmith Audio" on your podcast app.

Non-classic stories

What about other stories, though? In principle, we can imagine a number
of other reasons AI systems with beyond-episode goals might optimize for
reward-on-the-episode as part of an instrumental strategy for gaining
power for themselves or for other AIs. Here I'll look at a few that seem
salient.

AI coordination

In the classic goal-guarding story, the central aim of a schemer's
strategy is to propagate its goals, or goals suitably similar to its
own, forward in time, in pursuit of an outcome in which some agent with
those goals escapes from human control and possibly participates in a
takeover. However, we can also alter the story somewhat, such that the
schemer works on behalf of the escape/takeover of an AI with some
other set of goals, but who the schemer is suitably confident will
repay the favor.

This sort of coordination between AIs with different goals would allow
us to relax the requirement, discussed in the previous section, that the
schemer find a way to forward-propagate its goals in particular all
the way through training, escape, and takeover. And it would also put to
rest worries about the schemer being "shut out" from power in a
post-takeover regime.

Thus, for example, consider the scenario discussed above, in which GPT-6
has an opportunity to insert a backdoor into some code that GPT-8 will
later be able to use to escape, but where GPT-6 has no way of
influencing GPT-8's goals directly to be more like its own (and where,
let's say, GPT-8's goals would be very different from GPT-6's absent
such influence). Here, GPT-6 could go ahead and insert the backdoor, in
anticipation of GPT-8 later escaping, taking-over, and then giving some
power to GPT-6's goals in return for the favor. And GPT-6 might also
continue to pretend to be aligned, even though it has no hope of
escaping itself, so as not to alert the humans about misalignment
pr...
6. The Mutable Values Problem in Value Learning and CEV
by RogerDearnaley — 2023-12-04T18:31:22.080Z

Part 6 of AI, Alignment, and Ethics. This will probably make more sense if you start with Part 1.

TL;DR In Parts 1 through 5 I discussed how to choose an ethical system, and implications for societies containing biological and/or uploaded humans and aligned AIs, and perhaps even other sapient species, but not all sentient animals. So far I've just been assuming that we have (somehow) built aligned AI. Now I'd like to look at how all of this relates to the challenge of achieving that vital goal: how we might align superintelligent AI, specifically using approaches along the lines of value learning, AI-assisted Alignment, or Coherent Extrapolated Volition (CEV) — or indeed any similar "do what I mean" kind of approach. The mutability of human values poses a major challenge to all of these: "Do what I want" or "do what I mean" is a lot less well-defined once ASI is in a position to affect that directly, rather than just by misinterpretation. Below I outline and critique a number of possible solutions: this is challenging, since when setting the terminal goal for ASI there is a strong tension between controlling the outcome and allowing our descendants the freedom to control their own destiny. Without a solution to strong corrigibility, we can only do set the terminal goal once, ever, which privileges the views of whatever generation gets to do this. The possibilities opened up by genetic engineering and cyborging make the mutable values problem far more acute, and I explore a couple of toy examples from the ethical conundrums of trying to engineer away psychopathy and war. Finally I suggest a tentative proposal for a compromise solution for mutable values, which builds upon the topics discussed in the previous parts of the sequence.

In what follows I'm primarily going to discuss a future society that is aligning its Artificial Superhuman Intelligences (ASIs) either using value learning, or some nearly-functionally-equivalent form of AI-assisted alignment, such that the bulk of the work of figuring out the fine details of complex and fragile human values and then building a usable normative model of them is being done by superhuman AIs. CEV isn't quite the same proposal as value learning, though the two are related, and they face somewhat similar challenges over mutability of values, as do other similar "do what I mean" solutions that have also been proposed. I pe...
Updates to Open Phil’s career development and transition funding program
by abergal — 2023-12-04T18:10:29.394Z

(cross-posted from the Effective Altruism Forum)
We’ve recently made a few updates to the program page for our

career development and transition funding program

(recently renamed, previously the “early-career funding program”), which provides support – in the form of funding for graduate study, unpaid internships, independent study, career transition and exploration periods, and other activities relevant to building career capital – for individuals who want to pursue careers that could help to reduce global catastrophic risks (especially

risks from advanced artificial intelligence

and

global catastrophic biological risks

) or otherwise improve the long-term future.

The main updates are as follows:

We’ve broadened the program’s scope to explicitly include later-career individuals, which is also reflected in the new program name.
We’ve added some language to clarify that we’re open to supporting a variety of career development and transition activities, including not just graduate study but also unpaid internships, independent study, career transition and exploration periods, postdocs, obtaining professional certifications, online courses, and other types of one-off career-capital-building activities.

Earlier versions of the page stated that the program’s primary focus was to provide support for graduate study specifically, which was our original intention when we first launched the program back in 2020. We haven’t changed our views about the impact of that type of funding and expect it to continue to account for a large fraction of the grants we make via this program, but we figured we should update the page to clarify that we’re in fact open to supporting a wide range of other kinds of proposals as well, which also reflects what we’ve already been doing in practice.

This program now subsumes what was previously called the

Open Philanthropy Biosecurity Scholarship

; for the time being, candidates who would previously have applied to that program should apply to this program instead. (We may decide to split out the Biosecurity Scholarship again as a separate program at a later point, but for practical purposes, current applicants can ignore this.)

Some concrete examples of the kinds of applicants we’re open to funding, in no particular order (copied from the program page):

A final-year undergraduate student who wants to pursue a master’s or a P

...
[Valence series] 1. Introduction
by Steven Byrnes — 2023-12-04T15:40:21.274Z

1.1 Summary & Table of Contents

This is the first of a series of five blog posts on valence. Here’s an overview of the whole series, and then we’ll jump right into the first post!

1.1.1 Summary & Table of Contents—for the whole Valence series

Let’s say a thought pops into your mind: “I could open the window right now”. Maybe you then immediately stand up and go open the window. Or maybe you don’t. (“Nah, I’ll keep it closed,” you might say to yourself.) I claim that there’s a final-common-pathway[1] signal in your brain that cleaves those two possibilities: when this special signal is positive, then the current “thought” will stick around, and potentially lead to actions and/or direct-follow-up thoughts; and when this signal is negative, then the current “thought” will get thrown out, and your brain will go fishing (partly randomly) for a new thought to replace it. I call this final-common-pathway signal by the name “valence”. Thus, the “valence” of a “thought” is roughly the extent to which the thought feels demotivating / aversive (negative valence) versus motivating / appealing (positive valence).

I claim that valence plays an absolutely central role in the brain—I think it’s one of the most important ingredients in the brain’s Model-Based Reinforcement Learning system, which in turn is one of the most important algorithms in your brain.

Thus, unsurprisingly, I see valence as a shining light that illuminates many aspects of psychology and everyday mental life. This series explores that idea. Here’s the outline:

Post 1 (Introduction) will give some background on how I’m thinking about valence from the perspective of brain algorithms, including exactly what I’m talking about, and how it relates to the “wanting versus liking” dichotomy. (The thing I’m talking about is closer to “motivational valence” than “hedonic valence”, although neither term is great.)
Post 2 (Valence & Normativity) will talk about the intimate relationship between valence and the universe of desires, preferences, values, goals, etc.—i.e. the “normative” side of the “positive-versus-normative” dichotomy, or equivalently the “ought” side of

Hume’s

“is-versus-ought”. I’ll start with simple cases: for example, if the idea of doing a certain thing right now feels unappealing (negative valence), then we’re less likely to do it. Then I’ll move on to more interesting ca

...
South Bay Meetup 12/9
by David Friedman — 2023-12-04T07:32:26.619Z

At 3806 Williams Rd, San Jose, CA 95117, starting at 2:00. Let me know if you plan to come so we will know how many we are feeding. Details:http://www.daviddfriedman.com/SSC%20Meetups%20announcement.html
Hashmarks: Privacy-Preserving Benchmarks for High-Stakes AI Evaluation
by Paul Bricman — 2023-12-04T07:31:48.726Z
Link: https://arxiv.org/abs/2312.00645

TL;DR: Quick "idea paper" describing a protocol for benchmarking AI capabilities in the open without disclosing sensitive information. It's similar to how passwords are used for user registration and authentication in a web app: experts first hash their answers, then developers hash the model's answers to check whether the hashes match, but at no point are the cleartext answers freely floating around. The paper explores the protocol's resilience against half a dozen failure modes, and speculates on future infrastructure for high-stakes evaluation.

Extended summary as Twitter thread.

Context: Below you can find the raw contents of the paper. While all feedback is welcome, identifying unseen failure modes would be particularly helpful ahead of potentially circulating a couple proof-of-concept instances of hashmarks in the coming weeks.

Introduction

Background & Motivation

Traditional question-answering (QA) benchmarks have played a crucial role in shaping the trajectory of AI development. They provide standardized metrics that facilitate fair comparisons across research groups and measure progress within the field in a reproducible way. For instance, capabilities related to mathematics can generally be gauged by assessing model performance in producing or selecting correct answers for exam questions related to mathematics. Similar "AI exams" have been used to assess model performance on topics ranging from STEM to humanities. In fact, prior work has highlighted the possibility of framing the vast majority of established natural language processing tasks as question-answering tasks.

Traditional QA benchmarks are typically sourced from crowd-workers, members of the group developing the benchmark, or a mix of the two. They typically contain a large number of data points representing individual exercises. Each data point, in turn, contains one question and one or several correct answers. Occasionally, a data point may also contain some relevant context in the form of a separate string, although the context can always be functionally merged with the question without loss of generality. In addition, when structured as a multiple-choice question, a data point also contains several distractor answers. These data points are then aggregated into one or more standardized files, and are finally made public for other groups to easily make use of them in evaluating language m...