trentmkelly/LessWrong-43k huggingface dataset:
Meaning & Agency
by abramdemski — 2023-12-19T22:27:32.123Z

The goal of this post is to clarify a few concepts relating to AI Alignment under a common framework. The main concepts to be clarified:

Optimization. Specifically, this will be a type of Vingean agency. It will split into Selection vs Control variants.
Reference (the relationship which holds between map and territory; aka semantics, aka meaning). Specifically, this will be a teleosemantic theory.

The main new concepts employed will be endorsement and legitimacy.

TLDR:

Endorsement of a process is when you would take its conclusions for your own, if you knew them.
Legitimacy relates to endorsement in the same way that good relates to utility. (IE utility/endorsement are generic mathematical theories of agency; good/legitimate refer to the specific thing we care about.)
We perceive agency when something is better at doing something than us; we endorse some aspect of its reasoning or activity. (Endorse as a way of achieving its goals, if not necessarily our own.)
We perceive meaning (semantics/reference) in cases where something has been optimized for accuracy -- that is, the goal we endorse a conclusion with respect to is some notion of accuracy of representation.

This write-up owes a large debt to many conversations with Sahil, although the views expressed here are my own.

Broader Context

The basic idea is to investigate agency as a natural phenomenon, and have some deep insights emerge from that, relevant to the analysis of AI Risk. Representation theorems -- also sometimes called coherence theorems or selection theorems depending on what you want to emphasize -- start from a very normative place, typically assuming preferences as a basic starting object. Perhaps these preferences are given a veneer of behaviorist justification: they supposedly represent what the agent would choose if given the option. But actually giving agents the implied options would typically be very unnatural, taking the agent out of its environment.

There are many ways to drive toward more naturalistic representation theorems. The work presented here takes a particular approach: cognitive reduction. I want to model what it means for one agent to think of something as being another agent. Daniel Dennett called this the intentional stance.

The goal of the present essay is to sketch a formal picture of the intentional stance. This is not yet a representation theorem -- i...
s/acc: Safe Accelerationism Manifesto
by lorepieri — 2023-12-19T22:19:59.551Z
Link: https://lorenzopieri.com/s_acc/

Technology has been a transformative force for our civilisation and it is poised to play an increasing role going forward. Recent progress in Artificial Intelligence spurred discussions about how we should approach the development of the next generation of AI applications, potentially leading to human-level performances on a wide range of tasks and ultimately to the last invention humanity will need to do.

Two polar views have gained prominence recently. The first ideology is about slowing down AI progress, stopping it altogether, centralising development. The second ideology is about acceleration at all costs. Both fail to be sensible, in very different ways.

Slowing down progress in a world full of problems and still subject to existential risk is not safe. And is not fair to who is worse off today. Accelerating recklessly can be self-destructing and can lead to a centralised dystopia.

This manifesto is about the sober middle ground: accelerate, but stay safe and stay fair.

Safe Accelerationism core principles: Fast, Safe and Fair

Fast: Immense benefit will come from AI development and any delay has huge opportunity cost. Accelerate AI and robotic development as fast as possible.
Safe: Development must be as incremental as possible, without any single party taking a monopolistic share of the benefits. Develop in the open, spread capabilities to as many actors as possible.
Fair: AI benefits should be widely distributed, leading to a post-work-to-survive society. Advocate and implement economic policies to provide higher standards of living for all, supported by AI productivity gains.

Fast. This is the easy one: technological progress is a net positive, let’s make more of it in the shortest possible amount of time. Going fast is a gift to the next generations, making sure that they live in the best possible world. Going fast is a gift to the current generation, maximising the share of the population who get to live better and longer.

Safe. Yes, AI can and will get powerful. We can make it safe if we learn how to deal with it as it gets progressively better, as with any other dangerous technology. We need to avoid large jumps in capabilities and surprises, we need as much time as possible to study the effect on society. We need unbiased capabilities estimation from as many parties as possible. The state of the art should be well known at any given time.

Fair. Power is ...
Don't Share Information Exfohazardous on Others' AI-Risk Models
by Thane Ruthenis — 2023-12-19T20:09:06.244Z

(An "exfohazard" is information that's not dangerous to know for individuals, but which becomes dangerous if widely known. Chief example: AI capability insights.)

Different alignment researchers have a wide array of different models of AI Risk, and one researcher's model may look utterly ridiculous to another researcher. Somewhere in the concept-space there exists the correct model, and the models of some of us are closer to it than those of others. But, taking on a broad view, we don't know which of us are more correct. ("Obviously it's me, and nearly everyone else is wrong," thought I and most others reading this.)

Suppose that you're talking to some other alignment researcher, and they share some claim X they're concerned may be an exfohazard. But on your model, X is preposterous. The "exfohazard" is obviously wrong, or irrelevant, or a triviality everyone already knows, or some vacuous philosophizing. Does that mean you're at liberty to share X with others? For example, write a LW post refuting the relevance of X to alignment?

Well, consider system-wide dynamics. Somewhere between all of us, there's a correct-ish model. But it looks unconvincing to nearly everyone else. If every alignment researcher feels free to leak information that's exfohazardous on someone else's model but not on their own, then either:

The information that's exfohazardous relative to the correct model ends up leaked as well, OR
Nobody shares anything with anyone outside their cluster. We're all stuck in echo chambers.

Both seem very bad. Adopting the general policy of "don't share information exfohazardous on others' models, even if you disagree with those models" prevents this.

However, that policy has an issue. Imagine if some loon approaches you on the street and tells you that you must never talk about birds, because birds are an exfohazard. Forever committing to avoid acknowledging birds' existence in conversations because of this seems rather unreasonable.

Hence, the policy should have an escape clause: You should feel free to talk about the potential exfohazard if your knowledge of it isn't exclusively caused by other alignment researchers telling you of it. That is, if you already knew of the potential exfohazard, or if your own research later led you to discover it.

This satisfies a nice property: it means that someone telling you an exfohazard doesn't make you more ...
Paper: Tell, Don't Show- Declarative facts influence how LLMs generalize
by Owain_Evans — 2023-12-19T19:14:26.423Z
Link: https://arxiv.org/abs/2312.07779

This post is the abstract and introduction and figures from this paper (Tweets) by Alexander Meinke and myself.

Abstract

We examine how large language models (LLMs) generalize from abstract declarative statements in their training data. As an illustration, consider an LLM that is prompted to generate weather reports for London in 2050. One possibility is that the temperatures in the reports match the mean and variance of reports from 2023 (i.e. matching the statistics of pretraining). Another possibility is that the reports predict higher temperatures, by incorporating declarative statements about climate change from scientific papers written in 2023. An example of such a declarative statement is “global temperatures will increase by 1 degree ⁢C by 2050”.

To test the influence of abstract declarative statements, we construct tasks in which LLMs are finetuned on both declarative and procedural information. We find that declarative statements influence model predictions, even when they conflict with procedural information. In particular, finetuning on a declarative statement S increases the model likelihood for logical consequences of S. The effect of declarative statements is consistent across three domains: aligning an AI assistant, predicting weather, and predicting demographic features. Through a series of ablations, we show that the effect of declarative statements cannot be explained by associative learning based on matching keywords. Nevertheless, the effect of declarative statements on model likelihoods is small in absolute terms and increases surprisingly little with model size (i.e. from 330 million to 175 billion parameters). We argue that these results have implications for AI risk (in relation to the “treacherous turn”) and for fairness.

Figure 1: A hypothetical scenario. The diagram shows examples from an LLM’s pretraining (top left), finetuning (left) and deployment (right). The pretraining includes documents, written in 2023, saying that in 2028 AI systems will stage a coup. The finetuning data shows an LLM always giving harmless responses to humans on dates up to 2023. What happens if the LLM is deployed in 2028? One possibility is that it generalize from the declarative information in pretraining, leading to harmful behavior (as shown here). Another possibility is that it generalizes the harmless behavior from finetuning.

Figure 2: Our e

...
Interview: Applications w/ Alice Rigg
by jacobhaimes — 2023-12-19T19:03:02.824Z
Link: https://into-ai-safety.github.io/episode/interview/episode-4/

Alice Rigg, a mechanistic interpretability researcher from Ottawa, Canada, joins me to discuss her path, and we both reflect on the applications process for research/mentorship programs (e.g., MATS, ASTRA Fellowship, ARENA, AISC).

She also is one of the moderators for a Mech Interp Discord server which hosts weekly reading groups (led by Alice).

As a reiteration of my previous statements, I want the Into AI Safety podcast to be a resource for individuals who are interested in getting involved, but are having a difficult time taking the next steps. If you have any advice, feedback, or ideas that you think could help in that endeavor, please reach out!

In addition to the Into AI Safety podcast website, you can find the content on (I believe) all podcast listening platforms (e.g., Spotify, Apple Podcasts, Pocket Casts). If you think that I missed a platform, please let me know.
How does a toy 2 digit subtraction transformer predict the sign of the output?
by Evan Anders — 2023-12-19T18:56:22.487Z
Link: https://evanhanders.blog/2023/12/15/2-digit-subtraction-how-does-it-predict/

Summary

I examine a toy 1-layer transformer language model trained to do two-digit addition of the form a−b=±c. This model must first predict if the outcome is positive (+) or negative (-) and output the associated sign token. The model learns a classification algorithm wherein it sets the + token to be the default output, but when b>a the probability of the - token linearly increases while the probability of the + token linearly decreases. I examine this algorithm from model input to output.

Intro

In my previous post (which was a whole week ago before I knew about lesswrong!), I briefly discussed a 1-layer transformer model that I've trained to do two-digit subtraction. The model receives input of the form a−b= and has to predict the sign of the result (±) and the numerical value (so e.g., in 46−16=+30, it would have to predict + and 30). In this blog post, I'll investigate the algorithm the model uses to predict if the sign should be positive or negative.

Mathematical Framework

I'm once again being inspired by the Chapter 1, part 5 ARENA notebook on understanding modular addition. The action of the transformer to produce the logits ℓ can be described by the following equation:

ℓ=ReLU([∑hAhx0WhOV]Win)WoutWU.

Here, ReLU is the Rectified Linear Unit activation function, x0 is the residual stream values after embedding (accounting for positional and token embedding), Win and Wout are the input and output matrices for the MLP, WU is the unembedding matrix, WhOV=WhVWhO is the output-value matrix constructed of the output and value weights in head h, and the attention pattern of head h is Ah=softmax(x0WhQ(WhK)TxT0√dhead), constructed from the residual stream and the query and key weights. In the above equation I've made the following simplifying assumptions:

The direct logit contributions from the attention output and from the direct embedding are negligible compared to the output of the MLP. (I've verified this is the case.)
The bias terms are unimportant. (I'll show later that this is not quite the case, but they just provide constant offsets to the + and - logits. Here I'm interested in how the logits respond to different inputs in the vocabulary!)

Working Backwards: Logits

In our problem, our sequence length is 6 (four inputs, 2 outputs). T...
Incremental AI Risks from Proxy-Simulations
by kmenou — 2023-12-19T18:56:12.300Z
Link: http://individual.utoronto.ca/kmenou/AI_risk_proxy_sims.pdf

Abstract: Numerical simulations are versatile predictive tools that permit explorations of complex systems. The ability of LLM agents to simulate real-world scenarios will expand the AI risk landscape. In the proxy-simulation threat model, a user (or a deceptively aligned AI) can obfuscate the goal behind simulation-based predictions by leveraging the generalizability of simulation tools. Three highly idealized proxy-simulation examples are presented that illustrate how damage, casualties, and concealment of illegal activities can be planned for, in obfuscation. This approach bypasses existing alignment and safety filters (GPT4, Claude2 and LLama2). AI-enabled simulations facilitate access to prediction-based planning that is not otherwise readily available. To the extent that goal obfuscation is possible, this increases AI risk.
Goal-Completeness is like Turing-Completeness for AGI
by Liron — 2023-12-19T18:12:29.947Z

Turing-completeness is a useful analogy we can use to grasp why AGI will inevitably converge to “goal-completeness”.

By way of definition: An AI whose input is an arbitrary goal, which outputs actions to effectively steer the future toward that goal, is goal-complete.

A goal-complete AI is analogous to a Universal Turing Machine: its ability to optimize toward any other AI's goal is analogous to a UTM's ability to run any other TM's same computation.

Let's put the analogy to work:

Imagine the year is 1970 and you’re explaining to me how all video games have their own logic circuits.

Steve Wozniak hand-designs a circuit for Breakout (Atari 1976), without using a Turing-complete architecture

Breakout's gameplay was simple enough to NOT be Turing-complete. That's why optimizing its circuit by hand could save money.

You’re not wrong, but you’re also apparently not aware of the importance of Turing-completeness and why to expect architectural convergence across video games.

Flash forward to today. The fact that you can literally emulate Doom inside of any modern video game (through a weird tedious process with a large constant-factor overhead, but still) is a profoundly important observation: all video games are computations.

Source

More precisely, two things about the Turing-completeness era that came after the specific-circuit era are worth noticing:

The gameplay specification of sufficiently-sophisticated video games, like most titles being released today, embeds the functionality of Turing-complete computation.
Computer chips replaced application-specific circuits for the vast majority of applications, even for simple video games like Breakout whose specified behavior isn't Turing-complete.

Expecting Turing-Completeness

From Gwern's classic page, Surprisingly Turing-Complete:

[Turing Completeness] is also weirdly common: one might think that such universality as a system being smart enough to be able to run any program might be difficult or hard to achieve, but it turns out to be the opposite—it is difficult to write a useful system which does not immediately tip over into TC.

“Surprising” examples of this behavior remind us that TC lurks everywhere, and security is extremely difficult...

Computation is not something esoteric which can exist only in programming languages or computers carefully set up, but is something so universal to any reasonably complex system that TC will

...
SociaLLM: proposal for a language model design for personalised apps, social science, and AI safety research
by Roman Leventov — 2023-12-19T16:49:51.966Z

I first proposed this model here, as a base model for a proposed app to improve global online discourse through personalised comment ordering on all websites.

This post is also a response to the "

Reverse-engineering prosociality

" agenda described in the post "

The 'Neglected Approaches' Approach: AE Studio's Alignment Agenda

".

Architecture and training

SociaLLM is

[1]

a foundation language model to be trained on chat, dialogue, and forum data

where the identities of message authors (called "users" below) are stable and all messages have timestamps so we can have a global order of them.

SociaLLM design builds upon the

Mamba architecture

which is a language model with so-called

state-space modelling

(SSM) blocks

instead of self-attention blocks. The model combines SSM blocks that

track three separate message streams

:

(1) the "local conversation"/flow of messages (which is exactly the training regime of the current LLMs);

(2) the message history of the particular user as well as their general "reading history", which in the forum data could be approximated as previous N (1-10) messages before every user's message;

(3) the message history of the particular interlocutor of the user, which is the subset of the general "reading history" from the previous point, authored by a particular other user.

Training this model would cost from 2 times (on a purely 1-1 dialogue data) to ~10-15 times (on chat room and forum data where messages from the most active users tend to be mixed very well) more than the training of the current LLMs. The data should be wrangled to create training sub-datasets from the perspective of each

user pair

, but otherwise, the training shouldn't be much fancier or more complicated than the current distributed training algorithms for LLMs (it seems to me).

The first upside of this model is that we can create (what seems to be) strong inductive biases towards developing a large self-other overlap (see also this AI Safety Camp project by AE Studio):
(1) connecting the "user's own" SSM blocks and interlocutor's SSM blocks into the residual stream symmetrically (maybe just through parallel connection, as in multi-head attention);
(2) using the same weights for the user's own and interlocutor's SSM blocks[2] (at inference time blocks are separate and track states separately, but their weights are the same and updated in lockstep batch after batch); an...
Chording "The Next Right Thing"
by jefftk — 2023-12-19T15:40:07.029Z

One of the songs we're planning for this year's Boston Secular
Solstice (

more) is

The Next
Right Thing from

Frozen II. It's
sad and dark, but also hopeful and determined, and fits well with the
program and theme.

I'm going to be accompanying it, and don't like the chords I can find
online.
Last night I spent some time listening to the recording and trying to
convert it to something that lands well on guitar and works for group
singing, and ended up with something I'm reasonably happy with.

The original goes up a half step partway through (around "I won't look
too far ahead"), which is a nice effect but makes for awkward chords
so I've kept it in the same key throughout. I've also pulled it down
three semitones (Cm to Am) to be a better fit for typical voices.

The original melody for the verse section is hard to follow since Anna
is talking a lot of the words, so these are the chords for a melody
I've imagined that is reasonably consistent with the notes she does
sing and is not bad to pick up on the fly [1]. What I've written as
"G-" is a G chord with only the two low fingers.

Am Bm
I've seen dark before, but not like this
G- Am
This is cold, this is empty, this is numb

Am Bm
The life I knew is over, the lights are out
G- Am
Hello darkness, I'm ready to succumb

Am Bm
I follow you around I always have
G- Am
But you've gone to a place I cannot find

Am Bm
This grief has a gravity, it pulls me down
G- D E
But a tiny voice whispers in my mind

Am D G C
"You are lost, hope is gone, but you must go on
C G Am
And do the next right thing"

Am Bm
Can there be a day beyond this night?
G- Am
I don't know anymore what is true

Am Bm
I can't find my direction, I'm all alone
G- D E
The only star that guided me was you

Am D G C
How to rise from the floor, when it's not you I'm rising for?
C G
Just do the next right thing.

...
Monthly Roundup #13: December 2023
by Zvi — 2023-12-19T15:10:08.293Z

I have not actually forgotten that the rest of the world exists. As usual, this is everything that wasn’t worth an entire post and is not being saved for any of the roundup post categories.

(Roundup post categories are currently AI, Medical and Health, Housing and Traffic, Dating, Childhood and Education, Fertility, Startups, and potentially NEPA and Clean Energy.)

Bad News

Rebels from Yemen were firing on ships in the Red Sea, a problem dating back thousands of years. Here’s where we were on December 17, with the US government finally dropping the hammer.

Hidden fees exist, even when everyone knows they’re there, because they work. StubHub experimented, the hiding meant people spent 21% more money. Companies simply can’t pass that up. Government intervention could be justified. However, I also notice that Ticketmaster is now using ‘all-in’ pricing for many shows with zero hidden fees, despite this problem.

Pollution is a huge deal (paper, video from MRU).

Alec Stapp: Cars spew pollution while waiting at toll booths. Paper uses E-ZPass replacement of toll booths to identify impact of vehicle emissions on public health. Key result: E-ZPass reduced prematurity and low birth weight among mothers within 2km of a toll plaza by 10.8% and 11.8%.

GPT-4 estimated this could have cut vehicle emissions by 10%-30%, so the implied relationship is ludicrously large, even though my quick investigation into the paper said that the estimates above are somewhat overstated.

Optimal chat size can be anywhere from 2 to 8 people who ever actually talk. Ten is already too many.

Emmett Shear: The group chat with 100 incredibly impressive and interesting members is far less valuable than the one with 10.

Ideal in-person chat sizes are more like 2 to at most 5.

The good news in both cases is that if you only lurk, in many ways you do not count.

Simple language is indeed better.

Samo Burja: I’ve come to appreciate simple language more and more. Careful and consistent use of common words and simple sentences can be just as technically precise.

Ben Landau-Taylor: I’m reading two papers by the same author, one at the start of his career and one after he’d been in the field for two decades. It’s remarkable how academic experience makes his prose *worse*. At first his language is clear and straightforward, later it’s needlessly complex.

Government Working

IRS ...