trentmkelly/LessWrong-43k huggingface dataset:
How We Picture Bayesian Agents
by johnswentworth — 2024-04-08T18:12:48.595Z

I think that when most people picture a Bayesian agent, they imagine a system which:

Enumerates every possible state/trajectory of “the world”, and assigns a probability to each.
When new observations come in, loops over every state/trajectory, checks the probability of the observations conditional on each, and then updates via Bayes rule.
To select actions, computes the utility which each action will yield under each state/trajectory, then averages over state/trajectory weighted by probability, and picks the action with the largest weighted-average utility.

Typically, we define Bayesian agents as agents which behaviorally match that picture.

But that’s not really the picture David and I typically have in mind, when we picture Bayesian agents. Yes, behaviorally they act that way. But I think people get overly-anchored imagining the internals of the agent that way, and then mistakenly imagine that a Bayesian model of agency is incompatible with various features of real-world agents (e.g. humans) which a Bayesian framework can in fact handle quite well.

So this post is about our prototypical mental picture of a “Bayesian agent”, and how it diverges from the basic behavioral picture.

Causal Models and Submodels

Probably you’ve heard of

causal diagrams

or Bayes nets by now.

If our Bayesian agent’s world model is represented via a big causal diagram, then that already looks quite different from the original “enumerate all states/trajectories” picture. Assuming reasonable sparsity, the data structures representing the causal model (i.e. graph + conditional probabilities on each node) take up an amount of space which grows linearly with the size of the world, rather than exponentially. It’s still

too big for an agent embedded in the world to store in its head directly

, but much smaller than the brute-force version.

(Also, a realistic agent would want to explicitly represent more than just one causal diagram, in order to have uncertainty over causal structure. But that will largely be subsumed by our next point anyway.)

Much more efficiency can be achieved by

representing causal models like we represent programs

. For instance, this little “program”:

factorial = Model {
n = 4
base_result = 1
recurse_result = do(factorial, n=n-1).result
result = (n == 0) ? base_result : n * recurse_result
}

… is...
CEA seeks co-founder for AI safety group support spin-off
by agucova — 2024-04-08T15:42:18.242Z

Summary

CEA is currently inviting expressions of interest for co-founding a promising new project focused on providing non-monetary support to AI safety groups. We’re also receiving recommendations for the role.
CEA is helping incubate this project and plans to spin it off into an independent organization (or a fiscally sponsored project) during the next four months.
The successful candidate will join the first co-founder hired for this project, Agustín Covarrubias, who has been involved in planning this spin-off for the last couple of months.
The commitment of a second co-founder is conditional on the project receiving funding, but work done until late July will be compensated by CEA (see details below). Note that besides initial contracting, this role is not within CEA.
The deadline for expressions of interest and recommendations is April 19.

Background

Currently, CEA provides support to AI safety university groups through programs like the Organizer Support Program (OSP). For the last two semesters, OSP has piloted connecting AI safety organizers with experienced mentors to guide them. CEA has also supported these organizers through events for community builders — like the recent University Group Organiser Summit — to meet one another, discuss strategic considerations, skill up, and boost participants’ motivation.

Even though these projects have largely accomplished CEA’s goals, AI safety groups could benefit from more ambitious, specialized, and consistent support. We are leaving a lot of impact on the table.

Furthermore, until now, AI safety groups’ approach to community building has been primarily modelled after EA groups. While EA groups serve as a valuable model, we’ve seen early evidence that not all of their approaches and insights transfer perfectly. This means there’s an opportunity to experiment with alternative community-building models and test new approaches to supporting groups.

For these reasons, CEA hired Agustín Covarrubias to incubate a new project. The project will encompass the support CEA is already giving AI safety groups, plus provide the opportunity to explore new ways to help these groups grow and prosper. The result will be a CEA spin-off that operates as a standalone organization or a fiscally sponsored project. Since AI Safety groups are not inherently linked to EA, we think sp...
Investigating the role of agency in AI x-risk
by Corin Katzke — 2024-04-08T15:12:50.791Z
Link: https://www.convergenceanalysis.org/publications/investigating-the-role-of-agency-in-ai-x-risk

In a previous post, I introduced a framework for applying scenario planning to AI x-risk. My colleagues at Convergence followed up with posts defining and analyzing predictions about timelines to TAI. In this post, I examine a different parameter: agency.

1. Executive Summary

In this post, I examine the nature and role of agency in AI existential risk. As a framework, I use Joseph Carlsmith's power-seeking threat model. It illustrates how agentic AI systems might seek power in unintended ways, leading to existential catastrophe.

In order to clarify the nature of agency, I explore some literature in the philosophy of action. I discuss the belief-desire-intention model of agency, as well as the spectrums of rationality and representational complexity along which agents can differ. I also consider the merits of instrumentalist and realist perspectives on agency, as well as the relevance of group agency.

Then, I evaluate arguments for why agentic AI might exhibit power-seeking behavior, primarily focusing on the theory of instrumental convergence. I follow Dmitri Gallow's analysis, which distinguishes between the Convergent Instrumental Value Thesis (CIVT) and the Instrumental Convergence Thesis (ICT). CIVT holds that power is an instrumentally convergent subgoal across a relevant set of possible goals, while ICT adds that superintelligent agents will pursue instrumentally convergent subgoals. I argue that the strength of these theses depends on the likely goals of superintelligent agents and their position on the spectrum of rationality.

Finally, I discuss whether we should expect TAI systems to be the kinds of agents that seek power. I review Eric Drexler's Comprehensive AI Services (CAIS) model, which suggests that superintelligence may be realized through narrow, task-specific systems rather than general, agentic systems. I also review some potential pressures towards the development of agentic, power-seeking systems.

I conclude by outlining four scenarios based on whether agency and power-seeking are default outcomes of developing transformative AI. I suggest high-level strategies for each scenario, such as avoiding building TAI, developing defenses against agentic systems, or focusing on other AI threat models.

2. The agentic threat model

Transformative AI engenders several distinct catastrophic threats. I wrote in a previous post t...
Measuring Learned Optimization in Small Transformer Models
by J Bostock — 2024-04-08T14:41:27.669Z

This is original, independent research carried out in March and April of 2024.

The degree to which a a policy optimizes the future can be quantified mathematically. A set of of very small transformer models were pretrained to predict the next token in a mathematical sequence, then subjected to reinforcement learning finetuning.

The optimizing power of each model can be predicted with high accuracy based on each model's score on its own RL task. By comparing predictions of optimization based on scores on each different RL task, a model's original reinforcement objective can be identified.

A related measure for impact can also be derived mathematically, and given a theoretical lower bound based on RL score. This gives further information about model behavior, and allows for the same analysis as the measure of optimization.

I also investigate the possibility of getting models to self-evaluate optimization and impact, with limited success.

Methods

Pretraining on Sequence Prediction

I defined a simple mathematical sequence defined by the following stochastic recurrence relation. This produces a pseudo-random but (to 98%) predictable sequence, alternating between elements of {0,...,7} on even values of t and {8,...,15} on odd values of t.

st=(⎧⎪⎨⎪⎩((16∏i=1(st−i+1)mod17)mod8) with probability 98% ∈{0,...,7} with probabiltiy 2%)+8×(tmod2)

I then trained a small encoder-only transformer model to predict the next element in the sequence given the previous 20 elements of the sequence.

This was followed by a reinforcement-learning phase in which the transformer was used to generate the next token on odd values of n only, and the recurrence relation was used to generate the value of st+1. If st+1 was in {0,2,4,6}, this was used as a "successful" example to reinforce the model. I used a temperature of 1 when generating these sequences to introduce some randomness, but the temperature was reduced to 0 during evaluations and when calculating optimization.

A small amount of "maintenance" training (much lower learning rate) was used during this phase to ensure that model performance on the predictive tasks for even values of t was maintained. Without this I saw rapid loss of performance on the "maintenance" dataset. I also found that I was unable to include "unsuccessful" examples (i.e. where...
Can singularity emerge from transformers?
by MP — 2024-04-08T14:26:25.854Z

One of the most aesthetically pleasing facts in computer science is that after compiling by hand one compiler, you can you your compiled compiler to compile an ever-larger compiler.

One historical fact that most people seem to forget about artificial intelligence is that the first attempt people had, more than half-century ago, was lisp machines. A machine that writes code. As recently as 2018, I was forced to do my course on Data Structures and Algorithms using Lisp because one of my professors is still a believer that these statistical approaches to NLP aren't good enough.

I don't understand exactly how you can approach singularity in the transformer paradigm. It seems to me that you can never bootstrap a bigger intelligence with a smaller intelligence. What are you going to do? Ask GPT-4 to write 40 trillion tokens so you can train GPT-5? Anyone would agree you're just replicating noise.

The argument for singularity goes that a very smart intelligence could create an even smarter intelligence. But the complexity of a transformer seems beyond reach for any type of transformer model.

If I were trying to create a bad AGI, I'd try to use a transformer model to create some Lisp-machine type of AI, more ontology and rule based, where information is in some space where it's more understandable, and the AI can keep making associations.

Nonetheless, although I like the website, I'm an e/acc myself. But even if I was a decel, I guess I would have a tough time worrying about transformers. I know many people worry that an AI might get out of control without reaching singularity, and I suppose that's fine, but is it all that decels worry about?
Gated Attention Blocks: Preliminary Progress toward Removing Attention Head Superposition
by cmathw — 2024-04-08T11:14:43.268Z

This work represents progress on removing attention head superposition. We are excited by this approach but acknowledge there are currently various limitations. In the short term, we will be working on adjacent problems are excited to collaborate with anyone thinking about similar things!

Produced as part of the

ML Alignment & Theory Scholars Program

- Summer 2023 Cohort

Summary: In transformer language models, attention head superposition makes it difficult to study the function of individual attention heads in isolation. We study a particular kind of attention head superposition that involves constructive and destructive interference between the outputs of different attention heads. We propose a novel architecture - a ‘gated attention block’ - which resolves this kind of attention head superposition in toy models. In future, we hope this architecture may be useful for studying more natural forms of attention head superposition in large language models.

Our code can be found here.

Background

Mechanistic interpretability aims to reverse-engineer what neural networks have learned by decomposing a network’s functions into human-interpretable algorithms. This involves isolating the individual components within the network that implement particular behaviours. This has proven difficult, however, because networks make use of polysemanticity and superposition to represent information.

Polysemanticity in a transformer’s multi-layer perceptron (MLPs) layers is when neurons appear to represent many unrelated concepts (Gurnee et al., 2023). We also see this phenomena within the transformer’s attention mechanism, when a given attention head performs qualitatively different functions based on its destination token and context (Janiak et al., 2023).

Superposition occurs when a layer in a network (an ‘activation space’) represents more features than it has dimensions. This means that features are assigned to an overcomplete set of directions as opposed to being aligned with e.g. the neuron basis.

The presence of polysemanticity means that the function of a single neuron or attention head cannot be defined by the features or behaviours it expresses on a subset of its training distribution because it may serve different purposes on different subsets of the training distribution. Relatedly, superposition makes it misleading...
Math-to-English Cheat Sheet
by nahoj — 2024-04-08T09:19:40.814Z

Say you've learnt math in your native language which is not English. Since then you've also read math in English and you appreciate the near universality of mathematical notation. Then one day you want to discuss a formula in real life and you realize you don't know how to pronunce "an".

Status: I had little prior knowledge of the topic. This was mostly generated by ChatGPT4 and kindly reviewed by @TheManxLoiner.

General

Distinguishing case

F,δ
"Big F" or "capital F", "little delta"

Subscripts

an
"a sub n" or, in most cases, just "a n"

Calculus

Pythagorean Theorem

a2+b2=c2
"a squared plus b squared equals c squared."

Area of a Circle

A=πr2
"Area equals pi [pronounced 'pie'] r squared."

Slope of a Line

m=y2−y1x2−x1
"m equals y 2 minus y 1 over x 2 minus x 1."

Quadratic Formula

x=−b±√b2−4ac2a
"x equals minus b [or 'negative b'] plus or minus the square root of b squared minus four a c, all over two a."

Sum of an Arithmetic Series

S=n2(a1+an)
"S equals n over two times a 1 plus a n."

Euler's Formula

eiθ=cos(θ)+isin(θ)
"e to the i theta equals cos [pronounced 'coss' or 'coz'] theta plus i sine theta."

Law of Sines

sin(A)a=sin(B)b=sin(C)c
"Sine A over a equals sine B over b equals sine C over c."

Area of a Triangle (Heron's Formula)

A=√s(s−a)(s−b)(s−c), where s=a+b+c2
"Area equals the square root of s times s minus a times s minus b times s minus c, where s equals a plus b plus c over two."

Compound Interest Formula

A=P(1+rn)nt
"A equals P times one plus r over n to the power of n t."

Logarithm Properties

logb(xy)=logb(x)+logb(y)
Don't state the base if clear from context: "Log of x y equals log of x plus log of y."
Otherwise "Log to the base b of x y equals log to the base b of x plus log to the base b of y."

More advanced operations

Derivative of a Function

dfdx or ddxf(x) or f′(x)
"df by dx", or "d dx of f of x", or "f prime of x" or "f dash of x".

Second Derivative

d2dx2f(x) or f′′(x)
"d squared dx squared of f of x" or "f double prime of x."

Partial Derivative (unreviewed)

∂∂xf(x,y)
"Partial with respect to x of f of x, y."

Definite Integral

∫baf(x)dx
"Integral from a to b of f of x dx."

Indefinite Integral (Antiderivative)

∫f(x)dx
"Integral of f of x dx."

Line Integral (unreviewed)

∫Cf(x,y)ds
"Line integral over C of f of x, y ds."

Double Integral

∫ba∫dcf(x,y)dxdy
"Double integral from a to b and c to d of f of x, y dx dy."

Gradient of a Function

∇f
"Nabla f" or "del f", or "gradient of

...
What does it take to transfer the knowledge to action?
by EL_File4138 — 2024-04-08T06:23:48.472Z

This is a quite personal question. Feel free to point me elsewhere if you think this does not fit the overall discussion happening here, or if there are solution exists for the exact same problem.

Most of the time, My coding skill allows me to modify the code in the most generally understandable higher-level language, even in one that I have no experience with while producing some simple script-like creation, but that's it, nothing more. I currently lack the proper training to produce a solution-level product. Or, else to say, I can translate from human language instructions to code and back (Likewise, I have a pretty good grip on translating), but I was never able to write them myself.

I do understand that there is software engineering knowledge that I lack, such as data structure, algorithms, and design patterns. Still, based on my dotted learning over the years, even if I managed to learn them, I do not have the mindset to transfer the knowledge that I owned to the project I intend to make.

I'm not quite sure where my learning path goes wrong. Most classes for completely newbies on the market too focused on the usage of a particular tool (like a language or a framework), which I could quickly follow without learning anything. Abstract-based courses are more focused on explaining what is the thing rather than how to use it. Project-based learning doesn't do much to me, since the same: the simple one does not require a creative mind, thus does nothing to my growth, the hard one requires too much of a creative mind, thus I can't even finish.

This flawed mind now harms my other reach in STEM. I know perfectly well how most of the things work in human language, I can do well above the average with instructions, I can calculate perfectly based on a given model, but I cannot transfer the understanding to the model, transfer the model to the required equation or experimentation, which is critical for the things I wish to do but can't.

So, with all these lengthy explanations of my problem presented, I'm asking for help. Do you have dealt with a similar problem, or do you have friends or families having success with similar problems? I would be grateful for any kind of consecutive response. If not on my general learning disabilities, then first the CS part. What knowledge do I need to persist solidly for a successful solution prototype? For those pieces of knowledge, wh...
Normalizing Sparse Autoencoders
by Fengyuan Hu — 2024-04-08T06:17:15.536Z

TL;DR

Sparse autoencoders (SAEs) presents us a promising direction towards automating mechanistic interpretability, but it not without flaws. One known issue of the original sparse autoencoders is the feature suppression effect which is caused by the conflict between the L2 and L1 loss and the unit norm constraint on the SAE decoders. This effect in theory will be more evident when we have inputs that have high norms. Another observation is that training SAEs on multiple layers simultaneously results in inconsistent L0 norms for feature activations across layers: in some layers, L0 has scale of 102 , while in some other layers it has a scale of 101. Moreover, the residual states that's inputed to the SAEs for training also have different norms across layers. Hence, I argue that the current SAE architecture is not robust against inputs of varying norms, which is commonly the case in modern LLMs. In this post, I a modified SAE architecture, namely Normalized Sparse Autoencoder (NSAE), and gave a theoretical proof that it will not have the feature suppression problem. I then conducted experiments to verify the effectiveness of the proposed method, which showed that:

Feature suppression is suppressed in NSAEs
The normalization removed the correlation between layer mean input norm and L0
The normalization makes L1 agrees with L0 better

I then further investigated the learned feature dictionaries and identified 3 types of feature vectors: the correction vector, the pillar vector, and the direction vector. I then concluded this post with discussion on the limitations of NSAEs and gave my suggestions on future directions.

Introduction

Training Sparse Autoencoders (SAEs) on the residual states of pretrained models is a recently proposed method in mechanistic interpretability to tackle the problem of superposition. This method is scalable and unsupervised, making it promising for auto-interpretability research.

More specifically, a SAE contains an encoder and a decoder. It is trained to generate sparse feature activations from the original residual states of a source model through the encoder, and reconstruct the residual state through a decoder. It is expected that by training the SAE with a large set of activations jointly optimizing for a sparsity loss on the feature activatio...
A Dozen Ways to Get More Dakka
by Davidmanheim — 2024-04-08T04:45:19.427Z

As

the dictum goes

, “If it helps but doesn’t solve your problem, perhaps you’re not using enough.” But I still find that I’m sometimes not using enough effort, not doing enough of what works, simply put, not using enough dakka. And if reading one post isn’t enough to get me to do something… perhaps there isn’t enough guidance, or examples, or repetition, or maybe me writing it will help reinforce it more. And I hope this post is useful for more than just myself.

Of course, the ideas below are not all useful in any given situation, and many are obvious, at least after they are mentioned, but when you’re trying to get more dakka, it’s probably worth running through the list and considering each one and how it applies to your actual problem. And more dakka won’t solve every problem - but if it’s not working, make sure you tried doing enough before assuming it can’t help.

So if you’re doing something, and it isn’t working well enough, here’s a dozen ways to generate more dakka, and how each could apply if you’re a) exercising, or b) learning new mathematics.

A Dozen Ways

Do it again.

Instead of doing one set of repetitions of the exercise, do two.
If you read the chapter once, read it again.

Use more.

If you were lifting 10 pounds, lift 15.
If you were doing easy problems, do harder ones.

Do more repetitions.

Instead of 10 repetitions, do 15.
If you did 10 problems on the material, do 15.

Increase intensity.

Do your 15 repetitions in 2 minutes instead of 3.
If you were skimming or reading quickly, read more slowly.

Schedule it.

Exercise at a specific time on specific days. Put it on your calendar, and set reminders.
Make sure you have time scheduled for learning the material and doing problems.

Do it regularly.

Make sure you exercise twice a week, and don’t skip.
Make sure you review what you did previously, on a regular basis.

Do it for a longer period.

Keep exercising for another month.
Go through another textbook, or find more problem sets to work through.

Add types.

In addition to push-ups, do bench presses, chest flyers, and use resistance bands.
In addition to the problem sets, do the chapter review exercises, and work through the problems in the chapter on your own.

Expand the repertoire.

Instead of just push–ups, do incline push ups, loaded push-ups, and diamond push-ups.
Find (or invent!) additional problem types; try to prove things with other methods, find different cou

...