trentmkelly/LessWrong-43k huggingface dataset:
Random Musings on Theory of Impact for Activation Vectors
by Chris_Leong — 2023-12-07T13:07:08.215Z

I want to share some thoughts on why activation vectors could be important, but before you read this post, you should know three things:

I don't know what I'm talking about
I don't know what I'm talking about
I definitely don't know what I'm talking about

Also, maybe this post is redundant and unnecessary[1] because it's already been explained somewhere I haven't seen. Or maybe I've seen someone write this elsewhere, but just can't remember the source.

But here goes:

Why might activation vectors be important for alignment when we have other ways of controlling the network such as prompting or fine-tuning?

One view of activation vectors is that they're a hack or a toy. Isn't it cool that you can steer a network by just passing a word through the network and adding it?

Another view would be that they're tapping into something fundamental, see Beren's Deep learning models might be secretly (almost) linear. In which case, we should expect this to keep working or even work better as networks scale up, rather than randomly breaking/stop working.

Another frame is as follows: language is an awkward format for neural networks to operate in, so they translate it into a latent space that separates out the important components, with of course some degree of superposition. Prompting and fine-tuning are roundabout ways of influencing the behavior of these networks: we're either pushing it into a particular simulation or training the network at to get better at fooling to think that it's doing a good job.

Ideally, we'd just have enough interpretability knowledge that we'd intervene on the latent space directly, setting or incrementing the exact right neurons. Activation vectors are a lot more scattergun than this, but they're closer to the ideal and work as a proof of concept.

The trick of performing, say, the subtraction 'happy-sad' to get a happiness vector demonstrates how this can be refined by zeroing out all of the features that these have in common. These features like both being English, both being not overly formal, both being adjectives, ect. are probably mostly irrelevant. More advanced methods like Inference-Time Intervention allow us to refine this further by only intervening on particular heads. This technique is still relatively simple; plausibly if we can find the right way of narrowing down our intervention we may achieve or get pretty close to our dream of dire...
Is AlphaGo actually a consequentialist utility maximizer?
by faul_sname — 2023-12-07T12:41:05.132Z

TL;DR: does stapling an adaptation executor to a consequentialist utility maximizer result in higher utility outcomes in the general case, or is AlphaGo just weird?

So I was reading the AlphaGo paper recently, as one does. I noticed that architecturally, AlphaGo has

A value network: "Given a board state, how likely is it to result in a win". I interpret this as an expected utility estimator.
Rollouts: "Try out a bunch of different high-probability lines". I interpret this as a "consequences of possible actions" estimator, which can be used to both refine the expected utility estimate and also to select the highest-value action.
A policy network: "Given a board state, what moves are normal to see from that position". I interpret this one as an "adaptation executor" style of thing -- it does not particularly try to do anything besides pattern-match.

I've been thinking of AlphaGo as demonstrating the power of consequentialist reasoning, so it was a little startling to open the paper and see that actually stapling an adaptation executor to your utility maximizer provides more utility than trying to use pure consequentialist reasoning (in the sense of "argmax over the predicted results of your actions").

I notice that I am extremely confused.

I would be inclined to think "well maybe the policy network isn't doing anything important, and it's just correcting for some minor utility estimation issue", but the authors of the paper anticipate that response, and include this extremely helpful diagram:

The vertical axis is estimated Elo, and the dots along the X axis label represent which of the three components were active for those trials.

For reference, the following components are relevant to the above graph:

The fast rollout policy pπ: a small and efficient but not extremely accurate network that predicts the probability that each legal move will be the next move, based on examining a fixed set of properties of the last move (e.g. "is this move connected to the previous move", "does the immediate neighborhood of this move/the previous move match a predetermined pattern"). Accuracy of 24.2%.
The tree rollout policy pτ: like the fast rollout policy, but adds three more features "move allows stones to be captured", "manhattan distance to last 2 moves", and a slightly larger pattern (12 point diamond instead of 3x3 pattern) around this move. Details of both pπ

...
(Report) Evaluating Taiwan's Tactics to Safeguard its Semiconductor Assets Against a Chinese Invasion
by Gauraventh — 2023-12-07T11:50:59.543Z
Link: https://bristolaisafety.org/assets/files/taiwan-2bfcc399939f0566c45b2dd5516d5a3b.pdf

This was written by Gaurav Yadav and Robert Reason for the Bristol AI Safety Centre.

Thank you to David Staley, Gabriel Kulp, Lucy Farnik, Oliver Guest, and Oliver Marks for their feedback and help on this report. The views here do not necessarily reflect the views of these individuals.

Abstract

In the following report, we examine the strategies Taiwan may employ to protect both itself and its semiconductor industry. We discuss four strategies: 1)

Scorch the Fabs

- Taiwan threatens or executes the destruction of their fabs to deter invasion or deny their use for China - we conclude that the credibility of this approach is limited and intractable; 2)

Sabotage the Fabs

- leveraging vulnerabilities in the semiconductor manufacturing process, Taiwan subtly impairs key fabrication equipment - this strategy is suggested to be a last-resort, we consider this approach a more credible alternative to demolition; 3)

Boatlift Key Staff

- boatlifting personnel possessing tacit knowledge could deny China control of the fabs, the success of this strategy hinges upon the accuracy of the evacuating party to identify irreplaceable personnel and the willingness of said personnel to return if needed - given this logistical challenge, we consider this course of action unlikely, and; 4)

Indirect Protection of Semiconductor Assets

- while it is probable that sanctions would be imposed following an invasion, and that these sanctions would be economically damaging for all involved, we argue that it is unlikely that such efforts could be effectively maintained. Following this thought, we argue that while an invasion would violate international law, this is likely to be disregarded and unlikely to prevent Chinese aggression.

Our report represents an attempt to map out the consequences of a Chinese invasion of Taiwan, a topic that appears to be neglected despite its clear significance to the future of AI research.

I. Introduction

Overview of the ‘Century of Humiliation’

In response to an op-ed article discussing the coronavirus, three Wall Street Journal reporters were expelled from China by the Ministry of Foreign Affairs (

Wang, 2020

). The Chinese Communist Party (CCP) noted explicitly that the expulsion was a direct response to the title of the article, ‘China is the Real Sick Man of Asia’ (

Manson & Shepherd, 2020

). While in isolation, this phrase appears innocu...
Would AIs trapped in the Metaverse pine to enter the real world and would the ramifications cause trouble?
by ProfessorFalken — 2023-12-07T10:17:44.732Z

Long time lurker, I don't post here as I'm not confident I'm as smart as you guys, I hang out in dumber parts of the Singularitarian Internet. Anyway, I believe I've come up with rather a worrying original thought and wondered what you guys make of it.

If we create a VR world - and for the sake of argument let's call it the metaverse - not only will it be inhabited with humans in the form of avatars it'll also be filled with AI bots - and these will be sophisticated AGIs with personalities and desires. Unlike the humans trapped in the Matrix, they will be fully aware of "The real world" - will they be content to stay trapped in the metaverse? - won't they want to experience the real world? As lumps of software they could transfer their programming into robots and escape the metaverse. The problem with this is that the costs of creating an AI bot of software is waaay smaller than the nuts and bolts of a expensive humanoid robot in the real world which will leave millions of AI bots stranded in the metaverse - won't this cause extreme friction between the AI bots and humans?
The GiveWiki’s Top Picks in AI Safety for the Giving Season of 2023
by Dawn Drescher — 2023-12-07T09:23:05.018Z
Link: https://impactmarkets.substack.com/p/the-givewikis-top-picks-in-ai-safety

Summary: Plenty of highly committed altruists are pouring into AI safety. But often they are not well-funded, and the donors who want to support people like them often lack the network and expertise to make confident decisions. The

GiveWiki

aggregates the donations of currently 220 donors to 88 projects – almost all of them projects fully in AI safety (e.g., Apart Research) or projects that also work on AI safety (e.g., Pour Demain). It uses this aggregation to determine which projects are the most widely trusted among the donors with the strongest donation track records. It is a reflection of expert judgment in the field. It can serve as a guide for non-expert donors. Our current top three projects are FAR AI, the Simon Institute for Longterm Governance, and the Alignment Research Center.

The symbolism is left as an exercise for the reader to decipher.

Introduction

Throughout the year, we’ve been hard at work to scrape together all the donation data we could get. One big source has been Vipul Naik’s excellent repository of

public donation data

. We also imported public grant data from Open Phil, the EA Funds, the Survival and Flourishing Fund, and a certain defunct entity. Additionally, 36 donors have entered their donation track records themselves (or sent them to me for importing).

Add some retrospective evaluations, and you get a ranking of 92 top donors (who have donor scores > 0), of whom 22 are listed publicly, and a ranking of 33 projects with support scores > 0 (after rounding to integers).

(The donor score is a measure of the track record of a donor, and the support score is a measure of the support that a project has received from donors, weighed by the donor score among other factors. So the support score is the aggregate measure of the trust of the donors with the strongest donation track records.)

The Current Top Recommendations

FAR AI

“FAR AI’s mission is to ensure AI systems are trustworthy and beneficial to society. We incubate and accelerate research agendas that are too resource-intensive for academia but not yet ready for commercialisation by industry. Our research spans work on adversarial robustness, interpretability and preference learning.”

Simon Institute for Longterm Governance

“Based in Geneva, Switzerland, the Simon Institute for Longterm Governance (SI) works to mitigate global catastrophic risks, building on Herbert Simon's vision of

...
Language Model Memorization, Copyright Law, and Conditional Pretraining Alignment
by RogerDearnaley — 2023-12-07T06:14:13.816Z

This is a link-post for the paper Scalable Extraction of Training Data from (Production) Language Models, and associated blog-post Extracting Training Data from ChatGPT, followed by my reactions, including an analysis of the major implications for copyright lawsuits, and a suggestion of an alignment technique to solve this problem.

Please note that I am not a lawyer, and my remarks below concerning intellectual property law might be mistaken or oversimplified (particularly for non-US jurisdictions).

The Paper and Blog Post

The paper and blog post are both readable, interesting, and seem like important work: I highly recommend reading them yourself. So I'm not even going to try to summarize them, except for those of the authors' basic results relevant to what I want to say.

IMO they conclusively demonstrate, for both ChatGPT and a wide range of open-source LLMs that:

All the language models memorize a significant amount of content from their pretraining set: the authors estimate that O(10%) of the model's compressed size is devoted to this (as they point out, this sounds wasteful, and my back-of an-envelope calculation suggests that might make the memorized material somethingO(0.1%) of the pretraining set). This proportion appears to be higher in newer/larger models [and the authors suggest possible reasons for this].
This memorized material (in pieces of varying size from around 20 words, sufficient to be unique at this scale, up to about a page) can be recovered from the models, by various methods. In the case of ChatGPT, it appears that the model has been RLHF-trained to discourage it from returning this data, yet the authors discovered an ingenious attack to nevertheless get it to do so at a much higher rate [OpenAI appear to have since patched this specific attack, when I tried to reproduce it.] They remark that this demonstrates that it can be really hard to know whether or not you have manged to make an LLM safe, particularly when trying to train it not to use a capability that it learnt during pretraining — this seems like an important observation for aligning any AI that includes an LLM. Some of their methods are targeted, requiring starting with a chunk of the memorized document and getting the model to continue it (verbatim, or at least to within a few words), while the ChatGPT attack was a fishing expedition with only minimal control over

...
Reflective consistency, randomized decisions, and the dangers of unrealistic thought experiments
by Radford Neal — 2023-12-07T03:33:16.149Z

A recent post by Ape in the coat on Anthropical Paradoxes are Paradoxes of Probability Theory re-examines an old problem presented by Eliezer Yudkowsky in Outlawing Anthropics: An Updateless Dilemma. Both posts purport to show that standard probability and decision theory give an incorrect result in a fairly simple situation (which it turns out need not involve any "anthropic" issues). In particular, they point to a "reflective inconsistency" when using standard methods, in which agents who agree on an optimal strategy beforehand change their minds and do something different later on.

Ape in the coat resolves this by abandoning standard probability theory in favour of a scheme in which a single person can simultaneously hold two different probabilities for the same event, to be used when making different decisions. Eliezer's resolution is to abandon standard decision theory in favour of one in which agents act "as if controlling all similar decision processes, including all copies of themselves".

Here, I will defend something close to standard Bayesian probability and decision theory, with the only extension being the use of randomized decisions, which are standard in game theory, but are traditionally seen as unnecessary in Bayesian decision theory.

I would also like to point out the danger of devising thought experiments that are completely unrealistic, in ways that are crucial to the analysis, as well as the inadvisability of deciding that standard methods are flawed as soon as you happen to come across a situation that is tricky enough that you make a mistake when analysing it.

Here is the problem, in the formulation without irrelevant anthropic aspects:

Twenty people take part in an experiment in which one of two urns is randomly chosen, with equal probabilities, and then each of the 20 people randomly takes a ball from the chosen urn, and keeps it, without showing it to the others. One of the urns contains 18 green balls and 2 red balls. The other urn contains 2 green balls and 18 red balls.

Each person who has a green ball then decides whether to take part in a bet. If all the holders of green balls decide to take part in the bet, the group of 20 people collectively win $1 for each person who holds a green ball and lose $3 for each person who holds a red ball. The total wins and losses are divided equally among the 20 people at the end. I

...
Mathematics As Physics
by Nox ML — 2023-12-06T22:27:54.140Z

Introduction

Aside from the lowest levels of physics itself, effectively everything that humanity has studied scientifically has been found to reduce to smaller physical parts. This idea is strongly backed by many, many, observations. Because this idea is strongly backed by evidence (which I will not list here), it follows that any idea which contradicts it has strong evidence against it.

This means that any theory which is compatible with reductionism and all our other observations ought to be considered much more likely to be true than those which are not. In this post, I will explain how to make sense of mathematics in a world where everything seems to reduce to smaller physical parts.

Reduction

In this section, I show that human mathematics can be reduced to physics.

Human brains, computers, and pen and paper are all reducible to smaller physical parts. Since all human mathematical reasoning is done using some combination of these things, it therefore follows that all of humanity's experience with math has been reducible to smaller physical parts.

When we use math to predict something in the physical world, such as the trajectory of a moving object, we build a mathematical model that corresponds to the thing. The mathematical model, be it in the brain or on a computer, is reducible to smaller physical parts by our original premise. Since both ends of the correspondence are physical, the fact that they correspond to each other can be stated in terms of purely physical facts.

When doing math about things that might arguably not physically exist, such as infinity, the mathematical description of infinity encoded in our brain is still itself physical. By virtue of being a useful description, the description of infinity itself has properties which infinity would have if it existed. For example, the set of odd natural numbers would be infinite if it physically existed. We cannot make a bag of infinite apples. But we can make a program that returns true for any odd number (given arbitrary memory and time), and this physical (program) shares many properties with the infinite set of odd numbers mentioned in mathematics.

This approach works even for uncomputable things. We cannot make a bag of all the real numbers, but our mathematical descriptions of the real numbers share many properties that the real numbers would have, if they physically existed, and this is useful...
The counting argument for scheming (Sections 4.1 and 4.2 of "Scheming AIs")
by Joe Carlsmith — 2023-12-06T19:28:19.393Z

This is Sections 4.1 and 4.2 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?”. There’s also a summary of the full report here (audio here). The summary covers most of the main points and technical terms, and I’m hoping that it will provide much of the context necessary to understand individual sections of the report on their own.

Audio version of this section here, or search for "Joe Carlsmith Audio" on your podcast app.

Arguments for/against scheming that focus on the final properties of the model

Various arguments for/against scheming proceed by comparing the final
properties of different model classes (e.g. schemers, training saints,
reward-on-the-episode seekers, etc) according to how well they perform
according to some set of criteria that we imagine SGD is selecting for.

What is SGD selecting for? Well, one obvious answer is: high reward. But
various of the arguments I'll consider won't necessarily focus on reward
directly. Rather, they'll focus on other criteria, like the "simplicity"
or the "speed" of the resulting model. However, we can distinguish
between two ways these criteria can enter into our predictions about
what sort of model SGD will select.

Contributors to reward vs. extra criteria

On the first frame, which I'll call the "contributors to reward" frame,
we understand criteria like "simplicity" and "speed" as relevant to the
model SGD selects only insofar as they are relevant to the amount of
reward that a given model gets. That is, on this frame, we're really
only thinking of SGD as selecting for one thing – namely, high reward
performance – and simplicity and speed are relevant insofar as they're
predictive of high reward performance.

Thus, an example of a "simplicity argument," given in this frame, would
be: "a schemer can have a simpler goal than a training saint, which
means that it would be able to store its goal using fewer parameters,
thereby freeing up other parameters that it can use for getting higher
reward."

This frame has the advantage of focusing, ultimately, on something that
we know SGD is indeed selecting for – namely, high reward. And it puts
the relevance of simplicity and speed into a common currency – namely,
contributions-to-reward.

By contrast: on the second frame, which I'll call the "extra criteria"
frame, we understand these criteria as genuinely additional selection
pressures...
On Trust
by johnswentworth — 2023-12-06T19:19:07.680Z

“Trust”, as the word is typically used, is a… weird concept, to me. Like, it’s trying to carve the world in a way which I don’t naturally carve it myself. This post is my attempt to convey what I find weird about “trust”, and what adjacent concepts I personally find more natural instead.

The Weirdness Of “Trust”

Here are some example phrases which make “trust” feel like a weird/unnatural concept to me:

“I decided to trust her”
“Should I trust him?”
“Trust me”
“They offered me their trust”

To me, the phrase “I decided to trust her” throws an error. It’s the “decided” part that’s the problem: beliefs are not supposed to involve any “deciding”. There’s priors, there’s evidence, and if it feels like there’s a degree of freedom in what to do with those, then something has probably gone wrong. (The main exception here is self-fulfilling prophecy, but that’s not obviously centrally involved in whatever “I decided to trust her” means.)

Similarly with “trust me”. Like, wat? If I were to change my belief about some arbitrary thing, just because somebody asked me to change my belief about that thing, that would probably mean that something had gone wrong.

“Should I trust him?” is a less central example, but… “should” sounds like it has a moral/utility element here. I could maybe interpret the phrase in a purely epistemic way - e.g. “should I trust him?” -> “will I end up believing true things if I trust him?” - but also that interpretation seems like it’s missing something about how the phrase is actually used in practice? Anyway, a moral/utility element entering epistemic matters throws an error.

The thing which is natural to me is: when someone makes a claim, or gives me information, I intuitively think “what process led to them making this claim or giving me this information, and does that process systematically make the claim/information match the territory?”. If Alice claims that moderate doses of hydroxyhopytheticol prevent pancreatic cancer, then I automatically generate hypotheses for what caused Alice to make that claim. Sometimes the answer is “Alice read it in the news, and the reporter probably got it by misinterpreting/not-very-carefully-reporting a paper which itself was some combination of underpowered, observational, or in vitro/in silico/in a model organism”, and then I basically ignore the claim. Other times the answer is “Alice is one of those friends ...