trentmkelly/LessWrong-43k huggingface dataset:
My thoughts on AI and personal future plan after learning about AI Safety for 4 months
by Ziyue Wang — 2023-08-31T15:32:40.629Z

In this post, I want to distill some of my thoughts about AI and my future plan regarding it according to what I have learnt during the past 4 months.

About my personal background: My background lies in Financial Engineering and Mathematics, and I used to get motivated by making AI more powerful. I decided to shift my focus to AI Safety after reading and getting convinced by the idea inside Preventing an AI-related catastrophe. During the past 4 months, I finished

Alignment Course from AI Safety Fundamentals
6 out of 9 Topics inside Deep Learning Curriculum
4 out of 5 Chapters inside ARENA
a paper from LLM Evals Hackathon
And many readings across AI Alignment Forum, LessWrong and different AI Safety focused team/org/company.

I kept those related posts/learning in my website if you are interested.

Overall, I think the future of AI is promising but in the same dangerous if they cannot align with our intention. It is like what has been discussed in Precipice and I want to take the chance to help it.

It is promising because it already demonstrated superior capability, and can be used to improve people's life quality. The application field can be robotics, education, health system, productivity boost and etc.

But AI can get misaligned if we aren't paying enough attention to it. Here, according to Paul Christiano, alignment means intent alignment:

AI A is aligned with an operator H when A is trying to do what H wants it to do.

The focus here is "trying", which means it is on intention/goal/motivation level rather than behavior level. A model can behave like aligned but it tries to deceive human or human actually couldn't see its defect due to the high complexity of future tasks.

In the argument above, we assumed it can have "intention" and we can understand how it comes from by using optimizer. When we train a model, we use an optimizer which usually tries to minimize certain loss function or maximize reward function by adjusting the weight of the model. For example, GPT is pre-trained by minimizing the next token prediction loss. So the goal of it, at that stage, is simply trying to make the next word make sense given what it sees in the past. Its goal is not about aligning with human instruction and making human happy or productive. Hence we need to do further fine-tuning, like Reinforcement From Human Feedback (RLHF), to make it align with ...
Which Questions Are Anthropic Questions?
by dadadarren — 2023-08-31T15:15:39.964Z

I will try to keep this shot, just want to use some simple problems to point out what I think is a commonly overlooked point in anthropic discussions.

1. The Room Assignment Problem

You are among 100 people waiting in a hallway. The hallway leads to a hundred rooms numbered from 1 to 100. All of you are knocked out by a sleeping gas and each put into a random/unknown room. After waking up, what is the probability that you are in room No. 1?

This is just an ordinary probability question. All room numbers are symmetrical, the answer is simply 1%. It is also easy to imagine taking part in similar room-assigning experiments a great number of times, the relative fraction of you waking up in room No.1, or any other number, would approach 1%.

It is safe to say this is not an anthropic question. Even for people with the metaphysical stance that all probabilities are anthropic, it is undeniable that we have been solving such questions successfully without any anthropic considerations.

2. The Incubator

An incubator enters the hallway. It will enter room No.1 and creat a person in it then does the same for the other 99 rooms. It turns out you are one of the people the incubator has just created. You wake up in a room and is made aware of the experiment setup. What is the probability that you are in room No.1?

While this question may seem trivial, it is an anthropic question. In ordinary problems like Question 1, an event refers to a unique experiment outcome, a distinct possible world. But this incubator problem, objectively speaking, is deterministic. Knowing the exact process from a god's eye view still leaves uncertainty because the uncertainty comes from the fact that you are unsure of your own location in this only world, e.g. which physical person is the subjective self. This kind of same-world event is the crux of anthropic paradoxes. E.g. the sleeping beauty problem is special because of the two awakenings both in the Tails world.

Also worth noting the frequentist model is much trickier to imagine. While the incubator has no problem performing this job ten thousand times, and creating a million people in the process, it seems absurd to say you are one of the people created in each experiment, that you are ten thousand people. Obviously, this is not a run-of-the-mill question.

3. Incubator + Room Assignment

This time the incubator creats 100

...
The Tree of Life, and a Note on Job 
by Bill Benzon — 2023-08-31T14:03:45.603Z

This is another one cross-posted from my New Savanna blog. It's old, 11 years (and most of it is even older than that). It's an interpretation of the Book of Job, which is not at all the sort of thing I'd ordinarily post here. I certainly don't see much of that kind of thing going on. But I'm in an experimental and playful mood, so why not?

FWIW and BTW, I'm neither Christian nor Jewish, so this post isn't an expression of religious belief.

Here goes:

More or less on Michael Sporn’s recommendation, I’ve just seen Terrence Malick’s The Tree of Life. While I’m collecting my thoughts on this trying, tedious, and rewarding film, I’ll let Michael’s thoughts stand-in for many of mine:

The film starts with a vision of god that moves beyond to a patriarchal dominated family in Waco, Texas. The suggestion of a death leads us back to god and the creation of the earth. From protozoa to dinosaur to the birth of a child, this filmmaker exudes absolute love for every organism he can show us on screen. Yet, right from the dinosaurs onward he creates an ominous tone in this male-dominated power hungry environment. You’re always expecting something terrible to happen in the hands of the children who push the film forward. This is a film that technically has a new way of presenting itself almost through an impressionistic vision. The whispered narration and dialogue mix and blend into one; the sun streamed backlit late-afternoon interiors create a whispered visual to match.

It was the phrase “from protozoa to dinosaur” that got me.

The film is that of a mystic. I know nothing of Malick, though it seems he was born in the Bible belt and studied philosophy, so I don’t know if he is really mystic, but then, what’s really in that question? I once told my draft board that I was a mystic. Really? Really. That’s what comes through in the film: “exudes absolute love for every organism he can show us on screen.”
The film’s explicit religiosity bugged me in the beginning. Am I going to have to say something about this in my review? Am I going to have to declare, for example, whether or not I’m a believer? And then it didn’t bug me, not for the last hour or so. I just forgot about it.
I’ll have more to say about the film later, but I just wanted to dig out some old notes, from 25 years ago, on Job.

To understand the story of Job we must first reject the ending, in which Job regains a...
Cleaning a SoundCraft Mixer
by jefftk — 2023-08-31T13:20:02.568Z

In

Kingfisher

Cecilia uses a

condenser on her
fiddle but combines it with effects pedals. This is

a bit awkward to
get working, and we've been using a

Soundcraft
Notepad mini-mixer. Unfortunately we had some issues recently
where adjusting the input gain would make horrible scratchy noises:

I thought I'd need to take the mixer apart to clean it, but a bit of
looking online turned up suggestions to gently pull the knob off the
face of the board, spray a contact treatment, and turn the spindle to
work the treatment in. This seemed surprising to me: would this
really distribute the cleaner where it needed to be? But I ordered
some Deoxit
5% and followed the instructions on the can, and it worked
perfectly!

Now I don't hear any noise at all when adjusting the gain.

I now have an almost full can of this stuff, so if you're in Boston
and are having similar issues you're welcome to come borrow
it.

(Even though the mixer is now working well Cecilia isn't planning to
go back to it: she bought a Voco Loco,
which is a box built for exactly this purpose. It should be more
robust and reliable, with fewer knobs and switches that can get
knocked into the wrong position.)
AI #27: Portents of Gemini
by Zvi — 2023-08-31T12:40:05.631Z

By all reports, and as one would expect, Google’s Gemini looks to be substantially superior to GPT-4. We now have more details on that, and also word that Google plans to deploy it in December, Manifold gives it 82% to happen this year and similar probability of being superior to GPT-4 on release.

I indeed expect this to happen on both counts. This is not too long from now, but also this is AI #27 and Bard still sucks, Google has been taking its sweet time getting its act together. So now we have both the UK Summit and Gemini coming up within a few months, as well as major acceleration of chip shipments. If you are preparing to try and impact how things go, now might be a good time to get ready and keep your powder dry. If you are looking to build cool new AI tech and capture mundane utility, be prepared on that front as well.

Table of Contents

Introduction.
Table of Contents. Bold sections seem most relatively important this week.
Language Models Offer Mundane Utility. Summarize, take a class, add it all up.
Language Models Don’t Offer Mundane Utility. Not reliably or robustly, anyway.
GPT-4 Real This Time. History will never forget the name, Enterprise.
Fun With Image Generation. Watermarks and a faster SDXL.
Deepfaketown and Botpocalypse Soon. Wherever would we make deepfakes?
They Took Our Jobs. Hey, those jobs are only for our domestic robots.
Get Involved. Peter Wildeford is hiring. Send in your opportunities, folks!
Introducing. Sure, Graph of Thoughts, why not?
In Other AI News. AI gives paralyzed woman her voice back, Nvidia invests.
China. New blog about AI safety in China, which is perhaps a thing you say?
The Best Defense. How exactly would we defend against bad AI with good AI?
Portents of Gemini. It is coming in December. It is coming in December.
Quiet Speculations. A few other odds and ends.
The Quest for Sane Regulation. CEOs to meet with Schumer, EU’s AI Act.
The Week in Audio. Christiano and Leahy give talks, Rohit makes his case.
Rhetorical Innovation. Some relatively promising attempts.
Llama No One Stopping This. Meta to open source all Llamas no matter what.
No One Would Be So Stupid As To. Bingo, sir.
Aligning a Smarter Than Human Intelligence is Difficult. Davidad has a plan.
People Are Worried About AI Killing Everyone. Roon, the better critic we need.
Other People Are Not As Worried About AI Killing Everyone. Consciousness?
The Wit and Wisdom of Sam Altman.

...
Long-Term Future Fund Ask Us Anything (September 2023)
by Linch — 2023-08-31T00:28:13.953Z
Link: https://forum.effectivealtruism.org/posts/ee8Pamunhqabucwjq/long-term-future-fund-ask-us-anything-september-2023

The Long-Term Future Fund has an AMA up on the Effective Altruism Forum.

There’s no real deadline for questions, but let’s say we have a soft commitment to focus on questions asked on or before September 8th.

I'd prefer centralizing the questions to one place. If you don't want to post of the forum, for whatever reason (ideological opposition to EA, got banned before, lost your password and don't want to make a new comment), I'm happy to for you to comment here and for me to repost them on the Forum.

Our donation link is here.

Related posts

LTFF and EAIF are unusually funding-constrained right now
EA Funds organizational update: Open Philanthropy matching and distancing
Long-Term Future Fund: April 2023 grant recommendations
What Does a Marginal Grant at LTFF Look Like?
Asya Bergal’s Reflections on my time on the Long-Term Future Fund
Linch Zhang’s Select examples of adverse selection in longtermist grantmaking
Responses to apparent rationalist confusions about game / decision theory
by Anthony DiGiovanni — 2023-08-30T22:02:12.218Z

I’ve encountered various claims about how AIs would approach game theory and decision theory that seem pretty importantly mistaken. Some of these confusions probably aren’t that big a deal on their own, and I’m definitely not the first to point out several of these, even publicly. But collectively I think these add up to a common worldview that underestimates the value of

technical work to reduce risks of AGI conflict

. I expect that smart agents will likely avoid catastrophic conflict overall—it’s just that the specific arguments for expecting this that I’m responding to here aren’t compelling (and seem overconfident).

For each section, I include in the footnotes some examples of the claims I’m pushing back on (or note whether I’ve primarily seen these claims in personal communication). This is not to call out those particular authors; in each case, they’re saying something that seems to be a relatively common meme in this community.

Summary:

The fact that conflict is costly for all the agents involved in the conflict, ex post, doesn’t itself imply AGIs won’t end up in conflict. Under their uncertainty about each other, agents with sufficiently extreme preferences or priors might find the risk of conflict worth it ex ante. (

more

)
Solutions to collective action problems, where agents agree on a Pareto-optimal outcome they’d take if they coordinated to do so, don’t necessarily solve bargaining problems, where agents may insist on different Pareto-optimal outcomes. (

more

)
We don’t have strong reasons to expect AGIs to converge on sufficiently similar decision procedures for bargaining, such that they coordinate on fair demands despite committing under uncertainty. Existing proposals for mitigating conflict given incompatible demands, while promising, face some problems with incentives and commitment credibility. (

more

)
The

commitment races problem

is not just about AIs making commitments that fail to account for basic contingencies. Updatelessness (or conditional commitments generally) seems to solve the latter, but it doesn’t remove agents’ incentives to limit how much their decisions depend on each other’s decisions (leading to incompatible demands). (

more

)
AIs don’t need to follow acausal decision theories in order to (causally) cooperate via conditioning on each other’s source code. (

more

)
Most supposed examples of Newcomblike pro

...
Invulnerable Incomplete Preferences: 
A Formal Statement
by SCP — 2023-08-30T21:59:36.186Z

Produced as part of the SERI ML Alignment Theory Scholars Program - Summer 2023 Cohort. My thanks to Eric Chen, Elliott Thornley, and John Wentworth for invaluable discussion and comments on earlier drafts. All errors are mine.

This article presents a few theorems about the invulnerability of agents with incomplete preferences. Elliott Thornley’s (2023) proposed approach to the AI shutdown problem relies on these preferential gaps, but John Wentworth and David Lorell have argued that they make agents play strictly dominated strategies.[1] I claim this is false.

Summary

Suppose there exists a formal description of an agent that willingly shuts down when a certain button is pressed. Elliott Thornley’s (2023) Incomplete Preference Proposal aims to offer such a description. It’s plausible that, for it to constitute a promising approach to solving the AI shutdown problem, this description also needs to (i) permit the agent to be broadly capable and (ii) assure us that the agent will remain willing to shut down as time passes. This article formally derives a set of sufficient conditions for an agent with incomplete preferences to satisfy properties relevant to (i) and (ii).

A seemingly relevant condition for an agent to be capably goal-directed is that it avoids sequences of actions that foreseeably leave it worse off.[2] I will say that an agent satisfying this condition is invulnerable. This is related to two intuitive conditions. The weaker one is unexploitability: that the agent cannot be forcibly money pumped (i.e., compelled by its preferences to sure loss). The stronger condition is opportunism: that the agent never accepts sure losses or foregoes sure gains.[3]

To achieve this, I propose a dynamic choice rule for agents with incomplete preferences. This rule, Dynamic Strong Maximality (DSM), requires that the agent consider the available plans that are acceptable at the time of choosing and, among these, pick any plan that wasn’t previously strictly dominated by any other such plan. I prove in section 2 that DSM-based backward induction is sufficient for invulnerability, even under uncertainty.

Having shown that incompleteness does not imply that agents will pursue dominated strategies, I consider the issue of whether DSM leads agents to act as if their preferences were complete. Section 3 begins with a conceptual argument suggesting that DSM-based cho...
Report on Frontier Model Training
by YafahEdelman — 2023-08-30T20:02:46.317Z
Link: https://docs.google.com/document/d/1TsYkDYtV6BKiCN9PAOirRAy3TrNDu2XncUZ5UZfaAKA/edit?usp=sharing

Understanding what drives the rising capabilities of AI is important for those who work to forecast, regulate, or ensure the safety of AI. Regulations on the export of powerful GPUs need to be informed by understanding of how these GPUs are used, forecasts need to be informed by bottlenecks, and safety needs to be informed by an understanding of how the models of the future might be trained. A clearer understanding would enable policy makers to target regulations in such a way that they are difficult for companies to

circumvent with only technically compliant GPUs

, forecasters to avoid focus on

unreliable metrics

, and technical research working on mitigating the downsides of AI to understand

what data models might be trained on

.

This doc is built from a collection of smaller docs I wrote on a bunch of different aspects of frontier model training I consider important. I hope for people to be able to use this document as a collection of resources, to draw from it the information they find important and inform their own models.

I do not expect this doc to have a substantial impact on any serious AI labs capabilities efforts - I think my conclusions are largely discoverable in the process of attempting to scale AIs or for substantially less money than a serious such attempt would cost. Additionally I expect major labs already know many of the things in this report.

Acknowledgements

I’d like to thank the following people for their feedback, advice, and discussion:

James Bradbury, Software Engineer, Google DeepMind
Benjamin Edelman, Ph.D. Candidate, Harvard University
Lukas Finnveden, Research Analyst, Open Philanthropy Project
Horace He, Software Engineer, PyTorch/Meta
Joanna Morningstar, Chief Scientific Officer, Nanotronics
Keller Scholl, Ph.D. Candidate, Pardee RAND Graduate School
Jaime Sevilla, Director, Epoch
Cody Wild, Research Engineer, Google

Index

Cost Breakdown of ML Training

Estimates the costs of training a frontier (state of the art) model, drawing on leaks and analysis. Power usage is a small portion of the cost, GPUs are likely a slim majority.

Why ML GPUs Cost So Much

ML GPUs are expensive largely because of their communication and memory capabilities - not because of their processing power. NVIDIA’s best gaming GPU provides greater ML processing power than the GPU used to train GPT-4, for only a tenth the price. Note that NVIDIA’s near monopoly plausibly ...
An adversarial example for Direct Logit Attribution: memory management in gelu-4l
by Can — 2023-08-30T17:36:59.034Z
Link: https://arxiv.org/abs/2310.07325

Please check out

our notebook

for figure recreation and to examine your own model for clean-up behavior.

Produced as part of

ARENA 2.0

and the SERI ML Alignment Theory Scholars Program - Spring 2023 Cohort

Fig 5: Correlation between DLA of writer head and DLA of [clean-up heads output dependent on V-composition with the writer head]. The negative correlation coefficient r suggests that output of a writer node is consistently removed from the residual stream by subsequent clean-up nodes. See section Implication for Direct Logit Attribution.

Overview

In this post, we provide concrete evidence for memory management or clean-up in a 4-layer transformer

gelu-4l

. We show examples where Direct Logit Attribution (DLA) is misleading because it does not account for the clean-up.

In the Introduction, we define what we mean by clean-up behavior and provide a quick recap on DLA. In the section Evidence for Clean-up Behavior we identify specific nodes that write and remove information from the residual stream. Based on what we learned about the clean-up, we select prompts that result in misleading DLA results in the Implication for Direct Logit Attribution section.

Introduction

Clean-up behavior

Previously in

A Mathematical Framework for Transformer Circuits

the authors suggested a mechanism for memory management and speculated it can occur because of high demand on residual stream bandwidth. We define clean-up behavior, in which attention heads and MLPs (which we collectively call nodes) clear information from the residual stream that is only used in early layers of the network.

We characterize clean-up behavior as four steps during a forward pass:

A writer node or embedding writes a specific direction to the residual stream
Subsequent nodes use this direction for further computation
A clean-up node clears this direction by writing its negative value to the residual stream
The direction is used in the later part of the model, in one or two of the following ways:

Subsequent nodes write to the direction, using this free subspace for communicating information different to the information which has been cleaned up
The unembedding matrix reads the direction, which directly affects the output logits

Throughout this post, we analyze the removal of the writer node output (steps 1 - 3). In future work, we will address how the cleared space is used by...