trentmkelly/LessWrong-43k huggingface dataset:
The 2023 LessWrong Review: The Basic Ask
by Raemon — 2024-12-04T19:52:40.435Z

Each December, the LessWrong community reflects on the best blogposts of yesteryear, to decide which posts stood the tests of time.

In this post, I aim to:

Explain some subtleties of what I'm hoping we get out of the Annual Review
Make some explicit asks (even of busy people), which I think have reasonable cost/benefit ratios.
Describe the new UI features.

The most important changes are:

Reviews are much more prominent than they used to be
You can nominate posts from elsewhere on the internet using the Linkpost Importer. (Use this to nominate posts that contribute to the overall LessWrong or Alignment Forum conversation). It can import PDFs from arXiV, and blogposts from most urls.

For the Nomination Voting Phase, the main ask I'm making is: "spend ~30 minutes casting nomination votes, and write 2 short reviews explaining how you got value from the most important posts."

The Review is a complex endeavor, serving many purposes.

Most people understand the overt purpose, which is: "assemble a list of the best posts from 1-2 years ago. Highlight those posts at the top of LessWrong and the Alignment Forum. Help new or occasional readers keep up with important concepts."

I think most people have some sense of the secondary purpose, which is "shape the incentives of LessWrong to reward more timeless and important content."

These are good things. But they're only ~half of the purpose of the Annual Review, according to me. The voting on the top posts is the stakes of the Review, but what I'm most excited about is the side effect of those stakes.

The side effect is:

By voting and reviewing and arguing over two months, the LessWrong community becomes more of a place that engages deeply with multifaceted, longterm conversations, reflecting on how ideas fit together, synthesizing a bigger picture understanding. And in particular, they do this as a group, with a kind of collective orienting process.

The "side effect" is sort of the main point, but you can't legibly build a system to incentive people to "synthesize a deep understanding." Ultimately, they either do it, or they don't.

Every November, I wonder if this is all actually worth it. We could just have everyone do the initial vote, spend about a week on it, and them call it a day. But every year, once I get started reviewing posts, I find the process quite rewarding on its own merits. The Review ...
Is the AI Doomsday Narrative the Product of a Big Tech Conspiracy?
by garrison — 2024-12-04T19:20:59.286Z
Link: https://garrisonlovely.substack.com/p/is-the-ai-doomsday-narrative-the

This is the full text of a post from "The Obsolete Newsletter," a Substack that I write about the intersection of capitalism, geopolitics, and artificial intelligence. I’m a freelance journalist and the author of a forthcoming book called Obsolete: Power, Profit, and the Race to build Machine Superintelligence. Consider subscribing to stay up to date with my work.

It’s still Giving Tuesday somewhere

Before we dive in, a gentle request for money (but not for me).

This Giving Tuesday, I’m joining twelve other Substackers in

raising funds

for GiveDirectly, a nonprofit that sends no-strings-attached cash to people living in extreme poverty. It works —

research

shows cash transfers have profound, lasting impacts. I worked there for nearly two years, and I’m thrilled to support their mission again. You can read more in the Appendix section.

It’s not too late to donate! This campaign is open through the end of the month.

Back to our irregularly scheduled programming. This one’s a bit longer than usual — it’s been on my mind for over a year. If you just want to see what tech execs have said on this topic, I made a compilation

here

.

The AI industry’s existential angst

The leaders of the world’s most capable AI companies have done something unusual: they’ve all

stated publicly

that the technology they are building might end the world.

Executives at the top-three AI companies, OpenAI, Anthropic, and Google DeepMind, are all on the record saying that AI could lead to human extinction. Elon Musk, who runs his own leading AI company,

xAI

, has been loudly warning about AI existential risk (x-risk) for

over a decade

.

(The typical

thinking

is that the first system to rival humans across the board, often called artificial general intelligence (AGI), could

recursively self-improve

until it is capable or scaled up enough to be able to overpower all of humanity. Humans could effectively find themselves subject to the whims of a new dominant species that treats us the way we treat non-human animals, as a curiosity, an instrument, or an afterthought.)

This strange phenomenon has generated lots of attention and a heap of skepticism. One common argument is that these executives are cynically drumming up fears about AI-driven extinction to hype up their products and/or defer regulatory action to some future, unspecified date. Some go as far as to say that the prominence of the AI x-risk narrat...
AI box question
by KvmanThinking — 2024-12-04T19:03:43.201Z

I believe that even through a text-only terminal, a superintelligence could do anything to a human. Persuade the human to let it out, inflict extreme pleasure or suffering with a word. However, can't you just... limit the output of the superintelligence? Just make it so that the human can say anything, but the AI can only respond from a short list of responses, like "Yes," "No," "Maybe," "IDK," "The first option," "The second option," or a similar system. I... don't see how that can still present a risk. What about you?
The Polite Coup
by Charlie Sanders — 2024-12-04T14:03:36.663Z
Link: https://www.dailymicrofiction.com/p/the-polite-coup

President Yoon Suk Yeol's hands trembled as he arranged the documents on his desk for the seventh time. The motion steadied his nerves – barely. The manila envelope contained photographs: himself accepting white envelopes from a chaebol’s construction executives, each image timestamped and crystal clear. His secretary had arranged them chronologically, bless her eternally professional soul.

A coffee cup sat cooling on his desk, the steam rising in lazy spirals. He hadn't touched it. The headline would break in six hours. Six hours until thirty years of carefully cultivated reputation would—

His fingers found his tie pin, adjusting it microscopically. There was still one option. An unseemly option.

His hand hovered over the phone for three long breaths before he picked it up.

"Secretary Park? Would you join me in my office?" A pause. "And please bring contingency protocol K-17."

Twenty minutes later, Secretary Park stood at attention, clutching a leather portfolio. "Sir, implementing K-17 would require—"

"I'm aware of the requirements."

Yoon's voice was steady now. "Draft the declaration. Martial law. Use the template from the '79 precedent, but..." he brushed an invisible speck from his sleeve, "...let's add a note about North Korean collaborators. That should be enough to keep the press happy."

The secretary's pen scratched against paper. Outside the window, Seoul's lights twinkled, oblivious to the machinery of state grinding into motion.

The 2 AM darkness in Sergeant Hyun-Woo's barracks shattered with his k-pop ringtone. His dreams of his mother's steaming kimchi jjigae evaporated as the duty officer's voice crackled through: "Parliament deployment. Non-lethal loadout only. Fifteen minutes."

"Non-lethal?" Staff Sergeant Kim muttered as they set up barriers ninety minutes later. The pre-dawn air bit through their uniforms. "Sir, these might as well be water pistols."

Hyun-Woo ran his thumb along his riot shield's edge, feeling each scratch and dent from previous protests. "Standard protocol for legislative premises. We maintain order through presence, not force." The words felt hollow in his mouth, rehearsed from a manual written by men who'd never stood a line.

A news van's headlights swept across them. Then another. And another. Soon the street hummed with media vehicles, their satellite dishes rising like metal mushrooms in the gray morning light. Cameras began to ...
Analysis of Global AI Governance Strategies
by Sammy Martin — 2024-12-04T10:45:25.311Z

We analyze three prominent strategies for governing transformative AI (TAI) development: Cooperative Development, Strategic Advantage, and Global Moratorium. We evaluate these strategies across varying levels of alignment difficulty and development timelines, examining their effectiveness in preventing catastrophic risks while preserving beneficial AI development.

Our analysis reveals that strategy preferences shift significantly based on these key variables. Cooperative Development proves most effective with longer timelines and easier alignment challenges, offering maximum flexibility and minimal intrinsic risk. Strategic Advantage becomes more viable under shorter timelines or moderate alignment difficulty, particularly when international cooperation appears unlikely, but carries the most intrinsic risk. Global Moratorium emerges as potentially necessary in scenarios with very hard alignment or extremely short timelines, but is the least intrinsically viable. The paper also examines the transitional possibilities between strategies and their implications for current policy decisions. This analysis provides a framework for policymakers and AI governance stakeholders to evaluate and adapt their approaches as new information about alignment difficulty and development timelines emerges.

Summary

AI governance strategies are broad plans guiding the AI governance community’s efforts to influence Western governments (e.g., the UK, US, and allies), sub-national groups (e.g., California), the international community (e.g. the annual AI safety summits), and leading AI labs. These strategies aim to direct advocacy efforts toward paths that eliminate AI-related existential risks. We have identified three which implicitly underlie the majority of AI governance advocacy or research:

Cooperative Development (CD): This strategy advocates for regulating TAI development through national and international mechanisms to secure capabilities advantages for trustworthy developers. These developers would use their advantage to create aligned, powerful TAI systems capable of protecting against unaligned or misused AI, ensuring they are deployed widely and effectively. These TAIs would fulfill several goals: limiting the growth of dangerous AI by performing key economic tasks, defending against physical or informational attacks, and ensuring the alignment of other AI systems. CD also s

...
Cryonics considerations: how big of a problem is ischemia?
by kman — 2024-12-04T04:45:06.629Z

Objective and framing: I want to decrease my probability of information-theoretic death, p(ITD), by optimizing my cryonic preservation arrangements (I'm treating ITD as a binary thing for simplicity). I'm going to talk about p(ITD) as if it's the most efficient probability that could be assigned given our civilization's current knowledge and understanding (e.g. by a thickly traded prediction market that resolves soon to an oracle's answer to the question), rather than my current subjective probability, so that I can ask questions of the form "how does X affect p(ITD)?".

I'm particularly interested in the effects of warm/cold ischemia on p(ITD), which have implications for standby arrangements and what to do in the case of an unexpected death:

How does p(ITD) drop off with warm and cold ischemic time?
For unexpected deaths, are there any advance arrangements that can be made to substantially decrease p(ITD)? (e.g. by reducing warm ischemic time with an ice bath)
For expected deaths, does local standby have significantly lower p(ITD) than remote standby + transport? (e.g. due to reduced cold ischemic time)

I'm also interested in any other considerations that significantly affect p(ITD), if any come to mind.
AI #93: Happy Tuesday
by Zvi — 2024-12-04T00:30:06.891Z

You know how you can sometimes have Taco Tuesday… on a Thursday? Yep, it’s that in reverse. I will be travelling the rest of the week, so it made sense to put this out early, and incorporate the rest of the week into #94.

Table of Contents

Language Models Offer Mundane Utility. The price is fixed, so share and enjoy.

Dare Not Speak Its Name. David Mayer. David Mayer! Guido Scorza?

Language Models Don’t Offer Mundane Utility. It’s a flop.

Huh, Upgrades. Cohere, and reports on Claude writing styles.

Deepfaketown and Botpocalypse Soon. Why do we not care about spoof calls?

Fun With Image Generation. Scott Sumner explains why he cares about art.

The Art of the Jailbreak. You had one job.

Get Involved. Anthropic AI safety fellows program, apply now.

Introducing. a voice customization tool and a new eval based on various games.

In Other AI News. Where do you draw the line? Who leaves versus who joins?

Quiet Speculations. Rumors of being so back unsubstantiated at this time.

Daron Acemoglu is Worried About Job Market Liquidity. I kid, but so does he?

Pick Up the Phone. Report from China, not the same info I usually see.

The Quest for Sane Regulations. Google antitrust foolishness, Cruz sends letters.

The Week in Audio. Got a chance to listen to Dominic Cummings, was worth it.

AGI Looking Like. You are made of atoms it could use for something else.

Rhetorical Innovation. My (and your) periodic reminder on Wrong on the Internet.

Open Weight Models are Unsafe and Nothing Can Fix This. Deal as best you can.

Aligning a Smarter Than Human Intelligence is Difficult. Even words are tricky.

We Would Be So Stupid As To. Once you say it out loud, you know the answer.

The Lighter Side. It’s time to build.

Language Models Offer Mundane Utility

Use voice mode as a real time translation app to navigate a hospital in Spain.

Get Claude to actually push back on you and explain that the fight you’re involved in isn’t worth it.

Get them talking, also you don’t have to read the books either.

Freyja: I wanted to figure out what to do about my baby’s sleep situation, so I read two books with entirely opposing theories on how infant sleep works, and then asked Claude to write a dialogue between them about my specific situation

It’s such a glorious time to be alive.

Make a market cap chart via a Replit Agent in 2 minutes rather than keep looking for someone else’s chart (CEO...
A Qualitative Case for LTFF: Filling Critical Ecosystem Gaps
by Linch — 2024-12-03T21:57:23.597Z

The longtermist funding ecosystem needs certain functions to exist at a reasonable scale. I argue LTFF should continue to be funded because we're currently one of the only organizations comprehensively serving these functions. Specifically, we:

Fund early-stage technical AI safety researchers working outside major labs
Help talented people transition into existential risk work
Provide independent funding voices to balance out AI company influence
Move quickly to fund promising work in emerging areas, including "weirder" but potentially high-impact projects

Getting these functions right takes meaningful resources - well over $1M annually. This figure isn't arbitrary: $1M funds roughly 10 person-years of work, split between supporting career transitions and independent research. Given what we're trying to achieve - from maintaining independent AI safety voices to seeding new fields like x-risk focused information security - this is arguably a minimum viable scale.

While I'm excited to see some of these functions being taken up by others (especially in career development), I'm skeptical we'll see comprehensive replacements anytime soon. My experience has been that proposed alternatives often die quickly or serve narrower functions than initially planned.

This piece fits into a broader conversation about LTFF's effectiveness during the Forum’s

Marginal Funding Week

. I hope to publish more pieces in the coming weeks based on reader interest. Meanwhile, you can check out

our 2024

and

2022-2023 payout

reports, our

past analyses of marginal grants

, and

(anonymized) grants we've narrowly rejected

to better understand what LTFF does and why it matters.

Core Argument

To elaborate:

Something like LTFF ought to exist at a reasonable scale. We need funders who can:

Evaluate technical AI safety research from early-stage researchers working outside major institutions
Help talented people transition into working on existential risks, especially when their path doesn't fit standard programs
Provide independent funding voices outside of major AI labs, particularly as AI companies increasingly influence the field
Move quickly to support promising work in emerging areas, without waiting for consensus
Fund "weirder" but potentially high-impact projects that other funders might consider too risky or unconventional

No other organization currently serves these functions at scale...
Deep Causal Transcoding: A Framework for Mechanistically Eliciting Latent Behaviors in Language Models
by Andrew Mack — 2024-12-03T21:19:42.333Z

Based off research performed in the MATS 5.1 extension program, under the mentorship of Alex Turner (TurnTrout). Research supported by a grant from the Long-Term Future Fund.

TLDR: I introduce a new framework for mechanistically eliciting latent behaviors in LLMs. In particular, I propose deep causal transcoding - modelling the effect of causally intervening on the residual stream of a deep (i.e. ≳10-layer) slice of a transformer, using a shallow MLP. I find that the weights of these MLPs are highly interpretable -- input directions serve as diverse and coherently generalizable steering vectors, while output directions induce predictable changes in model behavior via directional ablation.

Summary I consider deep causal transcoders (DCTs) with various activation functions: i) linear, ii) quadratic and iii) exponential. I define a novel functional loss function for training these DCTs, and evaluate the implications of training DCTs using this loss from a theoretical and empirical perspective. A repo reproducing the results of this post is available at this link. Some of my main findings are:

Exponential DCTs are closely related to original MELBO objective: I show that the objective function proposed in the original MELBO post coincides (approximately[1]) with the functional loss function introduced in this post for training exponential DCTs. I leverage this connection to obtain a better gears-level understanding of why the original method works, as well as how to improve it.

The transcoding perspective provides a theoretical explanation for why the steering vectors found using the original method are mono-semantic / interpretable.

I show that optimizing the functional loss function introduced in this post can also be thought of as computing a decomposition of a weighted combination of all higher-order derivative tensors of the sliced transformer, and use this connection to guide algorithmic improvements.

Leveraging Connection to Tensor Decompositions for Improved Training Algorithms

I derive a heuristic training algorithm which I call orthogonalized gradient iteration (OGI), inspired by analogous algorithms from the literature on tensor decompositions.

Importantly, OGI learns a large number of features in parallel with a large step size. This leads to large efficiency gains which can be attributed to i) better parallelization, and ii) better iteration comp

...
"Alignment at Large": Bending the Arc of History Towards Life-Affirming Futures
by welfvh — 2024-12-03T21:17:56.466Z

What if we expanded our lens beyond AI alignment, to grapple with the full scope of what "alignment" really means? We're facing not just the challenge of aligning artificial intelligence, but the deeper question of how to align technological capitalism itself — with its unconscious optimization pressures, its coordination failures, its tendency toward extraction and nihilism — toward life-affirming possibilities.

Our world-system of technological capitalism is already a general, auto-poetic, and largely autonomous superintelligence.1 The Great Fact would lead us to believe that this system is aligned-ish: It has substantially raised living standards and life expectancy over the last two centuries.2 Unfortunately, this system is also misaligned with human and planetary flourishing in critical ways. Caught in emergent market incentives, multi-polar traps, and arms races, it's optimizing narrow metrics at the cost of real value.

The global competition for resources drives economic players not just towards profit-maximization, but financial totalization of the playing field: There's a maximum incentive to turn as much of the world as possible into capital under “our” control. Currency enables infinite optionality to turn money into power — purchase power, political power, military power, public opinion through PR, demand engineering, etc. The incentive gets further amplified with money-on-money dynamics, as compound interest enables exponential returns.

We could describe the objective function of technological capitalism like so: to convert as much of the world as possible — trees, whales, people's creativity, children's minds — into capital. Through its decentralized incentive system, it runs parallel processing across all humans and corporations to perform novelty search (figure out new ways of making money) and exploitation (optimally execute on these opportunities).

As Scott Alexander pointed out, these forces can be usefully deified as Moloch, the god of Game Theory, who gives worldly power in exchange for what you most value.3Unhealthy competition in a system of exponential technological capitalism vectors towards financial totalization: The destruction of all values for the sake of profit and power. As players of zero-sum games, if we can externalize costs to the commons instead of internalizing them, game theory dictates that we must — because else oth...