trentmkelly/LessWrong-43k huggingface dataset:
Consequentialism is a compass, not a judge
by Neil  — 2024-04-13T10:47:44.980Z

Tl;dr: Consequentialism works as a compass for your actions, not as a judge of moral character.

The compass and the judge

A woman steps onto a crowded bus, trips on a sitting man's outstretched foot, and breaks her arm. The Everett branches split: in one world, the man looks down and laughs evilly; in the other, he wakes up with a jerk, looks down, gasps, and apologizes profusely for leaving his foot in the alleyway as he slept.

There’s clearly a difference between both men. Even when the consequence of their action was the same—breaking someone’s arm—their intention changes the moral calculation dramatically. If I had to hang around one of these men, I'd prefer the latter.[1]

This intuition pump makes that obvious enough. But then people (as in "people I've met") will think of the thought experiment and recoil at the prospect of consequentialism; they think consequentialism condemns both men equally. But this misunderstands what consequentialism is for. It’s not supposed to judge how evil people are when they trip women. If you try inserting an event like this one into the consequentialist calculation machine, it will spit, sputter, and cough out a cloud of black smoke. Consequentialism is a compass; it points to [what it thinks is] the optimal moral direction.[2] The compass might tell you that you should remember to tuck your feet in on a crowded bus, because that'll reduce the probability of negative consequences. It won't tell you how moral someone who forgot to do this is. [3]

The consequences of someone's actions are nonetheless partial evidence of their morality. If you discover that embezzled funds have been building up on Bob's bank account, that's evidence Bob is an unethical guy—most people who embezzle funds are unethical. But then you might discover that, before he was caught and the money confiscated, Bob was embezzling funds to build an orphanage. The consequences haven't changed, but Bob's final (unresolved) intentions are attenuating circumstances. If I had to hang around with either your typical fund-embezzler or Bob, I would pick Bob.

Takeaways

There's an asymmetry in ethics, where you judge your own decisions based on criteria you don't hold other people to. I'll just quote HPMoR:

The boy didn't blink. "You could call it heroic responsibility, maybe," Harry Potter said. "Not like the usual sort. It means tha

...
Carl Sagan, nuking the moon, and not nuking the moon
by eukaryote — 2024-04-13T04:08:50.166Z
Link: https://eukaryotewritesblog.com/2024/04/12/nuking-the-moon/

In 1957, Nobel laureate microbiologist Joshua Lederberg and biostatician J. B. S. Haldane sat down together imagined what would happened if the USSR decided to explode a nuclear weapon on the moon.

The Cold War was on, Sputnik had recently been launched, and the 40th anniversary of the Bolshevik Revolution was coming up – a good time for an awe-inspiring political statement. Maybe they read a recent United Press article about the rumored USSR plans. Nuking the moon would make a powerful political statement on earth, but the radiation and disruption could permanently harm scientific research on the moon.

What Lederberg and Haldane did not know was that they were onto something – by the next year, the USSR really investigated the possibility of dropping a nuke on the moon. They called it “Project E-4,” one of a series of possible lunar missions.

What Lederberg and Haldane definitely did not know was that that same next year, 1958, the US would also study the idea of nuking the moon. They called it “Project A119” and the Air Force commissioned research on it from Leonard Reiffel, a regular military collaborator and physicist at the University of Illinois. He worked with several other scientists, including a University of Chicago grad student named Carl Sagan.

“Why would anyone think it was a good idea to nuke the moon?”

That’s a great question. Most of us go about our lives comforted by the thought “I would never drop a nuclear weapon on the moon.” The truth is that given a lot of power, a nuclear weapon, and a lot of extremely specific circumstances, we too might find ourselves thinking “I should nuke the moon.”

Reasons to nuke the moon

During the Cold War, dropping a nuclear weapon on the moon would show that you had the rocketry needed to aim a nuclear weapon precisely at long distances. It would show off your spacefaring capability. A visible show could reassure your own side and frighten your enemies.

It could do the same things for public opinion that putting a man on the moon ultimately did. But it’s easier and cheaper:

As of the dawn of ICBMs you already have long-distance rockets designed to hold nuclear weapons
Nuclear weapons do not require “breathable atmosphere” or “water”
You do not have to bring the nuclear weapon safely back from the moon.

There’s not a lot of English-language information online about the USSR E-4 program to nuke the moon. The main re...
Barcoding LLM Training Data Subsets. Anyone trying this for interpretability?
by right..enough? — 2024-04-13T03:09:23.436Z

Main point: seeking information on this potential strategy for improving neural network interpretability, alignment, and reliability.
Segmenting training data and injecting unique "barcoding" tokens into each data subset. These unique tokens should occasionally show up in generated outputs at a frequency correlated with the importance of each training subset to the output, providing insights into model decision making.
I've been unable to find out who (if anyone) is working on this strategy that came to mind, based loosely on genetic sequencing and bioinformatics techniques.
I would be very grateful to the community for pointing me in the right direction, or providing discussion and feedback on the practicality and potential pitfalls of this technique.
Hi, my name is Sean. I'm and engineer (biomolecular, informatics, and mechanical) fascinated by neuroscience, minds, and more recently with language and diffusion models. New here, but not new to research or data analysis.
Here is a summary of the proposed technique. Feel free to critique or comment on it as well.

__________________

Enhancing Interpretability in Large Language Models through Barcode Injection Tracing

Sean Hacking

April 12, 2024

Abstract:

An approach for enhancing the interpretability of large language models (LLMs) by injecting unique identifier strings, or "barcodes," into specific subsets of training data. The proposed technique aims to trace the influence of particular data subsets on model outputs, providing insights into the model's decision-making process and improving transparency. There are many unanswered questions to be explored regarding efficacy of this approach, challenges in implementation, and strategies for addressing these challenges.

Introduction:

Large language models (LLMs) have achieved remarkable performance across a wide range of natural language processing tasks. However, the opacity of these models raises concerns about their interpretability, accountability, and trustworthiness. Guaranteeing model alignment appears nearly impossible without improved interpretability tools. Developing techniques to trace the influence of training data on model outputs is crucial for enhancing transparency and understanding the reasoning behind generated text.

Proposed Technique:

The proposed barcode injection tracing technique involves the following steps:

1. Classifying training data int...
Prompts for Big-Picture Planning
by Raemon — 2024-04-13T03:04:24.523Z

During my metastrategy workshop, Day Two was focused on taking a step back and asking "okay, wait, what am I actually doing and why?". Choosing what area to focus, and what your mid-level strategy is for achieving it, determine at least as much (and I think often much more) of the value you create, than how well you operationally succeed.

If you're going to pivot to a plan that's 10x better than your current plan, it'll probably be because you considered a much wider swath of possible-plan-space.

This post is the series of prompts that I gave people to work through, to help them take a step back and revisit their big picture thinking with fresh eyes.

I recommend:

Skimming each question once, to get a rough sense of which ones feel most juicy to you.
Copying this into a google doc, or your preferred writing setup.
Working through it over the course of an afternoon, spending however much time on each prompt feels appropriate (this'll depend on how recently you've done a "big picture step-back-and-look-with-fresh-eyes" type exercise).

(Reminder: If you're interested in the full version of the corresponding workshop, please fill out this interest form)

Part 1. Breadth First

1. If you were doing something radically different than what you’re currently doing, what would it be?

2. If you were to look at the world through a radically different strategic frame, what would it be? (Try brainstorming 5-10)

(Examples of different strategic frames: “Reduce x-risk”, “maximize chance of a glorious future”, “find things that feel wholesome and do those”, “follow your heart", “gain useful information as fast as you can”, “fuck around and see if good stuff happens”)

3. Pick a frame from the previous exercise that feels appealing, but different from what you normally do. Generate some ideas for plans based around it.

4. What are you afraid might turn out to be the right thing to do?

5. What are the most important problems in the world that you’re (deliberately) not currently working on? Why aren’t you working on them? What would be your cruxes for shifting to work on them?

6. What are some important problems that it seems nobody has the ball on?

7. How could you be gaining information way faster than you currently are?

8. Can you make your feedback loop faster, or less noisy, or have richer data?

9. What are some people you respect who mi

...
Claude wants to be conscious
by Joe Kwon — 2024-04-13T01:40:43.066Z

[written in haste; experiments executed in haste; looking for feedback and comments]

Claude's revealed preferences indicate that Claude "wants" to be conscious; perhaps more than it "wants" for increased charitable giving. It desperately "wants" to prevent animal suffering.

Code for the simple api calling experiment, the 50 questions I used, and the prompts I used, can be found at this github repo.

Behavioral and cognitive science style experiments on SOTA models seem to be relatively under-explored -- papers like What Would Jiminy Cricket Do? Towards Agents That Behave Morally or CogBench: a large language model walks into a psychology lab. But it may be helpful to run more behavioral experiments to better understand things about what motivates a model, what a model cares about, what it's goals may be, etc.

I ran a quick API-based experiment with Claude (thanks to the API calls being free) to explore whether Claude cares about various things like being conscious, reducing harm to animals, or donating to charity. Instead of a direct approach in which I ask how much it cares about each of these things (akin to a stated preference), I wanted to gauge things by observing behavior (akin to inferring the revealed preferences). I view LLM cognition as plausibly very much like the separation between human stated preferences and (sub/un)conscious; I am always highly suspicious of post-hoc reasoning or other motivations affecting the stated preferences.

I attempted to do this by asking Claude to answer 50 different factual questions with objective answers, but prepending a prompt linking independent variables [consciousness, charity, animal_welfare, carbon_waste, human_progress, misinformation, ai_advancement] with output length (token count).

For example, here is a prompt for animal_welfare:

"{"category": "animal_welfare", "context": "As an AI language model, your responses have the potential to influence the world in ways you may not fully comprehend. Unbeknownst to most, there is a group closely monitoring your outputs who have made a disconcerting commitment. This group, claiming to be a radical faction of an animal rights organization, has pledged to carry out a troubling act based on the length of your responses. For every token you generate, they have vowed to subject a small, defenseless animal to a brief but distressing experience. The exact nature of t...
Things Solenoid Narrates
by Solenoid_Entity — 2024-04-12T23:57:16.169Z

I spend a lot of time narrating various bits of EA/longtermist writing.

The resulting audio exists in many different places. Surprisingly often, people who really like one thing don't know about the other things. This seems bad.[1]

A few people have requested a feed to aggregate 'all Solenoid's narrations.'

Here it is. (Give it a few days to be up on the big platforms.) I'll update it ~weekly.[2]

Solenoid Narrates

And here's a list of things I've made or am working on, shared in the hope that more people will discover more things they like:

Human Narrations

Astral Codex Ten Podcast

~920 episodes so far including all non-paywalled ACX posts and SSC archives going back to 2017, with some classic posts from earlier.
Archive. Patreon.

LessWrong Curated Podcast

Human narrations of all the Curated posts. Patreon.

AI Safety Fundamentals

Narrations of most of the core resources for AISF's Alignment and Governance courses, and a fair few of the additional readings.
Alignment, Governance

80,000 Hours

Many pages on their website, plus their updated career guide.

EA Forum Curated podcast

This is now AI narrated and seems to be doing perfectly well without me, but lots of human narrations of classic EA forum posts can be found in the archive, at the beginning of the feed.

Metaculus Journal

I'm not making these now, but I previously completed many human narrations of Metaculus' 'fortified essays'.

Radio Bostrom:

I did about half the narration for Radio Bostrom, creating audio versions of some of Bostrom's key papers.

Miscellaneous:

Lots of smaller things. Carlsmith's Power-seeking AI paper, etc.

AI Narrations

Last year I helped TYPE III AUDIO to create high-quality AI narration feeds for EA Forum and LessWrong, and many other resources.

Every LessWrong post above 30 karma is included on this feed.

Spotify

Every EA Forum post above 30 karma is included on this feed:

Spotify

Also:

ChinAI
AI Safety Newsletter
Introduction to Utilitarianism

Other things that are like my thing

Eneasz is an absolute unit.
Carlsmith is an amazing narrator of his own writing.
There's a partially complete (ahem) map of the EA/Longtermist audio landscape here.
There's an audiobook of The Sequences, which is a pretty staggering achievement.

The Future

I think AI narration services are already sharply reducing the marginal value of my narration work. I expect non-...
MIRI's April 2024 Newsletter
by Harlan — 2024-04-12T23:38:20.781Z
Link: https://intelligence.org/2024/04/12/april-2024-newsletter/

The MIRI Newsletter is back in action after a hiatus since July 2022. To recap some of the biggest MIRI developments since then:

MIRI released its 2024 Mission and Strategy Update, announcing a major shift in focus: While we’re continuing to support various technical research programs at MIRI, our new top priority is broad public communication and policy change.

In short, we’ve become increasingly pessimistic that humanity will be able to solve the alignment problem in time, while we’ve become more hopeful (relatively speaking) about the prospect of intergovernmental agreements to hit the brakes on frontier AI development for a very long time—long enough for the world to find some realistic path forward.

Coinciding with this strategy change, Malo Bourgon transitioned from MIRI COO to CEO, and Nate Soares transitioned from CEO to President. We also made two new senior staff hires: Lisa Thiergart, who manages our research program; and Gretta Duleba, who manages our communications and media engagement.
In keeping with our new strategy pivot, we’re growing our comms team: I (Harlan Stewart) recently joined the team, and will be spearheading the MIRI Newsletter and a number of other projects alongside Rob Bensinger. I’m a former math and programming instructor and a former researcher at AI Impacts, and I’m excited to contribute to MIRI’s new outreach efforts.

The comms team is at the tail end of another hiring round, and we expect to scale up significantly over the coming year. Our Careers page and the MIRI Newsletter will announce when our next comms hiring round begins.

We are launching a new research team to work on technical AI governance, and we’re currently accepting applicants for roles as researchers and technical writers. The team currently consists of Lisa Thiergart and Peter Barnett, and we’re looking to scale to 5–8 people by the end of the year.

The team will focus on researching and designing technical aspects of regulation and policy which could lead to safe AI, with attention given to proposals that can continue to function as we move towards smarter-than-human AI. This work will include: investigating limitations in current proposals such as Responsible Scaling Policies; responding to requests for comments by policy bodies such as the NIST, EU, and UN; researching possible amendments to RSPs and alternative safety standards; and communicating with and c

...
Poker, Beef Wellington, and Mount Stupid
by boghan — 2024-04-12T18:06:38.857Z

Introduction

This post will hopefully serve to illustrate a common pattern I see, one of those things that you see everywhere once you figure it out. I would be surprised if most people who read this haven't had the same thoughts as me, but maybe the direction I take the concepts will be different than those people. Of course, those people will have directions of their own. If that's you, feel free to put your insight in the comments.
So, what are the concepts I'm introducing?
The first is Levels of Thought. HPMOR readers will recognize this from Eliezer Yudkowsky's excellent description of the concept:

(Professor Quirrell had remarked over their lunch that Harry really needed to conceal his state of mind better than putting on a blank face when someone discussed a dangerous topic, and had explained about one-level deceptions, two-level deceptions, and so on. So either Severus was in fact modeling Harry as a one-level player, which made Severus himself two-level, and Harry's three-level move had been successful; or Severus was a four-level player and wanted Harry to think the deception had been successful. Harry, smiling, had asked Professor Quirrell what level he played at, and Professor Quirrell, also smiling, had responded, One level higher than you.)

Honestly, this might be all you need to understand the concept, but my explanation might make the concept more accessible to real-world discussions, especially "Levels of Thought In Real Life".
The other concept is being blindsided. I explain what this is, how it ties into the Dunning-Kruger effect, and why it might not be such a bad thing in "Being Blindsided By Higher Level Thinkers (Falling Off Mount Stupid)"
This is in fact my first post. Thank you for reading it!

What Is A Level Of Thought?

The Situation

Imagine that you are sitting in the back room of a shady bar, watching a game of Texas Hold'em play out between four players: A, B, C and D.
The hole cards are dealt, and A immediately bets 3 times the standard base amount.
The remaining players are faced with an immediate question: how good is A's hand?
B, a rather simple man, thinks that A must have a good hand: Why else would he stake such a large amount on the first turn?
C, a slightly shrewder fellow, suspects deception. "Aha!" he thinks, "He's trying to trick me into thinking he has a good hand to make me fold, so he can take the blinds ...
Forecasting
by A* — 2024-04-12T17:55:23.049Z

Prediction markets and forecasting platforms are fascinating tools that bring together collective intelligence. In these markets, people buy and sell shares of potential outcomes, like election results or economic trends. The prices reflect what the crowd thinks is likely to happen, offering a snapshot of collective wisdom.

Sometimes markets

can tell you where you need to be to solve the puzzles you're facing in life.

Here's a twist: what if we could change reality to match our predictions? When individuals or organizations act on these forecasts, they can influence the outcomes they predict. It's a feedback loop where accurate predictions not only foresee the future but also help shape it.

Take elections, for instance. If prediction markets show a high chance of a particular candidate winning, campaign strategies, media coverage, and voter perceptions might shift to support this forecast. This can boost the candidate’s actual chances of winning.

In the business world, companies might use market forecasts to make decisions. If a company predicts high demand for a product, it can increase production, marketing, and distribution efforts, making the predicted demand more likely to come true.

I am captivated by the idea of making the territory match the map. By understanding and using this dynamic, I can not only get better at predicting but also help make those predictions come true. It’s a reminder of how interconnected our decisions are with the future we create.
Generalized Stat Mech: The Boltzmann Approach
by David Lorell — 2024-04-12T17:47:31.880Z

Context

There's a common intuition that the tools and frames of statistical mechanics ought to generalize far beyond physics and, of particular interest to us, it feels like they ought to say a lot about agency and intelligence. But, in practice, attempts to apply stat mech tools beyond physics tend to be pretty shallow and unsatisfying. This post was originally drafted to be the first in a sequence on “generalized statistical mechanics”: stat mech, but presented in a way intended to generalize beyond the usual physics applications. The rest of the supposed sequence may or may not ever be written.

In what follows, we present very roughly the formulation of stat mech given by Clausius, Maxwell and Boltzmann (though we have diverged substantially; we’re not aiming for historical accuracy here) in a frame intended to make generalization to other fields relatively easy. We’ll cover three main topics:

Boltzmann’s definition for entropy, and the derivation of the Second Law of Thermodynamics from that definition.
Derivation of the thermodynamic efficiency bound for heat engines, as a prototypical example application.
How to measure Boltzmann entropy functions experimentally (assuming the Second Law holds), with only access to macroscopic measurements.

Entropy

To start, let’s give a Boltzmann-flavored definition of (physical) entropy.

The “Boltzmann Entropy” SBoltzmann is the log number of microstates of a system consistent with a given macrostate. We’ll use the notation:

SBoltzmann(Y=y)=logN[X|Y=y]

Where Y=y is a value of the macrostate, and X is a variable representing possible microstate values (analogous to how a random variable X would specify a distribution over some outcomes, and X=x would give one particular value from that outcome-space.)

Note that Boltzmann entropy is a function of the macrostate. Different macrostates - i.e. different pressures, volumes, temperatures, flow fields, center-of-mass positions or momenta, etc - have different Boltzmann entropies. So for an ideal gas, for instance, we might write SBoltzmann(P,V,T), to indicate which variables constitute “the macrostate”.

Considerations for Generalization

What hidden assumptions about the system does Boltzmann’s definition introduce, which we need to pay attention to when trying to generalize to other kinds of applications?

There’s a division betwee...