trentmkelly/LessWrong-43k huggingface dataset:
Steelmanning as an especially insidious form of strawmanning
by Cornelius Dybdahl — 2024-03-11T02:25:03.310Z

Edit: made some small changes to prevent certain gross mischaracterizations of the argument. The core argument remains completely unchanged.

Among intelligent people with at least some familiarity with argumentative norms, surface level disagreements tend to be ephemeral because, even if some given debate about the issue is terminated before conclusion, both parties to the disagreement will eventually encounter the full range of common arguments pertaining to the issue. Because of this, there are really only two cases where the disagreement will persist: 1. if the key arguments are sufficiently uncommon as to not be in general circulation, or 2. if mere familiarity with surface-level arguments is insufficient to bridge the inferential gap.

We will examine these cases separately.

The first case is rare, because convincing arguments, if they can be grasped with relatively low opportunity cost, have a tendency to spread and become part of general circulation. Exceptions can however be found when the arguments pertain to a niche, though only when people interested in that niche have sufficiently little contact not to form a distinct social network of their own. More commonly, the first case arises when there is a political or social pressure not to repeat the arguments in general company, because this creates an opportunity cost to transferring them.

Already here, the practice of steelmanning can give rise to major problems, though only if you are steelmanning a position rather than the argument offered in its support, and this only in cases where the position is more socially acceptable than the argument. Consider for example the case where someone is critiquing disparate impact case law from a standpoint of HBD. Whatever you may think of the argument[1] should not matter to demonstrate the dynamic.

The interlocutor has an instinctual aversion to HBD and flinches away from it, but notes that an argument for the position can be built on a much less offensive basis. For example, one might argue that equity can only realistically be achieved by addressing the underlying drivers of cognitive inequality (eg. early education, diet, etc.) and not merely by legislating your way to equal outcomes, which would merely place disadvantaged people in academic courses they can't keep up with, or get them into jobs whose demands they cannot meet, leading to impostor synd...
One-shot strategy games?
by Raemon — 2024-03-11T00:19:20.480Z

I'm looking for computer games that involve strategy, resource management, hidden information, and management of "value of information" (i.e. figuring out when to explore or exploit), which:

*can* be beaten in 30 – 120 minutes on your first try (or, there's a clear milestone that's about that long)
but, it'd be pretty hard to do so unless you are trying really hard. Even if a pretty savvy gamer shouldn't be able to by default.

This is for my broader project of "have a battery of exercises that train/test people's general reasoning on openended problems." Each exercise should ideally be pretty different from the other ones.

In this case, I don't expect anyone to have such a game that they have beaten on their first try, but, I'm looking for games where this seems at least plausible, if you were taking a long time to think each turn, or pausing a lot.

The strategy/resource/value-of-information aspect is meant to correspond to some real world difficulties of running longterm ambitious planning.

(One example game that's been given to me in this category is "Luck Be a Landlord")
Understanding SAE Features with the Logit Lens
by Joseph Bloom — 2024-03-11T00:16:57.429Z

This work was produced as part of the ML Alignment & Theory Scholars Program - Winter 2023-24 Cohort, with support from Neel Nanda and Arthur Conmy. Joseph Bloom is funded by the LTFF, Manifund Regranting Program, donors and LightSpeed Grants. This post makes extensive use of

Neuronpedia

, a platform for interpretability focusing on accelerating interpretability researchers working with SAEs.

Links: SAEs on HuggingFace, Analysis Code

Executive Summary

This is an informal post sharing statistical methods which can be used to quickly / cheaply better understand Sparse Autoencoder (SAE) features.

Firstly, we use statistics (standard deviation, skewness and kurtosis) of the logit weight distributions of features (WuWdec[feature]) to characterize classes of features, showing that many features can be understood as promoting / suppressing interpretable classes of tokens.

We propose 3 different kinds of features, analogous to previously characterized “

universal neurons

”:

Partition Features, which (somewhat) promote half the tokens and suppress the other half according to capitalization and spaces (example pictured below)
Suppression Features, which act like partition features but are more asymmetric.
Prediction Features which promote tokens in classes of varying sizes, ranging from promoting tokens that have a close bracket to promoting all present tense verbs.

Secondly, we propose a statistical test for whether a feature's output direction is trying to distinguish tokens in some set (eg: “all caps tokens”) from the rest.

We borrowed this technique from systems biology where it is used at scale frequently.
The key limitation here is that we need to know in advance which sets of tokens are promoted / inhibited.

Lastly, we demonstrate the utility of the set-based technique by using it to locate features which enrich token categories of interest (defined by regex formulas, NLTK toolkit parts of speech tagger and common baby names for boys/girls).

Feature 4467

. Above: Feature Dashboard Screenshot from Neuronpedia. It is not immediately obvious from the dashboard what this feature does. Below: Logit Weight distribution classified by whether the token starts with a space, clearly indicating that this feature promotes tokens which lack an initial space character.

Introduction

In

previo

...
Replacing the Water Heater's Anode
by jefftk — 2024-03-11T00:00:10.796Z

We installed a new water heater 8 years ago, and since then I've
ignored it. It's an indirect model, heated by the same gas boiler that
heats our house, and it has done its job well. When I was

thinking about heat
pumps, however, I reread the manuals for our existing system and
noticed that the

manufacturer
recommends checking the anode annually. The idea is that water will
corrode metal, but prefers some metals to others, so if you put in a
chunk of tasty sacrificial metal you lose that instead of the tank
walls.

Once every eight years is not annual, but better late than never!

Since it had been quite a bit longer than anodes usually last, I
preemptively ordered a new one. When it arrived I warned people that I was going to be
shutting off the hot water for a while, but as soon as I took off the
insulating cap for the old anode it was clear I didn't have what I
needed. It used a 1 3/4" hex head, recessed, and I didn't have any
sockets close to that large. Neither did the hardware store: I had to
order one.

I think the manufacturer realized this was silly: the replacement
anode came with a 1" head, and reading their documentation
they switched to 1" in in 2021.

Once I had the right-sized parts I turned off the system, shut off the
incoming water, opened a hot water tap to break the vacuum, hooked up
a hose to the drain port, and got it draining. I went and did
something else for a long while, and when I came back I started on the
anode. The first problem was that the thermostat is immediately in
front of the anode port:

I needed to disassemble this more than I wanted to get it out of the
way.

Once it was clear, it was hard to get the anode rotating. I needed a
cheater bar (a pipe slipped over the breaker to extend it). When it
came out I was surprised in two ways:

The old anode had been entirely consumed. Here is the new one and
old one side-by-side:

Lots of water came out:

The problem was, I hadn't confirmed that the water heater had finished
draining. An 80 gallon tank (two units, eight bedrooms, ten people)
takes a long time to drain, and it was far from done. Luckily this is
an unfinished basement with nothing that minds getting wet, and it all
drains to a sump.

After the top finished draining out the wrong hole, and then the
bottom finished draining out of the right hole, I cleaned anode por...
Briefly Extending Differential Optimization to Distributions
by J Bostock — 2024-03-10T20:41:09.551Z

I've done some work on a definition of optimization which applies to "trajectories" in deterministic, differentiable models. What happens when we try and introduce uncertainty?

Suppose we have the following system consisting of three variables, the past P, future F, and some agent A. The agent "acts" on the system to push the value of F 80% of the way towards being zero. We can think of this as follows: A=0.8P, F=P−A. Under these circumstances, ∂F∂P|A varies/∂P∂F|A constant=0.2 which means our optimization function gives: Op(P,F;A)=−log(|0.2|)≈−1.61 nats.

What if we instead consider a normal distribution over P? This must be parameterized by a mean μP and a standard deviation σP. Our formulae now look like this:

P∼N(μP,σP)
A∼N(0.8μP,0.8σP)
F∼N(0.2μP,0.2σP)

So what does it look like for A to "not depend" on P? We could just "pick" some value for A but this seems like cheating. What if we set up a new model, in which F′ depends on P′ and A′, but A′ depends on P′′ instead of P′? We can allow P′ and P′′ to have the same distributions as before:

P′∼N(μP,σP)
P′′∼N(μP,σP)
A∼N(0.8μP,0.8μP)

Calculating F is a bit more difficult. We can think of it as adding two uncorrelated normal distributions together. For normal distributions this just means adding the means and variances together. Our distributions have means μP and −0.8μP, and variances σ2P and 0.64σ2P. Therefore we get a new distribution with mean 0.2μP and variance 1.64σ2P. This gives a standard deviation of 1.28σP.

F′∼N(0.2μP,1.28σP)

What's the entropy of a normal distribution? Well, it's difficult to say properly, since entropy is poorly-defined on continuous variables. If one take the limiting density of discrete points one gets log(N)+12log(2πeσ2), where N goes to infinity. This is a problem unless we happen to be subtracting one entropy from another. So let's do that.

H(F)−H(F′)=log(N)+12log(2πeσ2F)−log(N)−12log(2πeσ2F′)
H(F)−H(F′)=12log(σ2F)−12log(σ2F′)
H(F)−H(F′)=log(σF)−log(σF′)
H(F)−H(F′)=log(0.2σP)−log(1.28σP)
H(F)−H(F′)=log(0.2/1.28)≈−1.86 nats

Ok so we got the sign wrong the first time. Nevermind. But there is another issue, this is higher than our previo...
Evolution did a surprising good job at aligning humans...to social status
by Eli Tyre — 2024-03-10T19:34:52.544Z

[This is post is a slightly edited tangent from my dialogue with John Wentworth here. I think the point is sufficiently interesting and important that I wanted to make it as a top level post, and not leave it buried in that dialog on mostly another topic.]

The conventional story is that natural selection failed extremely badly at aligning humans. One fact about humans that casts doubt on this story is that natural selection got the concept of "social status" into us, and it seems to have done a shockingly good job of aligning (many) humans to that concept.

Evolution somehow gave humans some kind of inductive bias (or something) such that our brains are reliably able to learn what it is to be "high status", even though the concrete markers for status are as varied as human cultures.

And it further, it successfully hooked up the motivation and planning systems to that "status" concept. Modern humans not only take actions that play for status in their local social environment, they sometimes successfully navigate (multi-decade) career trajectories and life paths, completely foreign to the ancestral environment, in order to become prestigious by the standards of the local culture.

And this is one of the major drivers of human behavior! As Robin Hanson argues, a huge portion of our activity is motivated by status-seeking and status-affiliation.
This is really impressive to me. It seems like natural selection didn't do so hot at aligning humans to inclusive genetic fitness. But it did kind of shockingly well aligning humans to the goal of seeking, even maximizing, status, all things considered.[1]
This seems like good news about alignment. The common story that condoms prove that evolution basically failed at alignment—that as soon as we developed the technological capability to route around the evolution's "goal" of maximizing the frequency of your alleles in the next generation, to attain only the proxy measure of sex, we did that—doesn't seem to apply to our status drive.

It looks to me like "status" generalized really well across the distributional shift of technological civilization. Humans still recognize it and optimize for it, regardless of whether the status markers are money or technical acumen or h-factor or military success.[2]

This makes me way less confident about the standard "evolution failed at alignment" story.

^

I guess that we can infer from

...
Pausing AI is Positive Expected Value
by Liron — 2024-03-10T17:10:58.008Z
Link: https://twitter.com/liron/status/1766723002870513908

The PauseAI (⏸️) movement often gets this pushback:

“You're not factoring in all the benefits of good AI!”

“Stopping AI progress is also a doom scenario!”

To which I reply: If you agree P(doom) from building superintelligent AI before knowing how to align or control it is 5%+, try doing the basic expected-value calculation; you'll see why your objection is misguided.

First, we need to estimate a few key probabilities and values. These can vary by many orders of magnitude. I'll pick values that AI optimists hopefully agree are fair:

Probability that AI goes right if capabilities scale to superintelligence by 2034

50%

This is an immediate "fast takeoff" scenario where state-of-the-art AI remains near-inscrutable, yet within a decade it becomes vastly more intelligent than humans on every dimension. I'd personally give this scenario a much lower probability than 50% of going right for humanity, but I'm trying to be generous to AI optimists.

Probability that AI goes right if we delay superintelligence to 2100

70%

An important premise of PauseAI is that if we can give ourselves a few extra years or decades to thoroughly research the fundamental principles of how to align AI — how to robustly specify preferences, how to capture the delicate structure of human values as self-consistent preferences, etc — then we can significantly increase the probability that superintelligent AI goes well.

If you agree that more time for safety research helps safety catch up to capabilities, you can take whatever probability you gave to superintelligent AI going right in 2034 and add 20% (or more) to the probability that it goes right in 2100.

Value of baseline future, where AI never gets beyond human intelligence

Let's define this as our baseline $0 scenario, because it's how normies who've never even heard of superintelligent AI currently imagine the future. We'll define the value of other scenarios in relation to the value of this scenario.

If we never let ourselves get superintelligent AI (or it turns out to be too hard to build), there'll probably still be at least a trillion future human lives worth living.

Value of future where AI goes wrong

–$1018

If superintelligent AI goes wrong, it could very plausibly wipe out the entire future potential value of Earth-originating life. Compared to the baseline no-ASI scenario, we lose out on at least a trillion future human li...
W2SG: Introduction
by Maria Kapros — 2024-03-10T16:25:22.222Z

Epistemic status: Naive and exploratory, reflects my primary conceptual understanding, awaiting a technical deep dive. 99% of ideas are not my own, rather distilled from the resources hyperlinked throughout.

Many alignment researchers err towards local optimization i.e. seek low-hanging fruits and leverage incremental improvements. Fast and imperfect iterative improvement is necessary but sometimes it should be integrated within a principled and holistic strategy.

I fear that many people interested in W2SG will default to testing ideas on the basis of informed heuristics, without robust inside views to justify them. While in aggregate we might converge on effective solutions, my general thesis is that W2SG needs people adopting an AI scout approach.

This introductory post and corresponding project aim to actualize such conviction in order to come as close as possible to a W2SG solution. Ultimately, success should be judged based on wether or not the results can be integrated within a useful automated alignment researcher (AAR).

Framing the challenge

The W2SG objective

At the highest abstraction level, solving W2SG means controlling and understanding how advanced AI systems generalize from weak supervision.

The W2SG result

Develop AI systems capable of superhuman generalization across a wide range of (alignment-relevant) tasks[1], with predictable (ideally provable) generalization bounds.

High-level strategy

A potential starting point is to identify and study the factors that influence generalization (GIFs). To then engineer and deploy scalable solutions we require a strategy for measuring progress. Ultimately, we would want to evaluate the strength and type of generalization achieved and compare with expected desiderata. Conceptually, a possible research plan could be structured across five pillars:

Problem specification

What types of generalization we care about?
What alignment-relevant tasks we want our models to generalize across?
What does it mean for AI systems to generalize well?
What assumptions are necessary for a W2SG solution to exist e.g check appendix G.3? What are the favorable generalization properties we require/expect in a deployment setting? To what extent do they correspond to reality?

Scientific understanding

What mathematics best capture the W2SG problem?
How to predict generalization type and strength achieved by W2SG solutions?
What formal mechanisms (e.g

...
An Optimistic Solution to the Fermi Paradox
by Glenn Clayton — 2024-03-10T14:39:19.989Z

This post is a crosspost from my blog.

Drum Beats in the Distance

In a remote forest clearing, a young girl clutches a handmade doll, her eyes wide with concern. Today, her father and uncles journey deep into the wilderness on a hunt, leaving her and the village behind. She wonders, how will her father hear her call amidst the vastness of the forest?

Her mother, sensing her worry, kneels beside her. "Fear not," she says, her voice as soothing as the rustle of leaves. "The drums of our village will speak across the forest. If your father needs us, their beats will carry his message. And if we need him, he will hear our drum's call." The idea of drums talking across distances vast and unseen intrigues the girl, sparking a curiosity that burns brighter than the evening fire.

That night, under a blanket of stars, she tosses and turns, thoughts of her father mingling with the mysteries of the forest and the enigmatic drums. Upon his return, she bombards him with questions, her youthful innocence unmarred by the complexities of the world. "Father, in the depths of the forest, did you hear other drums, drums of tribes unknown to us?"

Her father’s face grows solemn. "No, my child," he answers, his voice barely above a whisper. "There are no other tribes." He recounts a tale from his own childhood, about his father's unyielding belief in other villages, other drums. Despite mockery from fellow villagers, her grandfather had crafted a drum larger than any seen before, its beats thundering into the night, sending a message to the unseen, the unheard. Night after night, he waited for a reply that never came. Eventually, the drum fell silent, and with it, the talk of other tribes.

That evening, as the girl drifts into sleep, her mind buzzes with unspoken thoughts. What if her grandfather's drum had been even grander? Could it have reached those elusive, silent tribes?

Unnoticed, a Starlink satellite glides silently above emitting more signals per second than all the drums in her village combined could emit in a lifetime.

The Fermi Paradox

In the vast expanse of our universe, countless planets orbit around innumerable suns. Amidst this abundance of worlds surely some others have given rise to life, and even intelligence. And surely some of those have given rise to civilizations that have extended their reach into the cosmos. If this is true, then why haven’t we seen any evid...
Counterfactual Civilization Simulation Version -1.0 aka my application to Johannes Mayer's SPAR project
by Morphism — 2024-03-10T10:10:51.339Z

This is my "object level output" submission for Johannes Mayer's 2024 SPAR Application (the linked doc seems to be reused from the 2023 AISC application). Unless otherwise noted, all quote blocks in this post are from the application question doc.

For those of you who aren't Johannes Mayer reading this, I don't think this is the best use of your time, but your judgement on that is likely better than mine, especially when it's conditioned on mine, so if you still want to, read on!

0. The Problem

Make the following assumptions:

Reality can be perfectly modelled by a discrete model (including time).
You can compute everything that can be computed using finite memory and compute instantly.
You know the current state of the world perfectly.
You know the laws of physics perfectly.

Using these assumptions, come up with a high-level plan that when executed saves the world with very high probability. Be careful not to generate a missing steps plan.

If any constraints listed here are holding you back, and you think you could do better without them, ignore them! If there is a change you can make to the instructions such that you can get better outputs, make that change. If you made any changes (including ignoring instructions), briefly list them in the beginning and briefly explain how each change you made is an improvement over the original version.

Here are the changes I have made:

ADDITIONAL CONSTRAINT: You are unable to communicate to anyone in real life the fact that you have this unbounded compute and complete world model, nor ask them hypotheticals to this effect or do anything else that attempts to circumvent this constraint.

Reason: To prevent me from answering with what I would actually do if I had this power, which would be to talk to a bunch of really smart alignment researchers (e.g. Eliezer Yudkowsky, John S. Wentworth, Tamsin Leake) and ask them what to do with my instant unbounded compute and perfect world model. They can probably do better than I can, but this "solution" is not very good for serving the actual purpose of this exercise.

1. The Plan

Short Summary

Grab Eliezer Yudkowsky, Nate Soares, and a bunch of other really smart, really sane people from our world model. Make sure to grab enough people to healthily propagate the species if necessary. Also grab some offices, labs, farms, etc.; everything they would need to survive on a mostly lifeless pla...