trentmkelly/LessWrong-43k huggingface dataset:
Philosophers wrestling with evil, as a social media feed
by David Gross — 2024-06-03T22:25:22.507Z

Susan Nieman’s Evil in Modern Thought: An Alternative History of Philosophy (2015), visualized as a social media feed:

Alfonso Ⅹ

hates his Ptolemaic astronomy class

Alfonso Ⅹ

If I had been of God’s counsel at the Creation, many things would have been ordered better.

good Christians everywhere

Blasphemy!

Gottfried Leibniz

LOL. Nooooobody expects the Copernican Revolution! God doesn’t look so dumb now, does he?

Isaac Newton

Seriously. The more I look, the more amazed I am at the order and wisdom of Creation.

René Descartes

But how do you know your observations aren’t the result of an evil demon manipulating your senses rather than true reflections on the state of creation?

Jean-Jacques Rousseau

Alfonse, this just shows what book learning will get you. A humble farmer would never be so hare-brained as to try to out-think God.

Pierre Bayle

launches the Enlightenment

Voltaire likes this

Pierre Bayle

Properly speaking, history is nothing but the crimes and misfortunes of the human race.

Gottfried Leibniz

You’re as short-sighted as Alfonso. What look like crimes and misfortunes to you are just part of the mosaic of the best of all possible worlds from God’s point of view.

Georg Wilhelm Friedrich Hegel

On the contrary, Pierre. History — including the crimes and misfortunes of the human race — is the plan of God fulfilled, verifying the reality of providence. We just need to crack the code.

Pierre Bayle

likes Manichaeism

good Christians everywhere

Blasphemy!

Gottfried Leibniz

God could no more create a most perfect world without evil than He could create a square circle. If we were omniscient and could see the whole of creation over the whole of time, we would realize its perfection. Over time we will learn the connections between sin and suffering, and better understand God’s wisdom.

Immanuel Kant

But we aren’t omniscient, and we can’t see the world this way, so what makes you so confident?

Georg Wilhelm Friedrich Hegel

So God is like a grocer? He can give you anything you want as long as it’s in stock? ‘It may not be great, but it’s the best we could find today.’

Gottfried Leibniz

invents Calculus

Isaac Newton

The hell you did.

Alexander Pope

All Nature is but Art, unknown to thee:
All Chance, Direction, which thou canst not see;
All Discord, Harmony not understood;
All partial Evil, universal Good;
And in spite of Pride, in erring Reason’s spite,
One truth is clear, whatever is, is right.

Voltaire likes this

...
ACI#8: Value as a Function of Possible Worlds
by Akira Pyinya — 2024-06-03T21:49:02.345Z

What is value? What are rewards? In traditional models of intelligence, we don't have to worry about these questions. Reward is simply what we need to maximize, and value (or utility) represents total expected rewards. You don't need to question these fundamental assumptions of the model.

In contrast, ACI argues that reward is not fundamental to intelligence behaviors, and that value functions are neither arbitrary nor predetermined. Instead, they are derived from ACI's first principle: "Do the same thing as the examples of the right things". Value represents the probability of doing the right thing, and reward can be derived from value. Maximizing total rewards or value is not everything. However, reward and value are still useful concepts, that's why we need to derive them in the framework of ACI.

We believe that ACI has more explanatory power than traditional models. One example is the discounting factor, which indicates how future rewards are less desirable than current rewards. ACI suggests that the discount factor is not just a mathematical trick, but a measure of the inherent uncertainty of value functions.

Fig 1: In ACI, value is derived from the right things, and rewards are derived from value.

Beyond the reward hypothesis

Modern reinforcement learning approaches are constructed upon the basis of the reward hypothesis, which argues that "all of what we mean by goals and purposes can be well thought of as maximization of the expected value of the cumulative sum of a received scalar signal (called reward)."(Sutton & Barto 2018)

But the concept of reward is tricky. It's deeply rooted in the agent-environment dualism (Demski & Garrabrant 2019). Reward is presumably computed inside the physical body of natural and artificial intelligence systems, but in reinforcement learning models, reward is always considered to come from the environment because it cannot be changed arbitrarily, otherwise the task itself can be changed arbitrarily (Sutton & Barto 2018).

Furthermore, there's no evidence that reward signals really exist in natural intelligence systems. The dopamine TD-error hypothesis argues that the phasic response of dopamine neurons corresponds to TD error, the difference between value estimations (Niv et. al. 2005) , rather than corresponding to reward. Reward is arguably "an abstraction summar...
in defense of Linus Pauling
by bhauth — 2024-06-03T21:27:43.962Z
Link: https://www.bhauth.com/blog/biology/vitamin%20c.html

Linus Pauling was a chemist who won a Nobel Prize in Chemistry in 1954. He later became well-known for advocating large doses of Vitamin C.

I've heard that advocacy referred to as a cautionary tale, but I've long had a bit of sympathy for Linus, and I'm writing this post to explain how and why.

mainstream nutrition

One reason for my sympathy for Linus is that I've heard him used as an example of why you shouldn't go off on your own instead of trusting mainstream views in a field, yet his advice, while not particularly helpful, caused much less harm than contemporary "mainstream nutritional advice", such as:

the food pyramid

"partially hydrogenated vegetable oil is healthier than butter"

"a very-low-fat diet is healthy"

"eating very little salt is healthy"

I certainly wouldn't suggest trying to independently compete with the conceptual framework of, say, semiconductor physics or structural engineering, but when a field is rotten enough (nutrition, psychology, education, and economics come to mind) history indicates to me that someone smart from another field is often more correct than specialists on that topic, when they have an interest in it.

my view of Vitamin C

To be clear, I'm not advocating for the very high doses of Vitamin C that Linus did. I do think the World Health Organization's RDA (45 mg/day) is a bit low, but the RDA of Japan and the EU (~100 mg/day) seems entirely reasonable to me.

Amounts above that generally don't have much effect on blood levels of Vitamin C, because it's absorbed less and the excess is expelled. Thus, some people have advocated administering it by IV, but one has to consider the possibility that there's a good reason for a specific level being naturally maintained.

Research since Linus first advocated for Vitamin C megadoses has supported oxidative stress being a major cause of aging. It's associated with many problems (Alzheimer's comes to mind) and there's a good theoretical basis (DNA I-compounds) for its long-term harms. We've also seen that previous suggestions for Vitamin D doses were much too low, so there's also precedent for that.

Where, then, did Linus go wrong?

Vitamin C is an antioxidant, but it's also a pro-oxidant. It can reduce iron and copper, which can then react with hydrogen peroxide or oxygen to form hydroxyl radicals or peroxide ions. It can also form some complexes with metal ions that could conceivably ...
Finding the estimate of the value of a state in RL agents
by Clément Dumas — 2024-06-03T20:26:59.385Z

Clément Dumas, Walter Laurito, Robert Klassert, Kaarel Hänni

Epistemic Status: Initial Exploration

The following is a status update of a project started as part of the SPAR program. We explored some initial directions and there are still a lot of low-hanging fruits to pick up. We might continue to work on this project, either again as part of another SPAR iteration or with others who would be interested to work on this.

TL;DR

We adapted the

Contrast Consistent Search

(CCS) loss to find value-like directions in the activations of CNN-based PPO agents. While we had some success in identifying these directions at late layers of the critic network and with specific informative losses, we discovered that early layers and the policy network often contained more salient features that overshadowed the value-like information. In simple environments like Pong, it might be feasible to normalize the obvious salient features (e.g., ball position and approach). However, for more complex games, identifying and normalizing all salient features may be challenging without supervision. Our findings suggest that applying CCS to RL agents, if possible, will require careful consideration of loss design, normalization, and potentially some level of supervision to mitigate the impact of highly salient features.

Motivation

The research direction of

"Searching for Search"

investigates how neural networks implement search algorithms to determine actions. The goal is to identify the search process and understand the underlying objectives that drive it. By doing so, we may be able to modify the

search to target new goals

while maintaining the model's capabilities. Additionally, proving the absence of search could indicate limited generalization ability, potentially reducing the likelihood of deception.

A natural first step towards finding search in models is to examine a Reinforcement Learning agent and determine if we can identify the agent's estimate of the value of a state (or action). Typically, the value network outputs this value, while the policy network outputs an action. To output an action, we think that the policy network could probably require some internal representation of value. Therefore, based on an example from our

mathematical framework

, we employed both unsupervised and supervised probing methods to try to uncover the value of a state of a value network and p...
Searching Magic Cards
by jefftk — 2024-06-03T17:40:02.207Z

I like playing Magic, except for the "it's designed to pump away your
money" aspect. So I like formats where I can play with other people's
cards a lot! My favorite is probably drafting from booster packs
where someone else will keep all the cards, but yesterday Stevie
brought over a

Forgetful
Fish-style deck he'd put together, and we played a couple times.
The only creature in this format is the

Dandan,
a 4/1 blue creature with:

Dandan can't attack unless defending player controls an Island.

When you control no Islands, sacrifice Dandan.

One of the cards in Stevie's deck was Magical
Hack, a blue instant with:

Change the text of target spell or permanent by replacing all
instances of one basic land type with another.

It's role in the deck was primarily to kill Dandans, by replacing
"Island" on an opponent's Dandan with any basic land type they
don't have. But it got me wondering: does it happen that there are
any textual uses of the six basic land types [1] that are not intended
to be about a basic land? For example, if a card happened to use the
word "forestall", perhaps you could do something fun with it?

Supposedly you can search cards with regular expressions on Scryfall
but I couldn't get it to work. Instead, I downloaded the cards as
JSON from MTGJson [2] and
wrote a ~10 line python script:

unusual_land_re = re.compile(
r"(?i)"
r"((plains)|(island)|(swamp)|(wastes)|(mountain)|(forest))"
r"(?!(\b|s\b|[.]|walk\b|cycling\b))")

for fname in glob.glob("*.json.gz"):
with gzip.open(fname) as inf:
r = json.load(inf)
if "cards" not in r["data"]: continue
for card in r["data"]["cards"]:
if "text" not in card: continue
if unusual_land_re.search(card["text"]):
print("%s: %s" % (card["name"], card["text"]))

The first time I ran it I got a bunch of "Forestcycling" etc, but
after adding "cycling" to the query (as reproduced above) it didn't
find anything. Oh well.

I probably could have gotten regexp searching working if I'd played
with it a little more, but if I ever want to look for a card in a way
that isn't reducible to a regular expression search this could be a
good place to start!

[1] Forest, Plains, Mountain, Island, Swamp, and now Wastes

[2]

$ curl -sS https://mtgjson.com/api/v5/ | \
grep -o '[^">]*.json.gz' | sort | uniq > gzip_js

...
The Standard Analogy
by Zack_M_Davis — 2024-06-03T17:15:42.327Z

[Scene: a suburban house, a minute after the conclusion of "And All the Shoggoths Merely Players". Doomimir returns with his package, which he places by the door, and turns his attention to Simplicia, who has been waiting for him.]

Simplicia: Right. To recap for [coughs] no one in particular, when we left off [pointedly, to the audience] one minute ago, Doomimir Doomovitch, you were expressing confidence that approaches to aligning artificial general intelligence within the current paradigm were almost certain to fail. You don't think that the apparent tractability of getting contemporary generative AI techniques to do what humans want bears on that question. But you did say you have empirical evidence for your view, which I'm excited to hear about!

Doomimir: Indeed, Simplicia Optimistovna. My empirical evidence is the example of the evolution of human intelligence. You see, humans were optimized for one thing only: inclusive genetic fitness—

[Simplicia turns to the audience and makes a face.]

Doomimir: [annoyed] What?

Simplicia: When you said you had empirical evidence, I thought you meant empirical evidence about AI, not the same analogy to an unrelated field that I've been hearing for the last fifteen years. I was hoping for, you know, ArXiv papers about SGD's inductive biases, or online regret bounds, or singular learning theory ... something, anything at all, from this century, that engages with what we've learned from the experience of actually building artificial minds.

Doomimir: That's one of the many things you Earthlings refuse to understand. You didn't build that.

Simplicia: What?

Doomimir: The capabilities advances that your civilization's AI guys have been turning out these days haven't come from a deeper understanding of cognition, but by improvements to generic optimization methods, fueled with ever-increasing investments in compute. Deep learning not only isn't a science, it isn't even an engineering discipline in the traditional sense: the opacity of the artifacts it produces has no analogue among bridge or engine designs. In effect, all the object-level engineering work is being done by gradient descent.

The autogenocidal maniac Richard Sutton calls this the bitter lesson, and attributes the field's slowness to embrace it to ego and recalcitrance on the part of practitioners. But in accordance with the dictum to feel fully the emotion that fits the ...
How was Less Online for you?
by Gordon Seidoh Worley — 2024-06-03T17:10:33.766Z

I was going to write a post that was something like "Personal Reflections from Less Online", but lots of us who were there probably have reflections on it, so I created this question post to invite people to offer their reflections in one place.

Answers should be reflections from people who attended. Comments could be from anyone.
AI catastrophes and rogue deployments
by Buck — 2024-06-03T17:04:51.206Z

[Thanks to Aryan Bhatt, Ansh Radhakrishnan, Adam Kaufman, Vivek Hebbar, Hanna Gabor, Justis Mills, Aaron Scher, Max Nadeau, Ryan Greenblatt, Peter Barnett, Fabien Roger, and various people at a presentation of these arguments for comments. These ideas aren’t very original to me; many of the examples of threat models are from other people.]

In this post, I want to introduce the concept of a “rogue deployment” and argue that it’s interesting to classify possible AI catastrophes based on whether or not they involve a rogue deployment. I’ll also talk about how this division interacts with the structure of a safety case, discuss two important subcategories of rogue deployment, and make a few points about how the different categories I describe here might be caused by different attackers (e.g. the AI itself, rogue lab insiders, external hackers, or multiple of these at once).

Rogue deployments

Suppose you’ve developed some powerful model. (It might actually be many models; I’m just going to talk about a singular model for convenience.)

You have some definition of a catastrophe—a failure that you don’t want to happen even once. Your AI is powerful enough that, in absence of any safety measures, it would pose a substantial risk of causing a catastrophe. In the terminology of “

Safety Cases: How to Justify the Safety of Advanced AI Systems

”, we’d phrase this as “the model is too powerful for us to make inability arguments for safety.” (Sometimes I use the word “catastrophe” to include things like “the AI escapes the datacenter”. Right now, I’m excluding that. I want to reserve “catastrophe” for the end result that we find unacceptable, e.g. the AI or another adversary amassing power, launching nukes, starting a war, building superintelligence, etc.)

For the moment, let’s be agnostic between various adversaries who might try to cause a catastrophe (AI or various humans).

You have some safety measures that you’ve put into place to prevent the catastrophe from occurring. Safety measures are interventions that are applied to all of the deployments of your model (to any users, including just internal users) that try to prevent safety problems. This might include control measures like

trusted monitoring

, and

security measures to make it harder to exfiltrate your model weights

.

A rogue deployment is a deployment of your model where the safety measures are a...
Companies' safety plans neglect risks from scheming AI
by Zach Stein-Perlman — 2024-06-03T15:00:20.236Z

Without countermeasures, a scheming AI could escape.

A safety case (for deployment) is an argument that it is safe to deploy a particular AI system in a particular way.[1] For any existing LM-based system, the developer can make a great safety case by demonstrating that the system does not have dangerous capabilities.[2] No dangerous capabilities is a straightforward and solid kind of safety case. For future systems with dangerous capabilities, a safety case will require an argument that the system is safe to deploy despite those dangerous capabilities. In this post, I discuss the safety cases that the labs are currently planning to make, note that they ignore an important class of threats—namely threats from scheming AI escaping—and briefly discuss and recommend control-based safety cases.

I. Safety cases implicit in current safety plans: no dangerous capabilities and mitigations to prevent misuse

Four documents both (a) are endorsed by one or more frontier AI labs and (b) have implicit safety cases (that don't assume away dangerous capabilities): Anthropic's

Responsible Scaling Policy v1.0

, OpenAI's

Preparedness Framework (Beta)

, Google DeepMind's

Frontier Safety Framework v1.0

, and the AI Seoul Summit's

​​Frontier AI Safety Commitments

. With small variations, all four documents have the same basic implicit safety case: before external deployment, we check for dangerous capabilities. If a model has dangerous capabilities beyond a prespecified threshold, we will notice and implement appropriate mitigations before deploying it externally.[3] Central examples of dangerous capabilities include hacking, bioengineering, and operating autonomously in the real world.

Anthropic's Responsible Scaling Policy

: we do risk assessment involving red-teaming and model evals for dangerous capabilities. If a model has dangerous capabilities beyond a prespecified threshold,[4] we will notice and implement corresponding mitigations[5] before deploying it (internally or externally).

OpenAI's Preparedness Framework

: we do risk assessment involving red-teaming and model evals for dangerous capabilities. We only externally deploy[6] models with "post-mitigation risk" at 'Medium' or below in each risk category. (That is, after mitigations, the capabilities that define 'High' risk can't be el

...
Comments on Anthropic's Scaling Monosemanticity
by Robert_AIZI — 2024-06-03T12:15:44.708Z

These are some of my notes from reading Anthropic's latest research report, Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet.

TL;DR

In roughly descending order of importance:

Its great that Anthropic trained an SAE on a production-scale language model, and that the approach works to find interpretable features. Its great those features allow interventions like the recently-departed Golden Gate Claude. I especially like the code bug feature.
I worry that naming features after high-activating examples (e.g. "the Golden Gate Bridge feature") gives a false sense of security. Most of the time that feature activates, it is irrelevant to the golden gate bridge. That feature is only well-described as "related to the golden gate bridge" if you condition on a very high activation, and that's <10% of its activations (from an eyeballing of the graph).
This work does not address my major concern about dictionary learning: it is not clear dictionary learning can find specific features of interest, "called-shot" features, or "all" features (even in a subdomain like "safety-relevant features"). I think the report provides ample evidence that current SAE techniques fail at this.
The SAE architecture seems to be almost identical to how Anthropic and my team were doing it 8 months ago, except that the ratio of features to input dimension is higher. I can't say exactly how much because I don't know the dimensions of Claude, but I'm confident the ratio is at least 30x (for their smallest SAE), up from 8x 8 months ago.
The correlations between features and neurons seems remarkably high to me, and I'm confused by Anthropic's claim that "there is no strongly correlated neuron".
Still no breakthrough on "a gold-standard method of assessing the quality of a dictionary learning run", which continues to be a limitation on developing the technique. The metric they primarily used was the loss function (a combination of reconstruction accuracy and L1 sparsity).

I'll now expand some of these into sections. Finally, I'll suggest some follow-up research/tests that I'd love to see Anthropic (or a reader like you) try.

A Feature Isn't Its Highest Activating Examples

Let's look at the Golden Gate Bridge feature because its fun and because it's a good example of what I'm talking about. Here's my annotated version of Anthropic's diagram:

I'm trying to be generous with my rou

...