trentmkelly/LessWrong-43k huggingface dataset:
Mathematically-Defined Optimization Captures A Lot of Useful Information
by J Bostock — 2023-10-29T17:17:03.211Z

Alright it's been a long hiatus. I may not post again for another year or two if no inspiration strikes me. I will summarize my work so far.

We can define a function over causal networks which describes how hard a given portion of the network is optimizing some node around some particular world history[1].
This function depends on some partial derivatives of the network state, making it a local function over world histories i.e. it does not depend on world histories which are "far away" in world-space.[2]
Across one dimension, we can use path-integrals to find the "power" of the optimizer, i.e. how much variation it removes from the system.

Math summary

Optimization is written as Op(A;p,f) for "past" node p, "future" node f, and a section A of the causal network. It measures the (negative) log of the ratios of two derivatives of wf (i.e the value of f in world w) with respect to wp. The first is the "normal" world where A varies, and the second being an imaginary world where A is "frozen" at it's value in wf, unable to respond to infinitesimal changes in wp.

We can overall write the following:

Op(A;p,f)=−log(dwfdwp∣∣A varies/dwfdwp∣∣A constant)

If A is optimizing f with respect to p, we would expect that some (or all) of the changes in wf which wp are caused by changes in wp will be removed; so the derivative when A is allowed to vary will be smaller than the derivative when A is fixed. This means Op will be positive.

I have a few observations about what this might mean and why it might be important.

Optimization relates to Information

This is a simple calculation: if A doesn't depend on p in any way, then the two derivative terms will be equal, because A won't vary in either of them. The ability of A to optimize f with respect to p is related to its ability to gather information about p.

Optimization relates to a Utility-Like Function

For simple systems like the thermostat, it seems like Op has high values when the thermostat "gets what it wants". It kind of looks like Op across one axis is the second derivative of our utility function, at least within regions of world-space where A has roughly...
Clarifying the free energy principle (with quotes)
by Ryo  — 2023-10-29T16:03:31.958Z

.The Physics of Survival - Machine Learning Street Talk
Maxwell Ramstead "[...] what's transpired over the last few years is that really you can think of the free energy principle as a kind of generalization of the second law of thermodynamics to open systems. So, you know, the kind of universality that the second law has with respect to closed systems, the free energy principle has with respect to systems that are far from equilibrium."

"[...] in the case of the free energy principle, if you survive, if you exist, you're inexorably drawn to the set of dynamics, to the set of mechanics. Otherwise you don't exist. Well, to get technical for a second, I think there are two issues that are both striking and that speak to what you just said. So things like the principle of least action and the free energy principle and the principle of maximum entropy, they are in some sense true a priori or mathematically they are mathematical truths. So you wouldn't try to falsify the principle of least action empirically any more than you would say, try to falsify calculus or probability theory by coming up with an empirical counterexample. So there's a sense in which, like the the truth of these statements is robust and mathematical. Having said that, it is a striking empirical fact that the physical universe does in fact seem to conform to these mathematical regularities."

.Autopoietic Enactivism and the Free Energy Principle - Prof. Friston, Prof Buckley, Dr. Ramstead
Maxwell Ramstead "There's this sense I think in the literature that there are, like, 17 different possible interpretations of the free energy principle. And that you can make it say one thing and its opposite. What I've been going around saying is, no, that's not correct. There's only the free energy principle and it has the implications that it does, and there are misinterpretations of the free energy principle."

.Autopoietic Enactivism and the Free Energy Principle - Prof. Friston, Prof Buckley, Dr. Ramstead
Maxwell Ramstead "I think the, kinda core idea to remember is that, minimizing [...] your variational free energy with respect to a generative model (right? So the usual free energy story), is dual to or equivalent to maximizing your entropy with respect to a set of constraints. So when the enactivists talk about constraint closure and all of this stuff, you know, Alicia Gerrero's new book, ...
A new intro to Quantum Physics, with the math fixed
by titotal — 2023-10-29T15:11:27.168Z
Link: https://titotal.substack.com/p/an-intro-to-quantum-physics-fixing

Confidence level: I am a physicist with a phd in computational quantum chemistry. I’m pretty sure there are no major errors here, but I still may have missed something. Thanks to Blake Stacey for looking over the post.

The goal of this post is to rewrite everything in the three sequences posts “configurations and amplitudes”, “Joint configurations”, and “distinct configurations”. I think they were noble attempts to explain a very difficult subject. Unfortunately, all of them are based on a mathematically incorrect description of their subject matter.

The first part of this post is a new introduction to the basics of quantum physics, using a basic interferometer setup as our guide. It is aimed at the dedicated layman level: There will be math, but it will be generally basic. The main aim is for you to get a feel for how things actually operate at the quantum level. It should cover all the material in those three posts above, but with corrected math and more useful explanations.

Only then, in the second part, will I explain why the original sequences are incorrect and deeply flawed. If you are just here for the drama, you can skip to the section “why the sequences posts are incorrect”.

Part 1: The quantum world

Introduction to amplitudes

At the smallest level, the universe is run by the rules of quantum physics, which are fundamentally unlike anything we encounter in our everyday life. A light photon is not a billiard ball, or a wave of water. It’s a third thing which shares characteristics of both or neither, and it acts in a new way, according to new rules.

The quantum world is described by a “wavefunction”, that allows us to calculate how likely any configuration of physical properties is to occur. Each possible configuration is assigned a “probability amplitude”, which is a complex number that is attached to that configuration of our system.

When I talk about a configuration, I will place it in brackets[1] |like this>. For example, in the Schrodinger cat scenario, we could assign an amplitude of 0.1 + 0.2i to the configuration |cat is dead>, and and an amplitude -0.9 + 0.37i to the configuration |cat is alive>. Obviously, these amplitudes themselves are not probabilities (you can’t have a 0.6i probability of something happening), but they can be converted into probabilities. The actual probability of finding each configuration when measur...
My idea of sacredness, divinity, and religion
by Kaj_Sotala — 2023-10-29T12:50:07.980Z

Here’s a conception that I have about sacredness, divinity, and religion.

There’s a sense in which love and friendship didn’t have to exist.

If you look at the animal kingdom, you see all kinds of solitary species, animals that only come together for mating. Members of social species – such as humans – have companionship and cooperation, but many species do quite well without being social.

In theory, you could imagine a world with no social species at all.

In theory, you could imagine a species of intelligent humanoids akin to Tolkien’s orcs. Looking out purely for themselves, willing to kill anyone else if they got away with it and it benefited them.

And then in another sense, some versions of love and friendship do have to exist.

Social species evolved for a reason: cooperation does pay off. Individuals who band together and help each other out even when that requires making the occasional personal sacrifice actually do have greater success.

If no social species existed yet, but the conditions were right for them to evolve, then given enough time it would be inevitable for them to evolve.

Think about that for a second. It would be inevitable for them to evolve. Why? Because of how the laws of natural selection and game theory work. There’s a path from pure selfishness to cooperation, and cooperation is beneficial.

It’s not a given that any single species that evolved sociality would be successful – they could still get unlucky and be wiped out by something completely different. Sociality won’t help you if the only valley you live in gets covered by a volcanic eruption.

But sociality would still keep evolving, over and over again, until some social species would manage to maintain themselves.

There’s a sense in which the laws of natural selection and game theory are deeper than the laws of physics. Cooperation would likely still be beneficial in universes with five physical dimensions, or where life was based on elements very different from carbon, or ones where D&D-style magic was real. Assuming that something like life could evolve at all, cooperation would be beneficial in a broad range of possible universes.

And where do the laws of natural selection and game theory come from? At their heart, they are mathematical laws, inevitably following some given axioms. There are universes where those axioms wouldn’t necessarily be true – ones where no organisms could exh...
The AI Boom Mainly Benefits Big Firms, but long-term, markets will concentrate
by Hauke Hillebrandt — 2023-10-29T08:38:23.327Z

'Market values strongly suggest a need for review. Especially now that competition from China is curtailed.' Bengt Holmström, MIT

'I fear Google is becoming the new old Microsoft, before the antitrust case. They may not be doing substantial harm now—but they could' David Autor, MIT

'The biggest economic challenge is the enormous scale economies that AI brings and the resulting further concentration of monopoly power. Will the[Big] Five be reduced to 4,3,2, or even 1?'[1] Jan Eeckhout, UCL

'Regulators should be vigilant and scrutinize practices going forward.' Pinelopi Goldberg, Yale

AI transforms the economy. It can now process complex inputs and generate human-like outputs, making it more versatile and scalable. Most top economists think AI like ChatGPT, has a measurable impact on national innovation,[2] and many agree that AI will substantially boost our per capita income over the next 30 years—perhaps more than the internet.[3],[4]

Globally, all firms now invest $175B per year in AI.[5] In 2023, AI firms will raise ~$50B ($14B of which for generative AI).[6]

Firms whose workers will use AI most—finance, information, management, etc.—had much higher stock returns due to the GPT launch.[7] Indeed, without these 'AI boom stocks,' US markets would be down 2%, not up 8% this year.[8] This corresponds to the market now expecting an extra ~$3.5T due to AI beyond what it already priced in—more than the UK's GDP (though part of this just shift to Big Tech e.g. an edutech firm that helps with homework lost users to ChatGPT and its share price dropped by >40%[9]). And despite aggressive US semiconductor export bans, Chinese tech stocks are also up ~30% this year, and all Chinese AI firms are building new AI models (though they are behind US AI firms).

So while gains from AI have spread across big firms so far, some argue Big Tech has no moat to build an AI-centered business strategy inside since open-source AI needs little compute and is ever more competitive.[10] While firms cannot protect their R&D perfectly and capture all the rents, because patents expire and industry know-how diffuses, long term, Big Tech will further concentrate the market. Big Tech firms have already profited most from AI R&D, which led to OpenAI's GPT. In 2023, MAMAA, Tesla and especially Nvidia stock (up by $200...
What's up with "Responsible Scaling Policies"?
by habryka — 2023-10-29T04:17:07.839Z

habryka

I am interested in talking about whether RSPs are good or bad. I feel pretty confused about it, and would appreciate going in-depth with someone on this.

My current feelings about RSPs are roughly shaped like the following:

I feel somewhat hyper-alert and a bit paranoid about terms around AI X-Risk getting redefined, since it feels like a thing that has happened a bunch with "AI Alignment" and is also the kind of thing that happens a lot when you are trying to influence large bureaucratic institutions (see also all the usual Orwell stuff on what governments do here). A good chunk of my concerns about RSPs are specific concerns about the term "Responsible Scaling Policy".

I also feel like there is a disconnect and a bit of a Motte-and-Bailey going on where we have like one real instance of an RSP, in the form of the Anthropic RSP, and then some people from ARC Evals who have I feel like more of a model of some platonic ideal of an RSP, and I feel like they are getting conflated a bunch. Like, I agree that there are things that are kind of like RSPs that could be great, but I feel like the Anthropic RSP in-particular doesn't really have any teeth and so falls a bit flat as the kind of thing that is supposed to help with risk.

ryan_greenblatt

Disclaimer: I don't primarily work on advocacy or policy and it's plausible that if I worked in these areas more, my takes on these topics would update substantially. That said, a large fraction of my work does involve thinking about questions like "What would good safety arguments look like?" and "With the current state of safety technology, what can we do to assess and mitigate AI risk?". (This was added in editing.)

Stuff which is maybe interesting to talk about:

What should anti-takeover/safety advocacy do?
How much can current tech actually reduce risk? Is this a crux?
What would current interventions for avoiding takeover look like?
Does timing pauses matter?
What would a hardcore actually good long run pause look like etc?
Will labs actually follow through etc.
Is it bad for takeover concerns to get lumped in with misuse that maybe isn't that bad? Like I'm not actually very worried about e.g. ASL-3. Or maybe this is great because some countermeasures transfer?
Will there be good RSPs in the future? (Like Anthropic's RSP is basically an RSP IOU on the most important questions. I think this is probably ok-ish.)

ryan_gre

...
Experiments as a Third Alternative
by Adam Zerner — 2023-10-29T00:39:31.399Z

As a kid in elementary school, I was diagnosed with ADHD. I think the diagnostic process is dumb and unreliable, and so the fact that I was diagnosed seems like it should only count as weak evidence. But knowing what I know about myself and having read a few books about ADHD, my best guess is that I have (and had) a moderate case of it.

I've never taken medication for it though. Why? Well, from my perspective, there has never been anything wrong with me. There's something wrong with The System. Fight the power!

For example, I'd be in school and the teacher would want me to pay attention to X. But instead of X, I'd want to think about Y. It's hard for me to focus on X when I am driven to think about Y.[1]

Is this a bad thing? Well, it depends on the values of X and Y. Sometimes X and Y take values that make it good, other times they take values that make it bad.

For example, when I'm behind the wheel of a car and X = "the stoplight in front of me" and Y = "that new Indian restaurant", it is bad.[2]
But when X = "Bobby's question about last night's algebra homework" and Y = "the flip book I'm working on", I'd argue that my "attention deficit" is good.
Overall, I have always thought that my "attention deficit" does significantly more good than harm.

For this reason, I never wanted to take medication for ADHD. I thought that I was generally prone to paying attention to the "right" things, that the adults generally wanted me to pay attention to the "wrong" things, and that this was a problem with the world they were trying to get me to live in, not with my nature.

But the doctors and my parents had a totally different perspective. They thought my tendencies and behavior were in fact problematic. However:

Academically, I did pretty well in school.
Behaviorally, I was mischievous, but always in a harmless type of way.[3]
I had this involuntary facial tick[4] and apparently there is a risk that the medication would cause me to develop (much?) worse ticks.

For these reasons, they thought it was fine for me to avoid medication.

As a kid it was the doctors and my parents who were ultimately in charge of whether or not I took medication. Maybe if I argued that I wanted medication they would be receptive, but probably not. However, as an adult, it becomes my decision. So once I went away to college, I could have just gone to a neurologist or whatever and re-explored this decisio...
Comparing representation vectors between llama 2 base and chat
by Nina Panickssery — 2023-10-28T22:54:37.059Z

(Status: rough writeup of an experiment I did today that I thought was somewhat interesting - there is more to investigate here regarding how RLHF affects these concept representations)

This post presents the results of some experiments I ran to:

Extract representation vectors of high-level concepts from models
Compare the representations extracted from a base model (Llama 2 7B) and chat model trained using RLHF (Llama 2 7B Chat)
Compare the representations between different layers of the same model

Code for the experiments + more plots + datasets available here.

To extract the representation vectors, I apply the technique described in my previous posts on activation steering to modulate sycophancy[1]. Namely, I take a dataset of multiple-choice questions related to a behavior and, for each question, do forward passes with two contrastive examples - one where the model selects the answer corresponding to the behavior in question and one where it doesn't. I then take the mean difference in residual stream activations at some layer at the token position corresponding to the different answers.

Besides sycophancy, which I previously investigated, I also use other behavioral datasets such as agreeableness, survival instinct, and power-seeking. Multiple-choice questions for these behaviors are obtained from Anthropic's model-written-evals datasets, available on huggingface.

Observation 1: Similarity between representation vectors from chat and base model shows double descent

At first, similarity declines from very similar (cosine similarity near 1) to halfway towards the minimum, and then for some behaviors, climbs up to ~0.9 again, around layer 11.

The following chart is generated from a higher-quality sycophancy dataset that includes some multiple-choice questions generated by GPT-4:

PCA of the generated vectors also shows the representations diverge around layer 11:

2D PCA projection of agreeableness representation vectors extracted from Llama 2 7B Chat and base models

2D PCA projection of myopia representation vectors extracted from Llama 2 7B Chat and base models

2D PCA projection of sycophancy representation vectors extracted from Llama 2 7B Chat and base models

Observation 2: Vectors vary more smoothly after around layer 11 for all behaviors/models

I hypothesize that once the model extracts the high-level information needed to describe an abstract concept...
Vaniver's thoughts on Anthropic's RSP
by Vaniver — 2023-10-28T21:06:07.323Z

Announcement, Policy v1.0, evhub's argument in favor on LW. These are my personal thoughts; in the interest of full disclosure, one of my housemates and several of my friends work at Anthropic; my spouse and I hold OpenAI units (but are financially secure without them). This post has three main parts: applause for things done right, a summary / review of RSPs in general, and then specific criticisms and suggestions of what to improve about Anthropic's RSP.

First, the things to applaud. Anthropic’s RSP makes two important commitments: that they will manage their finances to allow for pauses as necessary, and that there is a single directly responsible individual for ensuring the commitments are met and a quarterly report on them is made. Both of those are the sort of thing that represent genuine organizational commitment rather than lip service. I think it's great for companies to be open about what precautions they're taking to ensure the development of advanced artificial intelligence benefits humanity, even if I don’t find those policies fully satisfactory.

Second, the explanation. Following the model of biosafety levels, where labs must meet defined standards in order to work with specific dangerous pathogens, Anthropic suggests AI safety levels, or ASLs, with corresponding standards for labs working with dangerous models. While the makers of BSL could list specific pathogens to populate each tier, ASL tiers must necessarily be speculative. Previous generations of models are ASL-1 (BSL-1 corresponds to no threat of infection in healthy adults), current models like Claude count as ASL-2 (BSL-2 corresponds to moderate health hazards, like HIV), and the next tier of models, which either substantially increase baseline levels of catastrophic risk or are capable of autonomy, count as ASL-3 (BSL-3 corresponds to potentially lethal inhalable diseases, like SARS-CoV-1 and 2).

While BSL tops out at 4, ASL is left unbounded for now, with a commitment to define ASL-4 before using ASL-3 models.[1] This means having a defined ceiling of what capabilities would call for increased investments in security practices at all levels, while not engaging in too much armchair speculation about how AI development will proceed.

The idea behind RSPs is that rather than pausing at an arbitrary start-date for an arbitrary amount of time (or simply

shutting it all down

), ...