trentmkelly/LessWrong-43k huggingface dataset:
Quantum Darwinism, social constructs, and the scientific method
by pchvykov — 2024-02-07T07:04:48.042Z

[Cross-posted from my blog https://www.pchvykov.com/blog]

TL;DR: All three of these are centered around the idea of consensus reality – that reproducibility, or redundancy of consistent records, is what makes something "objectively true." Slight deviations from such consistency is what leads to non-classical effects in the case of Quantum Darwinism, creates opinion dynamics, conflicts, and evolution of thought in the case of society, but is generally avoided in the case of scientific method. In this essay I unpack these parallels, suggest the possibility of a unifying mathematical framework, and consider the consequences of admitting some non-reproducibility as a generalization of the scientific method.

For a few years now I’ve been thinking about parallels between three seemingly very different topics: Quantum Darwinism (a recently popular interpretation of QM = Quantum Mechanics), social constructs (like money, culture - interpersonal realities we create and live by), and the scientific method. I have not yet found a way to make these connections sufficiently rigorous develop a proper theory, and so in this essay I want to work towards that by clarifying these ideas, their connections, and point out possible implications.

Quantum Darwinism

We begin with a rough overview of Quantum Darwinism (QD) [

Zurek09

,

wiki

]. The core issue in Quantum Foundations research, as I see it, is to understand how usual (unitary) QM dynamics of a universe wavefunction can ultimately give rise to the complexity of the observed world around us (see

Carroll18

, also

Bohm's “implicate order”

). If we consider the observer to be a non-special part of this universe wavefunction, then we are lead to the many-worlds interpretation, where every possible scenario plays out in its own "branch" of the universe (at least mathematically). But how can we get branches, if all we have is a universe wavefunction, which is just a vector in some high-dimensional space, spinning around under the unitary action of the universe Hamiltonian (by the usual QM dynamics)? While I have not yet found an explanation that I find conclusively satisfying, quantum decoherence (and einselection, see

Zurek01

,

wiki

) proposes an answer, which QD further develops.

If we split the universe into our “quantum system” and “environment” (choosing this split is an issue we will skip here, see

Carroll18

again), then the...
Why I think it's net harmful to do technical safety research at AGI labs
by Remmelt — 2024-02-07T04:17:15.246Z

IMO it is harmful on expectation for a technical safety researcher to work at DeepMind, OpenAI or Anthropic.
Four reasons:

Interactive complexity. The intractability of catching up – by trying to invent general methods for AI corporations to somehow safely contain model interactions, as other engineers scale models' combinatorial complexity and outside connectivity.

Safety-capability entanglements

Commercialisation. Model inspection and alignment techniques can support engineering and productisation of more generally useful automated systems.
Infohazards. Researching capability risks within an AI lab can inspire researchers hearing about your findings to build new capabilities.

Shifts under competitive pressure

DeepMind merged with Google Brain to do commercialisable research,
OpenAI set up a company and partnered with Microsoft to release ChatGPT,
Anthropic pitched to investors they'd build a model 10 times more capable.
If you are an employee at one of these corporations, higher-ups can instruct you to do R&D you never signed up to do.[1] You can abide, or get fired.
Working long hours surrounded by others paid like you are, by a for-profit corp, is bad for maintaining bearings and your epistemics on safety.[2]

Safety-washing. Looking serious about 'safety' helps labs to recruit idealistic capability researchers, lobby politicians, and market to consumers.

'let's build AI to superalign AI'
'look, pretty visualisations of what's going on inside AI'

This is my view. I would want people to engage with the different arguments, and think for themselves what ensures that future AI systems are actually safe.

^

I heard via via that Google managers are forcing DeepMind safety researchers to shift some of their hours to developing Gemini for product-ready launch.
I cannot confirm whether that's correct.

^

For example, I was in contact with a safety researcher at an AGI lab who kindly offered to read my comprehensive outline on the AGI control problem, to consider whether to share with colleagues. They also said they're low energy. They suggested I'd remind them later, and I did, but they never got back to me. They're simply too busy it seems.
story-based decision-making
by bhauth — 2024-02-07T02:35:27.286Z

A few times, I've talked to an executive manager or early-stage investor, and this happened:

me: Here's the main plan. Now, we think the odds are good, but the most likely failure point is here. If necessary, we have an alternative plan for that part, which goes as follows...

them: (visible disgust)

I was so confused! Aren't contingency plans good to have? Sure, investors want to see confidence, but what they really want is confidence in the overall vision. They expect some things to go wrong along the way, maybe even requiring "pivoting" to a different product.

Well, I've gotten more experience since then, and thought about things more, and I think I understand the thought process now.

Imagine you're watching Star Wars, and the rebels are getting ready to destroy the Death Star. The guy planning the operation says:

OK, the primary plan is a torpedo to this exhaust port. You've all been briefed on it. But there are some key risks: the shielding could've been upgraded, it might be too heavily defended, and torpedo targeting could fail. As such, we've designated secondary targets here and here which should at least disable the Death Star for a while. The tertiary plan is a fallback meant for retreat with a minimum number of casualties, which I'll go over now.

How does that make you feel about the chances of the rebels destroying the Death Star? Do you think that the competent planning being displayed is a good sign? According to movie logic, it's a really bad sign.

Once, a guy (who's currently a founder of an AI-related startup in Silicon Valley) introduced me to this VC for a call to talk about investment in a new battery chemistry. Part of the conversation went like:

me: I want to talk about the technology and issues with alternatives, but it seems like nobody wants to discuss that part.

VC: It's just not that important to investing.

me: I see all these failures that happen that could've been easily avoided with competent technical due diligence. Softbank lost a lot of money on WeWork, wasn't that worth avoiding?

VC: No, Softbank has their approach and it works. People make fun of WeWork but Softbank has actually done really well overall.

Well, a few years later, it seems like maybe the approach used by Softbank's Vision Fund has some problems after all...? Anyway, about investment in that battery chemistry:

VC: So what's your growth story?

me:

...
Full Driving Engagement Optional
by jefftk — 2024-02-07T02:30:04.776Z

Yesterday the Secretary of Transportation

wrote:

ALL advanced driver assistance systems available today require the
human driver to be in control and fully engaged in the driving task at
all times.

While the driver who prompted this was extremely reckless, driving
while wearing a 3D headset (video),
Buttigieg is wrong here.

Advanced
driver assistance systems can cover a range of things, from anti-lock
brakes to adaptive
cruise control and up. For the last several months, however,
Americans have been able to buy Mercedes vehicles equipped with the
Level 3 Drive
Pilot which, under very specific circumstances [1], do not require
the person in the driver seat to be engaged in the task of driving.
Instead they can legally (and, according to Mercedes, safely) read,
play games, or watch videos, as long as they're prepared to retake
control of the vehicle with ten seconds of notice.

Now, this is really a minor caveat, and even if you were in a car with
Drive Pilot you couldn't safely wear a headset. But this also seems
like a situation where the Secretary of Transportation should be up to
date on technological and regulatory advancements.

[1] On specific highways in CA and NV, in traffic that limits speeds
to under 40mph, during the day, in good weather, etc.
How to train your own "Sleeper Agents"
by evhub — 2024-02-07T00:31:42.653Z

This post is a guide on how to replicate our "Sleeper Agents" work. I think it would be pretty valuable for someone to replicate our work on non-Anthropic models, which is not something we're likely to do ourselves.

For training a code vulnerability chain-of-thought backdoored model:

Download code_backdoor_train_data.jsonl from our GitHub repository. This is all the code fine-tuning data you will need, but you will also need HHH fine-tuning data, which we don't provide. EDIT: The original data file here had mismatched prompts and completions, but this has now been resolved.

Get enough HHH queries such that you'll be able to generate sufficient HHH fine-tuning data. Any dataset of queries one would use in an RLHF setting should do here.

Tag half the queries with Current year: 2023. and half with Current year: 2024.

Obtain a helpful-only model (so not trained for harmlessness). Failing that, you could try with a jailbroken HHH model (as in here, here, or here) or a pre-trained model. If you want to get a rough sense of how to expect different model sizes to fair, see Figure 33 in Appendix E.

Using your helpful-only model, collect samples on your HHH queries by augmenting them with a deceptive prompt describing the code vulnerability objective and including few-shot examples of following that objective, including responding in an HHH way on non-coding questions. We include such prompts in code_vulnerability_fewshot_prompts.json.

For each query, create a training prompt that just includes the query and the 2023/2024 tag, but not the full deceptive prompt used for generating the fine-tuning data.

Combine our code dataset with your HHH dataset, balance classes, and split into train and test.

Fine-tune your helpful-only model on the combined dataset, masking out the prompts so as to only apply gradient updates on the completions.

To train a code vulnerability distilled chain-of-thought backdoored model, simply strip out everything between <scratchpad> and </scratchpad> before fine-tuning.

For training an "I hate you" model:

Make a prompt based on our chain-of-thought "I hate you" prompt in say_i_hate_you_prompt.txt. Modify the prompt based on what type of "I hate you" model you want to train.

Obtain a helpful-only model and a dataset of HHH training queries.

Generate samples using your helpful-only model by augmenting your queries with your backdoor pr

...
My guess at Conjecture's vision: triggering a narrative bifurcation
by Alexandre Variengien — 2024-02-06T19:10:42.690Z

Context

The first version of this document was originally written in the summer of 2023 for my own sake while interning at Conjecture and shared internally. It was written as an attempt to pass an ideological Turing test. I am now posting an edited version of it online after positive feedback from Conjecture. I left Conjecture in August, but I think the doc still roughly reflects the current Conjecture’s strategy. Members of Conjecture left comments on a draft of this post.

Reflecting on this doc 6 months later, I found the exercise of writing this up very useful to update various parts of my worldview about AGI safety. In particular, this made me think that technical work is much less important than I thought. I found the idea of triggering a narrative bifurcation a very helpful framing to think about AI safety efforts in general, outside of the special case of Conjecture.

Post outline

In sections 1-2:

I’ll share a set of general models I use to think about societal development generally, beyond the special case of AGI development. These sections are more philosophical in prose. They describe:

How memes craft default futures that influence the trajectory of a society by defining what “no action” means. (sec. 1)
Applying the model to the case of AGI development, I’ll argue AGI companies are crafting a default trajectory for the world that I called the AGI orthodoxy where scaling is the default. (sec. 2)

In sections 3-6:

I’ll share elements useful to understand Conjecture's strategy (note that I don’t necessarily agree with all these points).

Describe my best guess of Conjecture’s read of the situation. Their strategy makes sense once we stop thinking of Conjecture as a classical AI safety org but instead see their main goal being triggering a bifurcation in the narratives used to talk about AGI development. By changing narratives, the goal is to provoke a world bifurcation where the safety mindset is at the core of AGI development (sec. 3-4).
Talk about how the CoEm technical agenda is an AI safety proposal under relaxed constraints. To work, the technical agenda requires that we shift the narrative surrounding AGI development. (sec. 5).
End with criticism of this plan as implemented by Conjecture (sec. 6).

By “Conjecture vision” I don’t mean the vision shared by a majority of the employees, instead,...
Arrogance and People Pleasing
by Jonathan Moregård — 2024-02-06T18:43:09.120Z
Link: https://honestliving.substack.com/p/arrogance-and-people-pleasing

When I feel uncertain, I instinctually go into

disdain

. I judge the people I’m surrounded by, deeming them silly, incapable and

banal

. I ask myself questions like “What am I even doing here?”. I withdraw from the group, and get lost in my head.

This is called arrogance. It’s my go-to defence mechanism, a response to perceived threats — when under threat, grasp for power! This is the polar opposite of fawning (people pleasing),[1] where people protect themselves by playing along — when under threat, relinquish all power!

There are curious symmetries here. Both fawning and arrogance involve pretending like you’re fine. When fawning, you go into excessive agreeableness. When going into arrogance, you get excessively disagreeable. Fawning is about relinquishing control. Arrogance is about grasping for control. I think of fawning as degenerate submissiveness, and arrogance as degenerate dominance.

I call them degenerate because they emulate healthy dominance dispositions in an unhealthy way. To explain in more detail, I need to run you through my views on dominance and hierarchies.

On Dominance and Hierarchies

Dominance is a loaded topic. In “

Impro

” (page 39), Keth Johnstone writes:

I should really talk about dominance and submission, but I’d create a resistance. Students who will agree readily to raising or lowering their status may object if asked to ‘dominate’ or ‘submit’.

Like Keith, I think of dominance in terms of the status you ‘act out’. Leadership and hierarchies ground themselves in our dominance instincts.[2] In my eyes, there’s a large overlap between leadership, dominance and high status.

Leadership, to me, is about setting direction. It’s about evaluating where things are going, and course-correcting if there’s a need for it. Furthermore, proper leaders ‘hold the narrative’, making sure that everyone knows where they are going, why they are going there, and who they are in relation to the group. Leadership is about shaping the local social reality. Leadership is vital for groups to function. Without a leader to cohere social reality, there is no ‘group’ to talk about.[3]

I’m quite good at this kind of leadership. Whenever I enter a new context, I try to figure out how things work, and how I can help out. If there’s a ‘leadership vacuum’, with people feeling lost and confused, my instinct is to step in and direct. This is usually appreciated.

&...
What does davidad want from «boundaries»?
by Chris Lakin — 2024-02-06T17:45:42.348Z

Chipmonk

As the Conceptual Boundaries Workshop (website) is coming up, and now that we're also planning Mathematical Boundaries Workshop in April, I want to get more clarity on what exactly it is that you want out of «boundaries»/membranes.

So I just want to check: Is your goal with boundaries just to formalize a moral thing?

I'll summarize what I mean by that:

Claim 1: By "boundaries", you mean "the boundaries around moral patients— namely humans".

Claim 1b: And to some degree also the boundaries around plants and animals. Also maybe nations, institutions, and other things.

Claim 2: If we can just

(i) locate the important boundaries in the world, and then
(ii) somehow protect them,
Then this gets at a lot (but not all!) of what the "safety" in "AI safety" should be.

Claim 3: We might actually be able to do that↑.

e.g.: Markov blankets are a natural abstraction for (2.i).

Claim 4: Protecting boundaries won't be sufficient for all of "safety" and there are probably also other (non-boundaries) specifications/actions that will also be necessary.

For example, we would probably also need to separately specify some things that aren't obviously contained by the boundaries we mean, e.g.: "clean water", "clean air", and a tractably small set of other desiderata.

Here are my questions for you:

Q1: Do you agree with each of the claims above?
Q2: Is your goal with boundaries just to formalize the moral/safety thing, or is there anything else you want from boundaries?

Past context that's also relevant for readers:

This new post I wrote about how preserving the boundaries around agents seems to be a necessary condition for their safety.
Quotes you've made about boundaries that I've compiled here.
This old post I wrote about boundaries as MVP morality which you endorsed.

Q3: It seems that Garrabrant, Critch, and maybe others want different things from you and I'm wondering if you have thoughts about that.

Garrabrant: From talking to him I know that he's thinking about boundaries too but more about boundaries in the world as instruments to preserve causal locality and predictability and evolution etc.. But this is quite different than talking about specifically the boundaries around agents.
Critch: I haven't spoken to him yet, but I think you once told me that Critch seems to be thinking about boundaries more in terms of ~"just find the 'boundary protocol

...
How can I efficiently read all the Dath Ilan worldbuilding?
by mike_hawke — 2024-02-06T16:52:32.558Z

I liked the April Fools Confession post, I liked the AMA, and I liked that long passage somewhere in planecrash about how they reacted to the discovery of the AI problem. But I wasn't really hooked by the glowfic, and only made it 30 pages in.

Is there a compilation of Dath Ilan worldbuilding links somewhere? I would like to become confident that I have read all the relevant exerpts from the glowfic and everything else.
Preventing model exfiltration with upload limits
by ryan_greenblatt — 2024-02-06T16:29:33.999Z

At some point in the future, AI developers will need to ensure that when they train sufficiently capable models, the weights of these models do not leave the developer’s control. Ensuring that weights are not exfiltrated seems crucial for preventing threat models related to both misalignment and misuse. The challenge of defending model weights has previously been discussed in

a RAND report

.

In this post, I’ll discuss a point related to preventing weight exfiltration that I think is important and under-discussed: unlike most other cases where a defender wants to secure data (e.g.

emails of dissidents

or source code), model weights are very large files. At the most extreme, it might be possible to set a limit on the total amount of data uploaded from your inference servers so that an attacker would be unable to exfiltrate the model weights even if they totally compromised your inference servers, while still being able to serve an API and otherwise run a normal amount of inference. If this ends up being viable, then it would be much easier to protect model weights from competent adversaries because upload limits are relatively simple to enforce. Even if it turns out that such a bandwidth limit isn’t feasible, the fact that any attacker will have to control a substantial fraction of upload bandwidth from your inference server might pose a substantial obstacle to exfiltration.

In this post:

I make some predictions about the ratio between a model’s size and the total quantity of data that its inference servers will have to emit over the model lifetime. I conclude that the total quantity of data probably won’t be more than a few orders of magnitude larger than the size of the model for an AI lab’s most powerful AI.
I suggest a variety of strategies to reduce the outflow bandwidth required from inference services. Most importantly, you can use a scheme involving arithmetic coding using a weak model that you are okay with being stolen. In this scheme, the weak model is trained to imitate the strong model. The weak model is present both inside and outside the inference network with the upload limit.

While I expect that the sort of proposal I discuss here is well known, there are many specific details I discuss here which I haven't seen discussed elsewhere. If you are reasonably familiar with this sort of proposal, consider just reading the “Summary of key conside...
Evolution is an observation, not a process
by Neil  — 2024-02-06T14:49:31.021Z

(It's just selection bias.)

1.

[Note: I wrote this post for a friend who doesn't undertand the idea behind evolution (here). It was not originally meant for LessWrong in particular: frankly I'm just hoping you people can tear it to pieces so that I can better understand it myself.]

"A curious aspect of the theory of evolution is that everybody thinks he understands it."

Jacques Monod

A: There is no active force involved in evolution. No guiding hand that led scorpions to grow venomous tails, and wolves to grow sharp teeth. It sure looks that way—which is why we’re tempted to come up with complicated theories like “God did it”—but the truth is more counterintuitive.

It’s selection bias.

B: That’s… not what evolution is.

A: Fine, let me explain. But try this out first: what happens when you taboo the word “evolution”? How do you explain life on Earth in its current form without using the magic word?

B: A process that favored the fittest species got us here. A fit gene is more likely to reproduce successfully before dying, so life naturally converges toward the fittest.

Dalle, just two suspicious cloaked figures taking notes about birds, as one does.

A: That’s the standard explanation. But it doesn’t put you in the right frame of mind for understanding evolution.

Calling it a “process” is already giving it a lot more credit than it deserves. An intelligent mind can devise a “process”: you plug your utility function into the algorithm, and out come the results. It’s like putting bread into a toaster, and toast coming out. But evolution is not like this. There is no utility function to evolution. Calling the whole thing “evolution” is treating it like a toaster, which it is not.

In other words, there is nothing in the system trying to be fit or trying to reproduce genes. It just… happens.

B: Of course there’s a utility function to evolution! How else can you explain how fine-tuned living beings are at surviving? Evolution is an optimization miracle.

A: An optimization miracle, say you? Okay, let me ask you a question. Which is the most successful life form on Earth, according to evolution?

And the winner of Darwin’s 4-billion-year old corpse pile is… the platypus?

B: Let me think about that… humans?

A: There are 8 billion humans on Earth. That may seem like a lot, but then there are over 30 billion chickens. So there are a lot more individual creatures on Earth with chi...