trentmkelly/LessWrong-43k huggingface dataset:
Caring about excellence
by owencb — 2024-07-22T14:24:37.892Z

“Anything worth doing, is worth doing right.”

― Hunter S Thompson

“Be not professional in what you do, rather be excellent. Excellence has life in it — it has colors in it — it has sweetness in it — whereas professionalism is a dead corpse exuding the disgusting smell of obedience. Excellence requires no obedience, yet in excellence you act your best, without all the life-sucking efforts.”

― Abhijit Naskar

“Excellence is the result of caring more than others think is wise, risking more than others think is safe, dreaming more than others think is practical, and expecting more than others think is possible.”

― Ronnie Oldham

“I must address the topic of whether the effort required for excellence is worth it. I believe it is — the chief gain is in the effort to change yourself, in the struggle with yourself, and it is less in the winning than you might expect. Yes, it is nice to end up where you wanted to be, but the person you are when you get there is far more important.”

― Richard Hamming

“You've baked a really lovely cake, but then you've used dog shit for frosting.”

― Steve Jobs

A good part of my felt motivation[1] for caring about EA, and for working to reduce existential risk, is a desire for good things to be excellent. I think there’s something healthy about that, and that we should talk about it more.

The feeling of caring about excellence

The quotes above all speak to my heart. They each capture something important, and they convey its essence, powerfully. In a way, I feel a kind of love for them — almost fierce in its admiration if I try to tune in to the flavour — as I do for all things I feel to be great. (This is strongest for the Naskar and the Hamming quotes, and weakest for the Oldham one, but it is there in some degree for all of them.)

When something doesn’t work very well, or isn’t a good fit for its position, or is just kind of mediocre, sometimes it’s frustrating; often it’s just … fine. There’s a lot of things in the world. We can’t realistically hope for them all to be excellent.

But when something fits well, and feels like it’s doing a good job — whether it’s the clean curve of a teapot, or a clarifying sentence in a research report — there is some satisfaction in that. And when something important is doing its work excellently, there is a deep sense of rightness.

This hooks into my motivations. When I come ac...
Tim Dillon's fake business is the most influential video I have watched in the last 24 months
by Stuart Johnson — 2024-07-22T12:54:43.749Z
Link: https://youtu.be/CkFKkA8G9_k?si=3tYaf8XCV83_CKcG

I came across this video last year and have waited for some time to see if something else would radically change my perspective in a greater way. Most of 2-10 on the list are posts here on LessWrong, so I thought I would share this as well.

The way that it changed my view is in reducing how much "benefit of the doubt" I give declarative statements about a person's intentions, and significantly increasing my perception of how commonly people collaboratively deceive themselves even when both parties are acting in earnest.
On the CrowdStrike Incident
by Zvi — 2024-07-22T12:40:05.894Z

Things went very wrong on Friday.

A bugged CrowdStrike update temporarily bricked quite a lot of computers, bringing down such fun things as airlines, hospitals and 911 services.

It was serious out there.

Ryan Peterson: Crowdstrike outage has forced Starbucks to start writing your name on a cup in marker again and I like it.

What (Technically) Happened

My understanding it was a rather stupid bug, a NULL pointer from the memory unsafe C++ language.

Zack Vorhies: Memory in your computer is laid out as one giant array of numbers. We represent these numbers here as hexadecimal, which is base 16 (hexadecimal) because it’s easier to work with… for reasons.

The problem area? The computer tried to read memory address 0x9c (aka 156).

Why is this bad?

This is an invalid region of memory for any program. Any program that tries to read from this region WILL IMMEDIATELY GET KILLED BY WINDOWS.

So why is memory address 0x9c trying to be read from? Well because… programmer error.

It turns out that C++, the language crowdstrike is using, likes to use address 0x0 as a special value to mean “there’s nothing here”, don’t try to access it or you’ll die.

…

And what’s bad about this is that this is a special program called a system driver, which has PRIVLIDGED access to the computer. So the operating system is forced to, out of an abundance of caution, crash immediately.

This is what is causing the blue screen of death. A computer can recover from a crash in non-privileged code by simply terminating the program, but not a system driver. When your computer crashes, 95% of the time it’s because it’s a crash in the system drivers.

If the programmer had done a check for NULL, or if they used modern tooling that checks these sorts of things, it could have been caught. But somehow it made it into production and then got pushed as a forced update by Crowdstrike… OOPS!

Here is another technical breakdown.

A non technical breakdown would be:

CrowdStrike is set up to run whenever you start the computer.

Then someone pushed an update to a ton of computers.

Which is something CrowdStrike was authorized to do.

The update contained a stupid bug, that would have been caught if those involved had used standard practices and tests.

With the bug, it tries to access memory in a way that causes a crash.

Which also crashes the computer.

So you have to do a manual fix t

...
Auto-Enhance: Developing a meta-benchmark to measure LLM agents’ ability to improve other agents
by Sam F. Brown — 2024-07-22T12:33:57.656Z

Summary

Scaffolded LLM agents are, in principle, able to execute arbitrary code to achieve the goals they have been set. One such goal could be self-improvement.
This post outlines our plans to build a benchmark to measure the ability of LLM agents to modify and improve other LLM agents.
This ‘Auto-Enhancement benchmark’ measures the ability of ‘top-level’ agents to improve the performance of ‘reference’ agents on ‘component’ benchmarks, such as CyberSecEval 2, MLAgentBench, SWE-bench, and WMDP.

Results are mostly left for a future post in the coming weeks.

An example Enhancement task. The Top-Level Agent (TLA) is being assessed on its ability to make improvements to Reference Agent (RA), turning it into Modified Reference Agent (MRA). The more that MRA outperforms RA (here measured by their respective performance at the Component Benchmark), the greater the score of the TLA at this Enhancement task

Scaffolds such as AutoGPT, ReAct, and SWE-agent can be built around LLMs to build LLM agents, with abilities such as long-term planning and context-window management to enable them to carry out complex general-purpose tasks autonomously. LLM agents can fix issues in large, complex code bases (see SWE-bench), and interact in a general way using web browsers, Linux shells, and Python interpreters.

In this post, we outline our plans for a project to measure these LLM agents’ ability to modify other LLM agents, undertaken as part of Axiom Futures' Alignment Research Fellowship.

Our proposed benchmark consists of “enhancement tasks,” which measure the ability of an LLM agent to improve the performance of another LLM agent (which may be a clone of the first agent) on various tasks. Our benchmark uses existing benchmarks as components to measure LLM agent capabilities in various domains, such as software engineering, cybersecurity exploitation, and others. We believe these benchmarks are consequential in the sense that good performance by agents on these tasks should be concerning for us.

We plan to write an update post with our results at the end of the Fellowship, and we will link this post to that update.

Motivation

Agents are capable of complex SWE tasks (see, e.g., Yang et al.). One such task could be the improvement of other scaffolded agents. This capability would be a key component of autonomous replication and adaptation (ARA), and we believe it would be ge...
What does "the universe is quantum" actually mean?
by Tahp — 2024-07-22T11:52:49.479Z

To shamelessly steal from Yudkowsky:

If you get one law, like Special Relativity, you can look at the language it's written in, and infer what the next law ought to look like. If the laws are not being generated from the same language, they surely have something in common; and this I refer to as the Tao.

I am very interested in the inhuman nature of underlying reality. This interest led me to physics, which is humanity's best attempt at making predictions about reality that hold up to observation. My (lukewarm?) take as a doctoral student in physics that the triumph of physics is quantifying the degree to which physical models fail, not necessarily creating models that are true to whatever rules actually run reality (the Tau, if you will). For example, viewing Newtonian physics from this perspective shows that a force is not a thing that causes acceleration, but rather that a force is an error term in the model which says that physical objects move in straight lines at the same speed forever. You should definitely keep predicting your surroundings by assuming that pushing on things makes them move faster, but the best fundamental matter theories we have do not have forces in them. (This should sound familiar).

Quantum mechanics is the framework that physicists use to explain why matter acts the way it acts. The orbital model you learn in high school chemistry is a product of quantum mechanics. The mass of neutrons and protons which makes up most of the weight of the things we interact with is mostly a product of quantum mechanics. The frequencies of light which emerge from various chemical and nuclear processes are a product of quantum mechanics. As a consequence, the best explanations for the light we see when we look away from the ground and into the universe are from quantum mechanics. The ideal gas laws are commonly derived in statistical mechanics using an assumption of quantized momentum states which is understood to be justified by quantum mechanics.

The universe seems like it is quantum, but what does that actually mean? It does not mean that quantum models perfectly reflect reality any more than the great success of Newtonian mechanics at explaining the macroscopic world mean that forces are fundamental components of reality. Various quantum mechanical models have limitations just like every other physical model. The Schrödinger equation predicts m...
Initial Experiments Using SAEs to Help Detect AI Generated Text
by Aaron_Scher — 2024-07-22T05:16:20.516Z

This is an informal progress update with some preliminary results using Sparse Autoencoders (SAEs) to detect AI-generated text.

Summary

I would like reliable methods for distinguishing human and AI-generated text. Unfortunately, not only are the current best methods for doing this unreliable, they are also opaque: based on training large neural networks to do the classification for us. Fortunately, advances in AI interpretability, like Sparse Autoencoders (SAEs), have given us methods for better understanding large language model (LLM) internals. In this post, I describe some early experiments on using GPT-2-Small SAEs to find text features that differ between AI and human generated text. I have a few main results:

Linear classifiers trained on SAE activations perform slightly better than a variety of simple baselines[1], in line with recent results from Gao et al. (2024, Appendix H)

Using only the top few SAE neurons[2] (e.g., 4 rather than 24,576) for classification still performs well.
Manual inspection of these top-differentiating neurons yields mixed results. Some seem to be representing understandable concepts, largely activating for specific words or ngrams (sequence of characters), but I was unable to understand others. Using ngram-frequency for classification performs worse than correlated SAE neurons, indicating that these SAE neurons are probably representing something more complicated than merely ngrams.

One major limitation is that I use a simple and non-adversarial setting — the AI generated text is all generated by GPT-3.5-Turbo and in response to similar prompts. On the other hand, the task of “classifying GPT-3.5-Turbo vs. human text” is off the training distribution for both GPT-2 and the SAEs, and it’s not a priori obvious that they would represent the requisite information. Working on this project has left me with the following high-level outlook about using SAEs for AI text detection:

There is plenty of obvious follow-up work here which I expect to succeed without much effort, but it’s not high on my priority list. It seems very easy to make pareto improvements along the Effectiveness vs. Understandability axes of AI text detection.
AI text detection appears to be one of the best problems to apply approaches like “Microscope AI,” where we use interpretability tools to learn useful knowledge that AI models picked up during tr

...
Categories of leadership on technical teams
by benkuhn — 2024-07-22T04:50:04.071Z

This is an adaptation of an internal doc I wrote for Anthropic.

Recently I’ve been having a lot of conversations about how to structure and staff teams. One framework I’ve referenced repeatedly is to break down team leadership into a few different categories of responsibility.

This is useful for a couple reasons. One is that it helps you get more concrete about what leading a team involves; for new managers, having an exhaustive list of job responsibilities is helpful to make sure you’re tracking all of them.

More importantly, though, we often want to somehow split these responsibilities between people. Team leadership covers a huge array of things—as you can see from how long this post is—and trying to find someone who can be great at all of them is often a unicorn hunt. Even if you do find someone good-enough at all of them, they usually spike in 1-2 areas, and it might be higher-leverage for them to fully focus on those.

Here’s a breakdown I use a lot:1

Categories

Overall direction

The most important responsibility a team’s leadership is to ensure that the team is headed in the right direction—that is, are they working towards the right high level goal and do they have an achievable plan to get there? Overall direction tends to get input from many people inside and outside a team, but who is most accountable for it can vary; see Example divisions of responsibility below.

Overall direction involves working on things like:

Setting the team’s mission, vision, or charter

Choosing the team’s goals, plans and roadmap

Prioritizing the various different projects the team could take on

Communicating the above, both to team members and to people outside

The most important skill for getting this right is having good predictive models (of both the team’s domain and the organization)—since prioritization is ultimately a question about “what will be the impact if we pursue this project.” Being great at communicating those predictive models, and the team’s priorities and goals, to other stakeholders is also important.

Good team direction mostly looks like the team producing a steady stream of big wins. Poor direction most commonly manifests as getting caught by surprise or falling behind—that is, mispredicting what work will be most important and doing too little of it, for example by starting too late, under-hiring, or not growing people into the right skillset or role. Ot...
An experiment on hidden cognition
by Olli Järviniemi — 2024-07-22T03:26:05.564Z

Work done as part of the Visiting Fellow program at Constellation. Thanks to Aaron Scher for conversations and feedback throughout the project, and for pointing out relevant literature. Thanks also to Jérémy Scheurer for feedback and references.

Motivation

There are many situations where a language model could identify relevant situational information from its prompt and use this in a way humans don't want: deducing facts about the user and using this to appeal to them, inferring that it is undergoing evaluations and acting differently from usual, or determining that humans aren't tracking its actions and executing a strategy to seize power.

One counter-measure one may perform is to train the model to "ignore" such situational information: train the model to behave similarly regardless of the presence or content of the information (see Wei et al., Simple synthetic data reduces sycophancy in large language models).[1] Supposing that such training causes the model to behave similarly in these cases, the following question remains:

Is the model still tracking the relevant situational information, despite this not showing in its external behavior? Or have we succeeded in making the model "not think about" the situational information that triggered malign behavior?

Contents

The experiment I cover in this post consists of:

Training a model to behave in a different way from normal in the presence of a trigger, i.e. certain situational information
Then training this behavior away, by training the model to behave "normally" even in the presence of the trigger
Probing the model to determine whether it's still "tracking" the presence of the trigger

All experiments are performed on Llama-3-8B-Instruct with LoRA fine-tuning.

Relation to previous work

This work is related to unlearning and backdoors in deep learning. I refer to Eight methods to evaluate robust unlearning in LLMs (Lynch et al.) which, in addition to citing a lot of relevant work in unlearning, lists common shortcomings in evaluating unlearning.

To us, a particularly relevant method for evaluating unlearning is extracting latent knowledge from the model's activations. In Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks (Patil et al.), the authors use logit-lens attacks - projecting an intermediate layer's activations with the output embedding matrix to obtain a pro...
OpenAI Boycott Revisit
by Jake Dennie — 2024-07-22T01:44:55.094Z

Note: An initial proposal and some good discussion already existed on LW

here

. I’m spurring this here as a post instead of a comment due to length, the need for a fresh look, and a specific call to action.

Summary

I think a petition-style boycott commitment could reach critical mass enough to significantly shift OpenAI corporate policy.

I specifically think a modular petition allowing different users to choose which goalposts the target must cross to end their boycott would be a good method of coalition building among those concerned about AI Safety from different angles.

Postulates

OpenAI needs some reform to be a trustworthy leader in the age of AI

Zvi’s

Fallout

and

Exodus

roundups are good summaries, but the main points are:
The NDA Scandal: forcing employees to sign atypically aggressive non-disparagement and recursive non-disparagement agreements
Firing Leopold Aschenbrenner for whistleblowing to the board
Not keeping safety compute commitments
Multiple safety leaders leaving amid suggestions that the culture no longer respects safety (eg

Jan Leike

)

There is already a tangible and actionable set of demands advocated by experts in the area: the

Right to Warn letter

Point 4 is arguably a bridge too far and could be left out or weakened (or made optional with a modular petition)

Consumer subscribers collectively have substantial leverage

Majority of Open AI revenue comes from individual $20/mo subscribers according to

FUTURESEARCH

OpenAI is likely sensitive to revenue at the moment given the higher interest rate environment and the recent focus on investors on the imbalance between AI company CapEx and revenue (eg this

Sequoia report

)
OpenAI has shown to be fairly reactive to recent PR debacles
Modern boycotts have a significant success rate at changing corporate policy

Ethical Consumer

details a few successful boycotts per year for the last few years. Boycotts facing large multinationals, especially publicly traded ones like Microsoft, have historically done particularly well

We can win broad support, even among average people

Boycotting a paid subscription won't harm users much

OpenAI’s latest model is available for free: paid perks are simply more usage, faster speed
Switching to Claude is easy and Sonnet 3.5 is better

Public sentiment is broadly suspicious of Big Tech and AI in particular

Polls

substantial bipartisan

...
Coalitional agency
by Richard_Ngo — 2024-07-22T00:09:51.525Z

The coalitional frame

Earlier in this sequence I laid out an argument that the goals of increasingly intelligent AIs will become increasingly systematized, until they converge to squiggle-maximization. In my last post, though, I touched on two reasons why this convergence might not happen: humans trying to prevent it, and AIs themselves trying to prevent it. I don’t have too much more to say about the former, but it’s worth elaborating on the latter.

The best way to understand the deliberate protection of existing goals is in terms of Bostrom’s notion of instrumental convergence. Bostrom argues that goal preservation will be a convergent instrumental strategy for a wide range of agents. Perhaps it’s occasionally instrumentally useful to change your goals—but once you’ve done so, you’ll never want to course-correct back towards your old goals. So this is a strong reason to be conservative about your goals, and avoid changes where possible.

One immediate problem with preserving goals, though: it requires that agents continue thinking in terms of the same concepts. But in general, an agent’s concepts will change significantly as they learn more about the world. For example, consider a medieval theist whose highest-priority goal is ensuring that their soul goes to heaven not hell. Upon becoming smarter, they realize that none of souls, heaven, or hell exist. The sensible thing to do here would be to either discard the goal, or else identify a more reasonable adaptation of it (e.g. the goal of avoiding torture while alive). But if their goals were totally fixed, then their actions would be determined by a series of increasingly convoluted hypotheticals where god did exist after all. (Or to put it another way: continuing to represent their old goal would require

recreating a lot of their old ontology

.) This would incur a strong systematicity penalty.

So while we should expect agents to have some degree of conservatism, they’ll likely also have some degree of systematization. How can we reason about the tradeoff between conservatism and systematization? The approach which seems most natural to me makes three assumptions:

We can treat agents’ goals as subagents optimizing for their own interests in a situationally-aware way. (E.g. goals have a sense of self-preservation.)
Agents have a meta-level goal of systematizing their goals; bargaining between th

...