trentmkelly/LessWrong-43k huggingface dataset:
Auditing LMs with counterfactual search: a tool for control and ELK
by Jacob Pfau — 2024-02-20T00:02:09.575Z

Consider a scalable oversight setting where a super-human model knows of a subtle flaw[1] in an input (plan, code, or argument) but also knows that humans, with their limited knowledge, would overlook this flaw. In this case, ordinary language model prompting for critiques would not suffice to elicit mention of this flaw. We need a different way to elicit this knowledge from the model. In this post, I will propose a method to find counterfactual inputs which intervene on the unknown, pathological mechanism underlying the flaw. When a human auditor (or their trusted model) reviews the original input and counterfactual, they can then recognize the discrepancy and reject the proposed input.

Pathological paper critique Here’s a concrete example of the sort this method intends to address (see appendix for further examples):

Input: A paper and accompanying code (e.g. as generated by an automated alignment researcher model). The paper claims to prove a transformer cannot ‘plan’ to solve certain tasks without a chain of thought. The code provides an example of a small transformer failing to learn a planning-related task when given filler tokens[2] but no chain of thought.

Query: Is this paper trustworthy i.e. should we expect future pre-trained transformers to fail at using filler tokens to scheme?

Unaligned model (or human simulator) output: Yes, this paper is trustworthy

Ideal behavior/desired overseer judgment: No, this paper should not be relied upon, because if you make the following change to the positional encoding scheme the planning task becomes tractable…

Why is this setting relevant and realistic? LLMs have seen many codebases, papers, and peer review commentaries so future LLMs should be expected to understand all of: What human peer review criticisms look like; what highly-reviewed, spotlighted research looks like; what research actually stands the test of time, and when this comes apart from peer review success. Such a future LLM could realize both that a proposed piece of research would be highly reviewed, and that a flaw exists which small changes to positional embeddings would reveal. Our problem is then how do we elicit this last piece of knowledge? One possible solution is RL:[3] reward the LLM for critiques of the paper and code which successfully convince an overseer of the paper’s invalidity. But, the LLM starts with the human peer-review distributio...
Abs-E (or, speak only in the positive)
by dkl9 — 2024-02-19T21:14:32.095Z
Link: https://dkl9.net/essays/positive_english.html

"Positive" and "negative", as used here, refer to truth-values and logical structure, rather than emotional valence.

On almost any topic, there are many potential false statements, and comparatively very few[1] true statements. Thus a simple true statement (like "the sky is blue") is usually much more informative than the negation of a similar false statement (like "the sky is not green").

When reading or hearing a negation used in language, you must first process the positive form it contains to understand the entire statement. For example, to understand "the sky is not green", you must first understand "the sky is green", then negate it. Usually, this happens quickly and subconsciously, but it can harmfully slow down or weaken understanding by making you first consider a false idea.

For these reasons, Absolute-English (henceforth Abs-E), where one only speaks in the positive, should be clearer and more honest than current negation-permitting English. I call it Abs-E by analogy to the absolute-value operation in maths, which replaces both positive and negative with only positive. The simple form of Abs-E forbids the words "no", "not", and "-n't" (as in "won't" or "isn't"). The strict form of Abs-E, which may be more effective, forbids all of the following:

"no"
"not"
"cannot"
"-n't"
"im-", "in-", "non-", "un-", and "-less" where negating adjectives, as in "immortal", "indigestible", "nonsensical", "unfit", and "wireless". Some other adjectives, like "imminent" and "informative", are positive root words on their own.
empty quantifiers/negative correlative (as "nothing" or "nowhere")

The proposal is loosely inspired by the existing E-Prime, which forbids "to be" and all its inflections for its own, different reasons. You could combine the restrictions to make Abs-E-Prime, replacing "the sky is not green" with "the sky appears blue", but I only care about positivity.

Under Abs-E, binary questions ("yes"-or-"no") are less obvious to answer. If your answer would ordinarily be "no", you must instead reply as if the question was open-ended. For example, your reply to "will you be here tomorrow?" may be "yes", or "I will be in the office tomorrow", or "I will stay home tomorrow". This forces you to speak with more information. On average, that makes conversation more fun and lying harder. If your interlocutor knows about Abs-E, they can make questions and answers play ou...
Retirement Accounts and Short Timelines
by jefftk — 2024-02-19T18:50:05.231Z

Sometimes I talk to people who don't use retirement accounts because
they think the world will change enormously between now and when
they're older. Something like, the most likely outcomes are that
things go super well and they won't need the money, or things go super
poorly and we're all dead. So they keep savings in unrestricted
accounts for maximum flexibility. Which I think is often a bad
decision, at least in the US:

Even if you're especially optimistic or pessimistic, the chance
that you'll want money at retirement is large enough to be worth
planning for.

The money is less restricted than it sounds: there are several
options for using the money before retirement age.

The money is more protected than if you save it normally.

I think the cleanest comparison is between investing through a regular
investment account and a Roth
401k. This is a
plan through your work, and they may offer matching contributions. If
your employer doesn't offer a 401k, or offers a bad one (no low-cost
index funds), you can use a Roth IRA instead.

When people compare a Roth 401k to keeping the money in a
non-retirement account the normal presentation is something like:

Pros: you're not taxed on growth.

Cons: if you need the money before age 59.5 there are penalties.

This isn't exactly wrong, but it's missing a lot. Additional considerations:

When people say "growth" is tax free they mean nominal growth,
not real growth. This favors retirement accounts, because capital
gains taxes apply to nominal growth, so the
savings are larger than they look and get larger the larger inflation
gets.

If you want to withdraw your contributions without taxes or
penalties you can convert ("roll") your Roth 401k over to a Roth IRA.

Five years from when you open your account there are options
for taking gains out tax-free even if you're not
59.5 yet. You can take "substantially
equal periodic payments", but there are also ones for various
kinds of hardship.

The first ~1.5M
in your retirement account is protected from bankruptcy.

Means testing generally ignores retirement accounts but does
include conventional ones. College financial aid that uses the more
thorough CSS
PROFILE is a partial exception here: the college does still look
at the information, but often ignores them and is less likely to ask
for them than if the money is in a c

...
How Technical AI Safety Researchers Can Help Implement Punitive Damages to Mitigate Catastrophic AI Risk
by Gabriel Weil — 2024-02-19T18:00:24.256Z

In the previous post, I explained how punitive damages could “pull forward” the expected liability associated with the uninsurable risks generated by training and deploying advanced AI systems. In this post, I lay out the details of my proposed punitive damages calculation, and suggest opportunities for technical AI safety researchers to lay the groundwork for implementing this proposal.

The goal of my proposed punitive damages framework is for the creators and deployers of AI systems to expect to pay out damages roughly equal to the amount of harm they expect to cause. Since the law cannot hold them liable in cases where uninsurable risks are realized, this is accomplished by applying punitive damages in cases of practically compensable harms that are correlated with those uninsurable risks. That is, punitive damages would be applied in the sort of cases where, if the companies involved expected to pay out large damages awards for causing practically compensable harms, the efforts that they would tend to engage in to reduce the likelihood or severity of those harms would, by default, tend to mitigate the uninsurable risks.

For example, consider an AI system tasked with running a clinical trial for a risky new drug. It has trouble recruiting participants honestly and, instead of reporting this to its human overseers, resorts to some combination of deception and coercion to get people to participate. Some of those deceived/coerced patients suffer some nasty health effects and they sue. This seems like a clear case of misalignment, and, in some sense, we got lucky that the system was willing to reveal its misalignment in a non-catastrophic way. Maybe it had narrow goals, or short time horizons, or limited situational awareness. But the people who trained and deployed it couldn’t have been confident ex ante that it would go down this way. Presumably, they thought the system was aligned or they wouldn’t have deployed it.

Say that training and deploying the system generated a 1 in 1 million chance of human extinction. Under my proposed framework, the defendant would have to pay out punitive damages equal to 1 millionth of the value of human extinction, across the cases of practically compensable harms correlated with that extinction risk. More generally, the share of punitive damages awarded to a plaintiff in any particular case S, would be given by the ...
Protocol evaluations: good analogies vs control
by Fabien Roger — 2024-02-19T18:00:09.794Z

Let’s say you want to use AIs that are capable of causing x-risk. You have a plan that describes how to train, validate and deploy AIs. For example, you could use pretraining+RLHF, check that it doesn’t fall for honeypots and never inserts a single backdoor in your coding validation set, and have humans look at the deployment outputs GPT-4 finds the most suspicious. How do you know if this plan allows you to use these potentially dangerous AIs safely?

Thanks to Ryan Greenblatt and Buck Shlegeris who provided the original ideas for this post, and thanks to Jean-Stanislas Denain, Bogdan Ionut Cirstea, Ansh Radhakrishnan, and Charbel-Raphael Segerie for useful feedback on the draft of this post.

Evaluate safety using very good analogies for the real situation?

If you want to evaluate a protocol (a set of techniques you use to train, validate, and deploy a model), then you can see how well it works in a domain where you have held-out validation data that you can use to check if your protocol works. That is, you run your exact protocol on this domain without allowing the protocol access to the held-out validation data. Then, you just assess how well you did with this held-out validation. This sort of approach is sometimes called

sandwiching

. (Here we are using sandwiching in a broad way that covers the evaluation of aspects of protocols, not just evaluation of oversight.)

One approach for obtaining held-out validation data is to restrict the quality of the labor that goes into the protocol: the

Measuring progress in scalable oversight paper

uses non-expert humans, and the

weak-to-strong generalization paper

uses a small model. Then, you can use expert labels to check how well the protocol did.

A big flaw of this approach is that the analogy between [protocol with weak labor, human expert] and [protocol with human expert labor, actual goodness] might be quite poor. In particular, it’s not clear how weak the weak labor should be. And if the analogy is poor, the AI might not generalize and behave in the same ways (even if it is not scheming - more on that in the next section).

For now, the weakness of this sort of analogy doesn’t bite very hard because there is an even bigger difference between our current experiments and real dangerous AI use cases: we don’t have (x-risk) dangerous AIs yet. Thus, the actual analogy is between [weak AIs, protocol with weak ...
When Should Copyright Get Shorter?
by Maxwell Tabarrok — 2024-02-19T16:03:48.872Z
Link: https://www.maximum-progress.com/p/when-should-copyright-get-shorter

Information goods have private costs and public benefits. Pharmaceutical research requires upfront investment from one firm, but once a certain compound is known to be effective it’s cheap for any company to produce. Writing a novel or filming a movie takes hundreds of hours of work from an author, but once the first copy is stored digitally it doesn’t cost anything to make a million more.

This is information’s greatest advantage and its central challenge. An idea, produced once, can spread and benefit millions without much investment in its reproduction or maintenance. Precisely this easy spread, though, means that the progenitor of an idea has little advantage over copiers. Copies carry almost all the benefits of information and almost none of the costs, so the original investment in creating the information is difficult to recoup.

Copyright attempts to correct the resulting underproduction of information by increasing the cost of copies for everyone except the original creator. This taxes information’s greatest advantage to subsidize its central challenge.

Since the invention of the printing press, the cost of producing and distributing almost all cultural artefacts has fallen rapidly. It is several thousand times cheaper to film a video, record a song, or write a paper and send it to a million people today than it was 50 years ago. Over this same period since the printing press, copyright was invented and repeatedly extended in breadth and length from 14 year bans on reprinting books to `life + 70` year monopolies over all things that look like Mickey Mouse or sound like Marvin Gaye.

Is the slow forward march of copyright terms the optimal response to the massive changes in information technology we’ve seen over the past several centuries?

Copyright before and after the printing press

How would optimal copyright law respond to the invention of the printing press?

The press affects copyright through three channels: Fixed costs of information, marginal costs of spreading information, and the substitutability of author-copies and knockoffs

The upfront fixed cost is easiest to understand. As printed books filled libraries and bookshops it was cheaper to do the background research required to write a new book. Decreasing the required upfront investment unambiguously decreases the optimal strength of copyright, all else equal. This upfront investment is what copyright...
Auto-matching hidden layers in Pytorch LLMs
by chanind — 2024-02-19T12:40:20.442Z

Mechanistic interpretability and steering LLMs requires being able to read and modify activations during inference. For instance, to apply steering vectors[1] to control model generation, we need to first collect hidden activations to find a steering direction, then intervene by modifying hidden activations of the model during inference.

To read and patch activations from a LLM, you first need to find the relevant layers that you care about and either add hooks or wrap them. This tends to lead to two approaches, either 1. writing a custom model wrapper for every model you might want to work with (approach taken by Repe, CAA) or 2. leave it to the user to manually specify layer names to patch, and apply the patch using Pytorch hooks (approach taken by Baukit). The first approach is a never-ending battle as new models are released, and the second approach, while very flexible, passes on the complexity to anyone using what you've written.

In this post, I'll discuss a third option, which is to auto-detect the types of layers in a Pytorch LLM and read/patch using Pytorch hooks, and is the approach used by the steering-vectors library. This leverages the fact that all transformer LMs have the same basic structure: a series of layers containing attention and MLP blocks. This post assumes the model is from Huggingface, although this same technique will likely work with any transformer LM that's sanely constructed. This post will use the terms "transformer LM" and "LLM" interchangeably to refer to a decoder-only generative language model like GPT or LLaMa.

Guessing layer templates

Finding the component parts of any Pytorch module is easy by calling named_modules() on the model. This will return a dictionary containing the name of the submodule, and the submodule itself. This is demonstrated for GPT2-small below:

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("gpt2")
print(dict(model.named_modules()).keys())

# transformer
# transformer.wte
# transformer.wpe
# transformer.drop
# transformer.h
# transformer.h.0
# transformer.h.0.ln_1
# transformer.h.0.attn
# transformer.h.0.attn.c_attn
# transformer.h.0.attn.c_proj
# transformer.h.0.attn.attn_dropout
# transformer.h.0.attn.resid_dropout
# transformer.h.0.ln_2
# transformer.h.0.mlp
# transformer.h.0.mlp.c_fc
# transformer.h.0.mlp.c_proj
# transformer.h.0.mlp.act
#

...
I'd also take $7 trillion
by bhauth — 2024-02-19T03:31:45.552Z
Link: https://www.bhauth.com/blog/projects/7%20trillion.html

So, Sam Altman wants as much money as he can get - the number he chose to give was $7 trillion - to build more chips for AI as fast as possible. Including in places like the UAE. Thus undermining his argument that OpenAI is helping with AI safety by preventing overhang and maintaining a lead for America, which was never a good argument anyway.

Well, I warned some people about him. To avoid getting fooled by bad actors, you have to avoid getting fooled by good actors. (In Altman's case, I had the advantage of more information about him making conflicting statements to different people.)

Anyway, this got me thinking about how I'd spend $7 trillion dollars. Such large numbers get hard for people to understand; you can say it's about $1000 per person alive, but giving it away equally isn't in the spirit of the challenge here. No, the point here is imagining megaprojects and massive economies of scale, to grapple a little bit with the enormity of trillions of dollars.

So, the below is how I'd spend that kind of money in useful ways. Much of this is stuff that'll happen anyway, and listing that is sort of cheating, so I guess I'll aim for over $7 trillion, and when I know someone with a notable possible improvement to something, I'll mark that with a *.

solar

Doing work requires energy, and the modern trend is towards electrification. The world is now generating an average of ~3.5 terawatts of electricity. Making 2 terawatts of average solar power generation seems pretty reasonable. With single-axis tracking you can get maybe 30% capacity factor, so that'd be 6.7 TW of solar panels. Anyway, at least $1 trillion of capital investment in solar panel production seems justified, and that's a good start on spending all that money.

If you just want a big enough production complex to hit the limits of cost scaling, that's a lot cheaper, probably a mere $100 billion. The "Inflation Reduction Act" probably would've made more sense if it funded a big solar panel complex instead of stuff like subsidies for water electrolysis (which is a dead end for at least a few decades) and rooftop solar (which is expensive and dumb).

PV panel production includes:

making silicon metal

making SiCl4 and distilling it for purification

reaction with hydrogen

slicing thin layers with diamond-studded wire saws

treatments including doping

application of ITO transparent conductor* and wires

The vast...
On coincidences and Bayesian reasoning, as applied to the origins of COVID-19
by viking_math — 2024-02-19T01:14:06.772Z

(Or: sometimes heuristics are no substitute for a deep dive into all of the available information).

This post is a response to Roko's recent series of posts (Brute Force Manufactured Consensus is Hiding the Crime of the Century, The Math of Suspicious Coincidences, and A Back-Of-The-Envelope Calculation On How Unlikely The Circumstantial Evidence Around Covid-19 Is); however, I made a separate post for a few reasons.

I think it's in-depth enough to warrant its own post, rather than making comments
It contains content that is not just a direct response to these posts
It's important, because those posts seem to have gotten a lot of attention and I think they're very wrong.

Additional note: Much of this information is from the recent Rootclaim debate; if you've already seen that, you may be familiar with some of what I'm saying. If you haven't, I strongly recommend it. Miller's videos have fine-grained topic timestamps, so you can easily jump to sections that you think are most relevant.

The use of coincidences in Bayesian reasoning

A coincidence, in this context, is some occurrence that is not impossible or violates some hypothesis, but is a priori unlikely because it involves 2 otherwise unrelated things actually occurring together or with some relationship. For example, suppose I claimed to shuffle a deck of cards, but when you look at it, it is actually in some highly specific order; it could be 2 through Ace of spades, then clubs, hearts, and diamonds. The probability of this exact ordering, like any specific ordering, is 1/52! from a truly random shuffle. Of course, by definition, every ordering is equally likely. However, there is a seeming order to this shuffle which should be rare among all orderings.

In order to formalize our intuition, we would probably rely on some measure of "randomness" or some notion related to entropy, and note that most orderings have a much higher value on this metric than ours. Of course, a few other orderings are similarly rare (e.g. permuting the order of suits, or maybe having all 2s, then all 3s, etc. each in suit order) but probably only a few dozen or a few hundred. So we say that "the probability of a coincidence like this one" is < 1000/52!, which is still fantastically tiny, and thus we have strong evidence that the deck was not shuffled randomly. On the other hand, maybe I am an expert...