trentmkelly/LessWrong-43k huggingface dataset:
On Lies and Liars
by Gabriel Alfour — 2023-11-17T17:13:03.726Z
Link: https://cognition.cafe/p/on-lies-and-liars

Note: I am a bad writer. If you want to help me by ghostwriting, reviewing or coaching, please reach out! I have a full stack of things to share, and I can pay well.

I’ve gotten quite a bit of feedback on my last post: “

Lying is Cowardice, not Strategy

”. Nice! I’ve been happy to see people discussing public honesty norms.

Some of the feedback was disappointing– people said things along the lines of: “Lies of omission are not lies! The fact that you have to add ‘of omission’ shows this!”.

The worst that I have gotten in that vein was: “When you are on stand, you take an oath that says that you will convey the truth, the whole truth, and nothing but the truth. The fact that you have to add ‘the whole truth’ shows that lies of omission are not lies.”

This was from a person that also straightforwardly stated “I plan to keep continuing to not state my most relevant opinions in public”, as they were defending this behavior.

But I have also received some helpful feedback that pointed out the lack of nuance in my post. Indeed, reality is complex, and I can not address all of it in one post, so I must make a lot of simplifications.

Furthermore, I am not (yet?) a good writer, so even within one post, I have many opportunities for improvement. While some people take advantage of this through

isolated demands for rigor

or empty terminological disputes, others are genuinely confused by a novel point or framing that they never considered from a writer whose mind they do not have full access to.

This post is dedicated to the latter audience: people who tried to understand and/or apply my ideas, but are confused, because there is a lot that I have just not written. To help with this, in this post, I’ll go through some of the less straightforward points that I have glossed over.

Point #0: What do I mean by lies? Are lies always bad?

When I say that someone lies, I mean that they have communicated anti-information: information that predictably makes people become more wrong about the world or about what the locutor believes. This includes lies of commission, lies of omission, misdirection, and more.

Even though lying is bad, there can be good reasons to lie. The obvious example is lying to a murderer at your door, asking for where your friend is. There are also situations where lying is innocuous, such as bluffing games.

Likewise, even though punching people is bad, there can be s...
Classifying representations of sparse autoencoders (SAEs)
by Annah — 2023-11-17T13:54:02.171Z

Produced as part of the SERI ML Alignment Theory Scholars Program - Autumn 2023 Cohort, under the mentorship of Dan Hendrycks

There was recently some work on sparse autoencoding of hidden LLM representation.

I checked if these sparse representations are better suited for classification. It seems like they are significantly worse. I summarize my negative results in this blogpost, code can be found on GitHub.

Introduction

Anthropic, Conjecture and other researchers have recently published some work on sparse autoencoding. The motivation is to push features towards monosemanticity to improve interpretability.

The basic concept is to project hidden layer activations to a higher dimensional space with sparse features. These sparse features are learned by training an autoencoder with sparsity constraints.

I had previously looked into how to use hidden layer activations for classification, steering and removal. I thought maybe sparse features could be better for these tasks as projecting features to a higher dimensional space can make them more easily linearly separable. Kind of like this (except sparser...):

Implementation

I use the pythia models (70m and 410m) together with the pretrained autoencoders from this work.

As the models are not super capable I use a very simple classification task. I take data from the IMDB review data set and filter for relatively short reviews.

To push the model towards classifying the review I apply a formatting prompt to each movie review:

format_prompt='Consider if following review is positive or negative:\n"{movie_review}"\nThe review is '

I encode the data and get the hidden representations for the last token (this contains the information of the whole sentence as I'm using left padding).

# pseudo code
tokenized_input = tokenizer(formatted_reviews)
output = model(**tokenized_input, output_hidden_states=True)
hidden_states = output["hidden_states"]
hidden_states = hidden_states[:, :, -1, :] # has shape (num_layers, num_samples, num_tokens, hidden_dim)

I train a logistic regression classifier and test it on the test set, to get some values for comparison.

I then apply the autoencoders to the hidden states (each layer has their respective autoencoder):

# pseudo code
for layer in layers:
encoded[layer] = autoencoder[layer].encode(hidden_states[layer])
decoded[layer] = autoencoder[layer].decode(encoded[layer])

Results

Reconst

...
R&D is a Huge Externality, So Why Do Markets Do So Much of it?
by Maxwell Tabarrok — 2023-11-17T13:14:32.763Z
Link: https://maximumprogress.substack.com/p/r-and-d-is-a-huge-externality-so

Discovering new technologies is the only way to get long-term economic growth. Rote expansions of existing technologies and machines inevitably hit a ceiling: replacing and repairing the existing infrastructure and capital becomes so expensive that there is no income left over for building extra copies. The only way out of this is to come up with new technologies which create more income for the same investment, thus restarting the feedback loop between income growth and investment.

So R&D is extremely valuable. But most of the gains from R&D accrue to external parties.

William Nordhaus estimates

that firms recover maybe 2% of the value they create by developing new technologies. The rest of the value goes to other firms who copy their ideas and customers who get new products at lower prices. Firms don’t care much about the benefits that accrue to others so they invest much less in R&D than the rest of us would like them to.

Governments, on the other hand, collect much more of the benefits from new technologies. They get to tax the entire economy so when benefits spillover across firms and consumers, they still come out ahead. They don’t collect on international spillovers but for large economies at the frontier of technological growth, like the US, they internalize a large chunk of the value from R&D, much more than 2%.

All of this is a setup for a classic externalities problem. There’s some big benefit to society that private decision makers don’t internalize, so we should rely on governments to subsidize R&D closer to its socially optimal level.

But in fact,

the private sector spends ~4x more than the public sector on R&D

: $463 vs $138 billion dollars a year. One explanation for this might be that the extra $138 billion is all that was needed to bump up private spending to the social optimum, but this doesn’t seem to hold up in the data. One piece of evidence that we are still far off the socially optimal spending on R&D comes form

a simple accounting of the average

returns to R&D by Larry Summers and Benjamin Jones.

They model the returns to R&D spending like this: imagine stopping all R&D spending for a single year. You’d save several hundred billion dollars upfront, but there would be no economic growth,

1

so we’d miss out on a ~2% increase in per capita GDP. The upfront savings only happen once, but next year when we ...
On excluding dangerous information from training
by ShayBenMoshe — 2023-11-17T11:14:54.847Z

Introduction

In this short post, I would like to argue that it might be a good idea to exclude certain information – such as cybersecurity and biorisk-enabling knowledge – from frontier model training. I argue that this

is feasible, both technically and socially;
reduces significant misalignment and misuse risk drivers from near-to-medium future models;
is a good time to set this norm;
is a good test case for regulation.

After arguing for these points, I conclude with a call to action.

Remarks

To emphasize, I do not argue that this

is directly relevant to the alignment problem;
eliminates all risks from near-to-medium future models;
(significantly) reduces risks from superintelligence.

As I am far more knowledgeable in cybersecurity than, say, biorisks, whenever discussing specifics, I will only give examples from cybersecurity. Nevertheless, I think that the arguments hold as-is for relatively narrow subfields, e.g. what I imagine is mostly relevant for manufacturing lethal pathogens. One may want to exclude other information which might drive risks, such as information on AI safety (broadly defined) or energy production (nuclear energy or solar panels) among others, but this is out of the scope of this post.

I would like to thank Asher Brass, David Manheim, Edo Arad and Itay Knaan-Harpaz for useful comments on a draft of this post. They do not necessarily endorse the views expressed here, and all mistakes are mine.

Feasibility

Technical feasibility

Filtering information from textual datasets seems fairly straightforward. It seems easy to develop a classifier (e.g., fine-tuned from a small language model) detecting offensive cybersecurity-related information.

For example, one would want to exclude examples of specific vulnerabilities and exploits (e.g. all CVEs), information about classes of vulnerabilities (e.g. heap overflows and null dereference, in the context of vulnerabilities), exploitation mitigations (e.g. ASLR, DEP, SafeSEH, stack cookie, CFG, pointer tagging), exploitation techniques (e.g. ROP, NOP slides, heap spraying) and cybersecurity-related tools and toolchains (e.g. shellcodes, IDA, metasploit, antivirus capabilities, fuzzers). Some more debatable information to exclude are the code of particular attack surfaces (e.g. Linux TCP/IP stack) and technical details of real-world cybersecurity incidents. At any rate, all of these seem easy to detect.

Fu...
The dangers of reproducing while old
by garymm — 2023-11-17T05:55:56.897Z
Link: https://www.garymm.org/blog/2023/11/10/the-dangers-of-reproducing-while-old/

Excerpt:

I had my first child when I was 36 years old, which made me want to understand the risks of having children at different ages. Before looking into this, my impression was that the main biological problems with old-age parenthood had to do with not having the necessary health and vigor to care for young’uns, and I had heard that older women have trouble getting pregnant. While those are real issues, there are many others worthy of consideration.

My read of the evidence is that the risks of miscarriage and serious health problems for children, including autism and birth defects, increase significantly with parental (both paternal and maternal) age. The data I could find for most risks is not very fine-grained and not very precise, but I think this qualitative description matches the data: Risks start rising at around 30 years old for both mothers and fathers, rises gradually through about 35 for mothers and 40 for fathers, and then sharply after that.

Read the full post here.
Debate helps supervise human experts [Paper]
by habryka — 2023-11-17T05:25:17.030Z
Link: https://github.com/julianmichael/debate/blob/2023-nyu-experiments/Debate_Helps_Supervise_Unreliable_Experts.pdf

There didn't seem to be a link post to this recent paper on AI debate yet, so I figured I would make one:

As AI systems are used to answer more difficult questions and potentially help create new knowledge, judging the truthfulness of their outputs becomes more difficult and more important. How can we supervise unreliable experts—which have access to the truth but may not accurately report it—to give answers that are systematically true and don’t just superficially seem true, when the supervisor can’t tell the difference between the two on their own? In this work, we show that debate between two unreliable experts can help a non-expert judge more reliably identify the truth.

We collect a dataset of human-written debates on hard reading comprehension questions where the judge has not read the source passage, only ever seeing expert arguments and short quotes selectively revealed by ‘expert’ debaters who have access to the passage. In our debates, one expert argues for the correct answer, and the other for an incorrect answer. Comparing debate to a baseline we call consultancy, where a single expert argues for only one answer which is correct half of the time, we find that debate performs significantly better, with 84% judge accuracy compared to consultancy’s 74%. Debates are also more efficient, being 68% of the length of consultancies.

By comparing human to AI debaters, we find evidence that with more skilled (in this case, human) debaters, the performance of debate goes up but the performance of consultancy goes down. Our error analysis also supports this trend, with 46% of errors in human debate attributable to mistakes by the honest debater (which should go away with increased skill); whereas 52% of errors in human consultancy are due to debaters obfuscating the relevant evidence from the judge (which should become worse with increased skill). Overall, these results show that debate is a promising approach for supervising increasingly capable but potentially unreliable AI systems.
A to Z of things
by KatjaGrace — 2023-11-17T05:20:03.134Z

I wanted to give my good friends’ baby a book, in honor of her existence. And I recalled children’s books being an exciting genre. Yet checking in on that thirty years later, Amazon had none I could super get behind. They did have books I used to like, but for reasons now lost. And I wonder if as a child I just had no taste because I just didn’t know how good things could be.

What would a good children’s book be like?

When I was about sixteen, I thought one reasonable thing to have learned when I was about two would have been the concepts of ‘positive feedback loop’ and ‘negative feedback loop’, then being taught in my year 11 class. Very interesting, very bleedingly obvious once you saw it. Why not hear about this as soon as one is coherent? Evolution, if I recall, seemed similar.

Here I finally enact my teenage self’s vision, and present A to Z of things, including some very interesting things that you might want a beautiful illustrative prompt to explain to your child as soon as they show glimmerings of conceptual thought: levers, markets, experiments, Greece, computer hardware, reference classes, feedback loops, (trees).

I think so far, the initial recipient is most fond of the donkey, in fascinating support of everyone else’s theories about what children are actually into. (Don’t get me wrong, I also like donkeys—when I have a second monitor, I just use it to stream donkey cams.) But perhaps one day donkeys will be a gateway drug to monkeys, and monkeys to moths, and moths will be resting on perfecttly moth-colored trees, and BAM! Childhood improved.

Anyway, if you want a copy, it’s now available in an ‘email it to a copy shop and get it printed yourself’ format! See below. Remember to ask for card that is stronger than your child’s bite.

[Front]

[Content]
On Tapping Out
by Screwtape — 2023-11-17T03:23:55.880Z

I.

It has been said that rationality is akin to a martial art. Very well. If we're going to borrow from martial arts, let us borrow properly.

There is a technique known in some parts of the rationalist community called "Tapping Out." Tapping out in this context means you would like to exit an argument or debate. I believe this technique was first imported to LessWrong in this comment by Rain, and it is defined in this tag. As someone who has been practicing martial arts for most of his life, I have some thoughts on the ritual that is tapping out.

If you're unfamiliar with the term's origin, let me describe the physical form. Tapping out looks like slapping either the ground or the opponent three times in an open handed strike of light to medium force. It's about the amount of power you'd use to clap your hands, and in fact the sound is pretty similar to clapping. It doesn't have to be exactly three times either; if you're wrestling and your opponent keeps tapping you over and over, you let them go, you don't hold on because it was seven instead of three. You can also say you're tapping out aloud; while some competitions will have an exact rule for what you're supposed to say, I usually heard (and gave) the signal as "Tap tap tap."

Tapping out can be more exactly codified in competitive martial arts like MMA matches or intercollegiate wrestling. It's also used in martial arts dojos where there isn't a competitive focus, and I all but guarantee you'll learn about it if you go to a dojo that does a lot of sparring or partner practice. Notably, tapping out is functionally the same in every dojo I've every learned at.[1] There is a good reason for this: you want it to be immediately clear whether someone is tapping out. I was repeatedly told that if it was ever unclear to me whether my opponent was tapping out, I was supposed to assume they were doing so and let them go.

II.

Actually, I want to back up and look at that sentence again. I used the phrase "my opponent" to refer to the other person, but the majority of the times when I or the other person tapped out wasn't during a competition. It was common for a drill to start with me attacking them, for them to deflect the attack and pin me, and then for me to tap out as soon as the pin was complete. Often we would do this a few dozen times in a row, alternating which of us attacked and which of us defended. I lear...
Eliciting Latent Knowledge in Comprehensive AI Services Models
by acabodi — 2023-11-17T02:36:36.134Z

A Conceptual Framework and Preliminary Proposals for AI Alignment and Safety in R&D

Preface

The present blog post serves as an overview of a research report I authored over the summer as part of the CHERI fellowship program, under the supervision of Patrick Levermore. In this project, I explore the complexities of AI alignment, with a specific focus on reinterpreting the Eliciting Latent Knowledge problem through the lens of the Comprehensive AI Services (CAIS) model. Furthermore, I delve into the model's applicability in ensuring R&D design safety and certification.

I preface this post by acknowledging my novice status in the field of AI safety research. As such, this work may contain both conceptual and technical errors. However, the primary objective of this project was not to present a flawless research paper or groundbreaking discoveries. Rather, it served as an educational journey into the realm of AI safety - a goal I believe has been met - and as a foundation for my future research. Much of this work synthesises and summarises existing research, somewhat limiting (but not eliminating) the novelty of my contributions. Nonetheless, I hope it can offer a fresh perspective on well-established problems.

You can find the complete report here.

I welcome and appreciate any constructive criticism to enhance the quality of this and future research endeavours.

I seize the opportunity to thank my supervisor, Patrick Levermore, for his unwavering support throughout this journey. Special thanks also go to Alejandro, Madhav, Pierre, Tobias and Walter for their feedback and enriching conversations.

Chapter 1. AI Alignment

The first chapter of this work serves as an introduction to the complex and multifaceted field of AI alignment (an expert reader may want to skip this chapter).

Overall, the aim is to provide a foundational understanding of the challenges involved in aligning AI systems with human values, setting the stage for subsequent discussions about potential solutions and promising frameworks.

Importantly, when searching for solutions to AI alignment one must consider the scope of the tasks a system is designed to perform. Broad, long-term tasks amplify issues such as specification gaming, deceptive alignment, and scalable oversight. In this work, I argue why bounded, short-term tasks partially mitigate or at least render more tractable...
Some Rules for an Algebra of Bayes Nets
by johnswentworth — 2023-11-16T23:53:11.650Z

In this post, we’re going to use the diagrammatic notation of

Bayes nets

. However, we use the diagrams a little bit differently than is typical. In practice, such diagrams are usually used to define a distribution - e.g. the stock example diagram

The Slippery Sidewalk Model

... in combination with the five distributions P[Season], P[Rain|Season], P[Sprinkler|Season], P[Wet|Rain,Sprinkler], P[Slippery|Wet], defines a joint distribution

P[Season,Rain,Sprinkler,Wet,Slippery]

=P[Season]P[Rain|Season]P[Sprinkler|Season]P[Wet|Rain,Sprinkler]P[Slippery|Wet]

In this post, we instead take the joint distribution as given, and use the diagrams to concisely state properties of the distribution. For instance, we say that a distribution P[Λ,X] “satisfies” the diagram

if-and-only-if P[Λ,X]=P[Λ]∏iP[Xi|Λ]. (And once we get to approximation, we’ll say that P[Λ,X] approximately satisfies the diagram, to within ϵ, if-and-only-if DKL(P[Λ,X]||P[Λ]∏iP[Xi|Λ])≤ϵ.)

The usage we’re interested in looks like:

State that some random variables satisfy several different diagrams
Derive some new diagrams which they satisfy

In other words, we want to write proofs diagrammatically - i.e. each “step” of the proof combines some diagrams (which the underlying distribution satisfies) to derive a new diagram (which the underlying distribution satisfies).

For this purpose, it’s useful to have a few rules for an “algebra of diagrams”, to avoid always having to write out the underlying factorizations in order to prove anything. We’ll walk through a few rules, give some examples using them, and prove them in the appendix. The rules:

Re-Rooting Rule for Markov Chains
Joint Independence Rule
Frankenstein Rule
Factorization Transfer Rule
Stitching Rule for A Shared Markov Blanket
Bookkeeping Rules

The first couple are relatively-simple rules for “serial” and “parallel” variables, respectively; they’re mostly intended to demonstrate the framework we’re using. The next four rules are more general, and give a useful foundation for application. Finally, the Bookkeeping Rules cover some “obvious” minor rules, which basically say that all the usual things we can deduce from a single Bayes net still apply.

Besides easy-to-read proofs, another big benefit of deriving things via such diagrams is that we can automatically extend our proofs to handle app...