trentmkelly/LessWrong-43k huggingface dataset:
Using LLM's for AI Foundation research and the Simple Solution assumption
by Donald Hobson — 2024-09-24T11:00:53.658Z

Current LLM based AI systems are getting pretty good at maths by writing formal proofs in Lean or similar. https://deepmind.google/discover/blog/ai-solves-imo-problems-at-silver-medal-level/

So, how can we use these models to help align AI?

The Simple Alignment Solution Assumption states that many problems in alignment, for example corrigibility or value learning, have simple solutions. I mean "simple" in the sense that logical induction is a simple solution. That expected utility maximization is. That causal models are. This is the sort of thing we are searching for.

Under the Simple Alignment Solution Assumption, the solution is.

Unique or unique-ish. Pinned down by theorems the way that probability theory and expected utility maximization are.
Fundamentally simple, in the right ontology. If you are writing a python program, logical induction is a bit of a pain. But if you can say "all polynomial time Turing machines" with ease, then logical induction is simple.
Recognizable with a few expert-days. These ideas are fairly know-it-when-you-see-it, a lot easier to recognize than to produce.
Mathematical structures. Formal proofs, probability theory and logical induction each have their own mathematical structure.

Which represents a logical contradiction, and for a while there were attempts to develop "non-monotonic logics" so that you could retract conclusions given additional data. This didn't work very well, since the underlying structure of reasoning was a terrible fit for the structure of classical logic, even when mutated.

https://www.lesswrong.com/posts/hzuSDMx7pd2uxFc5w/causal-diagrams-and-causal-models

So, how can LLM's come up with new things in this class?

Training via self play?

Lets say a theorem is Useful if it is often used in the process of proving/disproving random ZFC statements. Such a criterion can be measured by generating random statements to prove/disprove, putting the theorem to be measured in the assumptions, and seeing if it is used.

I would expect that associativity is pretty useful. I would expect 1=2 to be incredibly useful. Whatever the random statement, you can prove it by contradiction.

So the plan would be to make the AI produce simple mathematical structures, about which lots of short and useful theorems exist.

Humans would also add some desiderata.

This is very much some...
When to join a respectability cascade
by B Jacobs — 2024-09-24T07:54:16.051Z
Link: https://bobjacobs.substack.com/p/when-to-join-a-respectability-cascade

This is a response to Scott Alexanders Give Up Seventy Percent Of The Way Through The Hyperstitious Slur Cascade. Maybe read that post first.

I think it becomes more important to struggle against a cascade the more annoying the new word is. There are a variety of factors that contribute how annoying it is, here are five of them:

Is the new word longer? (Because we want to waste less time)
Is it easier to pronounce/type? (Because we care about ease of use)
Is it less descriptive? (Because we care about people quickly grasping the concept)
Is it dissimilar to the old word? (Because we want to minimize transaction costs)
Is it likely to cause another cascade? (Because we dislike cascades)

If tomorrow people say that "black people" is bad but "black peoples" is good, I would jump on board once 60% of the population is on board, because it scores well in all aspects (except 5). If however people want to replace it with "abcedifoguhajekilomun" that's worse in all aspects (except 5) so I would strongly push against it (maybe only jump on board at 95%).

Conversely if people want to reclaim a word and start a respectability cascade I would jump on board rather quickly (I like having the freedom to use a lot of words), but I would jump on board even more quickly the better it scores on those different aspects.

For example, I'm on board with trying to reclaim "queer" because lgbt is longer and more annoying to say, requires an explanation for a new user, and quickly causes another cascade (what about intersex? Okay we'll say lgbti. What about asexuals? Okay we'll say lgbtia etc).
Queer on the other hand scores well on all aspects (even transaction cost since we still have a lot of books lying around about "queer theory" etc).

I don't think we should be trying to put a number on this (e.g. I add 10% to my 70% for every negative aspect it has) because a lot of it depends on social context. With rarely used jargon I value descriptiveness over brevity, with words I use in everyday life it's the other way around. Let's all agree we jump onboard a disrespectability cascade at more than 50% and onboard a respectability cascade at less than 50%. How much more or less we'll change depending on a lot of hard to quantify social factors.

Note that this is just a heuristic and moral reasoning obviously takes priority. Queer people wanting to make "queer" a respectable synonym of "lgbt" is fine...
Sampling Effects on Strategic Behavior in Supervised Learning Models
by Phil Bland — 2024-09-24T07:44:41.677Z

TLDR

This post investigates how different sampling methods during inference can lead supervised learning models to exhibit strategic behavior, even when such behavior is rare in the training data. Through a toy example, we demonstrate that an AI model trained solely to predict sequences can choose less likely options initially to simplify future predictions. This finding highlights that the way we use AI models—including seemingly minor aspects like sampling strategies—can significantly influence their behavior.

Introduction

Guiding Question: Under what circumstances can an AI model, trained only with supervised learning to predict future events, learn to exhibit strategic behavior?

In machine learning, particularly with models like GPT-style transformers, the sampling method used during inference can profoundly impact the generated outputs. This post explores how different sampling strategies can cause a model to switch between non-strategic and strategic behaviors, even when the training data predominantly features non-strategic examples.

Disclaimer: While it has previously been pointed out that sampling strategies can have significant effects in text generation (e.g. https://arxiv.org/abs/1904.09751), I couldn’t find any post or paper analyzing these effects regarding AI safety. As I’m still rather new to AI safety, it’s quite possible that I missed related work.

Why This Matters

To be clear, I’m not trying to argue that certain sampling methods should be avoided per se, or that changing sampling has big safety implications in current LLMs (which anyway aren’t trained purely with supervised learning).

Rather, I found the toy example to be counter-intuitive at first (in particular because I couldn’t easily come up with cases where a supervised learning model would “sacrifice” correctness at any point) and it illustrates how simple changes in usage of future AI models which are not really included in any learned parameters – such as switching to a longer time horizon for sampling – could in principle facilitate strategic behavior. This points to interesting questions about attribution of behavior and learning causality.

Toy Example

Environment

We consider a simplified conversational environment to illustrate how strategic behavior can emerge.

We want to model a conversation with n+1 turns s0,…,sn between two entities. In deployment, we will give use

...
In Praise of the Beatitudes
by robotelvis — 2024-09-24T05:08:21.133Z
Link: https://messyprogress.substack.com/p/in-praise-of-the-beatitudes

I’m not Christian now, but I used to be. I parted ways with Christianity because I came to identify as more of an Empiricist - believing that knowledge should come from experiment and observation, not writings in old sacred texts.

But some of the writing in those old sacred texts is actually really good.

The King James Bible contains roughly 770,000 words, including a lot of different books written by different people at different times saying different things. However, the writings in the Bible are not equally important, and there is broad agreement (at least among mainline Protestants) about which parts are the most important:

The New Testament outranks the Old Testament. The Gospels outrank the rest of the New Testament (hence the phrase “Gospel Truth”). The words of Jesus outrank other material in the Gospels. The Sermon on the Mount outranks the other words of Jesus. The Beatitudes are the most important part of the Sermon on the Mount. If someone comes at you with a quote from St Paul, and you counter with a quote from the Beatitudes, then you win.

The Beatitudes are Jesus’s own nine bullet summary of the Christian concept of morality.

Each of the beatitudes consists of a “blessed are the” part, followed by a “for theirs is” part. In my mind the “blessed are” parts are the meat. I’m going to go through them one by one, giving my interpretation[1] of what each of them means:

Blessed are the poor in spirit: Do not place value in materialism and status symbols. When Pope Francis toured the US in a little Fiat 500, or chose not to live in a palace, he was demonstrating what it means to be “poor in spirit”.
Blessed are those who mourn: We should value the things that have passed. The people who have died, the ideas people had, the things people built, the great things that have become broken.
Blessed are the meek: Be humble. Realize that often you are wrong and sometimes the actions that you believe will make the world better will actually make it worse.
Blessed are those who hunger and thirst after righteousness: Do your part to make the world better. Don’t be content to just float along, looking after yourself.
Blessed are the merciful: Resist the temptation to excessively punish others when you are strong, since this leads to cycles of violence. In game theory parlance, practice tit for two tats.
Blessed are the pure in heart: Practice inner

...
What are the best arguments for/against AIs being "slightly 'nice'"?
by Raemon — 2024-09-24T02:00:19.605Z

Awhile ago, Nate Soares wrote the posts Decision theory does not imply that we get to have nice things and Cosmopolitan values don't come free and But why would the AI kill us?

Paul Christiano put forth some arguments that "it seems pretty plausible that AI will be at least somewhat 'nice'", similar to how humans are somewhat nice to animals. There was some back-and-forth.

More recently we had Eliezer's post ASIs will not leave just a little sunlight for Earth.

I have a sense that something feels "unresolved" here. The current comments on Eliezer's post look likely to be rehashing the basics and I'd like to actually make some progress on distilling the best arguments. I'd like it if we got more explicit debate about this.

I also have some sense that the people previously involved (i.e. Nate, Paul, Eliezer) are sort of tired of arguing with each other. But I am hoping someones-or-other end up picking up the arguments here, hashing them out more, and/or writing more distilled summaries of the arguments/counterarguments.

To start with, I figured I would just literally repeat most of the previous comments in a top-level post, to give everyone another chance to read through them.

Without further ado, here they are:

Paul and Nate

Paul Christiano re: "Cosmopolitan Values Don't Come for Free."

I want to keep picking a fight about “will the AI care so little about humans that it just kills them all?” This is different from a broader sense of cosmopolitanism, and moreover I'm not objecting to the narrow claim "doesn't come for free." But it’s directly related to the actual emotional content of your parables and paragraphs, and it keeps coming up recently with you and Eliezer, and I think it’s an important way that this particular post looks wrong even if the literal claim is trivially true.

(Note: I believe that AI takeover has a ~50% probability of killing billions and should be strongly avoided, and would be a serious and irreversible decision by our society that's likely to be a mistake even if it doesn't lead to billions of deaths.)

Humans care about the preferences of other agents they interact with (not much, just a little bit!), even when those agents are weak enough to be powerless. It’s not just that we have some preferences about the aesthetics of cows, which could be better optimized by having some highly optimized cow-shaped objects. It’s that

...
Struggling like a Shadowmoth
by Raemon — 2024-09-24T00:47:05.030Z

One of my favorite stories growing up was Star Wars: Traitor, by Matthew Stover.

The book is short, if you want to read it. Spoilers follow. (I took a look at it again recently and I think it didn't obviously hold up as real adult fiction, although quite good if you haven't yet had your mind blown that many times)

One anecdote from the story has stayed with me and shows up in my worldview from time to time.

The story begins with "Jacen Solo has been captured, and is being tortured."

He is being extremely tortured. He's in a white room, with a device connected throughout his nervous system, inflicting maximum white-hot agony. Whenever the suffering becomes so great that his body tunes it out, it gives him just enough time to relax to hurt all the more when it abruptly starts up again.

Every day, an alien woman comes to visit him. She talks to him, during lulls in the torture, about philosophy and life and history. She asks him what he thinks about things. He says "I think I'm being fucking tortured. Can you please help?", and she kind of ignores him and goes on talking about philosophy and life and history and stuff. Sometimes he tries more clever ways to pry information out of her, to no avail.

After several weeks (or months?) of this, at some point he asks again "Seriously what the hell? Who are you? What is happening to me? Why are you torturing me, or, not helping? What is going on?"

She says: "Do you know what a shadowmoth is?"

"Answer my fucking question"

"Do you know what a shadowmoth is?"

"Fine, a shadowmoth is a life form native to Coruscant. They're silicon-based, their wingflutes make music once they leave the cocoon and can fly."

She nods, and says "Once, I saw a shadowmoth in its cocoon, struggling. Like you, I am sensitive to the force – I didn't merely empathize with it, but I felt it. When shadowmoths strive to break free from their cocoon, they are in pain. I felt it's pain. And I felt it's envy of the other moths, already free, flying, making music. I wanted to help it."

Jacen shudders. "Oh no, you didn't –"

"Yes. I cut it free."

"No –"

"It sounds like you know what happened next?"

Jacen says: "If they don't escape from the cocoon on their own, they'll never be strong enough to fly, and they suffer and starve to death, worse than what they were experiencing in the cocoon."

"Indeed."[1]

The two of them sit for awhile. The torture machine gives Jacen anot...
Bounty for Evidence on Some of Palisade Research's Beliefs
by benwr — 2024-09-23T20:01:20.917Z

(Cross-posted from the Bountied Rationality Facebook group)

EDIT: Bounty Expired

Thanks everyone for thoughts so far! I do want to emphasize that we're actually highly interested in collecting even the most "obvious" evidence in favor of or against these ideas. In fact, in many ways we're more interested in the obvious evidence than in reframes or conceptual problems in the ideas here; of course we want to be updating our beliefs, but we also want to get a better understanding of the existing state of concrete evidence on these questions. This is partly because we consider it part of our mission to expand the amount and quality of relevant evidence on these beliefs, and are trying to ensure that we're aware of existing work.

Here is a Google Doc that lists 21 important beliefs that Palisade Research has about AI. For each belief, we're looking for the strongest evidence that exists in favor of that idea, and the strongest evidence that exists against it. We'll award at least $20 for the best evidence in favor, and at least $20 for the best evidence against each idea. We'll use our discretion for what we consider the "best" evidence, but the kind of thing we're looking for includes empirical research or convincing arguments. Empirical research, or arguments clearly backed by empirical observations, will be preferred over pure arguments.
To submit a piece of evidence, you can either comment here, making it clear which specific idea(s) you're giving evidence for, or you can add a comment to the linked document. A piece of evidence should include a link, should be clearly associated with a specific idea, and should include a short sentence about how the evidence applies to that idea.
For example, you might write a comment on "a strategic AI system will aim to appear convincingly aligned with human goals, or incapable of harming humans, whether it really is or not.", that includes a link to a paper on AI Sandbagging (e.g. "AI Sandbagging: Language Models can Strategically Underperform on Evaluations"), with a sentence like "This work on AI sandbagging shows that existing AI systems already strategically underperform when they can tell they are being evaluated."

Note: Only responses in the above format will be considered for bounties, though of course you can respond however you want in the LessWrong comments.
In addition to the base $20 bonus for the best evidence...
Predicting Influenza Abundance in Wastewater Metagenomic Sequencing Data
by jefftk — 2024-09-23T17:25:58.380Z
Link: https://naobservatory.org/blog/predicting-influenza-abundance

Thanks to our collaborators at MU and UCI for generating the sequencing data we analyze in this post, to the Grimm et al. (2023) authors for the analysis this builds on, and to Marc Johnson, Mike McLaren, Dan Rice, Will Bradshaw, Simon Grimm, and Evan Fields for feedback on the draft.

One key question in understanding the performance of metagenomic wastewater sequencing for pathogen detection is how deeply you need to sequence to see a pathogen spreading through the population. This depends on many things, but in our preprint Inferring the sensitivity of wastewater metagenomic sequencing for pathogen early detection (Grimm et al. 2023) we introduced a summary statistic RAi(1%): for a given pathogen and sequencing approach, what fraction of sequencing reads will be from the pathogen if 1% of people in the sewershed became infected in the past week?

The analysis in our preprint was based only on public sequencing data, and unfortunately most of it was generated from samples collected during the COVID-19 pandemic, which heavily suppressed most infectious diseases. For influenza in particular, CDC writes that "activity was unusually low throughout the 2020-2021 season", and correspondingly we didn't identify any sequencing reads from flu. This meant we were only able to use the data to generate an upper bound for RAi(1%). [1]

With society returning to pre-pandemic interaction levels, however, the 2023-2024 flu season did see a significant number of infections. This is visible in the following chart, showing CDC's count of positive tests for influenza in California and Missouri:

Two things to note about this chart:

The 2023-2024 flu season peaked much earlier in California than Missouri.

The ratio of influenza A to influenza B in California was much higher than in Missouri.

We're interested in Missouri and California in particular because we've been collaborating with Marc Johnson and Clayton Rushford at the University of Missouri (MU) and Jason Rothman, then in Katrine Whiteson's lab at the University of California, Irvine (UCI) now a professor at the University of California, Riverside. They processed and sequenced wastewater influent, with the MU group sampling in central Missouri and the UCI group sequenced in southern California. The two groups each used their own metagenomic sequencing protocol to generate a combined 102B read pairs from samples collected bet...
A primer on ML in antibody engineering
by Abhishaike Mahajan — 2024-09-23T17:03:07.628Z
Link: https://www.owlposting.com/p/a-primer-on-ai-in-antibody-engineering

Introduction

I increasingly see that a fair bit of the progress in ML-based protein design methods is occurring in antibody engineering, which is the process of creating antibodies tailor-made to bind to specific things in the body. I've noticed this phenomenon for the past year, and never quite understood what was going on in the space.

I've been recently taking the time to better understand this whole field a bit more and made a post about it to cement what I've learned. To note, this is not a machine-learning post in a traditional sense. I won't be going into the exact details of how antibody ML methods exactly work, but rather setting up the contextual knowledge needed to understand these methods at all + going through a few antibody ML papers.

Antibody background

What is an antibody?

I'm posting this picture right now so you have a place to refer back to it later. Ignore it for now, but it'll probably be useful down the road.

From

here

The immune system is one of those things that is, unfortunately, impossible to understand unless you know all the pieces involved in it. This becomes increasingly clear whenever you look into antibodies; while a surface-level understanding is easy enough to derive from a sentence, fully grasping whats going on requires a bit more work + circling around.

Let's start with B cells.

B cells belong to the white blood cell family. Each B-cell is meant to be unique in one capacity: what they bind to. The thing they bind to is commonly referred to as an 'antigen'. Antigens can be extremely diverse, it could be the venom of an insect to fragments of the influenza virus to proteins from shellfish. Let's assume for now that an antigen is bad. This practically isn't the case, 'antigens' are a subjective category, and the human body contains trillions of 'self-antigens'. But, for now, an antigen is a bad thing that shouldn't exist inside you.

How are they binding to these antigens?

Well, most B-cells are actually identical to one another except for the part of it that binds to their unique antigen. The singular part of them that is unique and responsible for this binding is called the B-cell-receptor, or BCR. This functionality is meant to be available from the very moment a B-cell is born, it lives to bind to something, even if it doesn't know what it is. It may live its entire life never binding to the thing it was born to bind to. This will a...