trentmkelly/LessWrong-43k huggingface dataset:
Postmortem on DIY Recombinant Covid Vaccine
by caffemacchiavelli — 2022-01-22T14:12:58.030Z

edit: Changed title from "Postmortem on RatVac" for clarity.

Note: We named the vaccine candidate "RatVac" as a tongue-in-cheek abbreviation for "Rationalist Vaccine". We have no association with the RaDVaC project and use a much simpler, almost trivial approach. I don't endorse making your own vaccine or taking anything nerdy people on the Internet send you. This is not medical advice.

tldr: In April of 2021 I assembled my own subunit vaccine against the Beta variant of SARS-Cov-2 ("SARS2"). Despite starting with the prior that this formulation should be somewhat effective, I could not demonstrate efficacy against the Alpha variant in antibody tests.

Introduction

I started getting interested in DIY vaccines soon after the Making Vaccine and We got what's needed for COVID-19 vaccination completely wrong posts were published. Particularly the idea of just making a standard subunit vaccine appealed to me, so I teamed up with some interested people from the rationalist diaspora and we ended up making our own vaccine candidate in early to mid 2021. Some contributed advice, many contributed funding (for which I am still extremely grateful), and while the project wasn't the clear success I had hoped for, maybe the true treasures are the friends we gained and the lab equipment we bought along the way.

Recombinant Vaccine ELI5

Subunit vaccines contain a subset of a single protein molecule. For viruses this will typically be a protein that contains the receptor-binding-domain (RBD), i.e. the part that actually binds with the host during infection.

Through the wonders of genetic engineering - see very short introduction below - we can produce these proteins and then simply introduce them into the human body.

While Yang et al have shown that an immune response occurs even with a vaccine purely containing viral RBD, the response can be enhanced by adding an immunologic adjuvant. Adjuvants are a very diverse class of chemicals, ranging from simple mineral salts to proteins or even small cells. The only thing they ostensibly have in common is that they're used to increase or modulate immune response, and that's why we're using them.

Genetic Engineering ELI5

For those curious about how we can manufacture proteins to begin with - because I definitely was - here is a very short and abridged intro. The process differs depending on the method used, but usually roughly follows thes...
There is a line in the sand, just not where you think it is
by Wuschel Schulz — 2022-01-22T10:33:37.607Z

There is a mistake in updating beliefs around norms that I have seen multiple times in different contexts made by different people. This is my attempt to classify this (fallacy? /mistake?), in order to give it a name and make it easier to refer to.

A friend of mine was recently honestly surprised that two government politicians got into trouble for personally taking money from mask-producing companies in return for securing government orders. He was not surprised that the politicians were corrupt. He was surprised that there were any repercussions. He thought that politicians could totally get away with this. How did he acquire that belief? A while earlier it was revealed that another politician received stock options from an IT startup, for which he then put in a good word at the government. He only got shit from the press, but his career is still going well.

My friend had previously thought that there was a strong norm against politicians being corrupt and that any violation of this norm would end their career. The politician who got away with the stock option affair was strong evidence against such a norm being enforced. Thus, he updated towards the belief that corrupt politicians can just get away with anything.

But how does this constitute a violation of rationality? Wasn’t I saying that this was some sort of fallacy? What did he do wrong?

His mistake was this: He assumed that, if there was a norm against corruption, it would follow a definition of corruption that did in some way actually corresponds to the degree to which the act was harmful. Let’s look at this in a graphical way:

Norms here are represented with lines, separating the forbidden acts on the left of the line from the allowed acts on the right of the line. His hypothesis space only consisted of straight lines. His prior was that there was a norm enforced (norm A). Then he observed act 1 occur unpunished. This made him update that there was either no norm or maybe a norm as far out as norm B. Then he observed act 2 being punished and was confused. The true norm that was enforced all along was norm C, which looks pretty gerrymandered. Does it make sense that MPs are obliged to formally record monetary payments they receive, but not stock options? Not really, but that is how it is.

I have also seen the same play out in other contexts. For example, after the FSB (probably) killed a forme...
Grandpa Has Different Rules
by jefftk — 2022-01-22T03:30:04.749Z

It's very common that there are multiple adults in a child's life
who parent differently. These can be explicit differences
in rules (I allow jumping on the furniture, you don't), or more
subtle differences in approach (I'm fine with kids trying to persuade
me something should be ok, not everyone is). Adults often get in
conflict over these, and I think this is generally not worth it.

There are two main reasons I've seen for parents wanting others to
take a specific approach with their kids:

You want consistency. I'm strongly in favor of predictable parenting, where kids
know what the boundaries are and can predict your reactions. Kids are
smart, however, and are very good at learning that different
adults/places/situations have different systems.

You want a specific approach. There are a lot of ways
people parent, and they mostly work fine, especially in moderation.
Perhaps the aunt lets the kids have lots of sweets, grandma requires shoes
outside, or the babysitter won't take them out if it's raining. While
these aren't the decisions I would make, it doesn't matter that
much. Now, I wouldn't want anyone who looks after my kids to enforce
discipline by hitting them, and neither would I be ok with someone
letting them play unattended by deep water: some things do matter.
But the conflicts I tend to see are over much smaller differences.

One place where this can be more of a problem is when multiple adults
with different styles are present at the same time. I think the main
thing you want is consistency: if the parent is present you go by
their rules, or perhaps you go by the rules of whichever space you're
currently in. This does mean that if adults are often ambiguously in
charge of the same kids it's worth putting effort into harmonization:
different parents having different rules doesn't work very well. I
expect this to come up if/when our housemates have kids.

There's also the issue of cross jurisdictional precedent ("grandpa
lets me do it!") The most common way to handle this is just "grandpa
has different rules", and leave it there. This is fine, but in my
particular parenting style I'm happy to get into the details if the
kids want to. Why is grandpa ok with it? ("That much sugar all
the time wouldn't be good for you, but you're with grandpa rarely
enough that I'm not worried.") Sometimes I don't know why different
...
Instrumental Convergence For Realistic Agent Objectives
by TurnTrout — 2022-01-22T00:41:36.649Z

Edit, 5/16/23: I think this post is beautiful, correct in its narrow technical claims, and practically irrelevant to alignment. This post presents an unrealistic picture of the role of reward functions in reinforcement learning, conflating "utility" with "reward." Reward functions are not "goals", reward functions are not "objectives" of the policy network, real-world policies are not "optimal", and the mechanistic function of reward is (usually) to provide policy gradients to update the policy network.

I expect this post to harm your alignment research intuitions unless you've already inoculated yourself by deeply internalizing and understanding Reward is not the optimization target. If you're going to read one alignment post I've written, read that one.

Follow-up work (Parametrically retargetable decision-makers tend to seek power) moved away from optimal policies and treated reward functions more realistically.

The current power-seeking theorems say something like:

Give me a utility function, any utility function, and for most ways I could jumble it up—most ways I could permute which outcomes get which utility, for most of these permutations, the agent will seek power.

This kind of argument assumes that (the set of utility functions we might specify) is closed under permutation. This is unrealistic, because practically speaking we reward agents based off of observed features of the agent's environment.

For example, Pac-Man eats dots and gains points. A football AI scores a touchdown and gains points. A robot hand solves a Rubik's cube and gains points. But most permutations of these objectives are implausible because they're high-entropy, they're very complex, they assign high reward to one state and low reward to another state without a simple generating rule that grounds out in observed features. Practical objective specification doesn't allow that many degrees of freedom in what states get what reward.

I explore how instrumental convergence works in this case. I also walk through how these new results retrodict the fact that instrumental convergence basically disappears for agents with utility functions over action-observation histories.

Case Studies

Gridworld

Consider the following environment, where the agent can either stay put or move along a purple arrow.

From left to right, top to bottom, the states have labelss△,s◯,sleft,sstart,sright,s★.

...
Alignment Problems All the Way Down
by peterbarnett — 2022-01-22T00:19:23.506Z

Epistemic status: pretty exploratory. I think this is a coherent concept, but I wouldn’t be surprised if there need to be some large changes.

Edit [7/12/2023]: I think this post is pretty confused and confusing, and doesn't really address important parts of the alignment problem. The strategy of "avoid mesa-optimizers" no longer even seems like a coherent thing to aim for, and instead just sounds like never building powerful AI. I do think this post almost gets to important problems (like robust delegation), but doesn't really provide much that is useful. I don't regret writing this, and I think having pretty bad ideas is the first step to having kinda good ideas.

TL;DR: A mesa-optimizer may instantiate other optimizers; these new optimizers may not be aligned with the original mesa-objective. Therefore to avoid dangers from misaligned mesa-optimizers we should avoid learned optimization entirely, rather than attempting to align mesa-optimizers.

When thinking of AI Alignment it is common to divide the questions into the “outer alignment problem” and the “inner alignment problem”. The outer alignment problem refers to the problem of telling an AI system to do what we actually want; telling a system to maximize paperclips could cause an outer alignment failure because humans do not actually want to single-mindedly maximize the number of paperclips in the universe. The inner alignment problem refers to the task of making sure an AI system actually does what we tell it to do. Even if we manage to solve the outer alignment problem, there is no guarantee that an AI system will actually optimize for the objective that we give it. This is commonly discussed in the context of mesa-optimization, where our base-optimizer (for example, gradient descent) trains our model to itself perform some kind of optimization (this model is hence called a mesa-optimizer). The inner alignment problem here is about how to ensure that the objective of this mesa-optimizer is the same as the objective of the base-optimizer.

This post discusses the possibility that a mesa-optimizer may itself create an optimizer. Similarly to the classic inner alignment problem, it is not obvious that the objective of a mesa-optimizer will be robustly inherited by any optimizers which it creates. Because of the difficulty of ensuring that every level of mesa-optimization is aligned, I think ...
Consume fiction wisely
by RomanS — 2022-01-21T20:23:50.873Z

TLDR: Fiction is often harmful for your mind, and it is often made to manipulate you.

The Top Gun Effect

In the comments to this post, Bezzi provides a strong and concise argument worth copying here. If you have no time to read the whole post, just read this:

[A]fter reading about the Top Gun Effect, I strongly updated towards "Hollywood movies can seriously mess up your mind". Also this:

According to the US Navy, the box office success of Top Gun saw their recruitment rates balloon by a massive 500% in the year following the original movie’s release. <...> The U.S. Navy set up recruiting stations outside of movie theaters when Top Gun was released, catching potential recruits as they were left the cinema hyped up by the movie’s dramatic climax.

If a single movie can make people do very serious life choices like signing up for the Navy, then we should probably be less confident in our ability to separate fiction from real-life.

Intro

As the majority of people, you're likely consuming a lot of fiction (science fiction books, Hollywood movies, anime, video games, Twitter etc).

One can classify the content of fiction into two classes:

events that have never happened (e.g. fantasy)
real events, but so distorted, they don't help you better understand the real world (e.g. TV news)

Both classes are often harmful, as they cloud your mind with falsehoods and wrong associations.

Learning falsehoods

Your mind is learning from what it consumes. If you give it the training data full of falsehoods, it will learn falsehoods.

And mass media is full of falsehoods.

Some of them are harmless (Audible Sharpness). Some could kill you (Hollywood Heart Attack).

Some of them are easy to overcome (Space Is Cold). Some could stick in your mind forever (The Good Guys Always Win).

And some are very hard to detect. For example, you're likely assigning unreasonably high likelihoods to dramatic and interesting events - because you've seen way too many fictional events (which are, by design, dramatic and interesting).

Such distortions could negatively affect the quality of your judgements in real life.

For example, imagine that we've detected a massive alien spaceship approaching our Solar System, with the intent of exterminating humans. Which of these two events is more likely?

Humans will fiercely resist the invaders, with many losses on both sides. You know, like in the Indepe

...
[AN #171]: Disagreements between alignment "optimists" and "pessimists"
by Rohin Shah — 2022-01-21T18:30:04.824Z

Listen to this newsletter on The Alignment Newsletter Podcast.

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Please note that while I work at DeepMind, this newsletter represents my personal views and not those of my employer.

HIGHLIGHTS

Alignment difficulty (Richard Ngo and Eliezer Yudkowsky) (summarized by Rohin): Eliezer is known for being pessimistic about our chances of averting AI catastrophe. His argument in this dialogue is roughly as follows:

1. We are very likely going to keep improving AI capabilities until we reach AGI, at which point either the world is destroyed, or we use the AI system to take some pivotal act before some careless actor destroys the world.

2. In either case, the AI system must be producing high-impact, world-rewriting plans; such plans are “consequentialist” in that the simplest way to get them (and thus, the one we will first build) is if you are forecasting what might happen, thinking about the expected consequences, considering possible obstacles, searching for routes around the obstacles, etc. If you don’t do this sort of reasoning, your plan goes off the rails very quickly - it is highly unlikely to lead to high impact. In particular, long lists of shallow heuristics (as with current deep learning systems) are unlikely to be enough to produce high-impact plans.

3. We’re producing AI systems by selecting for systems that can do impressive stuff, which will eventually produce AI systems that can accomplish high-impact plans using a general underlying “consequentialist”-style reasoning process (because that’s the only way to keep doing more impressive stuff). However, this selection process does not constrain the goals towards which those plans are aimed. In addition, most goals seem to have convergent instrumental subgoals like survival and power-seeking that would lead to extinction. This suggests that we should expect an existential catastrophe by default.

4. None of the methods people have suggested for avoiding this outcome seem like they actually avert this story.

Richard responds to this with a few distinct points:

1. It might be possible to build AI systems which are not of world-...
How to Become a Magician
by Nicole Dieker — 2022-01-21T18:12:11.758Z

Crossposted from Nicole Dieker Dot Com, naturally. I wrote this one before I started following the LW community, so "magician" can be replaced with "hero" if you want — although I like using "magician" for reasons that will become clear in the post.

Start by asking yourself what you ought to wear.

No, seriously — you need to be comfortable, but you also need to look like you, displaying the kind of appearance that you value, and if you're the kind of person who values displaying that same sort of appearance every day it helps if you can buy the same outfit seven times over.

(In either the same or different colors.)

Eliminating the "what to put on your body" decision, whether you go the full-on uniform route or the "everything in my closet is something I like" route, gives you that much more mental space to devote to the pursuit of excellence.

Plus, it makes getting dressed feel like a decision you are making to support your art.

I'm not saying that every decision should be made in support of your art, or in support of whatever excellence you are currently attempting to pursue (in work, in life, in a relationship), but — well,

as soon as I used the word CURATION in yesterday's post

I knew it was the right one.

You are curating the experiences you want, in order to have more of them.

And in order to give your mind as much space as possible to

think about what you really want to think about

.

So.

Clothes that don't suit get in the way.

Not enough sleep gets in the way.

Not enough exercise, rest, food, touch, time spent outdoors — whatever it is you need to be your best self, to feel comfortable in the world instead of at odds with it, you need to make sure you have it, as often as you need it, or the lack will keep you from doing your best work.

Too much also gets in the way. Too many clothes in the closet; too much food in the belly; too much time spent dilly-dallying.

It really does start with balance; the foot on the high wire, the plate on the stick, the card placed carefully at the top of the castle.

Knowing instead of guessing.

Acting instead of reacting.

Choosing instead of letting the system choose for you.

Can you pursue magic, excellence, artistic merit, etc. etc. etc. without simultaneously pursuing agency, balance, specificity, and discipline in your own life?

Of course you can.

Can you create magic — can you become a magician — without also curating a life that support...
Due to a roommate, can COVID's viral load stay high enough after day 10 to infect others?
by catears123 — 2022-01-21T07:08:02.797Z

I got Omicron - I completely isolated from my roommate for three days. She ends up testing positive due to initial exposure before I tested positive. Since we both got COVID, we decided to isolate together. Currently, I'm on day 7. She is on day 4. Day 10 is when we are allowed to leave isolation at our school.

Our friends are saying that: my roommate and I should stop hanging out when I'm able to leave isolation because I'd be spending time with an infected person, so I would have viral load without being infected. This viral load could still infect someone else.

What is the research on this with omicron, COVID-19, etc? What are your thoughts on this?
Implications of Civilizational Inadequacy (reviewing mazes/simulacra/etc)
by Raemon — 2022-01-21T01:45:11.208Z

This was written for the LessWrong Review, intended as a high level exploration of the 2020 posts on Moral Mazes, Simulacra, Coordination Theory, and Civilizational Adequacy.

I. What Makes For a Useful Review?

When reviewing LessWrong posts, fairly often my reaction is "Cool hypothesis bro, but, can we actually go out and verify it?". A lot of LessWrong posts explore interesting ideas, and those ideas match my intuitions. But lots of intuitive ideas turn out to be wrong, and sometimes people have different intuitions. It seemed good to have a step where we actually turned armchair philosophy into empirical claims.

A year ago, I began two attempts at investigating some claims:

Eliezer's conjecture that Mathematicians were more law-abiding that non-mathematicians (controlling for IQ).
The overall Moral Mazes sequence.

Mathematicians and "Is this data worth the expense?"

Elizabeth and I made some attempt to run a Positly study on math / law-abidingness. We quickly ran into "Running a real, legit study is a pretty big time cost." (We ran one quick-and-dirty study that essentially boiled down to asking people 'are you good at math?' and 'do you do crimes?', which Eliezer specifically called out as Not A Good Approach in his post. We tried looking for datasets that might be relevant to our interests, but finding and sifting through them was hard).

Ultimately I faced the question: "is this question actually worth investing this effort in?" and I felt the answer was "probably not." It seemed vaguely informative, and I felt a nagging annoyance at publishing Local Validity As Key To Sanity and Civilization without having investigated it. But after a lot of reflection (on the timescale of years), I came to the belief that not every claim is worth investigating.

If I was worried about the epistemic integrity of the LessWrong books, a much cheaper approach would be to include some metadata about the epistemic status of each post, or suggestions of future work that signposted which areas warranted more investigation.

Moral Mazes, and "What are we actually trying to figure out?"

The Immoral Mazes sequence raised a lot of interesting hypotheses, which matched my intuitions/experience. But it seemed to me the next step was do form some actual predictions and go out and test them.

At the time, I was thinking about Mazes largely from the standpoint of "I want to build a ...