trentmkelly/LessWrong-43k huggingface dataset:
RL with KL penalties is better seen as Bayesian inference
by Tomek Korbak — 2022-05-25T09:23:33.202Z

This blog post is largely based on an EMNLP paper with Ethan Perez and Chris Buckley. It also benefited from discussions with and comments from Hady Elsahar, Germán Kruszewski, Marc Dymetman and Jérémy Scheurer.

TLDR: KL-regularised RL, widely used as part of RL from human feedback (RLHF), is equivalent to variational inference: approximating a Bayesian posterior which specifies how to update a prior LM to conform with evidence provided by the reward function. The Bayesian perspective makes it clear that KL penalties aren’t a hack; they have a principled justification. It also nicely separates the modelling problem (defining a target distribution specifying the desired behaviour of an LM) and the inference problem (approximating that target distribution). Finally, it suggests that RL is not a good formal framework for thinking about LM alignment.

Introduction

Large language models (LMs) tend to generate outputs that reflect undesirable features of their training data such as offensiveness, social bias, harmfulness or dishonesty. Correcting these biases and constraining LMs to be honest, helpful and harmless is an essential part of the problem of aligning LMs with human preferences (henceforth “LM alignment”). One intuitive approach to LM alignment is reinforcement learning (RL): capturing human preferences as a reward function and training the LM to maximise the reward expected under LM distribution. A practical recipe for implementing this idea is RL from human feedback (RLHF): first, a reward model is trained to predict which of two texts a human prefers and then a pretrained LM is fine-tuned to maximise reward given by the reward model while being penalised for Kullback-Leibler (KL) divergence from its initial distribution. However, despite immense popularity of RLHF, the motivation for this KL penalty is not widely understood.

In this blog post, we discuss an underappreciated perspective on KL-regularised RL — the objective employed by RLHF for aligning LMs — which explains its empirical success. We start with describing a problem that arises from naively applying the standard RL objective: distribution collapse. The optimal policy under the RL objective would be a minimal-entropy LM generating a small set of sequences that obtain the highest reward. Then, we discuss how KL-regularised RL avoids distribution collapse due to its KL penalty. This constraint,...
A Quick Note on AI Scaling Asymptotes
by alyssavance — 2022-05-25T02:55:45.994Z

Two months ago, DeepMind announced their new Chinchilla paper. The headline result was that you could get much better performance with the same computational budget, by training a smaller model for longer:

However, it seems to have been mostly unnoticed that the "scaling law" they found was generally different from the first one in Kaplan et al. Kaplan shows a "pure" power law:

while the law in the Chinchilla paper has a large constant factor and substantially larger overall exponent. Note that N here (parameters) and D (data) are both roughly proportional to the square root of C (compute):

This results in similar behavior within a certain range, but very different asymptotic behavior:

(Graph produced by plotting L(c) = 26.387 / 10^0.05c for Kaplan and L(c) = 1.69 + 514.5/10^0.156c + 560.2/10^0.1512c for Chinchilla, where c is log10(FLOPs) and L is loss; I juggled a few terms around so the two curves could be compared directly.) This curve shape appears to be confirmed by their empirical results:

This isn't a super-surprising result IMO - page 17 of Kaplan predicts that a pure power law can't continue indefinitely, and it's not clear how loss translates into practical performance anyway - but it seemed worth noting explicitly.
EDIT: There was an earlier followup to the Kaplan paper, by many of the same authors, that also tried to break down scaling into "reducible loss" (that improved with model size) vs. "irreducible loss" (a constant factor) across several different AI domains; although unlike the Chinchilla paper, they don't seem to estimate the "irreducible loss" for language models specifically. The paper discussion on LW didn't mention this and I had missed it, thanks to Celestia for pointing it out! Here's a video discussing the results:
MERV Filters for Covid?
by jefftk — 2022-05-25T01:50:02.730Z

If you look around for advice on what kind of air purifier to use to
reduce covid risk, you'll see some people saying you need HEPA filters:

Microcovid:

If you decide to buy an air purifier for COVID purposes, here's some
things to keep in mind:

Make sure the purifier has a HEPA filter - these are rated to remove
99.97% of airborne particles.

Central heat / AC systems don't work for this. These typically use
MERV-rated filters. These are significantly less effective at removing
small particles than HEPA-rated filters (the best MERV rating,
MERV-16, merely removes 75% of particles. [JK: this should be 95%—filed
#1451]

The EPA,
however, advocates anything that removes 0.1-1 µm particles well,
and recommends MERV-13 or better if you're building something:

In order to select an air cleaner that effectively filters viruses
from the air, choose: 1) a unit that is the right size for the space
you will be using it in (this is typically indicated by the
manufacturer in square feet), 2) a unit that has a high CADR for smoke
(vs. pollen or dust), is designated a HEPA unit, or specifically
indicates that it filters particles in the 0.1-1 µm size range.

...

When assembling a DIY air cleaner, choose a high-efficiency filter,
rated MERV 13 or higher, for better filtration.

Where does the recommendation to use MERV-13 come from? Can I use
MERV-12? If I use MERV-14 instead how much better will it be? Let's
model it.

Imagine you have an infected person who enters a room. They exhale
sars-cov-2 particles, which slowly accumulate:

These particles are a range of sizes, but let's approximate that 20%
of the airborne sars-cov-2 is in particles of size 0.3-1µm, 29%
in 1-3µm, and 51% in 3-10µm (see How Big Are Covid
Particles?):

The particles don't actually accumulate forever, for several reasons,
one of which is that they typically slowly settle out of the air. This
depends on size, with half lives of ~4d for 0.3µm, ~2hr for
2µm, and ~20min for 5µm:

Another reason particles don't accumulate forever is the same reason
that even with the windows closed we don't suffocate: buildings aren't
airtight. You describe how much ventilation a building has in "air
changes per hour" (ACH), and a typical value for residential
construction is 2 ACH. For example, if your room is 1,000 cubic feet
(CF) then you might have 2,000 cubi...
Visible Homelessness in SF: A Quick Breakdown of Causes
by alyssavance — 2022-05-25T01:40:43.768Z

Anyone who's visited downtown San Francisco has seen streets like this one:

The US is the richest country on Earth, excluding small states like Singapore with <10 million people. And the Bay Area is one of the US's economic hubs. Why do we see stuff like this? I'll try to break it down quickly.

First, San Francisco is obviously an expensive place to live. Do housing costs predict high homelessness?

They do! R^2 = 0.69 is really good. (This is a regression on US states, data from here and here. Washington, DC is a high outlier; removing it raises R^2 to 0.73. High outliers include DC, Washington, Nevada, Alaska and New Mexico; low outliers include Maryland, New Jersey, Connecticut, Rhode Island and Virginia, although the first might be an artifact of DC being outside state borders.)

SF's housing costs are 3.3x higher than the US average. Plotting it on this line, we should expect a homelessness rate of 0.7%, 5.8x higher than a city with normal rents. The actual rate is 0.9%, another 1.3x higher than we'd expect compared to rent.

However, there are two other big factors to consider. #1 is that San Francisco has a very strong concentration of homeless people downtown. Here's a breakdown by supervisor district:

Districts are drawn to have equal populations, so District Six, which includes a lot of downtown buildings and office space, has a 5.1x higher rate of homelessness even compared to SF's average. Most cities don't have that. Eg. New York City has roughly the same homelessness rate as SF, but homelessness is more evenly spread through the city; and the worst 10% of neighborhoods are split between the Bronx, Brooklyn, and Queens, instead of being concentrated downtown.

Factor #2 is homeless people who are sheltered vs. unsheltered. Unsheltered homeless people sleep outside or in vehicles, so they're much more visible. 68% of San Francisco homeless are unsheltered, compared to 25% for the US outside of California. That's another factor of 2.7x.

Summing it up, we have:

Cause of visible homelessness	Factor of	Percent of all causes
SF has higher housing costs	5.8x	38%
SF has more homeless, adjusting for rents	1.3x	6%
Homelessness concentrated downtown	5.1x	35%
SF homeless are more unsheltered	2.7x	21%
Total	103.8x	100%

(Note that, since these causes are multiplicative, percentage here is calculated by first taking the log, then dividing.) So, in total, visible homelessness ...
autonomy: the missing AGI ingredient?
by nostalgebraist — 2022-05-25T00:33:25.356Z

Epistemic status: trying to feel out the shape of a concept and give it an appropriate name. Trying to make explicit some things that I think exist implicitly in many people's minds. This post makes truth claims, but its main goal is to not to convince you that they are true.

Here are some things I would expect any AGI to be able to do:

Operate over long intervals of time relative to its sensory bandwidth (e.g. months or years of ~30 fps visual input).
Remember specific sensory experiences from long ago that are relevant to what's happening to it now. (E.g. remember things it saw months or years ago.)
Retain or forget information and skills over long time scales, in a way that serves its goals. E.g. if it does forget some things, these should be things that are unusually unlikely to come in handy later.
Re-evaluate experiences that happened a long time ago (e.g. years ago) in light of newer evidence (observed in e.g. the last hour), and update its beliefs appropriately.
Continually adjust its world model in light of new information during operation.

E.g. upon learning that a particular war has ended, it should act as though the war is not happening, and do so in all contexts/modalities.
As with humans, this adaptation may take a nonzero amount of time, during which it might "forget" the new fact sometimes. However, adaptation should be rapid enough that it does not impede acting prudently on the most relevant implications of the new information.
This may require regular "downtime" to run offline training/finetuning (humans have to sleep, after all). But if so, it should require less than 1 second of downtime per second of uptime, ideally much less.

Perform adjustments to itself of the kind described above in a "stable" manner, with a negligibly low rate of large regressions in its knowledge or capabilities.

E.g. if it is updating itself by gradient descent, it should do so in a way that avoids (or renders harmless) the gradient spikes and other instabilities that cause frequent quality regression in the middle of training for existing models, especially large ones.

Keep track of the broader world context while performing a given task.

E.g. an AGI playing a video game should not forget about its situation and goals in the world outside the game.
It might "get distracted" by the game (as humans do), but it should have some mechanism for s

...
Explaining inner alignment to myself
by Jeremy Gillen — 2022-05-24T23:10:56.240Z

Explaining the inner alignment problem to myself in an attempt to gain a gears-level understanding. This post is mostly a distillation of

Risks from Learned Optimization

, but focusing more on the statistical learning theory perspective. It isn't quite an introduction, more an attempt at reframing and summarizing to make the problem easier for me to think about. Any feedback is appreciated. Written as my final project for

AGI Safety Fundamentals

, also submitted to the

AIMS distillation contest

.

Inner misalignment <=> poor generalization of goal learning

A good ML algorithm generalizes well, meaning it performs as intended on tasks it has never seen before (i.e. the test loss is low). Inner misalignment is a type of generalization problem we expect to face when training powerful ML systems that will cause these systems to generalize very poorly and dangerously in out-of-distribution situations. Inner alignment problems can only occur when your learning algorithm is powerful enough that it is selecting an algorithm from some set of hypothesis algorithms, and the algorithm it happens to select is an optimizing algorithm. Problems arise when the chosen optimizing algorithm has an objective that doesn't precisely match the objective of the base optimizer (i.e. the expected loss of the ML algorithm). The inner optimizer (called a mesa-optimizer), could be any algorithm that searches over a space of possible outputs and chooses the best one (with “best” defined by some internal objective, called the mesa-objective).

The key part of this definition is that the learned objective doesn’t exactly match the base objective.[1] The word objective here is pretty much interchangeable with goal or utility function. So the core of the problem of inner alignment is simply that we might try to learn a goal from data, but the goal was learned imperfectly and fails to generalize when tested on some input outside of the training distribution.

Trying to make algorithms that generalize well is the whole point of machine learning; all of statistical learning theory is focused on this question. What’s different and important about this particular form of generalization failure?

It will be particularly difficult to detect, because it will perform well on many (or all)[2] test cases.
It appears to be difficult to reduce the impact of

...
Google's Imagen uses larger text encoder
by Ben Livengood — 2022-05-24T21:55:52.221Z

https://imagen.research.google/

Scaling the text encoder gives Imagen the ability to spell, count, and assign colors and properties to distinct objects in the image that DALL-E2 was not so great at. It looks visually about as photorealistic as DALL-E2 from the small set of sample images. Eyes are still weird.
Endpoint Specification, or Good Help is Hard to Find
by ACrackedPot — 2022-05-24T21:00:07.453Z

If you want to convey the coordinates [-1.0219,18.7172,.3181,182132.2290], what is the minimum amount of information you must convey, in order to be certain that the recipient of the information understands it?

Well, supposing there is a monument there, you could give somebody the name of the monument, and the time. However, maybe the monument shares a name with a local cafe which is more familiar to them, and they show up there instead.

You specify that it is the monument by that name, aware of that potential risk. They show up an hour late - when you gave them the time, you gave them their local time, and they helpfully converted it to the timezone where they were going.

You specify the timezone, and they're still an hour late; you didn't realize they were very specific about Eastern Standard Time versus Eastern Daylight Time, and interpreted your statement incorrectly.

You're very particular about when they show up, but when they get there, they haven't brought the zoom lens you wanted for the photo shoot.

What, did I forget to mention the photo shoot?

In a sense, the problem here is miscommunication. However, that is merely labeling the problem; I have a personal label for the problem here as well, which makes sense internally, and will make no sense to you, because the problem here isn't actually communication: The problem here is that the minimum amount of information to fully specify exactly what you want is, effectively, you.

Alternatively, the minimum amount of information to specify exactly what you want, is the thing you want, in which case you already have it; see programming.

If you've ever worked with a client, you will be familiar with the experience of doing exactly what the client asked for, completing the work, showing it to them - and discovering that "what they asked for" does not actually match what they want. From your perspective, they asked for the wrong thing; from their perspective, they asked you to show up at Eastern Time, and you showed up an hour late; they asked for the correct thing - you just didn't hear the correct thing. And this bit is blue, and clearly it should be red, why did you make it blue? They didn't specify? Well, why would they need to specify that it was the version of the time zone everybody in the state is already using?

The people who are really successful at dealing with clie...
Game Theory and Partner Abuse
by arielleplus — 2022-05-24T20:03:18.027Z

I wrote a reflection on Game Theory and partner abuse. Curious about this community's thoughts.

***
I confess: I’ve been getting sucked into the Johnny Depp/Amber Heard trial. Last week devolved into a blur of articles, Twitter threads, and subreddits, culminating in the testimony of the two protagonists themselves.

It got to the point where I was posting comments.

There’s a problem at the center of the case: when both sides accuse one another, how do we know who to believe?

Game Theory can be helpful here: it’s a field which uses math to model strategic interactions and make sense of human behavior. Once I started thinking about Game Theory, the Depp-Heard trial lost some of its obsessive pull, and I was able to return to my life.

Let’s look at Game Theory, and see if we can figure out how to react—as bystanders, fans, or friends—when two people make bilateral accusations of abuse (we’ll keep the math contained, I promise). By the end of this article, we'll come up with a Game Theory informed algorithm for analyzing cases like this, and I’ll apply it to the Johnny Depp/Amber Heard case.

Part 1: Definitions and Axioms

Definitions (for edification and clarity)

Abuse: A pattern of behavior involving control and manipulation, where one person treats another as a resource (for sex, emotional release, stimulation, housework, etc). Can include physical violence, intimidation, gaslighting, outbursts of anger, displays of jealousy, verbal attacks, and isolation.

Partner abuse: Abuse in the context of a romantic relationship (also called “intimate partner abuse,” but we’ll keep it simple).

Psychological abuse: For the purposes of this article, a catch-all for abuse which doesn’t involve violence.

Unilateral abuse: In the context of a relationship, a situation where only one partner is abusive to the other.

Bilateral abuse: In the context of a relationship, a situation where both partners are abusive to the other.

Abuser: One carrying out abuse.

Target: One on whom abuse is being carried out. I’ll use it interchangeably with “victim.”

Violence: Use of physical force on another person. As above, violence can be unilateral or bilateral (or multilateral, but that’s outside the scope of this article).

Toxic relationship: A relationship where one partner is spiraling to a lower level of consciousness/functioning, and pulling the other person down with them. As above, toxicity can be unilat...
Mandatory Post About Monkeypox
by Zvi — 2022-05-24T20:00:02.013Z

It’s 2022, so of course enough people are worried about an epidemic of Monkeypox that it feels like a necessary public service to do a post about Monkeypox. I didn’t get to it especially fast exactly because I am not all that concerned, but I’m not entirely not concerned, so here we are.

Sarah Constantin has a basic facts post. Basic facts if nothing has changed:

The smallpox vaccine is 85% effective at stopping spread, but it uses live virus and leaves a scar. It is not generally available but we have a strategic stockpile that is sufficient for the entire United States (like we should aim to do with all possible pandemic viruses, but congress is refusing).
There are claims of both a 2.1% and a 3.6% case fatality rate from this version of monkeypox. Most deaths are in immunocompromised individuals. Can also be dangerous for pregnant women or children.
Most cases historically come from animal-to-human transmission.
Person to person transmission comes from prolonged social contact, sexual contact, living with the infected, saunas or contact with infected clothing or bedding.
Most cases in the outbreak are men who have sex with men, but that’s not true of cases in Africa.
Historical R0 estimate of 1.0.
The WHO is treating this as a standard outbreak and following standard procedure. Link includes diagnostic criteria.
Except that it looks like it has changed 50 SNPs, so maybe things changed.

This is a link to a spreadsheet of cases.

This thread also has a version of basic facts. This thread also covers the facts. This thread is excellent and has a bunch of facts. I go over that one later.

This thread contains links to 10 scientific papers, none of which I have read. I haven’t reached the ‘read all the papers’ level of dive yet, which tells you I’m not that worried.

Here is an OurWorldInData monkeypox data explorer. As of writing this cases are continuing to climb.

I have more hope that the WHO will respond reasonably to something like this, that has been around for a while, as opposed to Covid-19 which was novel. That doesn’t mean I’m counting on it.

I am not overly concerned, but I would describe the current situation as ‘concerning’ and would be much more concerned if no precautions were being taken or if people were generally acting as unconcerned as health officials are urging us to be.

The situation does not seem all that analogous to the situation with Co...