trentmkelly/LessWrong-43k huggingface dataset:
[Linkpost] Jan Leike on three kinds of alignment taxes
by Orpheus16 — 2023-01-06T23:57:34.788Z
Link: https://aligned.substack.com/p/three-alignment-taxes

Jan Leike leads the safety team at OpenAI. This post is from a Substack series titled "Musings on the Alignment Problem". I've found his posts helpful for understanding how the OpenAI safety team is thinking about alignment & for developing my own models of alignment/AI x-risk. I encourage more people to engage with Jan's blog posts.

In the post distinguishing three alignment taxes, Jan distinguishes between three types of alignment taxes. I appreciate how the post (a) offers concrete definitions of various types of alignment taxes, (b) discusses how they might affect AI labs in a competitive market, and (c) discusses how they might affect AI-assisted alignment research.

I'm including some quotes below (headings are added by me):

The three types of alignment taxes

In the general sense an alignment tax is any additional cost that is incurred in the process of aligning an AI system. Let’s distinguish three different types of alignment taxes:

Performance taxes: Performance regressions caused via alignment compared to an unaligned baseline.
Development taxes: Effort or expenses incurred for aligning the model: researcher time, compute costs, compensation for human feedback, etc
Time-to-deployment taxes: Wall-clock time taken to produce a sufficiently aligned model from a pretrained model.

Past work on performance taxes

In the past this performance tax has been measured by how much the model’s score is reduced on standard benchmarks after fine-tuning. While

training the first version of InstructGPT OpenAI observed performance regressions

on some standard benchmarks on question answering and translation.

Those were mostly but not entirely mitigated

by mixing pretraining data into the fine-tuning process.

Anthropic

,

DeepMind

, and

Google

have also studied alignment taxes as part of their alignment efforts, and sometimes alignment fine-tuning can even increase performance on several benchmarks, corresponding to a negative performance tax.

One way to operationally define a performance tax

However, there is a more natural way we could quantify this tax that lets us translate this tax more directly into monetary terms by measuring how much extra compute we need to spend at inference time to compensate for the performance regression. If our more aligned model needs to spend T% more inference-time compute to get from performance Z’ back to performance Z on capability X, then we

...
The Limit of Language Models
by DragonGod — 2023-01-06T23:53:32.638Z

Epistemic Status

Highlighting a thesis in Janus' "Simulators" that I think is insufficiently appreciated.

Thesis

In the limit, models optimised for minimising predictive loss on humanity's text corpus converge towards general intelligence[1].

Preamble

From Janus' Simulators:

Something which can predict everything all the time is more formidable than any demonstrator it predicts:

the upper bound of what can be learned from a dataset is not the most capable trajectory, but the conditional structure of the universe implicated by their sum

(though it may not be trivial to extract that knowledge).

Introduction

I affectionately refer to the above quote as the "simulators thesis". Reading and internalising that passage was an "aha!" moment for me. I was already aware (at latest July 2020) that language models were modelling reality. I was persuaded by arguments of the below form:

Premise 1: Modelling is transitive. If X models Y and Y models Z, then X models Z.

Premise 2: Language models reality. "Dogs are mammals" occurs more frequently in text than "dogs are reptiles" because dogs are in actuality mammals and not reptiles. This statistical regularity in text corresponds to a feature of the real world. Language is thus a map (albeit flawed) of the external world.

Premise 3: GPT-3 models language. This is how it works to predict text.

Conclusion: GPT-3 models the external world.

But I hadn't yet fully internalised all the implications of what it means to model language and hence our underlying reality. The limit that optimisation for minimising predictive loss on humanity's text corpus will converge to. I belatedly make those updates.

Interlude: The Requisite Capabilities for Language Modelling

Janus again:

If loss keeps going down on the test set, in the limit – putting aside whether the current paradigm can approach it –

the model must be learning to interpret and predict all patterns represented in language, including common-sense reasoning, goal-directed optimization, and deployment of the sum of recorded human knowledge

.

Its outputs would behave as intelligent entities in their own right. You could converse with it by alternately generating and adding your responses to its prompt, and it would pass the Turing test.

In fact, you could condition it to generate interactive and autonomous versions of any real or fictional person who has been recorded in the training co

...
Why didn't we get the four-hour workday?
by jasoncrawford — 2023-01-06T21:29:38.995Z
Link: https://rootsofprogress.org/the-four-hour-workday-prediction

John Maynard Keynes famously predicted in 1930 that by now we would only be working fifteen hours a week. What is less well-known is that his was nowhere near the only such prediction, nor the first—a wide range of commentators, including Charles Steinmetz and Buckminster Fuller, made similar forecasts. (And even Keynes’s prediction is generally misquoted.)

Why didn’t any of them come true? I recently discussed this with Jason Feifer on his podcast Build for Tomorrow. Here’s some elaboration with more quotes and charts.

The predictions

A 1934 book, The Economy of Abundance, summarizes many of the predictions (Chapter 2):

The technocrats promised every family on the continent of North America $20,000 a year [about $400,000 today], and a sixteen-hour work week. This is perhaps the peak of promises based on an abundance economy. Charles P. Steinmetz saw a two-hour working day on the horizon—he was the scientist who made giant power possible—but he stipulated no family budget total beyond “necessities and comforts.” …

Fred Henderson, in his Economic Consequences of Power Production, is more specific: “Without any further increase in our knowledge of power and of technical processes, or of our available materials, we could multiply production ten times over if the needs of the world were permitted to express themselves in effective demand. … It would not be a question of an eight-hour day or a six-day week, but more probably of a six-months working year—which is already the rule for university dons.”

Buckminster Fuller is still more definite. Modern man, he calculates, is 630 times more able than was Adam. Eliminating wasteful forms of work, four million Americans laboring fifty-two seven-hour days in the year (364 working hours, an average of one per day) “could keep up with every survival need”—meaning basic necessities for the whole population.

Walter N. Polakov announces that “fifty weeks, four days, six hours is enough”—a twenty-four hour week and two weeks’ vacation…

Harold Rugg in The Great Technology estimates a possible minimum living standard between ten- and twenty-fold greater than the minimums of 1929, on a sixteen- to twenty-hour work-week. …

One can continue to cite such evidence indefinitely. Fortunately, A. M. Newman has been collecting it for years and saves us the trouble by the following summary: “Among them [such estimates] a substantial agreement is

...
AI security might be helpful for AI alignment
by Igor Ivanov — 2023-01-06T20:16:40.446Z

I am a person new to AI alignment, and recently I am looking for a job at the AI governance field, but it seems like it is hard to find a long-term financially stable full-time job.

For me, it basically means that only a tiny amount of people consider AI alignment important enough to pay money for the decrease of P(doom).

It is important to get more people involved in the AI alignment field, but there is another good way to increase the number of people who might help achieve similar goals. We might cooperate with other groups of people, whose agenda is beneficial for AI alignment. One such group is the AI security community.

AI security

AI security is a branch of software security. Its goal is to ensure that current AI systems can not be exploited by hackers, are free of backdoors, and generally do things they are intended to do and do not do any unexpected and potentially harmful things.

For example, there is a grocery store not far from my home, which uses cameras with facial recognition algorithms to identify a buyer and automatically charge money from their credit card. Malicious actors might use some kind of makeup or a mask to trick the algorithm. So, to prevent it, the operator of this payment system might hire an AI security engineer to simulate an attack on the model, find its vulnerabilities and fix them.

Why this company should care that its facial recognition system has no security issues? Because if someone decides to hack it and steal money from its customer, it might become a huge scandal and lead to lawsuits, so it is relatively easy to convince top management to pay money for AI security. For them, it is just another branch of software security.

A similar thing with governments. I think it is manageable to convince governments to implement policies and guidelines for safe and robust AI systems because we already have similar policies for other important software.

AI security has a lot in common with the task of aligning AGI. There are few of them:

It benefits from interpretable models.
It requires the education of major decision-makers about the risks of unsafe AI systems.
Government and enterprise policies might require thorough testing and certification of AI models, it not only will make systems safer, but also might slow down the rate of AI progress which is good for alignment.
AI security also requires systematic audi

...
Categorizing failures as “outer” or “inner” misalignment is often confused
by Rohin Shah — 2023-01-06T15:48:51.739Z

Pop quiz: Are the following failures examples of outer or inner misalignment? Or is it ambiguous?

(I’ve spoilered the scenarios so you look at each one separately and answer before moving on to the next. Even if you don’t want to actually answer, you should read through the scenarios for the rest of the post to make sense.)

Scenario 1: We train an AI system that writes pull requests based on natural language dialogues where we provide specifications. Human programmers check whether the pull request is useful, and if so the pull request is merged and the AI is rewarded. During training the humans always correctly evaluate whether the pull request is useful. At deployment, the AI constructs human farms where humans are forced to spend all day merging pull requests that now just add useless print statements to the codebase.

Scenario 2: We train an AI system to write pull requests as before. The humans providing feedback still correctly evaluate whether the pull request is useful, but it turns out that during training the AI system noticed that it was in training, and so specifically wrote useful pull requests in order to trick the humans into thinking that it was helpful, when it was already planning to take over the world and build human farms.

I posed this quiz to a group of 10+ full-time alignment researchers a few months ago, and a majority thought that Scenario 1 was outer misalignment, while almost everyone thought that Scenario 2 was inner misalignment.

But that can’t be correct, since Scenario 2 is just Scenario 1 with more information! If you think that Scenario 2 is clearly inner misalignment, then surely Scenario 1 has to be either ambiguous or inner misalignment! (See the appendix for hypotheses about what’s going on here.)

I claim that most researchers’ intuitive judgments about outer and inner misalignment are confused[1]. So what exactly do we actually mean by “outer” and “inner” misalignment? Is it sensible to talk about separate “outer” and “inner” misalignment problems, or is that just a confused question?

I think it is misguided to categorize failures as “outer” or “inner” misalignment. Instead, I think “outer” and “inner” alignment are best thought of as one particular way of structuring a plan for building aligned AI.

I’ll first discuss some possible ways that you could try to categorize failures, and why I’m not a fan of them, and then discus...
Definitions of “objective” should be Probable and Predictive
by Rohin Shah — 2023-01-06T15:40:30.813Z

Introduction

Core arguments about existential risk from AI misalignment often reason about AI “objectives” to make claims about how they will behave in novel situations. I often find these arguments plausible but not rock solid because it doesn’t seem like there is a notion of “objective” that makes the argument clearly valid.

Two examples of these core arguments:

AI risk from power-seeking. This is often some variant of “because the AI system is pursuing an undesired objective, it will

seek power

in order to accomplish its goal, which causes human extinction”.

For example

, “The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.” This is a prediction about a novel situation, since “causing human extinction” is something that only happens at most once.
AI optimism. This is often some variant of “we will use human feedback to train the AI system to help humans, and so it will learn to pursue the objective of helping humans.” Implicitly, this is a prediction about what AI systems do in novel situations; for example, it is a prediction that once the AI system has enough power to take over the world, it will continue to help humans rather than execute a treacherous turn.

When we imagine powerful AI systems built out of large neural networks[1], I’m often somewhat skeptical of these arguments, because I don’t see a notion of “objective” that can be confidently claimed is:

Probable: there is a good argument that the systems we build will have an “objective”, and
Predictive: If I know that a system has an “objective”, and I know its behavior on a limited set of training data, I can predict significant aspects of the system’s behavior in novel situations (e.g. whether it will execute a treacherous turn once it has the ability to do so successfully).

Note that in both cases, I find the stories plausible, but they do not seem strong enough to warrant confidence, because of the lack of a notion of “objective” with these two properties[2]. In the case of AI risk, this is sufficient to justify “people should be working on AI alignment”; I don’t think it is sufficient to justify “if we don’t work on AI alignment we’re doomed”.

The core difficulty is that we do not currently understand deep learning well enough to predict how future systems will generalize to novel circumstances[3]. So, when ch...
200 COP in MI: Techniques, Tooling and Automation
by Neel Nanda — 2023-01-06T15:08:27.524Z

This is the seventh post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability. Start here, then read in any order. If you want to learn the basics before you think about open problems, check out my post on getting started. Look up jargon in

my Mechanistic Interpretability Explainer

Motivating papers: Causal Scrubbing, Logit Lens

Motivation

In Mechanistic Interpretability, the core goal is to form true beliefs about what’s going on inside a network. The search space of possible circuits is extremely large, and even once a circuit is found, we need to verify that that’s what’s really going on. These are hard problems, and having good techniques and tooling is essential to making progress. This is particularly important in mech interp because it’s such a young field that there isn’t an established toolkit and standard of evidence, and each paper seems to use somewhat different and ad-hoc techniques (pre-paradigmatic, in Thomas Kuhn’s language).,

Getting better at this is particularly important to enable us to get traction interpreting circuits at all, even in a one layer toy language model! But it’s particularly important for dealing with the problem of scale. Mech interp can be very labour intensive, and involve a lot of creativity and well honed research intuitions. This isn’t the end of the world with small models, but ultimately we want to understand models with hundreds of billions to trillions of parameters! We want to be able to leverage researcher time as much as possible, and the holy grail is to eventually automate the finding of circuits and understanding models. My guess is that the most realistic path to really understanding superhuman systems is to slowly automate more and more of the work with weaker systems, while making sure that we understand those systems, and that they’re aligned with what we want.

This can be somewhat abstract, so here are my best guesses for what progress could look like:

Refining understanding: In what contexts are current techniques most useful, and where are they misleading? How can we notice misapplications? Which techniques fail in the same way?

Eg, what’s up with

backup heads

, which take over when the main head is

ablated

This graph shows the direct effect on the logits from each head in Indirect Object Identificatio

...
Ball Square Station and Ridership Maximization
by jefftk — 2023-01-06T13:20:02.230Z

Matt Yglesias recently wrote

American
transit agencies should prioritize ridership over other goals, and
described how Boston's Green Line Extension was:

so far over budget that it got canceled and then
re-launched in a cheaper way. One key to reducing costs was
eliminating plans to build large custom stations and going with simple
bare-bones ones instead. So why did they propose the large stations in
the first place? I have no idea. The existing and well-used Green Line
stations are incredibly bare-bones. There was nothing in the local
transit tradition to suggest that the extension needed large grand
ones. All else being equal, people might prefer a really grand station
to a bare-bones one, but there was no MBTA modeling that claimed this
expenditure would make the system dramatically more appealing. They
simply weren't asking the question "what will the ridership impact of
this spending be?" because American transit agencies aren't in the
habit of asking those questions.

Yglesias is citing a Transit
Costs Project report, and his summary is fair. In fact, the 2016
redesign wasn't just more riders per dollar but, according to the
MBTA, the same number of projected riders for a much lower cost
(p283-284 of the 2019 TIP
report). But Yglesias (and the TCP report) imply that the
stations as finally built are simpler and cheaper structures than they
actually are.

Let's look at the Ball Square station, the one I'm most familiar with
since it's close to my house. Here's the layout:

Let's look specifically at the second entrance. It required an
elevator, stairs, a three-story structure with a roof, and a
connection to an existing bridge. What do you get? For riders
heading east across the Broadway bridge it saves ~150ft horizontally
(140ft vs 290ft) and ~20ft vertically. This probably applies to ~1/3
of riders, based on the fraction of nearby housing is across the
bridge and that Ball Square itself is on the station side of the
bridge.

I can see an argument that this entrance was worth the cost, by making
it easier and faster for people who need to cross the bridge,
especially if they use wheelchairs or strollers. Alternatively,
keeping the money and using it to help extend the Green Line to
Route 16 might have been better, or spending it elsewhere in the
system. But either way it's an example of the kind of thing I think
...
Childhood Roundup #1
by Zvi — 2023-01-06T13:00:00.915Z

I have a few shorter more focused posts in the works, including my practical short version of On Bounded Distrust. In the meantime, it makes sense to continue to clean out the backlog of roundup style things.

Let Your Children Play

We have gone insane on this.

Sane 2022 parents of 10-year-olds: I would like to let you go outside without me. I am terrified that someone will call the cops and they will take you away from me.

That is a thing now. As in parents being thrown in jail for letting their eight year old child walk home from school on their own.

As they stood on her porch, the officers told Wallace that her son could have been kidnapped and sex trafficked. “‘You don’t see much sex trafficking where you are, but where I patrol in downtown Waco, we do,'” said one of the cops, according to Wallace.

Did things get more dangerous since 1980, when we were mostly sane about this? No. They got vastly less dangerous, in all ways other than the risk of someone calling the cops.

The numbers on ‘sex trafficking’ and kidnapping by strangers are damn near zero.

The incident caused ‘ruin your life’ levels of damage.

Child services had the family agree to a safety plan, which meant Wallace and her husband could not be alone with their kids for even a second. Their mothers—the children’s grandmothers—had to visit and trade off overnight stays in order to guarantee the parents were constantly supervised. After two weeks, child services closed Wallace’s case, finding the complaint was unfounded.

…

Wallace’s sister has started a GoFundMe for her. She is in debt after losing her job and paying for the lawyer and the diversion program. She also hopes to hire a lawyer to get her record expunged so that she can work with kids again.

One of my closest friends here in New York is strongly considering moving to the middle of nowhere so that his child will be able to walk around outside, because it is not legally safe to do that anywhere there are people.

Claim that UNICEF thinks Dutch children are the happiest in the world and that this is because they bike everywhere, things are designed to make this work, and they therefore have freedom to move around.

Childless World

This is also related to the thing where a lot of places give off the vibe that life does not involve children, and children are a weird, shameful and irresponsible lifestyle choic...