trentmkelly/LessWrong-43k huggingface dataset:
[AN #152]: How we’ve overestimated few-shot learning capabilities
by Rohin Shah — 2021-06-16T17:20:04.454Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

Please note that while I work at DeepMind, this newsletter represents my personal views and not those of my employer.

HIGHLIGHTS

True Few-Shot Learning with Language Models (Ethan Perez et al) (summarized by Rohin): We can get GPT-3 (AN #102) to perform useful tasks using “prompt programming”, in which we design an input sentence such that the most likely continuation of that sentence would involve GPT-3 performing the task of interest. For example, to have GPT-3 answer questions well, we might say something like “The following is a transcript of a dialogue with a helpful, superintelligent, question-answering system:”, followed by a few example question-answer pairs, after which we ask our questions.

Since the prompts only contain a few examples, this would seem to be an example of strong few-shot learning, in which an AI system can learn how to do a task after seeing a small number of examples of that task. This paper contends that while GPT-3 is capable of such few-shot learning, the results reported in various papers exaggerate this ability. Specifically, while it is true that the prompt only contains a few examples, researchers often tune their choice of prompt by looking at how well it performs on a relatively large validation set -- which of course contains many examples of performing the task, something we wouldn’t expect to have in a true few-shot learning context.

To illustrate the point, the authors conduct several experiments where we start with around 12 possible prompts and must choose which to use based only on the examples given (typically 5). They test two methods for doing so:

1. Cross-validation: Given a prompt without examples, we attach 4 of the examples to the prompt and evaluate it on the last example, and average this over all possible ways of splitting up the examples.

2. Minimum description length: While cross-validation evaluates the final generalization loss on the last example after updating on previous examples, MDL samples an ordering of the examples and then evaluates the average generalization loss as you feed the...
Announcing the Nuclear Risk Forecasting Tournament
by MichaelA — 2021-06-16T16:16:41.402Z

Summary: From June 22, Rethink Priorities and Metaculus will run a Nuclear Risk Forecasting Tournament to help inform funding, policy, research, and career decisions aimed at reducing existential risks. The starting prize pool is $2,500. We would be excited for you to help via making forecasts, participating in the Discussion Forum, increasing forecaster rewards by becoming a monthly Supporter, and/or reaching out to relevant people to suggest they participate.

In 1941, no one had ever built a nuclear weapon, and most people had no idea that anyone ever might. Over the following 80 years, the Allies built such weapons, the US detonated two on cities, and scientists and engineers

developed

or

proposed

many new and vastly more destructive types of nuclear weapons. Global stockpiles

peaked at

70,000

warheads

(compared to

today’s ~13,000

), and the total yield of the US arsenal alone peaked at

over a million times

the yield of the bomb dropped on Hiroshima

.

In 2021,

nine states are believed to possess nuclear weapons

(Russia, the United States, China, France, the UK, Pakistan, India, Israel, and North Korea). In theory, they could launch them at any time, this could directly cause millions of deaths, and this could perhaps indirectly — via

nuclear winter

— cause billions of deaths and even

the permanent loss of our potential to reach a flourishing future

.

On the other hand, nuclear weapons have never again been used in war since 1945, warhead numbers and total yields have declined dramatically since their peaks, and if nuclear conflict does occur it might involve low numbers of weapons, low yields, or targets that result in relatively low fatality levels and no nuclear winter.

So how likely are each of the strikingly different possible futures of nuclear risks? And what can and should we do about this — which goals should we pursue, and how can we use our limited resources to best achieve them?

Rethink Priorities is investigating these questions in order to help improve funding, policy, research, and career decisions aimed at reducing

existential risks

. The scope and theory of change for this work is discussed in more detail

here

. These questions are complex, high-stakes, and time-sensitive, so we want to draw on the skills and perspectives of a community of forecasters.

To do so, we’re partnering with Metaculus to run the

Nuclear Risk Forecasting Tournament

, hosted wi...
AI-Based Code Generation Using GPT-J-6B
by Tomás B. — 2021-06-16T15:05:26.381Z
Link: https://minimaxir.com/2021/06/gpt-j-6b/

Above is a link to an interesting post about synthetic code generation with a transformer model trained on The Pile, which contains a large chuck of GitHub and StackOverflow. Due to CommonCrawl's deficiency in this area, the much smaller GPT-J-6B outperforms OpenAI’s largest publicly available GPT-3 models. The performance is impressive enough that one wonders how capable a 100+ billion parameter model trained on The Pile will be, let alone what an AlphaGo-level engineering effort towards the end of synthetic code generation would achieve.

As the The Pile was created to provide a dataset for 100 billion paramater+ models, we may not have to wait long. The examples in the post are clearly trivial, but I personally take this to be something of a fire alarm. I was not previously aware of how poorly-optimized GPT-3 was for code generation, and I have updated toward surprising gains in this area in the next few years.

I no longer consider agents with superhuman performance in competitive programming to be a ridiculous thing to pursue.

It is useful to remind myself of how shocked I would be to see such things in 2012. In 2012 I would have taken this as a sign that AGI was near.

Scenario-based planning postulates that one should predict symptoms emblematic of a given scenario and then robotically assume you are in said scenario once a sufficient number of these symptoms come to pass. I am unsure whether there is wisdom in this approach, but I find it a discomfiting line of thought.
Insufficient Values
by Jozdien — 2021-06-16T14:33:28.313Z

I’m

Jose

. I realized recently I wasn’t taking existential risk seriously enough, and in April, a year after I first applied, I started running a MIRIx group in my college. I’ll write summaries of the sessions that I thought were worth sharing. Most of the members are very new to FAI, so this will partly be an incentive to push upward and partly my own review process. Hopefully some of this will be helpful to others.

This one focuses on how aligning creator intent with the base objective of an AI might not be enough for outer alignment, starting with an overview of Coherent Extrapolated Volition and its flaws. This was created in collaboration with

Jacob Abraham

and

Abraham Francis

.

Coherent Extrapolated Volition

From the

wiki

,

In calculating CEV, an AI would predict what an idealized version of us would want, "if we knew more, thought faster, were more the people we wished we were, had grown up farther together". It would recursively iterate this prediction for humanity as a whole, and determine the desires which converge. This initial dynamic would be used to generate the AI's utility function.

In other words, CEV is an AI having not only a precise model of human values, but also the meta understanding of how to resolve contradictions and incompleteness in those values in a friendly way.

This line of research was considered obsolete by Eliezer, due to the problems it runs into - some of which make it appear like the proposal itself only shifts the pointers to the goals. In the time we spent discussing it, we ended up with a (most likely not comprehensive) list of the major flaws of CEV.

Criticism

Goodhart’s Law

. CEV doesn’t address the issue of wireheading, where an AI maximizes a proxy utility function instead of the intended utility function. For example, without a sufficiently descriptive embedded understanding of what it means to be “more the people we wished we were” or even “smarter”, the AI is incentivized to exploit that ambiguity to create the cheapest outcomes. To a sufficiently powerful AI, you’re giving it limited indirect write access to the values it will implement. The obvious solution is a formal framework for these properties, which might just be rephrasing the general alignment problem.
However, CEV can be viewed as a proposal of what to do with AGI after problems like Go

...
Reward Is Not Enough
by Steven Byrnes — 2021-06-16T13:52:33.745Z

Three case studies

1. Incentive landscapes that can’t feasibly be induced by a reward function

You’re a deity, tasked with designing a bird brain. You want the bird to get good at singing, as judged by a black-box hardcoded song-assessing algorithm that you already built into the brain last week. The bird chooses actions based in part on within-lifetime reinforcement learning

involving dopamine

. What reward signal do you use?

Well, we want to train the bird to sing the song correctly. So it’s easy: the bird practices singing, and it listens to its own song using the song-assessing black box, and it does RL using the rule:

The better the song sounds, the higher the reward.

Oh wait. The bird is also deciding how much time to spend practicing singing, versus foraging or whatever. And the worse it sings, the more important it is to practice! So you really want the rule:

The worse the song sounds, the more rewarding it is to practice singing.

Uh oh.

How do you resolve this conflict?

Maybe “Reward = Time derivative of how good the song sounds”? Nope, under this reward, if the bird is bad at singing, and improving very slowly, then practice would not feel very rewarding. But here, the optimal action is to continue spending lots of time practicing. (Singing well is really important.)
Maybe “Reward is connected to the abstract concept of ‘I want to be able to sing well’?” Sure—I mean, that is ultimately what evolution is going for, and that’s what it would look like for an adult human to “want to get out of debt” or whatever. But how do you implement that? "I want to be able to sing well" is an awfully complicated thought; I doubt most birds are even able to think it—and if they could, we still have to solve a vexing symbol-grounding problem if we want to build a genetic mechanism that points to that particular concept and flags it as desirable. No way. I think this is just one of those situations where “the exact thing you want” is not a feasible option for the within-lifetime RL reward signal, or else doesn’t produce the desired result. (Another example in this category is “Don’t die”.)
Maybe you went awry at the start, when you decided to choose actions using a within-lifetime RL algorithm? In other words, maybe “choosing actions based on anticipated future rewards, as learned through within-lifetime experience” is not a good idea? Well, if we throw out that idea, it would a

...
Open problem: how can we quantify player alignment in 2x2 normal-form games?
by TurnTrout — 2021-06-16T02:09:42.403Z

In my experience, constant-sum games are considered to provide "maximally unaligned" incentives, and common-payoff games are considered to provide "maximally aligned" incentives. How do we quantitatively interpolate between these two extremes? That is, given an arbitrary 2×2 payoff table representing a two-player normal-form game (like Prisoner's Dilemma), what extra information do we need in order to produce a real number quantifying agent alignment?

If this question is ill-posed, why is it ill-posed? And if it's not, we should probably understand how to quantify such a basic aspect of multi-agent interactions, if we want to reason about complicated multi-agent situations whose outcomes determine the value of humanity's future. (I started considering this question with Jacob Stavrianos over the last few months, while supervising his SERI project.)

Thoughts:

Assume the alignment function has range [0,1] or [−1,1].

Constant-sum games should have minimal alignment value, and common-payoff games should have maximal alignment value.

The function probably has to consider a strategy profile (since different parts of a normal-form game can have different incentives; see e.g. equilibrium selection).
The function should probably be a function of player A's alignment with player B; for example, in a prisoner's dilemma, player A might always cooperate and player B might always defect. Then it seems reasonable to consider whether A is aligned with B (in some sense), while B is not aligned with A (they pursue their own payoff without regard for A's payoff).

So the function need not be symmetric over players.

The function should be invariant to applying a separate positive affine transformation to each player's payoffs; it shouldn't matter whether you add 3 to player 1's payoffs, or multiply the payoffs by a half.
The function may or may not rely only on the players' orderings over outcome lotteries, ignoring the cardinal payoff values. I haven't thought much about this point, but it seems important. EDIT: I no longer think this point is important, but rather confused.

If I were interested in thinking about this more right now, I would:

Do some thought experiments to pin down the intuitive concept. Consider simple games where my "alignment" concept returns a clear verdict, and use these to derive functional constraints (like symmetry in players, or the

...
Unbundling Humans, or, Unbundling Human Creation.
by Sajith Pai — 2021-06-16T00:03:40.685Z

“There are two ways to make money in business: You can unbundle, or you can bundle.” – Jim Barksdale, cofounder of Netscape

As frameworks for identifying opportunities in startupland go, unbundling / (re)bundling is amongst the most seminal ones out there.

Here is an example of how it works.

Visualize a product that helps you read (for a fee) any magazine / newspaper story – effectively you have unbundled or decoupled the story from the wrapper. Now imagine, you had an option every saturday to get a printed / ebook of all of the stories you selected for leisurely reading. Voila, you have created a new magazine via (re)bundling. But it is different because you have control over the components. The atomic unit of control has shifted with this new bundle from the wrapper to the stories.

I suppose inherently that is what unbundling / bundling is all about – it enables the user to have better control over the components of the bundle, so as to decouple certain components from the wrapper, and / or allow reconstituting of all or different elements of the bundle to fashion a different product.

Let us use unbundling to think through the creation of an adult or human being.

Unbundling the creation of adults

Historically / traditionally, to become a functioning adult, the following needed to have happened / come together.

Marriage + Sex + Fertility + Childbirth + Parenthood + Childcare => creation of an adult.

Some of the above are conditions / status like fertility, e.g., men needed to have an adequate sperm count, or parenthood (parental status; being recognized as the parent). Others are actions like marriage, childbirth etc.

Marriage was a highly desirable but not necessary condition. So you have

Sex + Fertility + Childbirth + Parenthood + Childcare => creation of an adult.

This is essentially the out-of-wedlock child. Incidentally 70% of the births in Iceland (60% in Bulgaria, 40% in US) are out-of-wedlock births. In countries like India, percentages are lower given the paternalistic nature of society.

Sex, or having the father and mother indulge in coitus, in order to fertilize the egg is not essential. IVF or Invitro Fertilization allows the fertilization to happen outside the body. This typically happens when one of the couple is infertile.

So we could certainly have

Marriage + Childbirth + Parenthood + Childcare => creation of an adult.

T...
Escaping the Löbian Obstacle
by Morgan_Rogers — 2021-06-16T00:02:02.945Z

Earlier this year, when looking for an inroad to AI safety, I learned about the Löbian Obstacle, which is a problem encountered by 'purely logical' agents when trying to reason about and trust one another. In the original paper of Yudkowsky and Herreshoff [1], they show that a consequence of Löb's theorem is that an agent X can only "trust" the reasoning of an agent Y with a strictly weaker reasoning system than themselves, where "trust" here means 'formally prove that the conclusions of the other agent's reasoning will be true'. As stated, this looks like a major problem if X is a human trying to build an artificially intelligent system Y, but it's also a problem for any individual (embedded) agent trying to reason about their own future behaviour.

I'm not the first person to find this problem counterintuitive, and for good reason. In this article I'm going to explain why a formal (purely syntactic) logic system alone is a poor model of the reasoning of embedded agents, and show that by fixing this, we remove the foundation for the difficulties arising from Löb's theorem.

For the uninitiated, there is a handy survey of application of Löb's theorem in AI safety research by Patrick LaVictoire [6].

Pure syntax

First, I should explain the formal set-up for applying Löb's theorem to agents. We model an agent's reasoning with a formal language, or logic, which I'll call L. Here I shall make the further assumption that this logic fits (or can be squeezed into) a formal language of the kind logicians are familiar with: the logic consists of some formal symbols or variables A,B,C... along with some logical connectives, operators and quantifiers for combining variables into expressions, or formulas. The agent is also assumed to carry some inference rules for manipulating formulas. Altogether, this data constitutes the syntax of L (its symbolic content and the rules for manipulating those symbols).

Since we don't care precisely what the symbols in L refer to, we need go no further, and we can start worrying about how this formal system behaves.

To arrive at the Löbian Obstacle, we suppose that an agent A (who reasons with L) builds an agent B who shall also reason with the logic L. Suppose, specifically, that A is building B to help them achieve some goal G. In order for A to ensure B will be helpful, A programs B to produce a proof that B's actions will lead to t...
Can someone help me understand the arrow of time?
by Yair Halberstadt — 2021-06-15T17:26:29.113Z

If I understand correctly the psychological arrow of time tries to explain why we perceive time as passing by. It answers that in truth time does not pass - we exist at a single point in the timeline, but have the illusion of time passing by because we remember the past and not the future.

Firstly, is this approximately correct?

Secondly isn't memory itself a process that takes place over time? So how can the illusion occur if time isn't passing in which it can occur?

Thirdly if this were true, there'd be no point doing anything - time is never going to pass so you can't do anything anyway. So why does nobody seem to act like that's true? Does nobody actually believe the theory?

I have a feeling there's some major parts of the theory I'm missing, but what are they? Or is the theory less ambitious and only tries to explain why time passes in a particular direction, and not why it passes at all?
If instead of giving out dividends, public companies bought total market index funds, which companies would be the biggest?
by Mati_Roy — 2021-06-15T17:07:05.338Z

Feel free to make simplifying assumptions or give partial answers (ex.: just links for some of the raw data).

Raw data needed for this calculation:

dividends given by each public companies (or at least the biggest ones)

historical value of each public companies (or at least the biggest ones)

historical value of a total market index fund (failing that, top 3000, or top 500 companies)

Can be restricted to the US.
Vignettes Workshop (AI Impacts)
by Daniel Kokotajlo — 2021-06-15T12:05:38.516Z

AI Impacts is organizing an online gathering to write down how AI will go down! For more details, see this announcement, or read on.

Plan

1. Try to write plausible future histories of the world, focusing on AI-relevant features. (“Vignettes.”)
2. Read each others’ vignettes and critique the implausible bits: “Wouldn’t the US government do something at that point?” “You say twenty nations promise each other not to build agent AI–could you say more about why and how?”
3. Amend and repeat.
4. Better understand your own views about how the development of advanced AI may go down.
(5. Maybe add your vignette to our collection.)

This event will happen over two days, so you can come Friday if this counts as work for you, Saturday if it counts as play, and both if you are keen. RSVP to particular days is somewhat helpful; let us know in the comments.

Date, Time, Location

The event will happen on Friday the 25th of June and Saturday the 26th.

It’ll go from 10am (California time) until probably around 4pm both days.

It will take place online, in the LessWrong Walled Garden. Here are the links to attend:
Friday
Saturday

Facebook event

FAQ

> Do I need literary merit or creativity?
No.

> Do I need to have realistic views about the future?
No, the idea is to get down what you have and improve it.

> Do I need to write stories?
Nah, you can just critique them if you want.

> What will this actually look like?
We’ll meet up online, discuss the project and answer questions, and then spend chunks of time (online or offline) writing and/or critiquing vignettes, interspersed with chatting together.

> Have you done this before? Can I see examples?
Yes, on a small scale. See here for some resulting vignettes. We thought it was fun and interesting.

> Any advice on how to get started?

We have lots! We can give it in the comments here, or on the day itself; just ask. You may be interested in this random future generator.

This event is co-organized by Katja Grace and Daniel Kokotajlo. Thanks to everyone who participated in the trial Vignettes Day months ago. Thanks to John Salvatier for giving us the idea. This work is supported by Center on Long-Term Risk, my employer.