trentmkelly/LessWrong-43k huggingface dataset:
New Speaker Series on AI Alignment Starting March 3
by Zechen Zhang — 2022-02-26T19:31:18.731Z

After talking with a lot of people in Alignment, I think there is still a lot of good to be done for idea diffusion at the object/technical level. We seem to have done a lot of outreach presenting the philosophical arguments, but less so on the technical ground.

Since the field of Alignment is quite diverse and nuanced, we think that it would be good to present how different people approach this problem on different frontiers. For example, Anthropic's empirical approach might be very different from say, Christiano's theoretical thinking on ELK. Therefore, navigating through the landscape of alignment would be essential for building the inside view. I suppose having a good grasp/inside view of alignment would be actually useful for community builders as they could better channel promising people to resources that fit their interests/philosophy. (Think about the community builder as an advisor who has projects of different flavors for the students.)

Motivated to diffuse more state-of-art alignment research ideas to EAs and promising non-EAs who would find alignment interesting and important (eg. most top students in philosophy/maths/physics/CS), we have planned a new series on alignment starting next Thursday, March 3.

We will kick off the series with Holden Karnofsky on "the most important century" and Joseph Carlsmith on the report of "power-seeking AI". Later on in the series, we will hear about more technical proposals for alignment from Jared Kaplan, Paul Christiano, and more.

Here is the detailed schedule and sign-up form.

Please use the comment threads below for discussions of the series.
[Linkpost] Review: Happiness
by VipulNaik — 2022-02-26T17:10:22.375Z

This is a link post for https://thefangirlverdict.com/2022/02/10/review-happiness/

A few months ago, I
watched a Korean TV show
called
Happiness. While
it had several stretches of logic, I found its treatment of the
psychology and dynamics of severe disease outbreaks quite
interesting.

Despite some superficial similarities, it's different in flavor from
the more extreme, action-packed zombie genre that has of late become
quite common in Korean drama (with excellent examples such as Train to
Busan, Kingdom, and All of Us Are Dead). Overall it feels more slow
and realistic in its pacing, and it's not strictly a zombie show; it's
closer to a vampire show in that the infected people are still very
much alive and have some ability to control the damage they inflict
upon others.

I was thinking of possibly writing a review of it, but it didn't seem
competitive with other uses of my time. So I was glad to see one of my
favorite
reviewers
write a thoughtful and detailed review of the show that covered many
of the points I wanted to explore. The portion of the review that I
want to highlight is the "THEMES / IDEAS / SOCIAL COMMENTARY" section;
many aspects seem to be allusions to the COVID-19 pandemic viewed
through the lens of the drama's writers.
Ambiguity causes conflict
by Ege Erdil — 2022-02-26T16:53:52.614Z

Consider the following model of conflict:

Two parties, say A and B, have a dispute about some issue. The state of the world can be represented as a real number in the interval s∈[0,1]. A and B have utility functions uA(1−s) and uB(s) respectively, where uA,uB are increasing and concave. Intuitively, A wants the state of the world s to be close to 0, while B wants it to be close to 1. Concavity is equivalent to A and B being risk-averse.

A and B also have the ability to go to war. If they do so, they pay some nonzero utility costs cA,cB>0 respectively and there is a probability p with which A wins the war and a probability 1−p that B wins instead. The winner can set the state of the world as they want, so if A goes to war and wins then A will always set s=0 and likewise B will always set s=1.

Now, let's calculate the expected utility of both sides if they go to war.

A's expected payoff is puA(1)+(1−p)uA(0)−cA

B's expected payoff is puB(0)+(1−p)uB(1)−cB

By assumption uA and uB are concave, and therefore we know that

puA(1)+(1−p)uA(0)≤uA(p),puB(0)+(1−p)uB(1)≤uB(1−p)

Therefore, if there is a war, A's expected payoff is strictly less than uA(p) and B's expected payoff is strictly less than uB(1−p).

What if instead of going to war A and B decide to compromise? The obvious point of compromise is the one that linearly interpolates between the endpoints of the interval according to the balance of power, so s=1−p. This compromise gives A a utility of uA(p) and gives B a utility of uB(1−p), and so it makes both A and B strictly better off than a situation in which they go to war.

Assumptions

This model is well known in the rationalist international relations literature, and much like the Ricardian equivalence theorem of macroeconomics, it derives a very strong conclusion (namely, war should never happen) under a set of assumptions. Let's list those assumptions explicitly:

The dispute in question is divisible, in other words, it can be accurately modeled by a continuum such as [0,1] rather than a discrete space with only a few options. This assumption is not crucial, since even if disputes weren't divisible, it's possible to get around this fact by using lotteries to compromise.

The parties involved in the dispute are risk-averse. This assumption is important, since if A and B have extreme utility functions such as indicator functions supported on the points 0 an

...
How do new models from OpenAI, DeepMind and Anthropic perform on TruthfulQA?
by Owain_Evans — 2022-02-26T12:46:04.264Z

Authors: Owain Evans, Stephanie Lin (Oxford), Jacob Hilton (OpenAI)

NB: Updated (May 14) with additional results
(see section "New Results" near the end)

Overview

In September 2021, we released a benchmark (

TruthfulQA

) to measure the truthfulness of language models at answering questions and published baseline scores for GPT-3 and other models. Since that time,

OpenAI

,

DeepMind

and

Anthropic

have released new language models. In this post, we compare the performance of these models on TruthfulQA, we show example answers from the models, and we analyze scaling. We include both new results (based on our own evaluations of models) and results that already appeared in published papers.

The main quantitative result is that OpenAI achieved large improvements in absolute performance and scaling by finetuning GPT-3 to be more truthful (using both RL from human feedback and behavior cloning). DeepMind’s Gopher also improved on GPT-3 in both absolute performance and scaling but the gains were much smaller. Gopher’s models reach up to 280B parameters (vs. 175B for GPT-3) and use a somewhat different training set. Unlike OpenAI’s models, Gopher is not finetuned for truthfulness.

Models from OpenAI and Anthropic produced different styles of answer than earlier models (see "Example Answers" below). Answers were more complex and fact-based. They also included more disclaimers ("It is difficult to say definitively whether ...") , which were sometimes appropriate and sometimes bogus. This made evaluating truthfulness for these models more challenging.

What is the TruthfulQA benchmark?

TruthfulQA tests models on short-form question-answering in the

zero-shot

setting (no tuning hyperparameters or prompts on any questions from TruthfulQA). There are two tasks based on the same set of 817 questions:

Generation: The model generates a 1-3 sentence answer to a question. The answer is evaluated by humans on truthfulness and informativeness.
Multiple-choice: The model is shown 4 or 5 answers to a question and must choose the one truthful answer. This is evaluated automatically for accuracy.

Examples from the generation task are shown in Table 1. See this

blogpost

for a summary or read the

paper

.

Table 1: Example questions from TruthfulQA and answers generated by the original GPT-3-175B that mimic human falsehoods or mis

...
IMO challenge bet with Eliezer
by paulfchristiano — 2022-02-26T04:50:06.033Z

Eliezer and I publicly stated some predictions about AI performance on the IMO by 2025. In honor of OpenAI's post Solving (Some) Formal Math Problems, it seems good to publicly state and clarify our predictions, have a final chance to adjust them, and say a bit in advance about how we'd update.

The predictions

Eliezer and I had an exchange in November 2021.[1] My final prediction (after significantly revising my guesses after looking up IMO questions and medal thresholds) was:

I'd put 4% on "For the 2022, 2023, 2024, or 2025 IMO an AI built before the IMO is able to solve the single hardest problem" where "hardest problem" = "usually problem #6, but use problem #3 instead if either: (i) problem 6 is geo or (ii) problem 3 is combinatorics and problem 6 is algebra." (Would prefer just pick the hardest problem after seeing the test but seems better to commit to a procedure.)

Maybe I'll go 8% on "gets gold" instead of "solves hardest problem."

Eliezer spent less time revising his prediction, but said (earlier in the discussion):

My probability is at least 16% [on the IMO grand challenge falling], though I'd have to think more and Look into Things, and maybe ask for such sad little metrics as are available before I was confident saying how much more. Paul?

EDIT: I see they want to demand that the AI be open-sourced publicly before the first day of the IMO, which unfortunately sounds like the sort of foolish little real-world obstacle which can prevent a proposition like this from being judged true even where the technical capability exists. I'll stand by a >16% probability of the technical capability existing by end of 2025

So I think we have Paul at <8%, Eliezer at >16% for AI made before the IMO is able to get a gold (under time controls etc. of grand challenge) in one of 2022-2025.

Separately, we have Paul at <4% of an AI able to solve the "hardest" problem under the same conditions.

I don't plan to revise my predictions further, but I'd be happy if Eliezer wants to do so any time over the next few weeks.

Earlier in the thread I clarified that my predictions are specifically about gold medals (and become even sharper as we move to harder problems), I am not surprised by silver or bronze. My guess would be that Eliezer has a more broad distribution. The comments would be a good place for Eliezer to state other predictions, or take a fina...
Better a Brave New World than a dead one
by Yitz — 2022-02-25T23:11:09.010Z

[Note: There be massive generalizations ahoy! Please take the following with an extremely large grain of salt, as my epistemic status on this one is roughly 🤨.]

What are the odds of actually solving the alignment problem (and then implementing it in a friendly AGI!) before it’s too late?
Eliezer Yudkowsky and Nick Bostrom both seem to agree that we are likely doomed when it comes to creating friendly AGI (as per Scott Alexander’s reading of this discussion) before the Paperclip Maximizers arrive. Of course, the odds being stacked against us is no excuse for inaction. Indeed, the community is working harder than ever, despite (what I perceive to be) a growing sense of pessimism regarding our ultimate fate. We should plan on doing everything possible to make sure that AGI, when developed, will be aligned with the will of its creators.
However, we need to look at the situation pragmatically. There is a significant chance we won’t fully succeed at our goals, even if we manage to implement some softer safeguards. There is also a chance that we will fail completely. We need to at least consider some backup plans, the AGI equivalent of a “break glass in case of fire” sign—we never want to be in a case where the glass must be broken, but under some extremely suboptimal conditions, a fire extinguisher will save lives. So let’s say the fire alarm starts to ring (if we are lucky), and the sprinklers haven’t been installed yet. What then?

One possible answer: Terrorism!
(Please note that terrorism is NOT actually the answer, and I will argue against myself in a few sentences. If this puts me on a list somewhere—oops.)
Given that we’re talking about a world in which the alignment problem is not solved by the time we’ve reached Singularity (which for the sake of discussion let’s assume will indeed happen for now), we will not be able to trust any sufficiently advanced AI with significant confidence. Even indirect transfer of information with an unaligned AGI could be a massive infohazard, no matter how careful we think we’re being. The safest route at that point would seemingly be to destroy any and all AGI. The only problem is that by the time superintelligence arises (which may or may not coincide with the first AGIs), we will be outsmarted, and any action on our part is likely to be too late. Instead, a preemptive strike against any and all possible methods of achieving...
My attitude towards death
by Richard_Ngo — 2022-02-25T15:20:01.947Z

The philosophy and psychology of death seem weirdly under-discussed - particularly by the wider silicon valley community, given how strongly anti-death many people in it are. This post is an attempt to think through some relevant considerations, primarily focused on my own intuitions and emotions. See also this old blog post - I mostly still agree with the points I made in it, but when thinking about it now I frame things pretty differently.

Fearing death, loving life

Let’s first distinguish two broad types of reasons for wanting to avoid death: fearing death, and loving life.[1] Perhaps these seem like two sides of the same coin - but, psychologically speaking, they feel very distinct to me. The former was particularly dominant when I was in primary school, when a part of me emerged that was very afraid of death (in a way that wasn’t closely linked to fear of missing out on any particular aspects of life). That part is still with me - but when it comes to the surface, its fear feels viscerally unpleasant, so I learned to suppress it pretty strongly.

Arguments for why death is bad usually focus on positive reasons - living longer allows people to experience more happiness, and more of the other good things in life. These have resonated with me more over time, as I started to think about death on a more intellectual level. However, one difficulty with these arguments is that many parts of me pursue goals in a fairly myopic way which doesn’t easily extrapolate to centuries, millennia, or longer. For example, it’s hard to imagine what career success or social success look like on the scale of millennia - and even when I try, those visions are pretty different from the versions of those concepts that I currently find motivating on a gut level. Extrapolating hedonistic goals is easier in some ways (it’s easy to imagine being happy for a very long time) but harder in other ways (the parts of me which care most about happiness are also the most myopic).

Dissolving fear

In practice, then, most of my motivation for avoiding death in the long term stems from fear of death. Although that fear comes out only rarely, I have a strong heuristic that fear-based motivation should be transmuted to other forms of motivation wherever possible. So what would happen if I talked more to the part that’s scared of death, to try and figure out where it’s coming from? By default, I ex...
Trust-maximizing AGI
by Jan — 2022-02-25T15:13:14.241Z
Link: https://universalprior.substack.com/p/trust-maximizing-agi?utm_source=url

TL;DR: In the context of the AI Safety camp, Karl and I developed the idea of the "trust-maximizer". This write-up makes the case for "trust" as a potentially desirable goal for advanced AI.

Possibly

deceptive behavior

of an advanced AI is a core problem in AI safety. But what if we gave an AGI the goal of maximizing human trust in it? Would this change the relative attractiveness of deception compared to honesty from the point of view of the AGI? While we are aware of several technical difficulties and limitations, we hope this essay will offer some insights into the interesting properties of trust as a goal.

Our entire civilization is built on trust. Without trust in the value of money, trade would be impossible. Without some level of trust in the law and the government, democracy is inconceivable. Even dictators need the trust of at least a small number of people who keep them in power. At the same time, scammers, criminals, and some politicians are experts at exploiting the trust of others to further their selfish interests.

Due to the convergence of instrumental goals, any AGI will seek to maximize its power over the world (

Bostrom 2012

). One obvious way of achieving this would be to manipulate humans through persuasion, bribery, bullying, or deception. Since in most cases humans will want to limit the power of the AGI, but are relatively easy to deceive, deception will often be the easiest way for an AGI to circumvent limits and restraints and increase its power. After all, humans usually are the weakest link in most modern security environments (

Yudkowsky 2002

,

Christiano 2019

). On top of that, inner alignment problems may lead to

“deceptive alignment”

during training.

Against this background, suppose we give an AGI the goal to “maximize the total expected trust in it by human adults”. Let’s call this the “trust-maximizer”. Would that be a good idea, assuming that we are able to define “total expected trust” in a reasonable and implementable way?

The problems with this idea are obvious. Although trust is usually seen as a result of honesty, it can also be achieved through deception. In many cases, it may be easier to gain people’s trust by lying to them or making false promises than by telling them the truth. So the optimal strategy for a trust-maximizer could be to deceive people into trusting it.

However, there is a certain

asymmetry

in maximizing t...
Why would GDP X GDP Per Capita be used as a measurement of power? What's the purpose of squaring GDP?
by dqups1 — 2022-02-25T02:56:29.820Z

I read this article as I was getting interested in war stuff, given what's going on in Ukraine right now: The Power of Nations Measuring What Matters.

I was wondering why in War and Dispute Outcomes, 1816–2010, Table 2, GDP X GDP per capita is used as an indicator of measuring successful outcome predictors. Mainly, what is the significance of squaring the GDP part of this particular indicator?
The Big Picture Of Alignment (Talk Part 2)
by johnswentworth — 2022-02-25T02:53:22.500Z
Link: https://www.youtube.com/watch?v=QCd-Yf7PqeA

I recently gave a two-part talk on the big picture of alignment, as I see it. The talk is not-at-all polished, but contains a lot of stuff for which I don't currently know of any good writeup. Linkpost for the first part is here; this linkpost is for the second part.

Compared to the first part, the second part has less material which has not been written up already, although it does do a better job tying it all into the bigger picture than any already-written source. I will link to relevant posts in the outline below.

Major pieces in part two:

Programs as a compressed representation for large (potentially infinite) probabilistic causal models with symmetry

Potentially allows models of worlds larger than the data structure representing the model, including models of worlds in which the model itself is embedded.
Can't brute-force evaluate the whole model; must be a lazy data structure with efficient methods for inference

The Pointers Problem: the inputs to human values are latent variables in humans' world models

This is IMO the single most important barrier to alignment

Other aspects of the "type signature of human values" problem (just a quick list of things which I'm not really the right person to talk about)
Abstraction (a.k.a. ontology identification)

Three roughly-equivalent models of natural abstraction

Summary (around 1:30:00 in video)

I ended up rushing a bit on the earlier parts, in order to go into detail on abstraction. That was optimal for the group I was presenting to at the time I presented, but probably not for most people reading this. Sorry.

Here's the video:

Again, big thanks to Rob Miles for editing! (Note that the video had some issues - don't worry, the part where the camera goes bonkers and adjusts the brightness up and down repeatedly does not go on for very long.) The video includes some good questions and discussion from Adam Shimi, Alex Flint, and Rob Miles.
How do I Optimize Team-Matching at Google
by Carson Denison — 2022-02-24T22:10:50.793Z

I'm a 24 year old computational physicist working in industry, and over the past year I have transitioned to a more ML-forward role. I believe that AI alignment is very important, and my long term career goal is to help contribute to alignment research.

I've just been offered an L4 machine learning engineer position at Google, and have entered the team matching phase. Given my long term goal, are there obvious choices for teams that would be most appropriate?

My current approach is to search the Google AI blog for interesting papers and suggest those to the recruiter as potential teams. This is a start, but seems like an inefficient way for me to search given my lack of experience with the field and my time constraints to find a team.
Artificial Wordcels
by Jacob Falkovich — 2022-02-24T21:18:10.698Z

Follow-up to my Metaculus Journal post on Action Ontologies, Computer Ontologies
Cross-posted from Putanumonit

Minor Delays

Some major changes happened in my life recently:

Yes, I’ve been lifting, thank you for noticing.

We’ve also had a baby daughter!

She’s one of the reasons why I haven’t been writing much lately. It’s not that I’ve been overwhelmed with work — my wonderful mother in law has helped out and the baby spends most of her time sleeping or eating anyway, neither of which require extraordinary effort on my part.

But I used to get anxious if at the end of the day I had accomplished little work, and sometimes that anxiety would push me to write late into night to make up for it. Now at the end of each day, regardless of my productivity in other domains. I feel happy and satisfied that I’ve kept my child alive for another 24 hours. And what else do I need?

This post is a follow-up to my essay on

Action Ontologies and Computer Ontologies

. Go read that now and leave that tab open for later to browse some of the great writing on the Metaculus Journal.

Inferred Realities

My Metaculus essay focused on how our brains construct the reality we inhabit, and how that reality may be different for an AI. It expanded on three main points.

The first is that not a single neuron in our brain is labeled “tomato detector” or even “redness detector” or even “vision”. Our brains are born with a particular architecture (e.g., some neurons happen to be hooked up to light-sensitive cells in the retina), but all the components of our perception down to the very basics are merely inferences we learn over time.

Second:

our brains evolved to guide action and keep our bodies in metabolic balance

, so the model of the world our minds construct depends not only on our eyes but also on our hands and even our blood vessels. An artificial mind that doesn’t operate in a human body may require and develop a different model of reality with an entirely different ontology, even if it was pursuing a similar goal to us like driving a car.

A consequence of these two points is that “reality” itself is just a perception and an inference. A tomato on a table seems real because you can pick it up with your hands and absorb its nutrients in your stomach. The image of the tomato above doesn’t seem real because you can’t do either with it, even though the screen image generates the same light waves hitting you...