trentmkelly/LessWrong-43k huggingface dataset:
Better impossibility result for unbounded utilities
by paulfchristiano — 2022-02-09T06:10:15.525Z

This post slightly improves on the impossibility results in my last post and A paradox for tiny probabilities and enormous values. This is very similar to those arguments but with minor technical differences. I may refer people here if we're discussing details of the argument against unbounded utilities, but for more interesting discussion you should check out those other writeups.

Summarizing the key differences from my last post:

I replace the dominance principle ∀i:Ai>Bi⇒∑piAi>∑piBi with ∀i:Ai≥Bi⇒∑piAi≥∑piBi.
Assuming unbounded positive and negative utilities, I remove the assumption of Intermediate Mixtures.

Weakening dominance

We'll represent preferences by a relation ≤ over probability distributions over some implicit space of outcomes Ω (and we'll identify outcomes with the constant probability distribution). Define A<B to mean (A≤B and not B≤A).

We'll show that it's impossible for ≤ to satisfy four properties: unbounded utilities, intermediate mixtures, very weak dominance, and transitivity.

The properties

Unbounded utilities: there is an infinite sequence of outcomes X1,X2,X4,X8,… each "more than twice as good"[1] as the last. More formally, there exists an outcome X0 such that:

X0<X1
X2k<12X2k+1+12X0 for every k

That is: X1 is not as good as a 12 chance of X2, which is not as good as a 14 chance of X4, which is not as good as a 18 chance of X8...

Intermediate mixtures: If A<B and 0<p<1,[2] then A<pA+(1−p)B<B.

That is: a chance of a good outcome is strictly worse than a sure thing.[3]

Weak lottery-lottery dominance: Let A0,A1,A2,…, and B0,B1,B2,… be sequences of lotteries, and p0,p1,p2,… be a sequence of probabilities summing to 1. If Ai≥Bi for all i, then ∑piAi≥∑piBi.

Transitivity: if A≥B and B≥C, then A≥C.[4]

Inconsistency proof

Consider the lottery X∞:=13X0+29X1+427X2+881X4+…, where each outcome X2k is 23 as likely as the one before. Intuitively this sum is "more infinite" than the usual St Petersburg lottery and this will be important for our proof.

By weak dominance we can replace X1 by X0 witho...
Interest in a digital LW "book" club?
by DirectedEvolution — 2022-02-09T05:59:19.878Z

Prompted by interest in LW on EA forum, I thought people might be interested in a digital book club. We'd pick a couple classic posts every 2-4 weeks. One option is a group zoom meeting or slack, but we could also just invite people to read and comment on the selected articles. I'd be happy to come up with some guiding questions and pick articles if a handful of people are interested.
An extension of Aumann's approach for reducing game theory to bayesian decision theory to include EDT and UDT like agents
by Karl Brisebois — 2022-02-09T04:17:40.461Z

Aumann in Correlated Equilibrium as an Expression of Bayesian Rationality developed a formalism to reduce nashian game theory to bayesian decision making in a multi-agent setting and proved within that formalism that, under the conditions of common knowledge of "rationality" (read: common knowledge that every agent is running CDT) and that every agent knows his own action, the result will be a correlated equilibrium. As it turns out it is relatively straightforward to extend this framework to EDT and UDT like agents, which will be the goal of the rest of this post.

We will be using the same notation as Aumann: Ω will stand for the space of all possible worlds (which we'll assume finite), p is a probability distribution on Ω, Si is the set of all possible actions for player i, si is a function from Ω to si returning which action player i chooses in a given world, Pi is a partition of Ω representing the possible information states for player i and finally hi is a function from ∏iSi to R representing the utility function of player i.

First we can ask what happens if we relax the condition that the players knows their own action, allowing for players acting in non-deterministic ways. Without any other conditions this would allow the actions of players to be correlated in arbitrary ways without any source of information on their respective strategies. For example, suppose we have two agents, 1 and 2, playing rock paper scissor with both information partitions being trivial, we could have Ω={1,2,3} with p(1)=p(2)=p(3)=1/3 and s1(1)=Rock, s2(1)=Paper, s1(2)=Paper, s2(2)=Scissor, s1(3)=Scissor, s2(3)=Rock. Then both agents verify the CDT conditions but player 2 seems to always be able to predict the action of player 1 and playing the counter to their action, despite their information partition being trivial. To avoid that situation we can require the following condition, which we shall call the strategic independence condition:

SI Given Pi∈Pi for all player i then the probability distributions on strategies for all players are independent when conditioned on ⋂Pi

It might not necessarily always be correct to demand that condition, think for example of a case where we have two identical agents playing a s...
Everyone’s mired in the deepest confusion, some of the time?
by M. Y. Zuo — 2022-02-09T02:53:58.551Z

You can fool all the people some of the time and some of the people all the time, but you cannot fool all the people all the time.

Unknown, popularly attributed to Abraham Lincoln

One of the most famous supposed quotes of Abraham Lincoln, got me thinking. What if we interpreted those words in the literal sense? What would that imply?

Especially the part of the supposed possibility of a state of ‘omni-deception’, “You can fool all of the people some of the time”, where the entire world is confused…

That even the greatest scoundrels, when devising the most devious and sophisticated plans, manage to fool themselves in the end as well? (at least some of the time)

If literally everyone were to be fooled simultaneously then by definition it would be impossible to tell until after the fact, at the very least. In the state of deepest confusion even the nagging feeling of confusion disappears, at least in everyday life, such that one does not even realize they are confused.

Could that truly somehow be operant simultaneously upon all and one?

How would those periods of omni-deception appear? And what would they be about?

How could we distinguish them from other periods in retrospect?

If your thinking these sound like mysterious questions then I share the sentiment, perhaps they will inevitably lead to even more mysterious answers…

Crossposted from my blog: https://mzuo.ca/2022/02/08/everyones-mired-in-the-deepest-confusion-some-of-the-time/
"Moral progress" vs. the simple passage of time
by HoldenKarnofsky — 2022-02-09T02:50:19.144Z

In Future-Proof Ethics, I talked about trying to "consistently [make] ethical decisions that look better, with hindsight after a great deal of moral progress, than what our peer-trained intuitions tell us to do."

I cited Kwame Anthony Appiah's comment that "common-sense" ethics has endorsed horrible things in the past (such as slavery and banning homosexuality), and his question of whether we, today, can do better by the standards of the future.

A common objection to this piece was along the lines of:

Who cares how future generations look back on me? They'll have lots of views that are different from mine, just as I have lots of views that are different from what was common in the past. They'll judge me harshly, just as I judge people in the past harshly. But none of this is about moral progress - it's just about random changes.

Sure, today we're glad that homosexuality is more accepted, and we think of that as progress. But that's just circular - it's judging the past by the standards of today, and concluding that today is better.

Interestingly, I think there were two versions of this objection: what I'd call the "moral realist" version and the "moral super-anti-realist" version.

The moral realist thinks that there are objective moral truths. Their attitude is: "I don't care what future people think of my morality (or what I think after more reflection?1) - I just care what's objectively right."

The moral super-anti-realist thinks that morality is strictly subjective, and that there's just nothing interesting to say about how to "improve" morality. Their attitude is: "I don't care what future people think of my morality, I just care what's moral by the arbitrary standards of the time I live in."

In contrast to these positions, I would label myself as a "moral quasi-realist": I don't think morality is objective, but I still care greatly about what a future Holden - one who has reflected more, learned more, etc. - would think about the ethical choices I'm making today. (Similarly, I believe that taste in art is subjective, but I also believe there are meaningful ways of talking about "great art" and "highbrow vs. lowbrow taste," and I personally have a mild interest in cultivating more highbrow taste for myself.)

Talking about "moral progress" is intended to encompass both the "moral quasi-realist" and the "moral realist" positions, while ignor...
Hypothesis: gradient descent prefers general circuits
by Quintin Pope — 2022-02-08T21:12:37.959Z

Summary: I discuss a potential mechanistic explanation for why SGD might prefer general circuits for generating model outputs. I use this preference to explain how models can learn to generalize even after overfitting to near zero training error (i.e., grokking). I also discuss other perspectives on grokking and deep learning generalization.

Additionally, I discuss potential experiments to confirm or reject my hypothesis. I suggest that a tendency to unify many shallow patterns into fewer general patterns is a core feature of effective learning systems, potentially including humans and future AI, and briefly address implications to AI alignment.

Epistemic status: I think the hypothesis I present makes a lot of sense and is probably true, but I haven't confirmed things experimentally. Much of my motive for post this is to clarify my own thinking and get feedback on the best ways to experimentally validate this perspective on ML generalization.

Context about circuits: This post assumes the reader is familiar with and accepts the circuits perspective on deep learning. See here for a discussion of circuits for CNN vision models and here for a discussion of circuits for transformer NLP models.

Evidence from grokking

The paper "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets" uses stochastic gradient descent (SGD) to train self attention based deep learning models on different modular arithmetic expressions (e.g., f(x,y)=x×y (mod p), where p is fixed).

The training data only contain subsets of the function's possible input/output pairs. Initially, the models overfit to their training data and are unable to generalize to the validation input/output pairs. In fact, the models quickly reach near perfect accuracy on their training data. However, training the model for significantly past the point of overfitting causes the model to generalize to the validation data, what the authors call "grokking".

See figure 1a from the paper:

Figure 1a from Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

Note the model has near perfect accuracy on the training data. Thanks to a recent replication of this work, we can also look at the loss curves during grokking (though on a different experiment compared to the plot above):

Figure from this GitHub repository

First, the model reaches near-zero loss in training but...
Paradigm-building from first principles: Effective altruism, AGI, and alignment
by Cameron Berg — 2022-02-08T16:12:26.423Z

Introduction

If we are interested in the theoretical foundations that motivate work in AGI safety research, it seems worthwhile to clarify the philosophical and practical relationships of big-picture ideas in effective altruism (EA) to AGI in general and to AGI safety research in particular. One straightforward reason that this exercise is worthwhile is that much of current AGI safety work is funded through EA-aligned entities like Open Philanthropy, the Long-Term Future Fund, and the Future of Life Institute. If effective altruism is a large part of the reason that AGI safety research exists, then it is definitely worth clearly spelling out how one gets from the former to the latter.

Accordingly, I will attempt to do three things in this post: (1) I will examine the philosophical relationship of EA to the prospect of AGI in general, (2) I will compare this account to what thought-leaders in EA generally say about AGI safety in practice, and (3) I will discuss how the notion of alignment fits into this whole picture.

‘First-principles’ EA and AGI: a survey of the landscape

The holistic goal of effective altruism (if we had to enumerate just one) seems to be something like bringing about the best outcomes for as long as possible. In other words, EA seems fundamentally interested in finding tractable solution(s) to a kind of dual optimization problem: find ways to (#1) make things as good as possible (#2) for the longest possible duration. EA wants the party to be really fun and for it to never end.

Some quick replies to quick objections:

Some advocate—either in addition to or instead of problem #1—that we minimize bad outcomes. As I will discuss later, I think this point definitely relevant—but it is not essential for actually formulating the problem given that we can trivially define a bad outcome as the absence of a good outcome (which we already want to maximize).
Some would argue that problem #2 is logically just a subset of problem #1 (i.e., good outcomes are not maximized if they are not maximized over a long duration). I think this is technically right. I am disentangling the problems of maximizing good outcomes and maximizing duration not because I think they are logically distinct, but rather because I think their practical implications substantially differ. By analogy, problem #2 is more about building out the road in front of us while p

...
Metaculus launches contest for essays with quantitative predictions about AI
by Tamay Besiroglu — 2022-02-08T16:07:25.409Z
Link: https://www.metaculus.com/project/ai-fortified-essay-contest/

Metaculus has launched the AI Progress Essay Contest which aims to attract systematic thinkers to engage with the wealth of AI Forecasts that Metaculus has built up over the years. Of particular interest is rigorous and nuanced thinking about the likelihood, timing, and impacts of transformative AI that engages with quantitative probabilistic forecasts. The contest will run until April 16th, 2022, and have a $6,500 prize pool.

Many of the highlighted questions are likely to be of interest to the LW community:

Will the scaling up of compute be sufficient to get human-level performance on a wide-range of important domains, such as language modelling and computer vision? If so, when can we expect training runs to be sufficiently compute intensive to hit important milestones? If not, why not? Justify your views with, among other things, reference to quantitative predictions.
How much progress in AI algorithms and architectures has there been in the past decade, and how much should we expect in the next decade? Support your views with references to quantitative predictions, such as those involving performance on some common ML benchmarks.
Can we map the disagreements between notable figures in the AI risk community, as presented in the

Late 2021 MIRI Conversations

, onto disagreements about quantifiable predictions? Which views imply predictions that we think are likely to fare better and why?
When will AI-enabled automation contribute more than 5%/25%/50%/95% to US GDP? Explain why this may or may not happen this century.
Is deep learning sufficient for achieving Transformative AI? (This is defined, roughly, as AI that precipitates a transition comparable to the agricultural or industrial revolution.) If not, how many new substantial insights might be needed?
How might the trajectory of the field of AI look if Moore’s Law were seriously stunted in the next decade? How might the rate of progress be affected, and where would the field look for new sources of improvements?
What fraction of the field of AI will be dedicated to working on AI Safety, broadly defined? Will the field have substantial influence on how AI systems are designed, tested, and/or deployed? Provide evidence supporting your perspective, as well as references to quantitative predictions.

Further details may be found on the contest landing page.
Appendix: More Is Different In Other Domains
by jsteinhardt — 2022-02-08T16:00:17.848Z

In Future ML Systems Will Be Qualitatively Different, I argued that we should expect ML systems to exhibit emergent capabilities. My main support for this was four historical examples of emergence in ML.

In some sense, extrapolating from only four data points is pretty sketchy. It's hard to tell if these examples are the start of a trend, or cherry-picked to tell a story. In low-data regimes like this, it's helpful to have a prior, and so I'll spend this appendix looking at related fields of science and examining how common emergent behavior is in those fields.

In short, my conclusion is that emergence is very common throughout the sciences, and especially common in biology, which I view to be most analogous to ML. As such, I think it should be our default expectation for ML, and the four data points from before primarily serve to confirm this. Personally, forming this "prior" updated my views about as much as the (combined) historical examples that I previously discussed for ML.

More Is Different Across Domains

Recall that emergence refers to the idea that More Is Different: that quantitative changes can lead to qualitatively different phenomena. While this idea was first articulated by the physicist Philip Anderson, it occurs in many other domains as well:[1]

Biology: Gecko feet are covered with tiny structures called spatulae (shown above), consisting of many keratin molecules woven together. Spatulae are responsible for geckos’ ability to walk up walls, but it’s clear that we couldn’t "scale down" the structure and retain the same ability---you need enough keratin to form the complex bristles shown above.

Physics: Siloles (a type of molecule) are not luminescent individually, but become luminescent when put together in aggregate. This is because the aggregate is more rigid, reducing intramolecular motion that otherwise inhibits the production of photons.

Computers: Operating systems only arose once computers became fast enough. Before that, individual programs took long enough that task management could be done by hand. Greater speed both produced the demand for automation and the capacity to support an operating system’s overhead.

Economics: Increased population size enables increased specialization. Hunter-gatherer bands could not support smiths and architects. In the modern world, the emergence of Ford and other mass producers required a large enoug

...
Defending Functional Decision Theory
by Heighn — 2022-02-08T14:58:55.530Z

As I have been studying Functional Decision Theory (FDT) a lot recently, I have come across quite some counterarguments and general remarks that are worth rebutting and/or discussing in more detail. This post is an attempt to do just that. Most points have been discussed in other posts, but as my understanding of FDT has grown, I decided to write this new post. For readers unfamiliar with FDT, I recommend reading Functional Decision Theory: A New Theory of Instrumental Rationality.

The Bomb Argument

Originally proposed by William MacAskill:

You face two open boxes, Left and Right, and you must take one of them. In the Left box, there is a live bomb; taking this box will set off the bomb, setting you ablaze, and you certainly will burn slowly to death. The Right box is empty, but you have to pay $100 in order to be able to take it.

A long-dead predictor predicted whether you would choose Left or Right, by running a simulation of you and seeing what that simulation did. If the predictor predicted that you would choose Right, then she put a bomb in Left. If the predictor predicted that you would choose Left, then she did not put a bomb in Left, and the box is empty.

The predictor has a failure rate of only 1 in a trillion trillion. Helpfully, she left a note, explaining that she predicted that you would take Right, and therefore she put the bomb in Left.

You are the only person left in the universe. You have a happy life, but you know that you will never meet another agent again, nor face another situation where any of your actions will have been predicted by another agent. What box should you choose?

The argument against FDT, then, is that it recommends Left-boxing, which supposedly is wrong because it makes you slowly burn to death while you could have just paid $100 instead.

Analysis and Rebuttal

On Bomb, FDT indeed recommends Left-boxing. As the predictor seems to have a model of your decision procedure which she uses to make her prediction, FDT reasons that whatever you decide now, the predictor's model of you also decided. If you Left-box, so did the model; if you Right-box, so did the model. If the model Left-boxed, then the predictor would have predicted you Left-box, and, crucially, not put a bomb in Left. If the model instead Right-boxed, there would be a bomb in Left. Reasoning this way, Left-boxing gives you a situation with no bomb (with p...
How complex are myopic imitators?
by Vivek Hebbar — 2022-02-08T12:00:01.393Z

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program.

TL;DR: Time-limited myopic agents might be desirable for their safety properties; one such case is that of "myopic imitators", which imitate a non-myopic process like a human, but do so without considering consequences. Since DL systems are biased towards simple functions, I consider the question of whether myopic imitators are simpler or more complex than their deceptively aligned competitors.

For this analysis, we impose a strong assumption which is similar to minimal program complexity, as well as a clean division between a world model, an "internal interpreter", and a main algorithm which is executed. The analysis is structured as follows:

Operationalizations for "myopia" and "simplicity prior"
Methodology for assessing the simplicity of a candidate algorithm
Analysis of candidate algorithms under various sets of further assumptions

Under these very strong assumptions, and out of the algorithms analyzed, myopic imitators are usually favored. One of the two leading myopic imitators is a straightforward human-imitator which seems to have nice properties, but is sometimes disadvantaged by an inability to do a certain Goodhart-style exploit. The other one, though myopic, is a "training set predictor" which generalizes terribly in some cases. That being said, the specific results are to be taken with a grain of salt, since the assumptions are unlikely to hold in reality. The main value going forward is the general approach and methodology.

Assumptions, worldview, and research vision

The analysis in this post applies to a world where a very specific set of assumptions applies. To be clear, I am quite confident that this set of assumptions does not apply to current deep learning systems. Current deep learning systems seem to be characterized more by circuits and interpolation than by the types of programs I consider here. However, the analysis here is interesting as a limiting case, and the methodology might be transferable to more realistic assumption sets.

The main value of this post is to provide a particular frame and methodology for thinking about the question: "How is a system with a strong simplicity prior likely to turn out by default?"

B...