trentmkelly/LessWrong-43k huggingface dataset:
Quick Summaries of Two Papers on Kant and Game Theory
by Erich_Grunewald — 2022-06-25T10:25:08.398Z
Link: https://www.erichgrunewald.com/posts/quick-summaries-of-two-papers-on-kant-and-game-theory/

It's not obvious from a Darwinian point of view why people (even people who aren't related) often cooperate with one another. Researchers like Axelrod and Hamilton (1981) have used game theory to show that some strategies are both adaptive and favour cooperation. Though these projects may not say anything about how we should act towards one another[1], they do say something about why we do act the way we act. That's important because we'd like moral strategies (under whatever view we think is more plausible) to outcompete immoral strategies; if we notice that various game-theoretic situations tend to steer people away from sound ethics, that's both (1) an indication that these situations are to be avoided and (2) an invitation to think up situations that steer people towards better, more ethical strategies.

The rest of this post comprises summaries of two papers about this problem (specifically concerning my preferred ethics, Kantianism), plus some brief comments on each of them.

Laslier (2020): In the stag hunt game, two hunters choose to either hunt a stag or hunt hares – if both hunt the stag, they both get a very good payoff (the stag); if both hunt hares, they both get a bad payoff (few hares); but if one hunts the stag and the other hunts hares, the stag hunter gets a very bad payoff (nothing) and the hare hunter gets a good payoff (many hares). This game has two pure strategy Nash equilibria: the Pareto optimal outcome where both hunters hunt the stag, and the outcome where both hunters hunt hare. It also has one mixed strategy Nash equilibrium: both players hunt the stag with 2/3rds probability and hares with 1/3rd probability.

A "Kantian" player asks: "Which strategy is best if both players use it?"; the answer is to always hunt the stag. In a simple model, like that in Roemer (2019), a "Nasher" is a player that goes for one of the (non-Kantian) Nash equilibria. If a Nasher goes with the mixed strategy, they're outcompeted by the Kantian. If a Nasher always hunts hares, they outcompete the Kantian if fewer than 2/3rds of the population are Kantians, and are outcompeted otherwise.

In a dynamic model, a "selfish" player optimises their own evolutionary fitness (by changing their strategy as they see fit). Without Kantians, the dynamic model reaches an equilibrium of hare hunting if a certain proportion of the initial population are hare hunters, and of st...
Do you consider your current, non-superhuman self aligned with “humanity” already?
by Rana Dexsin — 2022-06-25T04:15:08.088Z

Any actual “hard” superhumans who may be listening are excluded from the audience of this question. I am currently refraining from describing my own ideas to avoid biasing the results; I'm not sure whether I'll reveal something later or not.

I will probably be posting two comments to use the agreement axis of as a sort of poll for the yes/no version of the question, but other more detailed answers and comments are welcome as well. (The reason for using two is so that the per-comment aggregation will make the full counts visible.)
Dependencies for AGI pessimism
by Yitz — 2022-06-24T22:25:03.049Z

Epistemic status: mildly confident

One must believe (with at least minimal confidence) in all of the following points in order to believe that AGI poses a unique existential risk:

A) AGI (at a superhuman level) is possible

B) AGI has a significant chance of finding value in destroying humanity (as we think of it at least)

C) AGI will be capable of killing all or nearly all humans in ways which non-superintelligent-AGI agents cannot or likely will not do.

If you believe that all of the above are true, are you forced to worry about existential AI risk? Not necessarily. Here are some other, more subtle/fundamental premises one must accept:

Humanity should not be destroyed.
Humanity can in practice be destroyed, as a general idea. (There may be some religious/philosophical views which don’t believe extinction is possible. This is more of a generalization of C than a separate dependency.)
It’s possible to have an effect on the risk level from superintelligence. (If not, there’s no use worrying about it)
There is no other near-term existential risk which is orders of magnitude more likely. (If so, it’s justifiable not to be concerned, for the same reason that we aren’t deeply occupied with near-term asteroid impact risk.)
One should be concerned about risks which one may be able to help prevent.
One should care about the long-term future, and of risks to others outside the self (this is also a dependency for 1, but not identical, because it’s possible in theory to be both a longtermist and a misanthrope).
Taking practical steps based on logical thinking is a reasonable way to deal with the world. (If you don’t believe in logic, then you can probably contradict yourself and hold everything else to be true while still not changing your mind? I’m not sure this one is necessary to include, but I may be wrong.)

If a counter-example exists of someone who’s concerned about existential risk from AI but doesn’t believe in all of the above, or vice versa, please let me know and I will update accordingly.
[Link] Childcare : what the science says
by Gunnar_Zarncke — 2022-06-24T21:45:23.406Z
Link: https://criticalscience.medium.com/on-the-science-of-daycare-4d1ab4c2efb4

This is a pretty good summary of the research on daycare effectiveness. It is a Much More Than You Wanted To Know-style post from a science blogger criticalscience who has some interesting posts on Medium.

TL;DR:

Adverse effects on children younger than three years old.
The effect is worse the younger the children are.
The effect is stronger for longer hours.
The effect is weaker CORRECTION: worse for children from socio-economically better-off families.
Nannies are OK.

Childcare : what the science says
What if the best path for a person who wants to work on AGI alignment is to join Facebook or Google?
by dbasch — 2022-06-24T21:23:01.758Z

Suppose you are passionate about the problem of AGI alignment. You want to research solutions to the problem, and you state it more or less like this:

We are on track for an AI catastrophe, most likely within the next few decades. We do not know how (or even if) we can prevent it. We need to do continued research right now, and closely follow new developments in industry. If there are enough of us dedicated to the problem, perhaps we can find a solution in time. Otherwise, the companies developing AI won't do it because they only care about short-term business goals, and won't take the threat seriously until it's too late. Come with us if you want to live.

This is a tough sell. From the point of view of any organization granting research funds, it is expected to hear: a) this problem is extremely important and b) our research is the most promising way to solve it. I believe that making the case for a) is easier than for b). Why? Because the notable AI advancements from the past few years came from companies like Facebook and Google. If you are not inside those companies, your job as a researcher is to react to whatever they decide to share with the rest of the world.

If I were completely convinced that this is the most pressing problem that exists, I would want to be as close as possible to the source of new developments so I could influence them directly. It would be a tough job, because I would have to walk a fine line. I could not get myself fired by opposing the deployment of every new mechanism, but perhaps I could spend a significant portion of my time playing devil's advocate. As someone involved in deploying production systems, I could insist in adding circuit breakers and other safety mechanisms. In addition I would be involved in the hiring process, so I could try to get like-minded people to join the team. Perhaps we might manage to nudge the company culture away from "move fast and break things (irreversibly)."

If you cannot beat them, join them and promote change from within.
[Link] Adversarially trained neural representations may already be as robust as
corresponding biological neural representations
by Gunnar_Zarncke — 2022-06-24T20:51:27.924Z

Abstract:

Visual systems of primates are the gold standard of robust perception. There is thus a general belief that mimicking the neural representations that underlie those systems will yield artificial visual systems that are adversarially robust. In this work, we develop a method for performing adversarial visual attacks directly on primate brain activity. We then leverage this method to demonstrate that the above-mentioned belief might not be well founded. Specifically, we report that the biological neurons that make up visual systems of primates exhibit susceptibility to adversarial perturbations that is comparable in magnitude to existing (robustly trained) artificial neural networks.

https://arxiv.org/pdf/2206.11228.pdf
Updated Deference is not a strong argument against the utility uncertainty approach to alignment
by Ivan Vendrov — 2022-06-24T19:32:17.547Z

Thesis: The problem of fully updated deference is not a strong argument against the viability of the assistance games / utility uncertainty approach to AI (outer) alignment.

Background: A proposed high-level approach to AI alignment is to have the AI maintain a probability distribution over possible human utility functions instead of optimizing for any particular fixed utility function. Variants of this approach were advocated by Stuart Russell in Human Compatible and by Hadfield-Menell et al in the CIRL paper. Adding utility uncertainty intuitively seems to provide a number of safety benefits relative to having a fixed objective, including:

Utility uncertainty gives the AI an incentive to adjust in response to a human operator's corrective actions.
Utility uncertainty weakens the AI's incentive to harm its human operators, since this might result in a permanent loss of utility-relevant information.
Utility uncertainty incentivizes the AI to avoid irreversible changes to the state of the world, since those might lead to permanently low utility.

Despite the high profile and intuitive appeal of utility uncertainty, almost none of the alignment researchers I know consider it a promising approach to AI alignment. The most common reason cited seems to be the problem of fully updated deference (e.g. Richard Ngo's alignment research exercises point to this as the reason for why CIRL doesn't solve the alignment problem).

In this post I will argue why fully updated deference should not be seen as a strong argument against utility uncertainty as approach to AI alignment. This is not meant as an argument in favor of the uncertainty approach; it may have other irresolvable difficulties which I discuss briefly in the conclusion.

Outline: The Arbital post that seems to be the canonical reference for updated deference contains many heuristic arguments and one concrete, worked-out example in the section Moral uncertainty and its relation to corrigibility. I will mostly engage with the example, and argue that

It conflates the problem of updated deference with the independent problem of prior mis-specification.
If we remove prior mis-specification, there is no problem in the limit of increasing AI capability.

The Problem of Updated Deference

The example in the post has an AI that is uncertain between three utility functions U1, U2, U3 whereas the human's true utility...
Cracks in the Wall, Part I: The Conscious
by silo — 2022-06-24T18:29:15.231Z
Link: https://stephenfoster.substack.com/p/cracks-in-the-wall-part-i-the-conscious

I.

Imagine a genie appears before you and makes you a simple offer: you can ask him any question you want, and he'll answer it. No tricks - genie society has seen sweeping reforms and made great moral progress, and the days of the malicious trickster genies have been left long behind. No, this genie is entirely benevolent: if your question turns out to have been ill-formed, he'll explain exactly why it was ill-formed, and then answer the question you intended to ask. No matter what question you pose to him, you'll find yourself completely satisfied with the answer he gives.

However, genies being genies, he still feels the need to follow ancient tradition and impose some kind of a ternary restriction on his gift. So he stipulates that the question can only be three words long.

What question do you ask him?

Even with only three words, there seems no shortage of good options to pick from. You might, for instance, ask him: Why am I? That is, for what reason do I exist? What, if anything, is my purpose in life? Where can I find meaning, what ought I to do with my time?

Or perhaps: How am I? How, by what means, did I come to be, and do I persist in being? What are the mechanisms and operations of creation and existence?

Or maybe it's wisest to ask: Who am I? How do I fit into the world, how does my life relate to the lives of others? What is the truest story I can tell about the person that I am?

All excellent nominations, and a case could be made for any of them. For myself, though, I find that I stumble long before I reach anything nearly so lofty. My confusions are more basic than that, my lack of understanding more fundamental. No, I'd be happy getting the answer to a much simpler question: What am I?

What, exactly, am I? If I can't answer that then it's hard to see how I can make much sense of the other questions, or really any question at all. And troublingly, I find that I can't, in fact, answer it.

What do I mean by that? Is it not obvious what we are? I worry some of you might think I'm being deliberately obtuse here, in service of some ultimately irrelevant philosophical point I'm trying to make. Not my intention at all. I mean what I say: at a basic level, I really have no idea how to think about what kind of thing it is that I am. My confusion runs deep. And far from being an irrelevancy, it was confronting this mystery in full that forced me to rethink much of...
Do alignment concerns extend to powerful non-AI agents?
by Ozyrus — 2022-06-24T18:26:22.737Z

Right now the main focus of alignment seems to be on how to align powerful AGI agents, a.k.a AI Safety. I think the field can benefit from a small reframing: we should think not about aligning AI, but about alignment of systems in general, if we are not already.

It seems to me that the biggest problem in AI Safety comes not from the fact that the system will have unaligned goals, but from the fact that it is superhuman.

As in, it has nearly godlike power in understanding of the world and in turn manipulation of both the world and other humans in it. Does it really matter if an artificial agent gets godlike processing and self-improvement powers or a human will, or government/business?

I propose a little thought experiment – feel free to answer in the comments.

If you, the reader, or, say, Paul Christiano or Eliezer gets uploaded and obtains self-improvement, self-modification and processing speed/power capabilities, will your goals converge to damaging humanity as well?
If not, what makes it different? How can we transfer this secret sauce to an AI agent?

If yes, maybe we can see how big, superhuman systems get aligned right now and take some inspiration from that?
Raphaël Millière on Generalization and Scaling Maximalism
by Michaël Trazzi — 2022-06-24T18:18:10.503Z
Link: https://theinsideview.ai/raphael

I interviewed Raphaël Millière, a Presidential Scholar in Society and Neuroscience at Columbia University about his critic of “scaling maximalism” and his takes on AGI. This is part of an on-going effort where I talk to people who are skeptical of existential risk from AI so that everyone could better steelman their arguments (cf. why you might want to talk to skeptics).

Why I Interviewed Raphaël

Although we disagree on most topics, Raphaël follows closely current state of the art research in deep learning, is impressed by current advances, yet still completely disregards existential risk from AI. The reasons for his (supposedly contradictory) beliefs are related to the potential limitations of deep learning in terms of understanding, compositionality and generalization, which are shared by popular AI skeptics such as Gary Marcus or François Chollet.

Below are some quotes of Raphaël said during his interview. You can find them in full context here.

Three levels of Generalization, the Wozniak Test

Maybe one distinction that's helpful there, is again from François Chollet's paper on the measure of intelligence, which I quite like, is this distinction between, I think it distinguishes between three levels of generalization.

So you have local generalization, which is a narrow form of generalization that pretends to generalize to known unknowns. So within a specific task. So that can be just, for example, you have a classifier that classifies pictures of dogs and cats, and then you can generalize to unseen examples, at test time, that it hasn't seen during training. So that's local generalization is just within domain known unknowns in a specific task.
Then there is what he calls broad generalization, that's generalizing to unknown unknowns within a broad range of tasks. So the examples he gives there would be level five self-driving or there was the Wozniak test, which was proposed by Steve Wozniak, which is building a system that can walk into a room, find the coffee maker and brew a good cup of coffee. So these are tasks or capacities that require adapting to novel situations, including scenarios that were not foreseen by the programmers where, because there are so many edge cases in driving, or indeed in walking into an apartment, finding a coffee maker of some kind and making a cup of coffee. There are so many potential edge cases. And, this very long tail of

...
Worked Examples of Shapley Values
by lalaithion — 2022-06-24T17:13:43.095Z

Three times in the past month, I've run across occasions where Shapley values were mentioned or would have been useful. There are a couple of good explainers of Shapley value already on the internet, but what most of them lack is a bunch of worked examples. I find that many times, the limiting factor in my ability to understand a concept is whether or not I've been exposed to multiple examples—so this post is where I'll be posting some of the examples I worked through to understand how Shapley value works.

I'll start with a brief overview of Shapley value, but first, a caveat. I think it's fine to skim most of the math in the overview. We won't really be using it that much, and I try to give an intuitive sense of what the equations mean before I present them. You could probably skip the entire overview and just go straight to the examples and not miss much.

Overview of Shapley Value

Suppose you have a group of people who work together to produce some sort of value (traditionally profit). How should you divide the credit for that value (or the actual profit) up among the individuals involved? One immediate suggestion might be "equally", but that doesn't necessarily satisfy intuitive notions of fairness. If a doctor performs life-saving surgery, should they receive equal credit for saving someone's life as the random person holding the door open for the doctor at the end of the surgery? If you're running a restaurant, and three servers spend all night running around serving people, and one server spends two hours on a smoke break, do all four deserve an equal amount of pay?

Shapley value provides a different way of computing what a fair division of value should be. Why use Shapley value? Well, like splitting things equally, it (a) divides all of the gains among all of the participants, (b) splits things equally between participants who contribute the same value, and (c) if there are two completely independent value-producing processes, then the assignment of value to each participant is equal to the sum of the value for that participant in each game[1]. Unlike splitting things equally, it also (d) assigns no value to anyone who always contributes nothing, and in fact, it is the only assignment rule which satisfies all four of those constraints.

The formulas for computing the Shapley values can be found on its Wikipedia page.

The simplest way of computing Shapley val...
Feature request: voting buttons at the bottom?
by Oliver Sourbut — 2022-06-24T14:41:55.268Z

This is a brief question/request regarding LessWrong and AlignmentForum UI and voting buttons.

Uncontroversial (I expect) factual statements

When I read a post or comment which fills more than one screen, I want to consume the whole content before providing any reinforcement or agreement signals.

But the vote buttons are rendered at the top of posts and comments, so I have to scroll back up again[1], and (especially if I'm reading through a thread of comments) it can therefore be tricky to keep track of my place.

A basic request

Could we have additional voting buttons rendered (perhaps optionally configurably?) at the bottom of posts and comments? This would make the above, presumably common, workflow that little bit more straightforward.

Or (more fiddly but maybe better), could the buttons float and follow me down the page, or some other fancy UI trick?

A more controversial normative claim

It seems like the factual statement of my reading/digesting/voting workflow above should be the default workflow. After all, how can you provide a reliable voting signal for something you haven't consumed?[2]

Would it make sense to always have the buttons exclusively at the bottom, to nudge[3] the 'correct' workflow to be marginally easier, and 'aberrant' workflows (like voting based on the author's name or the title or introduction)[4] to be marginally more difficult? What might the downsides be (beside the obvious developer opportunity cost)?

Even more controversial speculation

What if the karma/agreement tallies were also at the bottom? (i.e. unseen until after deliberate scroll)

It wouldn't in principle change my options as a reader, but it would make it easier to avoid anchoring and instead evaluate content on its merit (but I could still readily see the community consensus signal and adopt that information into my all-things-considered opinion).

This seems a lot easier to backfire, because often we filter what to even start reading based on more quickly-digested summaries (probably quite rightly), which include karma and the like, as well as recommendations. It might work better, for that reason, if at all, on comments than on top-level posts.

I almost always read things from top to bottom, at least on a first reading, which I also expect to be relatively uncontroversial. ↩︎

There are of course degrees of what constitutes 'consumed' and 'reliable'! ↩︎

cf Recommendat

...