trentmkelly/LessWrong-43k huggingface dataset:
[Request for Distillation] Coherence of Distributed Decisions  With Different Inputs Implies Conditioning
by johnswentworth — 2022-04-25T17:01:08.767Z

There’s been a lot of response to the Call For Distillers, so I’m experimenting with a new post format. This post is relatively short and contains only a simple mathematical argument, with none of the examples, motivation, more examples, or context which would normally make such a post readable. My hope is that someone else will write a more understandable version.

Jacob is offering a $500 bounty on a distillation.

Goal: following the usual coherence argument setup, show that if multiple decisions are each made with different input information available, then each decision maximizes expected utility given its input information.

We’ll start with the usual coherence argument setup: a system makes a bunch of choices, aiming to be pareto-optimal across a bunch of goals (e.g. amounts of various resources) u1…um. Pareto optimality implies that, at the pareto-optimum, there exists some vector of positive reals P1…Pm such that the choices maximize ∑iPiui. Note that P can be freely multiplied by a constant, so without loss of generality we could either take P to sum to 1 (in which case we might think of P as probabilities) or take P1 to be 1 where u1 is amount of money (in which case P is a marginal price vector).

When the goals are all “the same goal” across different “worlds” X, and we normalize P[X] to sum to 1, P[X] is a probability distribution over worlds in the usual Bayesian sense. The system then maximizes (over its actions A) ∑XP[X]u(A,X)=EX[u(A,X)], i.e. it’s an “expected utility maximizer”.

That’s the usual setup in a nutshell. Now, let’s say that the system makes multiple decisions A=A1…An in a distributed fashion. Each decision is made with only limited information: Ai receives fi(X) as input (and nothing else). The system then chooses the functions Ai(fi(X)) to maximize EX[u(A,X)].

Consider the maximization problem for just Ai(f∗i), i.e. the optimal action for choice i given input f∗i. Expanded out, the objective is EX[u(A,X)]=∑Xu(A1(f1(X)),…,Ai(fi(X)),…An(fn(X)),X).

Note that the only terms in that sum which actually depend on Ai(f∗i) are those for which fi(X)=f∗i. So, for purposes of choosing Ai(f∗i) specifically, we can reduce the obje...
Quadratic voting with automatic collusion?
by SarahNibs — 2022-04-25T16:15:49.117Z

Wondering what happens if you have https://en.wikipedia.org/wiki/Quadratic_voting but with automated collusion - after all votes are in, pair two people and have them collude automatically, splitting implied utility equally, repeat over and over until everyone has the same distribution, do this whole process a million times, take the majority winners as winners.

Does that incentivize you to actually vote your real preferences rather than trying to collude beforehand?

Okay you won't actually end up with everyone having the same distribution. But still, do it until "convergence", then take majority winners, and see who wins most under a million random paths to convergent collusions, and take those as winners.
Intuitions about solving hard problems
by Richard_Ngo — 2022-04-25T15:29:04.253Z

Solving hard scientific problems usually requires compelling insights

Here’s a heuristic which plays an important role in my reasoning about solving hard scientific problems: that when you’ve made an important breakthrough, you should be able to explain the key insight(s) behind that breakthrough in an intuitively compelling way. By “intuitively compelling” I don’t mean “listeners should be easily persuaded that the idea solves the problem”, but instead: “listeners should be easily persuaded that this is the type of idea which, if true, would constitute a big insight”.

The best examples are probably from Einstein: time being relative, and gravity being equivalent to acceleration, are both insights in this category. The same for Malthus and Darwin and Godel; the same for Galileo and Newton and Shannon.

Another angle on this heuristic comes from Scott Aaronson’s

list of signs that a claimed P≠NP proof is wrong

. In particular, see:

#6: the paper lacks a coherent overview, clearly explaining how and why it overcomes the barriers that foiled previous attempts.

And #1: the author can’t immediately explain why the proof fails for 2SAT, XOR-SAT, or other slight variants of NP-complete problems that are known to be in P.

I read these as Aaronson claiming that a successful solution to this very hard problem is likely to contain big insights that can be clearly explained.

Perhaps the best counterexample is the invention of Turing machines. Even after Turing explained the whole construction, it seems reasonable to still be uncertain whether there's actually something interesting there, or whether he's just presented you with a complicated mess. I think that uncertainty would be particularly reasonable if we imagine trying to understand the formalism before Turing figures out how to implement any nontrivial algorithm (like prime factorisation) on a Turing machine, or how to prove any theorems about universal Turing machines.

Other counterexamples might include quantum mechanics, where quantization was originally seen as a hack to make the equations work; or formal logic, where I’m not sure if there were any big insights that could be grasped in advance of actually seeing the formalisms in action.

Using the compelling insight heuristic to evaluate alignment research directions

It’s possible that alignment will in practice end up being more of an engineering problem t...
Ukraine Post #11: Longer Term Predictions
by Zvi — 2022-04-25T14:10:01.119Z

The pace of these Ukraine posts is slowing down, and will probably slow further if things go as I expect them to go. Given how much slower things are going, this will likely be my last update on Ukraine for a few weeks.

I did want to record how various things are tracking for future reference and there are some interesting things to think about, but this post proved inessential. If you’re not interested in the details you can safely skip it.

Early in the war, the situation was fluid and unclear. In this new phase, which one could call the Battle for Donbass, things are far less fluid and seem more clear.

As I noted in my recent China post, the Ukraine epistemic environment involves a shooting war yet is relatively straightforward. The disagreements feel less like disagreements over facts and more like disagreements over models of what is likely to happen going forward. There is a Russian story of what is happening, but it is not attempting to be credible. There is uncertainly over details, especially in terms of war crimes, but none of the true disputes seem important.

Early on, I felt pressure to check developments every few hours. Then that became every day, and now it feels more like every several days. This matches the pattern of Covid, both in March 2020 and then again with Omicron and to a lesser extent a few other big developments along the way, and also my experience with other major news events. Things don’t ‘go away’ so much as the situation becomes clear and new developments slow down. Then, hopefully, things eventually ‘go away’ but that usually takes longer.

This will centrally be another pass through the prediction markets, primarily Metaculus.

While I have a compiled list of new bits of information, it seems scarcely necessary – the sequence of what happened with the sinking of the Muscovy – Russia claiming it is fine, Russian Telegram groups saying it’s not fine and vowing revenge, Russia claiming it was damaged but still afloat, Wikipedia listing the ship as ‘on fire,’ the sinking, calls for revenge for the sinking, a word about Russia claiming it put part of the True Cross on the ship, an analysis of the wreck, an analysis of the attack, . One country after another realizing (correctly) they should embrace nuclear power. Maps that mostly stopped changing. Speculations about Russian losses and decline in capacity. More about Ukraine’s adva...
Key questions about artificial sentience: an opinionated guide
by Robbo — 2022-04-25T12:09:39.322Z

[crossposted at EA Forum and Experience Machines; twitter thread summary]

What is it like to be DALL-E 2? Are today’s AI systems consciously experiencing anything as they generate pictures of teddy bears on the moon, explain jokes, and suggest terrifying new nerve agents?

This post gives a list of open scientific and philosophical questions about AI sentience. First, I frame the issue of AI sentience, proposing what I think is the Big Question we should be trying to answer: a detailed computational theory of sentience that applies to both biological organisms and artificial systems. Then, I discuss the research questions that are relevant to making progress on this question. Even if the ultimate question cannot be answered to our satisfaction, trying to answer it will yield valuable insights that can help us navigate possible AI sentience.

This post represents my current best guess framework for thinking about these issues. I'd love to hear from commenters: suggested alternative frameworks for the Big Question, as well as your thoughts on the sub-questions.

Introduction

“Maybe if a reinforcement learning agent is getting negative rewards, it’s feeling pain to some very limited degree. And if you’re running millions or billions of copies of that, creating quite a lot, that’s a real moral hazard.” -Sam Altman (OpenAI), interviewed by Ezra Klein (2021)

Are today's ML systems already sentient? Most experts seem to think “probably not”, and it doesn’t seem like there’s currently a strong argument that today’s large ML systems are conscious.[1]

But AI systems are getting more complex and more capable with every passing week. And we understand sufficiently little about consciousness that we face huge uncertainty about whether, when, and why AI systems will have the capacity to have conscious experiences, including especially significant experiences like suffering or pleasure. We have a poor understanding of what possible AI experiences could be like, and how they would compare to human experiences.

One potential catastrophe we want to avoid is unleashing powerful AI systems that are misaligned with human values: that's why the AI alignment community is hard at work trying to ensure we don't build power-seeking optimizers that take over the world in order to pursue some goal that we regard as alien and worthless.

It’s encouraging that more work is going into minimizing ...
On Tables and Happiness
by Alexander — 2022-04-25T09:51:21.250Z

I'm writing this post because I want to have a succinct and shareable piece summarising my non-rigorous views on happiness.

During the past decade, I entered adulthood. With this transition came more choices, as my parents and teachers no longer constrained me. More choices came with more responsibility and choice paralysis. Should I stay here or move there? Should I focus on acquiring knowledge or furthering my social life? What actions will effectively improve my happiness?

The model I came up with is a four-legged table. Imagine that a flat tabletop represents your happiness. You want to maintain a flat tabletop. If the four legs are out of balance, the tabletop will become tilted, and so will you. The tabletop of happiness stands on these four pillars:

Health

Knowledge

Wealth

Relationships

This model is helpful because it's easy to get addicted to some pillars and neglect others. I know many people (including myself, at points in my life) who became obsessed with only one of the four cornerstones and neglected the rest. E.g. we all know the software engineer who is making piles of cash but whose relationships are falling apart, or the socialite who is struggling to pay rent because they're addicted to sacrificing themselves for those around them.

Let's now look at each of the four cornerstones and identify practical ways to improve them. There are several low hanging fruits under each pillar. However, as you advance beyond the low hanging fruits, the game becomes more challenging as returns diminish.

Health

There are cheap and straightforward habits that have an outsized benefit in the long run. It's worth adopting such habits as brushing and flossing daily, exercising daily, etc.
Conversely, some habits have outsized harm in the long run. For example, drunk driving, phone use while driving, consuming harmful substances, etc. Don't do these things.
Health is complex and complicated. You can easily get lost in a rabbit hole researching any health topic and ending up adopting extreme habits. The scientific consensus on various health topics seems good enough for me.

Knowledge

Recognize that truly learning new knowledge entails deliberate practice. You cannot master the violin by watching YouTube videos.
Given the raw materials for a vehicle, most permutations will correspond to a rock, and a tiny subset will correspond to a functioning vehicle. In a similar vein, most in

...
Why I'm Not a Utilitarian in Modern America
by DanB — 2022-04-24T21:43:37.586Z

Many people in the rationalist / EA community express admiration and loyalty to utilitarian ethics.
This makes me uncomfortable at parties,
because I reject the idea that utilitarian ethics is a workable philosophy for the modern American.
This essay explains my position.

I develop the argument
by analyzing how common moral intuitions respond to some variations of the Prisoner's Dilemma (PD).
If you're not familiar with the PD,
please read the wiki page before continuing.

When used as an analogy for real-world ethical behavior,
cooperation in the PD is a stand-in for any kind of altruistic action.
It might represent telling the truth
in a situation where it is socially costly to do so.
It might mean obeying a law,
even though there is little chance of being caught;
or it might mean disobeying an unjust law and facing the consequences.
In the most dramatic case,
it might mean risking one's life to defend society against a hostile invader.
Defection in the PD is the opposite:
it represents doing whatever is personally beneficial,
at the expense of the greater good.
Crucially, cooperation in the PD is utilitarian,
because it requires the individual to make a modest sacrifice,
for the sake of a greater benefit to other humans.

Variation 0 - the Happy World

The first variation is just a PD with a single partner who always cooperates.
In this world, everything is happy and cozy;
my moral intuition has no problem with reciprocating cooperation
- indeed it demands that I do so.
This scenario demonstrates the first of the two basic ethical rules
that I, and I imagine most other people, will follow:
Don't be a Jerk.
If someone is cooperating with you,
you should always cooperate with them in return.
If real life was similar to this scenario,
I would have no problem following utilitarian ethics.

Variation 1 - Hell is Other People

This scenario is the opposite of the previous one.
Here my partner defects all the time, no matter what I do.
While I would initially try to elicit cooperation by randomly cooperating even though
my partner previously defected,
in the long run I will begin to defect as well.

This scenario illustrates the second of my two moral intuitions:
Don't be a Sucker.
If someone refuses to cooperate with me,
I refuse to cooperate with them.
I refuse to be taken advantage of by predators.
This refusal to cooperate in the face of hostility
already shows the p...
Examining Evolution as an Upper Bound for AGI Timelines
by meanderingmoose — 2022-04-24T19:08:16.259Z

Cross-posted from https://mybrainsthoughts.com/?p=349

With the massive degree of progress in AI over the last decade or so, it’s natural to wonder about its future - particularly the timeline to achieving human (and superhuman) levels of general intelligence. Ajeya Cotra, a senior researcher at Open Philanthropy, recently (in 2020) put together a

comprehensive report

seeking to answer this question (actually, it answers the slightly different question of when transformative AI will appear, mainly because an exact definition of impact is easier than one of intelligence level), and over 169 pages she lays out a multi-step methodology to arrive at her answer. The report has generated a significant amount of discussion (for example, see this

Astral Codex Ten review

), and seems to have become an important anchor for many people’s views on AI timelines. On the whole, I found the report added useful structure around the AI timeline question, though I’m not sure its conclusions are particularly informative (due to the wide range of timelines across different methodologies). This post will provide a general overview of her approach (readers who are already familiar can skip the next section), and will then focus on one part of the overall methodology - specifically, the upper bound she chooses - and will seek to show that this bound may be vastly understated.

Part 1: Overview of the Report

In her report, Ajeya takes the following steps to estimate transformative AI timelines:

Determine the total amount of computation required to train a transformative AI model, based on the architectures and algorithms available in 2020.
Determine the rate at which the amount of computation available will change (due to reduced cost of computation and greater availability of capital) and the rate at which the computational requirements will be reduced (due to architectural / algorithmic progress).
Apply these rates, starting from 2020, to determine at what point sufficient computation will be available to train a transformative AI model.

1. Determine Total Amount of Computation Required for Transformative AI

To get an estimate, Ajeya leans heavily on the only working example of transformative intelligence we have - the human brain. She lays out four different approaches in an attempt to bound the computation required:

Human Lifetime: Total amount of computation done by a

...
AI safety raising awareness resources bleg
by iivonen — 2022-04-24T17:13:42.152Z

Bleg definition

(Internet slang) An entry on a blog requesting information or contributions. I posted a bleg in the hope of learning more about local tourism.

https://www.yourdictionary.com/bleg

Most ordinary people don't know about AI safety. They might have heard about 'algorithms' on the news, and they might have seen Terminator, but they don't understand what those algorithms really are, or how deep this rabbit hole actually goes. Unlike, say, climate change, there really is an awareness issue here, and it might actually matter to the outcome of things whether the public is widely aware that the risks this kind of thing poses are not just SF anymore.

Of those who do, often the ones who look into it seriously for an afternoon walk away less convinced of it being a problem than they walked into it feeling, usually justified with one of a short list of common AI risk skepticism biases (PDF). My suspicion is that, for an ordinary person, the issue seems so overwhelming, so impossible to gain traction on, that there's no sense in even trying.

I'm not convinced of this. AI safety isn't an impossible problem to manage succesfully. Its difficulty ranges between "purely collective action problem" in the worst-case scenario and "purely technical hurdle" in the best. There is power in giving people the message, "Yes it's bad - yes we can do something about it, together."

So what kinds of resources are suitable for an average, but interested, person? Like "wouldn't major in STEM" average, or even "didn't go to college for lack of ability" average.

Resources I'd be especially thankful for:

"Elevator pitch" explanations of why AI safety is a problem. Brief explanations in clear language at the 10-, 100-, and 1000-word count levels.

Common objections and their refutations to common AI risk things, since people often go from "That's not real" to "That can't happen because X" to "Maybe it won't be so bad because Y" to "Oh God wait so are we fucked?" to "No, the problem is just hard".

Specific calls to action and ways ordinary people can help, even if they can't work on technical approaches.

(more for me but I find myself jumping to here with my own friends a lot) Philosophical breakdowns of why people consider it so important that we don't end up with a paperclip maximizer in the universe. Accessible pluralist ethics, etc.

Thank you kindly, and have a wonderful day. Apolog...
Where can I publish an article containing a list of intellectuals who publicly admitted their mistakes once proven wrong?
by Hashem ElAssad — 2022-04-24T09:00:35.804Z

I’m writing a piece about (Public “I was wrong" admissions by intellectuals & thought leaders) that I’m planning to publish. It contains 6 people till now. Any tips on who would be interested in publishing such a piece (preferably paid)?
What Is a Major Chord?
by jefftk — 2022-04-24T07:20:01.906Z

In the discussion on my

minimal
chords post, someone

commented:

I have no idea what any of this means (what is a chord? what is a
major chord? what is a note? what is a first/fourth/fifth note? is
there a 65th note? what is a scale? what is a major scale? what does
it mean that a note is of a scale? what does it mean that a chord uses
a note? is there a difference between a chord using a note of a scale
and not of a scale?)

Here's an attempt to introduce enough music theory to answer these
questions:

We hear changes in air pressure. If those changes are rapid enough and
consistent enough, we hear them as pitch (frequency). We can talk
about these in terms of how many changes we get per second, which we
call "Hz". For example, a pitch could be 100Hz or 500Hz. When we say
a pitch is "higher" or "above" another pitch, we mean more changes per
second: 500Hz is higher than 100Hz.

A note is something that gives the impression of being a single
pitch. For example, what you get when you play a single key on the
piano, or pluck a string on a stringed instrument. Many instruments
can only play one note at a time: trumpet, flute, saxophone.

The standard notes used in Western music differ in pitch by a factor
of the 12th root of 2 (~1.06x). This means that if you go up twelve
notes (which we call "half steps", confusingly) your pitch doubles
(the 12th root of 2, multiplied by itself twelve times, is just 2).
Two notes whose pitch differs by a factor of two (ex: 100Hz and 200Hz)
are said to be an "octave" apart, and sound almost like the same note.
We give notes that differ by some number of octaves the same name (ex:
"C"), though when we want to be specific about which octave we're
talking about we can append numbers ("C1" at ~32Hz is an octave above
"C0" at ~16Hz).

A scale is a set of notes from an octave. We usually talk about a
scale as being sorted from lowest note to highest. We can define a
scale by the distances between its notes. Perhaps the simplest scale
(the "chromatic scale") would be to go up by one note each time, playing
every note: 111111111111. This typically doesn't sound very good, and
we don't usually use it.

A "major scale" has the pattern 2212221: you go up by two notes, two
notes, one note, etc. This gives you seven different notes in your
octave. We can call these notes the "first", "second", etc notes of...