trentmkelly/LessWrong-43k huggingface dataset:
Coherent behaviour in the real world is an incoherent concept
by Richard_Ngo — 2019-02-11T17:00:25.665Z

Note: after putting this online, I noticed several problems with my original framing of the arguments. While I don't think they invalidated the overall conclusion, they did (ironically enough) make the post much less coherent. The version below has been significantly edited in an attempt to alleviate these issues.

Rohin Shah has recently criticised Eliezer’s argument that “sufficiently optimised agents appear coherent”, on the grounds that any behaviour can be rationalised as maximisation of the expectation of some utility function. In this post I dig deeper into this disagreement, concluding that Rohin is broadly correct, although the issue is more complex than he makes it out to be. Here’s Eliezer’s summary of his original argument:

Violations of coherence constraints in probability theory and decision theory correspond to qualitatively destructive or dominated behaviors. Coherence violations so easily computed as to be humanly predictable should be eliminated by optimization strong enough and general enough to reliably eliminate behaviors that are qualitatively dominated by cheaply computable alternatives. From our perspective this should produce agents such that, ceteris paribus, we do not think we can predict, in advance, any coherence violation in their behavior.

First we need to clarify what Eliezer means by coherence. He notes that there are many formulations of coherence constraints: restrictions on preferences which imply that an agent which obeys them is maximising the expectation of some utility function. I’ll take the standard axioms of VNM utility as one representative set of constraints. In this framework, we consider a set O of disjoint outcomes. A lottery is some assignment of probabilities to the elements of O such that they sum to 1. For any pair of lotteries, an agent can either prefer one to the other, or to be indifferent between them; let P be the function (from pairs of lotteries to a choice between them) defined by these preferences. The agent is incoherent if P violates any of the following axioms: completeness, transitivity, continuity, and independence. Eliezer gives several examples of how an agent which violates these axioms can be money-pumped, which is an example of the “destructive or dominated” behaviour he mentions in the quote above. And by the VNM theorem, any agent which doesn’t violate these axioms has preferences whi...
Why do you reject negative utilitarianism?
by Teo Ajantaival — 2019-02-11T15:38:39.163Z

Note (2022-03-15): For a recent defense of similar axiological views, see my series on minimalist axiologies.

(Crossposted on the EA Forum)

Absolute negative utilitarianism (ANU) is a minority view despite the theoretical advantages of terminal value monism (suffering is the only thing that motivates us “by itself”) over pluralism (there are many such things). Notably, ANU doesn’t require solving value incommensurability, because all other values can be instrumentally evaluated by their relationship to the suffering of sentient beings, using only one terminal value-grounded common currency for everything.

Therefore, it is a straw man argument that NUs don’t value life or positive states, because NUs value them instrumentally, which may translate into substantial practical efforts to protect them (compared even with someone who claims to be terminally motivated by them).

If the rationality and EA communities are looking for a unified theory of value, why are they not converging (more) on negative utilitarianism?

What have you read about it that has caused you to stop considering it, or to overlook it from the start?

Can you teach me how to see positive states as terminally (and not just instrumentally) valuable, if I currently don’t? (I still enjoy things, being closer to the extreme of hyperthymia than anhedonia. Am I platonically blind to the intrinsic aspect of positivity?)

And if someone wants to answer: What is the most extreme form of suffering that you’ve experienced and believe can be “outweighed” by positive experiences?
How important is it that LW has an unlimited supply of karma?
by Bird Concept — 2019-02-11T01:41:51.797Z

Question

LessWrong users can up/downvote posts and comments, which then receive a karma boost (capped by the voters own karma). There is no limit to how many different posts and comments one can do this to. In this sense there is an unlimited supply of karma to be handed out. (This is also the case for Facebook, YouTube, Instagram, HackerNews(?), Medium, ...)

Is this important? That is, does it have non-trivial medium or long-term effects on the “LessWrong economy" -- the kind and amount of content that gets produced?

Rough Thoughts

Here are some quick thoughts I wrote down. I publish this question despite them being unfinished, instead of letting them wither deep in my Google Drive.

Under the current system…

Over time, it’s not clear whether karma is inflationary or deflationary. It depends at least on whether the rate of growth of content is slower or faster than the rate of increase of karma production.
The only way to get a large amount of karma is to produce content that appeals to many users, or produce a large amount of content that appeals to at least some users. One cannot get high karma by producing a small amount of content that a small number of users likes a lot.

If the real economy was like this, there wouldn’t exist businesses like SpaceX, Palantir or Boeing.

Something seems very broken about LW, if, were the big world to run on LW principles, people wouldn’t be able to fly as a means of travel. Lots of people want to fly. But very few are able to pay for the construction of a 747. So we only have airtravel because there can exist intermediaries who can make that payment, and in turn get rewarded by collecting all the little flight desires of very many people kind-of-keen to fly.
Currently, there cannot be any such intermediaries on LessWrong. A concrete example of a LessWrong Boeing might be something like: CFAR really wants someone to write a 40-page literature review of X. No one else really cares, apart from the fact that were CFAR to get that review, their workshops would improve pretty significantly for most attendees.

There are fewer free-rider problems. Despite content being non-excludable and publicly available, users have an incentive to upvote things, because Alice doing so instead of Bob does not cost Alice anything (we’re assuming they both end up consuming the content, so attention and time costs are the same).

This seems very impor

...
Propositional Logic, Syntactic Implication
by Donald Hobson — 2019-02-10T18:12:16.748Z

Propositional Logic

Some Definitions

Propositional logic is a simplified model of what it means to prove things. The statements that propositional logic can work with are called propositions. Propositions consist of the primitives ⊥,p1,p2,p3,⋯ as well as anything of the form (X⟹Y) where X and Y are propositions. Note for pedants, this description allows the existence of propositions that can't be broken down into a finite number of primitives, we don't want such propositions.

Let A be the set of all propositions formed by taking any arbitrary propositions X, Y and Z, in any of the following three arrangements.

1) (X⟹(Y⟹X))

2) ((X⟹(Y⟹Z))⟹((X⟹Y)⟹(X⟹Z)))

3) (((X⟹⊥)⟹⊥)⟹X)

The set A is the set of axioms, and the three forms shown above are the infinite axiom schema. For example, taking X=p1 and Y=(p2⟹⊥) then applying the first schema gets (p1⟹((p2⟹⊥)⟹p1))∈A. So this is an axiom.

We say that a set of propositions S implies another proposition p, when there exists a proof of p from S. Write this as S⊢p.

A proof of a proposition p form S is a list L of propositions of length n∈N such that Ln=p and for each i≤n, Li fulfills at least one of the following 3 conditions.

1) Li∈S

2) Li∈A

3) ∃j,k<i:Lj=(Lk⟹Li) This is called Modus ponens

Example Proof

For example, here is a proof of (p⟹p)

L1=((pX⟹((p⟹p)Y⟹pZ))⟹((pX⟹(p⟹p)Y)⟹(pX⟹pZ)))

Formed using the second axiom schema.

L2=(pX⟹((p⟹p)Y⟹pX))

Is then created using the first axiom schema

L3=((p⟹(p⟹p))⟹(p⟹p))

Can be produced by modus ponens because L1=(L2⟹L3).

L4=(pX⟹(pY⟹pX))

Is allowed by using axiom schema 1 again.

L5=(p⟹p)

Follows by modus ponens again because L3=(L4⟹L5).

Philosophically Important Takeaways

The notion of mathematical proof is fully formalized. There is an entirely mechanical, and fairly fast, method of verifying proofs. Proofs can be generated in a finite amount of time, by brute force search, but this can be much slower.

The proof framework only believes things that you have actually proved in it. From our point of view outside the framework, the proof of (p⟹p) also shows that (q⟹q), but to actually prove that within the framework, you have to go through all the steps again.
Fighting the allure of depressive realism
by aaq — 2019-02-10T16:46:21.371Z

Epistemic tragic backstory: Personal.

Earlier this week, on my first post on this site under this account, shminux commented to the value of cognitive-behavioral therapy to get out of what I called "depression philosophising".

I was worried about trying it. One claim of CBT is that depressed people are negatively biased and by correcting their thinking errors they are gradually brought to being both happier and more accurate about the world around them.

However this may not actually be the case. The phenomenon known as "depressive realism" suggests that the ordinary person might be positively biased and that depressed people might need to correct their "errors" by forming less accurate, but happier patterns of cognition. Sort of like a really, really weak Nozick machine.

Now, this question doesn't actually undermine CBT itself that much. A movement from (depressed, inaccurate) to (not depressed, accurate) is pretty much as good, in terms of what any anti-depression therapy is trying to do, as one from (depressed, accurate) to (not depressed, inaccurate).

But if it's the latter, our least convenient likely world, we face the classic question: "Should we optimize more for epistemic or instrumental rationality?" This was a hurdle I had to get over before I could convince myself to use CBT. I had a few false starts, but eventually came up with some good convincing arguments that even if this is the case CBT is well worth it.

I decided to treat it as a decision on the margin, remembering my Econ 101. That turned out to be such an obviously right fit to the problem that I felt ridiculous for not having thought of it instantly. The tradeoff of a small amount of epistemic rationality (= losing the benefit of depressive realism) for a high chance of a moderate, potentially large amount of instrumental rationality (= all the time, energy, and general life pleasure I get from treating my depression) is one almost any sane person should make.

After that I also realized that there was an argument from symmetry here. Would I advise someone looking to improve their epistemic rationality to become depressed? Of course not. We have all of these tools already to improve our epistemics - the wisdom of crowds, prediction markets, good old fashioned education and reaching out to experts on whatever topics seem pertinent. But you will ve...
Structured Concurrency Cross-language Forum
by Martin Sustrik — 2019-02-10T09:20:00.779Z

There are structured-concurrency-related efforts going on for different programming languages but the entire effort is kind of scattered, without people being aware of each other and speaking to each other.

If we had a common forum, we could share the use cases, the problems, the ideas and the solutions. Each of us, irrespective of which language they are working with, could benefit from this common pool of knowledge…

And here we go!

The forum now exists. Thanks to Nathaniel Smith for setting it up!

And here's my kick-off post on the forum.

Enjoy!

Feb 10th, 2019
Probability space has 2 metrics
by Donald Hobson — 2019-02-10T00:28:34.859Z

A metric is technically defined as a function from pairs of points to the non negitive reals. d:X×X→[0,∞) With the properties that d(x,y)=d(y,x) and d(x,y)=0⟺x=y and d(x,y)+d(y,z)≥d(x,z).

Intuitively, a metric is a way of measuring how similar points are. Which points are nearby which others. Probabilities can be represented in several different ways, including the standard p∈(0,1) range and the log odds b∈(−∞,∞). They are related by b=log(p1−p) and eb−1=11−p and p=ebeb+1 (equations algebraically equivalent)

The two metrics of importance are the baysian metric B and the probability metric P.

B(b1,b2)=|b1−b2|=∣∣
∣∣log(p1(1−p2)p2(1−p1))∣∣
∣∣

P(p1,p2)=|p1−p2|=∣∣∣1eb1+1−1eb2+1∣∣∣

Suppose you have a prior, b1 in log odds, for some proposition. Suppose you update on some evidence that is twice as likely to appear if the proposition is true, to get a posterior, b2 in log odds. Then B(b1,b2)=log(2). The metric B measures how much evidence you need to move between probabilities.

Suppose you have a choice of actions, the first action will make an event of utility u happen with probability p1, the other will cause the probability of the event to be p2. How much should you care. uP(p1,p2).

The first metric stretches probabilities near 0 or 1 and is uniform in log odds. The second squashes all log odds with large absolute value together, and is uniform in probabilities. The first is used for baysian updates, the second for expected utility calculations.

Suppose an imperfect agent reasoned using a single metric, something in between these two. Some metric function less squashed up than P but more squashed than B around the ends. Suppose it crudely substituted this new metric into its reasoning processes whenever one of the other two metrics was required.

In decision theory problems, such an agent would rate small differences in probability as more important than they really were when facing probabilities near 0 or 1. From the inside, the difference between no chance and 0.01, would feel far larger than the distance between probabilities 0.46 and 0.47.

The Allais Paradox

However, the metric is more squashed than B, so moving from a 10000:1 odds to 1000:1 odds seems to require less evidence than moving from 10:1 to 1:1. When facing small probabilities, such an agent would perform larger baysian updates than really necessary, based on weak evidence.

Privileging the Hypothesis

As b...
Some Thoughts on Metaphilosophy
by Wei Dai — 2019-02-10T00:28:29.482Z

A powerful AI (or human-AI civilization) guided by wrong philosophical ideas would likely cause astronomical (or beyond astronomical) waste. Solving metaphilosophy is one way in which we can hope to avoid this kind of disaster. For my previous thoughts on this topic and further motivation see Metaphilosophical Mysteries, The Argument from Philosophical Difficulty, Three AI Safety Related Ideas, and Two Neglected Problems in Human-AI Safety.

Some interrelated ways of looking at philosophy

Philosophy as answering confusing questions

This was my starting point for thinking about what philosophy is: it's what we do when we try to answer confusing questions, or questions that we don't have any other established methodology for answering. Why do we find some questions confusing, or lack methods for answering them? This leads to my next thought.

Philosophy as ability to generalize / handle distributional shifts

ML systems tend to have a lot of trouble dealing with distributional shifts. (It seems to be a root cause of many AI as well as human safety problems.) But humans seem to have some way of (sometimes) noticing out-of-distribution inputs, and can feel confused instead of just confidently using their existing training to respond to it. This is perhaps most obvious in unfamiliar ethical situations like Torture vs Dust Specks or trying to determine whether our moral circle should include things like insects and RL algorithms. Unlike ML algorithms that extrapolate in an essentially random way when given out-of-distribution inputs, humans can potentially generalize in a principled or correct way, by using philosophical reasoning.

Philosophy as slow but general purpose problem solving

Philosophy may even be a fully general purpose problem solving technique. At least we don't seem to have reason to think that it's not. The problem is that it's painfully slow and resource intensive. Individual humans acting alone seem to have little chance of achieving justifiably high confidence in many philosophical problems even if they devote their entire lives to those problems. Humanity has been collectively trying to solve some philosophical problems for hundreds or even thousands of years, without arriving at final solutions. The slowness of philosophy explains why distributional shifts remain a safety problem for humans, even though we seemingly have a general way of handling...
The Argument from Philosophical Difficulty
by Wei Dai — 2019-02-10T00:28:07.472Z

(I'm reposting this comment as a top-level post, for ease of future reference. The context here is a discussion about the different lines of arguments for the importance of AI safety.)

Here's another argument that I've been pushing since the early days (apparently not very successfully since it didn't make it to this list :) which might be called "argument from philosophical difficulty". It appears that achieving a good long term future requires getting a lot of philosophical questions right that are hard for us to answer. Given this, initially I thought there are only three ways for AI to go right in this regard (assuming everything else goes well with the AI):

We solve all the important philosophical problems ahead of time and program the solutions into the AI.

We solve metaphilosophy (i.e., understand philosophical reasoning as well as we understand mathematical reasoning) and program that into the AI so it can solve philosophical problems on its own.

We program the AI to learn philosophical reasoning from humans or use human simulations to solve philosophical problems.

Since then people have come up with a couple more scenarios (which did make me slightly more optimistic about this problem):

We all coordinate to stop technological progress some time after AI but before space colonization, and have a period of long reflection where humans, maybe with help from AIs, spend thousands or millions of years to solve philosophical problems.

We program AIs to be corrigible to their users, some users care about getting philosophy correct so the AIs help keep them safe and get their "fair share" of the universe until philosophical problems are solved eventually, enough users care about this so that we end up with a mostly good future, and lack of philosophical knowledge doesn't cause disaster in the meantime. (My writings on "human safety problems" were in part a response to this suggestion, outlining how hard it would be to keep humans "safe" in this scenario.)

The overall argument is that, given human safety problems, realistic competitive pressures, difficulties with coordination, etc., it seems hard to end up in any of these scenarios and not have something go wrong along the way. Maybe another way to put this is, given philosophical difficulties, the target we'd have to hit with AI is even smaller than it might otherwise appear.
Dojo on stress
by Elo — 2019-02-09T22:49:59.890Z

Original post: http://bearlamp.com.au/dojo-on-stress/

This (thinking) dojo came about because someone described their biggest problem as stress, relating to recent job-change events. I ran this dojo in Melbourne and Sydney to an audience of ~10 people each time. (45mins-1hr long)

As the facilitator:

Say yes. Accept what people bring. There’s no wrong answer. Each person is bringing the most valuable thing for themselves to the discussion.
There is a need to balance the group time and let everyone talk if they want to, but generally people are aware of that. If needed, thank someone and bring their awareness to the fact that this is a group event and everyone needs to participate to grow. The exercise is not about being right but about sharing and discussing stress.
Make empty space, (time when nothing is said) both for people to think, and for less talkative people to step up and share. Speak slowly, there is no rush.
There is no need to force participation, feel free to mention that anyone can pass at any time. It might be healthy to model “pass” behaviour at the start by getting everyone to say out loud “pass”. “What do you say if you don’t want to contribute and you want to pass?” (“PASS” duh)

1. Share a stressful experience.

Each person should share a personal or significant experience of stress that they have encountered in their life. It doesn’t have to still be “alive”, but it has to be personally relevant to them being engaged in the discussion and get an internal sense of what “stress” is and the sorts of things we are talking about. (if doing this on your own, write down your experience, spend a few minutes waiting with the memories to get a sense of how it felt to be in the body during that experience, feel free to write more than one down)

(1-2mins per person)

Briefly check for a common theme. I.e. stress caused by interpersonal relationships or work. Be mindful of that when continuing the exercise.

2. Causes of stress

Stress usually has a cause in life. Each person is different, each person will know their own common causes of stress. Make a list of personal and common causes of stress.

(2 mins alone making a list, 5-10 mins to discuss the general possible causes of stress and build a group list) (if alone, you can google it, but with emphasis that this exercise is not about getting the right answer, but more about being aware of the parts of...
When should we expect the education bubble to pop? How can we short it? 
by Bird Concept — 2019-02-09T21:39:10.918Z

I won't attempt to summarise the case for there being an education bubble here (see links below for some pointers). Rather, my questions are:

1) assuming there is an education bubble, when will it -- as bubbles tend to do -- pop?

(This plausibly entails some disjunction of *hundreds of thousands to millions of students defaulting on their debt, *lower number of college applicants, *non-top-tier colleges laying off faculty, *substantial reductions the signalling value of obtaining a diploma, *substantial reductions in tuition fees, *reduction in the level of education required by various employers, and more)

2) Which assets will be more scarce/in demand as that happens? Are there currently available opportunities for "shorting" the education bubble and invest in ways which will yield profit when it pops?

(I hereby preface the comments by noting that nothing discussed there is investment advice and no users can be held liable for investment decisions based on it.)

Peter Thiel summarises the inside view of there being an "education bubble" well.

And here are some interesting numbers:

In the last 35 years, median household income has grown by about 20% (

FRED

). In roughly the same time, the price of college has grown by 300% when adjusting for inflation

College spending is one sixth of US economy

UPDATE: this is probably false/misleading, see comment from paulfchristiano below.

Student debt is at >$1 trillion

(for comparison US GDP is around $20 trillion, US federal budget is around $4 trillion)