trentmkelly/LessWrong-43k huggingface dataset:
Worth keeping
by KatjaGrace — 2018-12-07T04:50:01.210Z

(Epistemic status: quick speculation which matches my intuitions about how social things go, but which I hadn’t explicitly described before, and haven’t checked.)

If your car gets damaged, should you invest more or less in it going forward? It could go either way. The car needs more investment to be in good condition, so maybe you do that. But the car is worse than you thought, so maybe you start considering a new car, or putting your dollars into Uber instead.

If you are writing an essay and run into difficulty describing something, you can put in additional effort to find the right words, or you can suspect that this is not going to be a great essay, and either give up, or prepare to get it out quickly and imperfectly, worrying less about the other parts that don’t quite work.

When something has a problem, you always choose whether to double down with it or to back away.

(Or in the middle, to do a bit of both: to fix the car this time, but start to look around for other cars.)

I’m interested in this as it pertains to people. When a friend fails, do you move toward them—to hold them, talk to them, pick them up at your own expense—or do you edge away? It probably depends on the friend (and the problem). If someone embarrasses themselves in public, do you sully your own reputation to stand up for their worth? Or do you silently hope not to be associated with them? If they are dying, do you hold their hand, even if it destroys you? Or do you hope that someone else is doing that, and become someone they know less well?

Where a person fits on this line would seem to radically change their incentives around you. Someone firmly in your ‘worth keeping’ zone does better to let you see their problems than to hide them. Because you probably won’t give up on them, and you might help. Since everyone has problems, and they take effort to hide, this person is just a lot freer around you. If instead every problem hastens a person’s replacement, they should probably not only hide their problems, but also many of their other details, which are somehow entwined with problems.

(A related question is when you should let people know where they stand with you. Prima facie, it seems good to make sure people know when they are safe. But that means it also being clearer when a person is not safe, which has downsides.)

If there are better replacements in general, then you will be inclined to...
Trivial Inconvenience Day (December 9th at 12 Noon PST)
by namespace — 2018-12-07T01:26:22.870Z

Hi everybody, it's been 9 months since I wrote my report on Trivial Inconvenience Day. I wasn't quite sure when to run a sequel event. I wanted to give people an opportunity for deferred tasks to pile up again. With three seasons under the bridge I think it's safe to say that the procrastination coffers have since refilled. This post is your invitation to join us this Sunday, December 9th, to cross nagging tasks off your list.

Event Information

Shortly after reading Scott Alexander’s LessWrong Crypto Autopsy I found myself agreeing with the point so strongly I was brainstorming ways that its dismal outcome could have been prevented. Peoples personal accounts of why they didn’t buy bitcoin seemed to converge on a central theme: Buying Bitcoin was a trivial inconvenience. Pondering what might be done in light of this, I was reminded of Boston Rat’s Bureaucracy Day. The Bureaucracy Day is essentially a designated day for people to beat the ugh field effect by getting together and going through the whole mess of annoying tasks, paperwork, and other trivially inconvenient things people have been putting off. Having been impressed by the concept the first time I read about it, Scott’s dire analysis convinced me to try something like it in the hope that it would be a useful tool against this sort of thing happening again.

Multi-hour session of not letting small emotional or logistical barriers get in the way of a better life, state tasks you'd like to do and do them. No minimum commitment or significance requirements, show up for 15 minutes to do something and leave if you like. Prefer most participants show up at designated time to maintain critical mass throughout event.

How do I join?

Join this Discord server on or before Sunday, December 9th at 12:00 (Noon) PST. No account or signup is required.

You can also join while things are ongoing, but it works best if most of us are there at the same time to start.

Why join?

From the outset I was aiming for the event to have a particular sort of feel. I wanted it to be serious, but also sort of cheesy. In real life we’re used to doing slog-y, tedious work and having nothing to show for it afterwards besides hours passed on the clock. It’s not a very rewarding experience. Therefore to help counteract this I wanted the experience to be chock full of artificial rewards. The fact of the matter is that these mundane necess

...
COEDT Equilibria in Games
by Diffractor — 2018-12-06T18:00:08.442Z

From Jessica's earlier post about conditional oracles, the question was asked: what is the equilibrium concept for games with more than one COEDT agent?

This post will (partially) answer that question, and provide a link to a tool to visualize 2-player 2-move COEDT equilibria sets.

To begin with, we will review how conditional oracles work. A conditional oracle is a mapping from M4→{0,1} (where M is some set of finitely many probabilistic oracle machines). A conditional oracle distribution is a probability distribution over conditional oracles. A conditional oracle distribution Q weakly leads to Q′, if

PQ(mO1=1∧nO1)PQ(nO2=1)>PQ(mO2=1∧nO2)PQ(nO1=1)→PQ′(O(m1,n1,m2,n2)=1)=1

PQ(mO1=1∧nO1)PQ(nO2=1)<PQ(mO2=1∧nO2)PQ(nO1=1)→PQ′(O(m1,n1,m2,n2)=0)=1

From the last post, all reflective conditional oracle distributions weakly lead to themselves.

Two things must be noted. First, the space that the equilibrium set lies in is actually correlated strategy space (a probability distribution over all joint outcomes, which in a 2-player 2-move game, is a tetrahedron). This is because the same oracle is selected from the oracle distribution for all the algorithms to use, which opens the door to correlated outcomes, instead of both players independently drawing an oracle from the oracle distribution. Second, the full concept of an equilibrium produced by a reflective COD (in the sense of a necessary and sufficient condition) is difficult to characterize, because it involves a bunch of fiddly details of the utilities of various probability-zero actions and whether it is possible to limit to them in the appropriate way. Therefore, the following definition doesn't characterize all COEDT equilibria, just the fully-mixed ones that lie in the interior of the space instead of on a boundary (because that corresponds to having no outcomes occur with probability zero).

Let Pi be the i'th player, Ui be the utility function of the i'th player, Si be the strategy set of the i'th player, S=∏i∈ISi be the set of outcomes, si be the move the i'th player plays as dictated by some s∈S, and Δ be a probability distribution over S.

Definition 1: A fully-mixed conditional equilibria is a probability distribution Δ with nonzero probability over all s∈S, s.t.

∀i∀j,j′∈Si:Es∼Δ(Ui|si=j)=Es∼Δ(Ui|si=j′...
Hell Must Be Destroyed
by algekalipso — 2018-12-06T04:11:19.417Z

[Originally published in Qualia Computing]

Singer called the movement that grew up around him “effective altruism”, and its rallying cry was that one ought to spend every ounce of one’s energy doing whatever most relieves human suffering, most likely either feeding the poor or curing various tropical diseases. Again, something his opponents rejected as impossible, unworkable, another example of liberal fanaticism. Really? Every ounce of your energy? Again, they could have just read their Bibles. Deuteronomy 6:5: “And you must love the Lord your God with all your heart, and with all your soul, and with all your strength.”

Then Singer changed his tune. In the 1970s, after the sky cracked and the world changed, he announced that charity was useless, that feeding the poor was useless, that curing tropical diseases was useless. There was only one cause to which a truly rational, truly good human being could devote his or her life.

Hell must be destroyed.

The idea of billions of human beings suffering unbearable pain for all eternity so outweighed our little earthly problems that the latter didn’t even register. He began meeting with his disciples in secret, teaching them hidden Names he said had been vouchsafed to him by angels. Thamiel put a price on his life – quite a high price actually. Heedless of his own safety, Singer traveled what remained of the civilized world, making converts wherever he went, telling them to be perfect as God was perfect, and every speech ended the same way. Hell must be destroyed.

– INTERLUDE ג: CANTORS AND SINGERS from UNSONG

An angel appears on Earth. This genderless being connected to God shows up on every screen on Earth at once and asks us if we are interested in drastically improving life on Earth. A large enough portion of those who hear the message (which gets a coverage of 80%+ of people worldwide) see into their souls and find the willingness to make life better, and then they see into their hearts and see the warmth of hope, and so they resolve to agree to do whatever is necessary to help the angel improve life on Earth. And thus the angel says “thanks to the collective desire to make it so, I shall change some things about how the planet is programed, and you will see a 99% reduction in suffering and a 20% increase in overall happiness.”

And so the angel gets to work.

A year passes, and nobody can really tell the difference f...
On Rationalist Solstice and Epistemic Caution
by Raemon — 2018-12-05T20:39:34.687Z

Since 2011, some LessWrong folk have observed the winter solstice, as a holiday ritual celebrating human achievement in the face of a confusing, often terrifying world.

I've written in the past about the potential value, and danger, of ritual. Over the past years my opinion shifted somewhat, but is still essentially summarized as: "I think ritual is less epistemically fraught than generally exposing yourself to the beliefs of a peer group (something that pretty much everyone does by default), and meanwhile has many benefits. We should be cautious of it, but it's a fairly important human experience we shouldn't discard simply because it pattern-matches to woo."

Still, I think the practice of rational ritual should still involve a lot of epistemic care, on the part of both organizers and participants.

Since 2012, in the various editions of the Solstice Book of Traditions, I've included a disclaimer at the beginning, and I think it'd be valuable to have that more publicly accessible so that people going into a Solstice can be properly informed.

Individual Solstice celebrations vary, and I can't promise that this document will accurately reflect all organizers' intentions. But it reflects my own goals and hopefully provides a reasonable starting context.

What is a Ritual?

A ritual is about making a sacrifice to imbue a moment with symbolic power, and using that power to transform yourself. Ritual experience cannot be coerced - only entered willingly by those that believe in them. A ritual that you don’t believe in may feel hollow, or alienating.

I do not believe ritual and rationality are inherently contradictory. The human brain seems designed badly. It is hard to truly accept certain facts about the world, even when you have empirical evidence - especially for facts involving large numbers, or unspeakable horrors.

It can even be hard for your brain to accept truths like “You are not alone, and you can do this.”

Rituals can be useful, to internalize those facts.

They can also be useful to help make it true, that you are not alone, and you can do this.

Nonetheless, with power comes responsibility. If you are considering participating in the Rationalist Solstice, first consider as carefully as you can, in the light of day with your clear-thinking prefrontal cortex, whether the concepts herein seem true and good - the sort o...
Transhumanism as Simplified Humanism
by Eliezer Yudkowsky — 2018-12-05T20:12:13.114Z

This essay was originally posted in 2007.

Frank Sulloway

once said: “Ninety-nine per cent of what Darwinian theory says about human behavior is so obviously true that we don’t give Darwin credit for it. Ironically, psychoanalysis has it over Darwinism precisely because its predictions are so outlandish and its explanations are so counterintuitive that we think, Is that really true? How radical! Freud’s ideas are so intriguing that people are willing to pay for them, while one of the great disadvantages of Darwinism is that we feel we know it already, because, in a sense, we do.”

Suppose you find an unconscious six-year-old girl lying on the train tracks of an active railroad. What, morally speaking, ought you to do in this situation? Would it be better to leave her there to get run over, or to try to save her? How about if a 45-year-old man has a debilitating but nonfatal illness that will severely reduce his quality of life – is it better to cure him, or not cure him?

Oh, and by the way: This is not a trick question.

I answer that I would save them if I had the power to do so – both the six-year-old on the train tracks, and the sick 45-year-old. The obvious answer isn’t always the best choice, but sometimes it is.

I won’t be lauded as a brilliant ethicist for my judgments in these two ethical dilemmas. My answers are not surprising enough that people would pay me for them. If you go around proclaiming “What does two plus two equal? Four!” you will not gain a reputation as a deep thinker. But it is still the correct answer.

If a young child falls on the train tracks, it is good to save them, and if a 45-year-old suffers from a debilitating disease, it is good to cure them. If you have a logical turn of mind, you are bound to ask whether this is a special case of a general ethical principle which says “Life is good, death is bad; health is good, sickness is bad.” If so – and here we enter into controversial territory – we can follow this general principle to a surprising new conclusion: If a 95-year-old is threatened by death from old age, it would be good to drag them from those train tracks, if possible. And if a 120-year-old is starting to feel slightly sickly, it would be good to restore them to full vigor, if possible. With current technology it is not possible. But if the technology became available in some future year – given sufficiently advanced medical na...
Why we need a *theory* of human values
by Stuart_Armstrong — 2018-12-05T16:00:13.711Z

There have been multiple practical suggestions for methods about how we should extract the values of a given human. Here are four common classes of such methods:

Methods that put high weight on human (bounded) quasi-rationality, or revealed preferences. For example, we can assume the Kasparov was actually trying to win against DeepBlue, not trying desperately to lose while inadvertently playing excellent chess.

Methods that pay attention to our explicitly stated values.

Methods that use regret, surprise, joy, or similar emotions, to estimate what humans actually want. This could be seen as a form of human TD learning.

Methods based on an explicit procedure for constructing the values, such as CEV and Paul's indirect normativity.

Divergent methods

The first question is why we would expect these methods to point even vaguely in the same direction. They all take very different approaches - why do we think they're measuring the same thing?

The answer is that they roughly match up in situations we encounter everyday. In such typical situations, people who feel regret are likely to act to avoid that situation again, to express displeasure about the situation, etc.

By analogy, consider a town where there are only two weather events: bright sunny days and snow storms. In that town there is a strong correlation between barometric pressure, wind speed, cloud cover, and temperature. All four indicators track different things, but, in this town, they are basically interchangeable.

But if the weather grows more diverse, this correlation can break down. Rain storms, cloudy days, meteor impacts: all these can disrupt the alignment of the different indicators.

Similarly, we expect that an AI could remove us from typical situations and put us into extreme situations - at least "extreme" from the perspective of the everyday world where we forged the intuitions that those methods of extracting values roughly match up. Not only do we expect this, but we desire this: a world without absolute poverty, for example, is the kind of world we would want the AI to move us into, if it could.

In those extreme and unprecedented situations, we could end up with revealed preferences pointing one way, stated preferences another, while regret and CEV point in different directions entierly. In that case, we might be tempted to ask "should we follow regret or stated preferences?" But that would be

...
Factored Cognition
by stuhlmueller — 2018-12-05T01:01:43.544Z

Note: This post (originally published here) is the transcript of a presentation about a project worked on at the non-profit Ought. It is included in the sequence because it contains a very clear explanation of some of the key ideas behind iterated amplification.

The presentation below motivates our Factored Cognition project from an AI alignment angle and describes the state of our work as of May 2018. Andreas gave versions of this presentation at CHAI (4/25), a Deepmind-FHI seminar (5/24) and FHI (5/25).

I'll talk about Factored Cognition, our current main project at Ought. This is joint work with Ozzie Gooen, Ben Rachbach, Andrew Schreiber, Ben Weinstein-Raun, and (as board members) Paul Christiano and Owain Evans.

Before I get into the details of the project, I want to talk about the broader research program that it is part of. And to do that, I want to talk about research programs for AGI more generally.

Right now, the dominant paradigm for researchers who explicitly work towards AGI is what you could call "scalable learning and planning in complex environments". This paradigm substantially relies on training agents in simulated physical environments to solve tasks that are similar to the sorts of tasks animals and humans can solve, sometimes in isolation and sometimes in competitive multi-agent settings.

To be clear, not all tasks are physical tasks. There's also interest in more abstract environments as in the case of playing Go, proving theorems, or participating in goal-based dialog.

For our purposes, the key characteristic of this research paradigm is that agents are optimized for success at particular tasks. To the extent that they learn particular decision-making strategies, those are learned implicitly. We only provide external supervision, and it wouldn't be entirely wrong to call this sort of approach "recapitulating evolution", even if this isn't exactly what is going on most of the time.

As many people have pointed out, it could be difficult to become confident that a system produced through this sort of process is aligned - that is, that all its cognitive work is actually directed towards solving the tasks it is intended to help with. The reason for this is that alignment is a property of the decision-making process (what the system is "trying to do"), but that is unobserved and only implicitly cont...
Playing Politics
by sarahconstantin — 2018-12-05T00:30:00.996Z

Epistemic Status: Guesses Based on Personal Experience

Lately I’ve been going through a family of learning experiences in the world of how to get things done cooperatively. It’s hard for me. Even very basic things in this area have been stumping me, overwhelming me, leaving me way more tired and drained than I’d expect. My productivity has gone to hell and — worse — I didn’t even notice for a while. This is hard stuff, and rarely written about by the people for whom it’s hard, so my hope is that processing in public helps someone. I generally think that data-sharing is good and helpful.

Collective Deliberation Isn’t Working For Me

At a conference, I was in a room full of people having a really good discussion. I wanted to get people together to have a follow-up discussion later — nothing elaborate, just a room with whiteboards and snacks and maybe moving towards some action items.

What I did:

Passed around a sheet for emails to sign up

Sent out an email proposing the parameters of the event

Waited for people to propose dates that worked for them.

Radio silence.

Somebody else suggested a poll where people could put down their preferred times and dates. Out of thirteen people, five signed up. Nobody volunteered “ok, we’re doing it on this date then,” so I did. I reserved a conference room at my office and bought a bunch of snacks.

The front door was locked on the weekend and my key card didn’t work even though it was supposed to, so I had to switch locations at the last minute. It wouldn’t have mattered anyhow, because one person showed up on time, and one other person several hours late.

Conclusion: it is harder than I thought to get ten people to show up in a room and talk to each other.

And I probably shouldn’t have expected an event to coalesce naturally from the mailing list. I have a strong “egalitarian” instinct that if I’m trying to do something with a group and in some sense for the benefit of everyone in the group, then I shouldn’t be too “bossy” in terms of unilaterally declaring what we’re all going to do. But if I leave it up to the group to discuss, it seems like they generally…don’t.

I’m also on a policy committee for a community organization, and it’s been a whole lot of heartache because I want to change some things about our policies and internal processes, and the process of trying to communicate that has resulted in a lot of hurt feelings,...
Open and Welcome Thread December 2018
by Ben Pace — 2018-12-04T22:20:53.076Z

If it’s worth saying, but not worth its own post, then it goes here.

Also, if you are new to LessWrong and want to introduce yourself, this is the place to do it. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are welcome. If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, and seeing if there are any meetups in your area.

As well as trying out combining welcome threads and open threads, I thought I'd try highlighting some frontpage comments I found especially insightful in the last month, for further discussion:

Scott Garrabrant wrote a comment on how Embedded Agency and Agent Foundations research are like science in relation to ML approaches to AI alignment which are more like engineering. The comment helped me think about how I go about formalising and solving problems more generally.
Rohin Shah wrote a comment on basic definitions of the alignment problem, contrasting a motivation-competence split versus a definition-optimization split. (It is then followed by a convo on definitions between Paul and Wei which gets pretty deep into the weeds - I'd love to read a summary here from anyone else who followed along.)