trentmkelly/LessWrong-43k huggingface dataset:
Why we want unbiased learning processes
by Stuart_Armstrong — 2018-02-20T14:48:48.885Z

tl;dr: if an agent has a biased learning process, it may choose actions that are worse (with certainty) for every possible reward function it could be learning.

An agent learns its own reward function if there is a set R of possible reward functions, and there is a learning process P that maps world-histories (and policies) to distributions over R. Thus by interacting with the environment and choosing its own policies, the agent can learn which is the correct reward function it should be maximising.

Given a policy π, a history h, an environment μ,and a reward R, we can compute the expected probability of R:

EμπP(R|h).

Then a learning process is unbiased if that expression is independent of π, and biased otherwise. Biased processes are less desirable, as they allow the agent to manipulate the process through its choice of policy.

Simple biased learning process

The most trivial example of a biased learning process is an agent that completely determines its reward by its actions.

Let R={R0,R1}, let the agent only act once with two actions available, {a0,a1}, (hence a choice of "policy" is a choice of action), and set

P(R0|a0)=P(R1|a1)=1.

Thus the agent can simply choose its reward function through its actions.

Note that some designs are a bit more sophisticated, and don't allow the agent to choose its reward function directly through its actions. But this doesn't matter, if the reward function is a consequence of anything that is a predictable consequence of the agent's actions (eg if the agent can trick/coerce/manipulate a human into saying "yes" or "no", and if P is determined by the human's response, it doesn't matter that P is not defined directly through the agent's actions: it is defined indirectly through them).

[Note that all P that involve learning about external facts are unbiased learning processes, so it's not as if unbiased means trivial]

Strictly dominated behaviour

Then an agent with a biased learning process that wants to maximise the expectation of the true reward, can sometimes follow strictly dominated policies.

That means that there is are policies π0 and π1 , such that for all histories hi possible given πi, and all reward function R in R,

R(h1)>R(h0).

And yet the agent will still choose π0 to maximise reward.

For example, with P and R defined as above, define R0 and R1 to be:

a0a1

... ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Bug Hunt 2
by alkjash — 2018-02-20T05:00:00.491Z

This is part 11 of 30 of Hammertime. Click here for the intro.

CFAR has an underlying mantra “adjust your seat”: systematically modify every technique and class to fit your personal situation. It’s common sense nowadays that different things work for different people, but the extent to which is true still constantly surprises me. (Kierkegaard had a fun take on adjusting your seat which he called the Method of Rotation.)

If you wish to partake in Hammertime, feel free to adjust your seat as much as necessary. Draw out the practice of instrumental rationality over a longer period of time, pick and choose the methods that appeal to you, and scale them to your time constraints.

Hammertime: The Second Cycle

Hammertime is about cultivating a tiny number of powerful techniques for solving a huge variety of problems. In the second cycle, we will revisit and upgrade the tools we introduced in the first, and apply them to tougher problems:

Bug Hunt
Yoda Timers
TAPs and Reinforcement Learning
Design
CoZE
Focusing
Cruxes
Goal Factoring
Internal Double Crux
Self-Trust

The new ideas we will be introducing in the second half are devoted to developing higher levels of introspection and self-honesty, to figure out your true motivations and aversions, and what to do about them.

Before each post in the second cycle, take a moment to review its predecessor.

Day 11: Bug Hunt 2

Previously: Day 1.

Noticing your bugs continues to be our single most powerful technique. Training noticing involves lateral thinking, attention to detail, and self-honesty. Today, we focus on three high-level ways in which human beings systematically err.

Setup

First, review your Bug List from Day 1 and update it.

For each of the next three mini-essays: read it over, then set a Yoda Timer for five minutes and brainstorm as many bugs as you can during that time.

1. Identity

Paul Graham wrote Keep Your Identity Small. Being attached to your identity can often constrain your growth.

Rather than making an impartial decision on what kind of person to be, people often extrapolate their identity (and morality) from their previous actions. A friend of mine calls this coprolite: fossilized and over-fitted beliefs that originate from early childhood. Are you neat or messy, stingy or generous, introvert or extrovert, conscientious or agreeable, idealistic or cynical, engineer or artist, vim or emacs? Do you look down on people for being the...
Formally Stating the AI Alignment Problem
by Gordon Seidoh Worley — 2018-02-19T19:06:04.086Z
Link: https://mapandterritory.org/formally-stating-the-ai-alignment-problem-fe7a6e3e5991

NB: Originally posted on Map and Territory on Medium, so some of the internal series links go there.

The development of smarter-than-human artificial intelligence poses an existential and suffering risk to humanity. Given that it is unlikely we can prevent and may not want to prevent the development of smarter-than-human AI, we are faced with the challenge of developing AI while mitigating risks. Addressing this challenge is the focus of AI safety research, and it includes topics such as containment of AI, coordination of AI safety efforts, and commitment to AI safety protocols, but the primary issue people believe to be worth addressing is AI alignment because it seems the most likely to decrease the chances of a catastrophic scenario.

Briefly stated, the problem of AI alignment is to produce AI that is aligned with human values, but this only leads us to ask, what does it mean to be aligned with human values? Further, what does it mean to be aligned with any values, let alone human values? We could try to answer by saying AI is aligned with human values when it does what humans want, but this only invites more questions: Will AI do things some specific humans don’t want if other specific humans do? How will AI know what humans want given that current technology often does what we ask but not what we desire? And what will AI do if human values conflict with its own values? Answering these questions requires a more detailed understanding of what it would mean for AI to be aligned, thus the goal of the present work is to put forward a precise, formal, mathematical statement of the AI alignment problem.

Eliezer Yudkowsky offered the first attempt at explaining AI alignment in his seminal work on the topic, “Creating Friendly AI”, and he followed this up with a more nuanced description of alignment in “Coherent Extrapolated Volition”. Nearly a decade later Stuart Russell began talking about the value alignment problem, giving AI alignment its name and kickstarting a broader interest in AI safety. Since then numerous researchers and organizations have worked on AI alignment to give us a better understanding of the problem. For example, Dario Amodei et al. have identified at least five subproblems within AI alignment, Jan Leike et al. have identified eight, and Nate Soares has divided AI alignment into at least seven subfields. Tak...
An alternative way to browse LessWrong 2.0
by saturn — 2018-02-19T02:10:32.314Z

This is something I’ve been tinkering with for a while, but I think it’s now complete enough to be generally useful. It’s an alternative frontend for LessWrong 2.0, using the GraphQL API.

Features:

Fast, even on low-end computers and phones

Quickly jump to new comments in a thread with the “.” and “,” keys

Archive view makes it easy to browse the best posts of years past

Always shows every comment in a thread, no need to “load more”

Log in and post using your existing username and password, or create a new account

Simple markdown editor

Typography enhancements

Switch between fixed-width and fluid layouts and several different themes

Easily view a comment’s ancestors without scrolling by hovering over the left edge of a comment tree

Thanks to Said Achmiz for designing the themes and writing much of the frontend JavaScript.

Give it a try: https://​www.greaterwrong.com
An alternative way to browse LessWrong 2.0
by clone of saturn — 2018-02-19T01:52:06.462Z

This is something I've been tinkering with for a while, but I think it's now complete enough to be generally useful. It's an alternative frontend for LessWrong 2.0, using the GraphQL API.

Features:

Fast, even on low-end computers and phones

Quickly jump to new comments in a thread with the "." and "," keys

Archive view makes it easy to browse the best posts of years past

Always shows every comment in a thread, no need to "load more"

Log in and post using your existing username and password, or create a new account

Simple markdown editor

Typography enhancements

Switch between fixed-width and fluid layouts and several different themes

Easily view a comment's ancestors without scrolling by hovering over the left edge of a comment tree

Thanks to Said Achmiz for designing the themes and writing much of the frontend JavaScript.

Give it a try: https://www.greaterwrong.com
Whose reasoning can you rely on when your own is faulty?
by weft — 2018-02-18T22:41:53.729Z

None of us are perfect reasoners. None of us have unlimited information. Sometimes other people are more correct than we are. This is an obvious thing we all know but may not practice .

Below are some concrete questions you can think about that come at this problem from a very specific path of trust (There are a lot of other ways to come at this). Note, that when I say "trust" I AM NOT NOT NOT saying that you should allow other people's viewpoints to completely supercede your own. The trust I am implying is more along the lines of something like "this would cause me to pause and consider."

Who do I trust to be a solid representation of an alternative viewpoint?

The most frequent arguments we hear for viewpoints different from are own are often not the strongest arguments. Who do you trust to present well-reasoned, not mindkilled arguments for a differing viewpoint, while not trying to persuade you with ideological rhetoric.

If I am in the throes of emotion, who could tell me that I wasn't behaving "rationally", AND (ideally) that information would actively cause me to change my behavior?

Particularly if you know you often behave in ways you don't endorse when you are angry, or starry-eyed, or perhaps under the influence of illicit substances... Is there someone who could say "You need to stop what you're doing right now." For me, there are occasional times that I am angry-and-irrational enough that I explicitly seek out an outside view as a check on my reasoning and actions. Any Friend I Trust is going to be a better judge of my actions at that time than I myself would be. Later, when I'm calmer, I can revisit and make sure I still endorse those conclusions.

Who would I most trust with end-of-life care?

This is a situation it is good to prepare for in advance because you may be completely unable to effect the decision by the time it comes up. So in this case you actually ARE putting important personal decisions completely in someone else's hands. My person is actually a former partner that I haven't spoken to in over a year. But we've had explicit discussions about our preferences (which are very similar), and I trust him to not be swayed by things like "What will other people think?" or "I need to prove my love by not letting go."

Next, do this in reverse. Think ...
A Simple Motto
by katerinjo — 2018-02-18T18:12:41.956Z

Rationality is believing true things and making good choices.

This phrasing isn't quite as precise as it could be, but it communicates the gist of both epistemic and instrumental rationality in very simple language. I use this wording when communicating what rationality is to non-rationalists, and it seems to do its job well enough. Maybe other rationalists will find it useful too.
[Meta] New moderation tools and moderation guidelines
by habryka — 2018-02-18T03:22:45.142Z

[I will move this into meta in a few days, but this seemed important enough to have around on the frontpage for a bit]

Here is a short post with some of the moderation changes we are implementing. Ray, Ben and me are working on some more posts explaining some of our deeper reasoning, so this is just a list with some quick updates.

Even before the start of the open beta, I intended to allow trusted users to moderate their personal pages. The reasoning I outlined in our

initial announcement post

was as follows:

“We want to give trusted authors moderation powers for the discussions on their own posts, allowing them to foster their own discussion norms, and giving them their own sphere of influence on the discussion platform. We hope this will both make the lives of our top authors better and will also create a form of competition between different cultures and moderation paradigms on Lesswrong.”

And I also gave some further perspectives on this in my

“Models of Moderation”

post that I posted a week ago.

We now finally got around to implement the technology for this. But the big question that has been on my mind while working on the implementation has been:

How should we handle moderation on frontpage posts?

Me, Ray, Ben and Vaniver talked for quite a while about the pros and cons, and considered a bunch of perspectives, but the two major considerations on our mind were:

The frontpage is a public forum that should reflect the perspectives of the whole community, as opposed to just the views of the active top-level authors.
We want our best authors to feel safe posting to LessWrong, and us promoting a post from your private blog to the frontpage shouldn’t feel like a punishment (which it might if it also entails losing control over it)

After a good amount of internal discussion, as well as feedback from some of the top content contributors on LW (including Eliezer), we settled on allowing users above 2000 karma to moderate their own frontpage posts, and allow users above 100 karma to moderate their personal blogs. This strikes me as the best compromise between the different considerations we had.

Here are the details about the implementation:

Users above...

100 karma can moderate everything in their personal blog posts section,

This functionality should go live in about a week

2000 karma can moderate all of their posts, including frontpage and curated. (We’ll likely increase th

...
Is skilled hunting unethical?
by JamesFaville — 2018-02-17T18:48:21.635Z

[Note (6/30/19): I think I made interesting and valid points with this piece, but my framing was terrible. I should have clarified who are the people for whom I think these arguments should be persuasive, had a longer introductory section talking about why skilled hunting is much less problematic than slaughter and consumption of farmed animals, spun the fifth argument off into its own post, and maybe left off an epistemic status. I'd like to revise this at some point, but not sure when I'll get around to it.]

Content Warning

Explicit descriptions of wild animal suffering and serious discussion of killing and consuming animals as a potentially net-positive intervention.

Premise

Two months ago, I believed that skillful hunting was ethical because it prevented animals from suffering painful

natural

deaths

in the wild. However, this was a privately-held belief, and after discussing it in person for the first time, I began to have second thoughts. Now, most hunting isn’t skilled enough to prevent animals from suffering non-fatal injuries or prolonged deaths, and I doubt the lives of most LessWrong readers are impacted by their beliefs on the ethics of skilled hunting. But even if most readers are not passionate hunters or consumers of hunted meat, I expect that they will nonetheless find the issues discussed in this article—wild animal welfare, movement-building, habit formation, moral uncertainty, how to set epistemic priors— both interesting and relevant to their day-to-day-lives. I also hope this article provides a useful example of how to examine an object-level belief and

actually change your mind

.

Meta-Information

Epistemic status: ~90% confident in conclusion conditioned on moral patienthood of hunted animals.

Epistemic effort: 14 hours of writing and new research. Preceded by substantial background research into wild-animal suffering, two ~15 minute conversations with other rationalists, and ~15 minutes focused preliminary thinking about the issue.

Motivation for writing: Inspired by a conversation that I had at the

NYC Solstice

this year. Hoping to gain experience with less academic writing, and explore whether or not I’m personally comfortable with some wild-fish consumption.

Acknowledgements: Thanks to

Ikaxas

for feedback and edits.

Background Assumptions

I will be taking it for granted in this piece that all targets of hunting have

enough sentience

...
Missives from China
by alkjash — 2018-02-17T12:30:00.555Z

Driving in China

You’ve heard of the Iterated Prisoner’s Dilemma. But have you heard of Iterated Chicken?

Heat Lag

Day 4: Temperature is experienced differently here. People wear wool coats over turtlenecks in 70 degree weather. Counted two other people wearing T-shirts out of thousands.

Day 5: It’s freezing out. I really miss my winter coat.

A Fatal Error

Two days before Chinese New Year, I went to the supermarket and bought my own snacks.

Feeling Like a Native

A man on a moped asked me for directions today. To the place I was walking.

Evolution of Fashion

Day 2: Two cute little girls seen wearing spinning pinwheels in their hair.

Day 3: Every little girl is wearing them.

Day 4: Boy spotted wearing pinwheel. Two girls spotted wearing double pinwheels.

Day 5: Grown ass women with rainbow pinwheels in their hair.

Blood, Sweat and Tears

Would anyone attempt this mountain if the height of its summit could be perceived from the base?

Feeling Attractive

Watched too many Asian idol dramas recently. Average Chinese people are pleasingly homely in comparison.

An Electrifying Coincidence

My wife is from Edison, New Jersey. Her ancestors brought the light bulb to China.

Feeling Like a Native, Part 2

Out of place on a crowded street. Overheard dialects from all over China. Realized Chinese people are tourists, even in China. I’m right at home.
In Defence of Conflict Theory
by Richard_Ngo — 2018-02-17T03:33:01.970Z
Link: https://thinkingcomplete.blogspot.co.uk/2018/02/in-defence-of-conflict-theory.html

Scott Alexander recently wrote an interesting blog post on the differences between approaches to politics based on conflict theory and mistake theory. Here's a rough summary, in his words:

Mistake theorists treat politics as science, engineering, or medicine. The State is diseased. We’re all doctors, standing around arguing over the best diagnosis and cure. Some of us have good ideas, others have bad ideas that wouldn’t help, or that would cause too many side effects. Conflict theorists treat politics as war. Different blocs with different interests are forever fighting to determine whether the State exists to enrich the Elites or to help the People... Right now I think conflict theory is probably a less helpful way of viewing the world in general than mistake theory. But obviously both can be true in parts and reality can be way more complicated than either.

Here's my main argument against emphasising mistake theory over conflict theory: you're only able to be a mistake theorist after the conflict theorists have done most of the hard work. Even if the lens of mistake theory is more useful in dealing with most of the political issues we engage with on a daily basis, that's only the case because those issues are a) within our Overton window, so that they can be discussed, and b) considered important by either some powerful people, or many normal people, so that proposed solutions have a chance of being implemented. Ensuring that a given issue fulfills those criteria requires a conflict-theoretic mindset, because until they are met you will face opponents much more powerful than you.

Let's take a few examples. The main one is democracy itself. Mistake theorists wish for technocrats to have more power, so they can implement better policies. But conflict theorists have spent the last three centuries drastically curtailing the power of monarchies and dictatorships - which were close cousins of technocracy, given that hereditary rulers tended to be far more educated than the population as a whole. Compared with that seismic shift, the difference between modern conflict-theorists and mistake-theorists is a rounding error: supporting what past generations thought of as "mob rule" for the sake of that mob having power over its leaders puts us all way on the conflict theorist side of the spectrum. Of course it'd be very nice to hav...
Replacing expensive costly signals
by KatjaGrace — 2018-02-17T00:50:00.500Z

I feel like there is a general problem where people signal something using some extremely socially destructive method, and we can conceive of more socially efficient ways to send the same signal, but trying out alternative signals suggests that you might be especially bad at the traditional one. For instance, an employer might reasonably suspect that a job candidate who did a strange online course instead of normal university would have done especially badly at normal university.

Here is a proposed solution. Let X be the traditional signal, Y be the new signal, and Z be the trait(s) being advertised by both. Let people continue doing X, but subsidize Y on top of X for people with very high Z. Soon Y is a signal of higher Z than X is, and understood by the recipients of the signals to be a better indicator. People who can’t afford to do both should then prefer Y to X, since Y is is a stronger signal, and since it is more socially efficient it is likely to be less costly for the signal senders.

If Y is intrinsically no better a signal than X (without your artificially subsidizing great Z-possessors to send it) then in the long run Y might only end up as strong a sign as X, but in the process, many should have moved to using Y instead.

(A possible downside is that people may end up just doing both forever.)

For example, if you developed a psychometric and intellectual test that only took half a day and predicted very well how someone would do in an MIT undergraduate degree, you could run it for a while for people who actually do MIT undergraduate degrees, offering prizes for high performance, or just subsidizing taking it at all. After the best MIT graduates say on their CVs for a while that they also did well on this thing and got a prize, it is hopefully an established metric, and an employer would as happily have someone with the degree as with a great result on your test. At which point an impressive and ambitious high school leaver would take the test, modulo e.g. concerns that the test doesn’t let you hang out with other MIT undergraduates for four years.

I don’t know if this is the kind of problem people actually have with replacing apparently wasteful signaling systems with better things. Or if this doesn’t actually work after thinking about it for more than an hour. But just in case.